Data Mining Predicted Swine Flu 18 Days before WHO

According to reports, Washington state surveillance firm Veratect Corp. raised the first warning about a possible swine flu outbreak in Mexico more than two weeks before the World Health Organization (WHO) made its own announcement. (Source: pnwlocalnews.com)It also appears as that federal and international health officials had access to the Veratect Corp. warning. Emails relating to the possible outbreak spreading in Mexico were sent to 10 officials of the Centers for Disease Control (CDC) and prevention. It's not known why health officials didn't act sooner than they did.

Data Mining Technique Detected Outbreak

Kirkland, Washington-based Veratect uses a data mining technique to automatically search tens of thousands of web sites daily for early signs of looming medical problems or civil unrest anywhere in the world.

When items of interest are found, the results are turned over to a team of 35 analysts who post legitimate threats on the company's web site. Government agencies, businesses and other groups have access to the Veratect web site.

Veratect has been unsuccessful in trying to sell its services to the CDC, the WHO and the Department of Homeland Security, though the CDC and the WHO were given access to the Veratect web site on a trial basis earlier this year.

Flu Detected Weeks Before WHO's Announcement

18 days before WHO issued the alert about a possible Swine flu pandemic, Veratect reported on a strange outbreak of respiratory disease in La Gloria, Mexico, noting that local residents thought the flu outbreak was related to contamination from pig breeding farms nearby. (Source: mcclatchydc.com)

The information was available to the CDC as well as many state and local health authorities. Records show that an epidemiologist at the Pan American Health Organization -- part of the WHO -- viewed the message about the La Gloria outbreak on April 10 and on April 11.

An automated email was sent to 10 people at the CDC on April 16 to notify them that a disease with an 'unspecified number of atypical pneumonia cases' was detected at a hospital in Oaxaca.

Data Mining Technique for Detecting Outbreaks Untested

When there was evidence of the outbreak spreading, Veratect's chief scientist called people he knew at the CDC's emergency operations center on April 20 to make them aware of what was happening in Mexico.

During that time, the CDC was focusing its attention on possible swine flu events happening in Texas and California. Apparently the CDC was not aware that the outbreak was spreading in Mexico. Four days later, WHO made its announcement about the outbreak.

Veratect hopes to be part of an effective global health monitoring system that is badly needed. The use of data mining to track possible disease outbreaks is, however, currently unfounded.

HP and Oracle team up on 'data warehouse appliances' that re-architect database-storage landscape

Oracle CEO Larry Ellison today introduced the company’s first hardware products, a joint effort with Hewlett-Packard, to re-architect large database and storage configurations and gain whopping data warehouse and business intelligence performance improvements from the largest data sets.

The Exadata Programmable Storage Server appliance and the HP Oracle Database Machine, a black and red refrigerator-size full database, storage and network data center on wheels, made their debt at the Oracle OpenWorld conference in San Francisco. Ellison called the Machine the fastest database in the world.

HP Chairman and CEO Mark Hurd called the HP Oracle Database Machine a “data warehouse appliance.” It leverages the architecture improvements in the Exadata Programmable Storage Server, but at the much larger scale and with other optimization benefits. [Disclosure: HP is a sponsor of BriefingsDirect podcasts.]

The hardware-software tag team also means Oracle is shifting its relationships with storage array vendors, including EMC, Netezza, NetApp and Terradata. The disk array market has been hot, but the HP-Oracle appliance may upset the high end of the market, and then bring the price-performance story down market, across more platforms.

I think we can safely say that HP is a preferred Oracle storage partner, and that Oracle wants, along with HP, some of those high-growth storage market profits for their own. There’s no reason to not expect a wider portfolio of Exadata appliances and more configurations like the HP Oracle Database Machine to suit a variety of market segments.

“We needed radical new thinking to deliver high performance,” said Ellison of the new hardware configurations, comparing the effort to the innovative design for his controversial America’s Cup boat. “We need much more performance out of databases than what we get.”

This barnburner announcement may also mark a market shift to combined and optimized forklift data warehouses, forcing the other storage suppliers to find database partners. IBM will no doubt have to respond as well.

The reason for the 10x to 72x performance improvements cited by Ellison are do to bringing the “intelligence” closer to the data, that is bringing the Exadata Programmable Storage Server appliance into close proximity to the Oracle database servers, and then connecting them through InfiniBand connections. In essence, this architecture mimics some of the performance value created by cloud computing environments like Google, with its MapReduce technology.

Ellison said that rather than large data sets moving between storage and database servers, which can slow up performance at 1TB and larger databases, the new Exadata-driven configuration moves only the query information across the networks. The current versions of these optimized boxes use Intel dual-core technology, but they will soon also be fired up by six-way Intel multi-core processors.

Talk about speeds and feeds …. But the market driver in these moves is massive data sets that need to be producing near real-time analytics paybacks. We’re seeing more and more data, and varyinf kinds of data, brought into data warehouses and being banged on by queries of applications and BI servers from a variety of business users across the enterprise.

HP and Oracle share some 150,000 joint customers worldwide, said HP Executive Vice President, Technology Solutions Group Ann Livermore. That means that these database boxes will have an army of sales and support personnel. HP will support the Machine hardware, Oracle the software. Both will sell it.

50% of data warehouse projects to fail in 2005-2007

Through 2007, more than 50% of data warehouse projects will have limited acceptance, or will be outright failures, as a result of a lack of attention to data quality issues, according to Gartner. CIOs said in a recent Gartner survey that BI implementation will be a significant factor in delivering IT’s contribution to business growth. However, most businesses are failing to use BI strategically. Gartner analysts said integration of business and IT requirements is critical to any successful BI strategy.

Benefits and Harms of Knowledge Management into Organization

Benefits of KM
As eighty percent of knowledge in organization is tacit knowledge, so calculating ROI (return of investment) is very difficult and it may not be possible also. The impact of successful knowledge management program can be seen in terms of new and better product development, higher customer satisfaction, reduce in input cost, higher productivity etc..

KM creates a platform for extensive data mining and business intelligence knowledge. It assists today’s managers with better decision support systems. With sharing of data and information across the organization and getting the details from customers and suppliers, managers can forecast the future trend and take better decision. Knowledge creation process does not limit itself within the organization. Conference and seminars brings new learning to the organization and introduces ones knowledge and expertise to others. It encourages partnership and collaboration to develop new products and services leveraging the experience of different organizations. Because of the strong networking culture Knowledge Management creates it helps in developing collaborative decision making and threat analyzing capability of any organization. Organizations enter into new line of business with their knowledge and collaborations with others.
Today's business is not located near availability of raw materials or capital. They are flowing to the areas where knowledge is available. Companies’ future valuation heavily depends on its future market potential based on today's knowledge creating capability.


The dark side of KM

The implementation of Knowledge Management may face a widespread resistance because people do not like the very idea of their knowledge being managed, particularly in non-profit organizations, where a business case for knowledge management may not resonate with the values of the organization. Furthermore other factors such as the extensive use of Information Technology and complex information share systems may alienate individuals who are not computer literates.
Culture factors such as a high degree of competitiveness among employees can be a barrier on sharing information through an information system. Furthermore, the individuals may hesitate to share their knowledge out of fear of criticism or of misleading the community members (Ardichivili et al, 2003).
Organizational resistance to internal Knowledge Management efforts also stems from hierarchical structures that function to reinforce norms of competition by creating winners and losers (De Long and Seeman, 2000). When management decides to initiate knowledge management techniques, it is probable that they resist, because they do not like changes due to various reasons such as job losses.
De Long and Seeman (2000) argue that the successful implementation of Knowledge Management and its integration with the strategic aims may transform to a source of conflict. Top managers and executives may conflict in order to benefit from the success applying Knowledge Management systems in business. Though the relative literature focus on the early stages of knowledge management and on the process of implementing it, it seems that even in a case of a success a conflict may arise. This means that from the very beginning of knowledge management managers and executives shall define whose property the plan is and to delegate power in order not to have any conflicts that may disrupt a successful knowledge management plan.

Piyush patel
Software Engineer

Data Mining in 21st Century

Data Mining in Economic Crisis

In this time of economic turmoil the whole world is slowly plunging into recession. We have seen bailouts from governments worth billions of dollars for giant companies in US, Europe and Asia. The businesses are finding hard times to operate and so ability to make smarter and intelligent business decisions are imperative to overcome in such harsh conditions.

Businesses who have already invested in business intelligence solutions will be in a better position to undertake right measures to survive and continue its growth. However, many may argue that most of the giant companies were using Data mining and BI solutions but still could not avoid the collapse. The important thing to note here is that data mining solutions provides an analytical perspective into the performance of an organization based on historical data but the economic impact on an organization is linked to many issues and in many cases to external forces and unscrupulous activities. The failure to predict this does not undermine the role of data mining for organizations but on the contrary makes it more important especially for regulatory bodies of governments to predict and identify such practices in advance and take necessary measures to avoid such circumstances in future.


Application of Data Mining

Data Mining has been able to grasp the attention of many in the field of scientific research, businesses, banking sector, intelligence agencies and many others from the early days of its inception. However its use was not as easy as it is now. The rapid growth of various tools and software during the past few years enable it to be used more widely than ever before. The ease with which one can carry out complex data mining techniques using these tools is simply outstanding.

Data Mining is used by businesses to do improve its marketing and to understand the buying patterns of its clients. Attrition Analysis, Customer Segmentation and Cross Selling are the most important ways through which data mining is showing the new ways in which businesses can multiply its revenue.

Data Mining is now used in the banking sector for credit card fraud detection by identifying the patterns involved in fraudulent transactions. It is also used to reduce credit risk by classifying a potential client and predicting bad loans.

Data Mining is used by intelligence agencies like FBI and CIA to identify threats of terrorism. After the 9/11 incident it has become one of the prime means to uncover terrorist plots. However this led to concerns among the people as data collected for such works undermines the privacy of a large number of people.



Posted By Piyush Patel
Software Engineer

HP teams with Microsoft, VMware to expand appeal of desktop virtualization solutions

As the sour economy pushes more companies into the arms of virtual desktop infrastructure (VDI) for cost cutting, the vendor community is eagerly wiping out obstacles to adoption by broadening the appeal of desktops as a service for more users, more types of applications and media.

This became very clear this week with a flurry of announcements that package the various parts of VDI into bundled solutions, focus on the need to make rich applications and media perform well, and expands the types of endpoints that can be on the receiving end of VDI.

Hewlett-Packard (HP) expanded its thin-client portfolio with new offerings designed to extend virtualization across the enterprise, while providing a more secure and reliable user experience. The solutions bundle software from Microsoft and VMware along with HP's own acceleration and performance software, as well as three thin client hardware options.

I can hardly wait for HP to combine the hardware and software on the server side, too. I have no knowledge that HP is working up VDI appliances that could join the hardware configurations on the client side. But it sure makes a lot of sense.

Seriously, there are few companies in the better position to bring VDI to the globe, given what technologies they gain with Mercury and Opsware, along with internal development ... Oh, and there's EDS to make VDI hosting a service in itself. Look for a grand push from HP into this enterprise productivity solutions area.

Leading the pack in this latest round of VDI enhancements are the three thin clients -- the HP gt7720 Performance Series, and the HP t5730w and t5630w Flexible Series. These offer new rich multimedia deployment and management functionality -- rich Internet applications (RIA), Flash, and streaming media support -- that enhance the Microsoft Windows Embedded Standard. [Disclosure: HP is a sponsor of BriefingsDirect podcasts.]

The Palo Alto, Calif. company also announced several other new features:

• The HP Remote Desktop Protocol (RDP) Enhancements software that will provide out-of-the-box multimedia and USB device redirection.
  
  
• Collaboration between HP and VMWare to allow HP Remote Graphics (RGS) software to deliver a rich multimedia and collaborative VM virtual desktop experience.
  
  
• HP Client Automation thin-client support, which allows administrators identify issues before they occur and allows them to automatically update administrative requirements.

The thin clients feature Microsoft Internet Explorer 7, Windows Media Player 11 and the ability to run applications locally, they also include Microsoft Remote Desktop Protocol 6.1, which enables devices to connect and take advantage of the latest security and enterprise management technologies from Windows Server 2008.

RDP Enhancements multimedia and USB redirection enable users to easily run web applications, videos and other files within a virtual desktop environment, while avoiding frame skipping and audio or video synchronization issues. The software downloads the processing directly to the thin client, creating an enhanced multimedia experience while lowering the load on the server, which results in increased server scalability.

This also creates a near-desktop experience for VMware View environments, including support for the latest VMware View Manager 3 broker with no need for additional employee training. Users simply log in on the thin client to take advantage of its multimedia features, such as training videos, and USB device support.

HP and VMware also are working together to enable VMware View Manager’s universal access feature to leverage RGS for remote desktop sessions.

RGS is designed for customers requiring secure, high-performance, collaborative remote desktop access to advanced multimedia streaming and workstation-class applications. The software includes expanded, real-time collaboration features to allow multiple workers from remote locations to see and share content-rich visualizations, including 2-D design, 3-D solid modeling, rendering, simulation, full-motion video, heavy flash animation and intense Web 2.0 pages.

Not suprisingly, a lot of the technology being used in these VDI bundles originated with secure CAD/CAM virtual workstation implementations, where graphics and speed are essential. If it works for developers in high-security areas, it should work for bringing ERP apps and help desk apps to the masses of workers who don't need a full PC on every desktop. They just need an interactive window into the apps and data.

Expected to be available in early May, the new thin clients will be priced from $499 to $799. More information is available through HP or authorized resellers or from http://www.hp.com/go/virtualization. I would expect that EDS is going to have some packages that drive the total cost down even more.

Welcome to BI-AJ! Business Intelligence - Agile Initiatives

Welcome to my blog on software development with primary focus on data warehousing and business intelligence projects.

Often data warehouse and business intelligence initiatives use long delivery cycles with tangible results, whilst businesses rapidly change and adjust to the market. The consequence of long delivery cycles usually means that whenever a solution is in place the business has already changed and the effect of the BI initiative is reduced. Agile development methods are especially useful for rapid software development with requirements that are hard to define and within rapidly changing businesses, many times with a high success factor. However, agile development methodologies have not yet been acknowledged as delivery models for developing business intelligence systems. Many claim that small iterations are not suitable for data warehousing because of the demand for a strong and reliable infrastructure. The purpose of this blog is to present that the agile foundations for sure can be utilized with great success for developing and deploying business intelligence systems within small iterations.

Through-out a number of posts I will give my thoughts and ideas on how to develop software in the context of DW/BI using small iterations and a pragmatic perspective on development. I do have clear influences in agile development, however I try to stay pragmatic to rules and structures as all development initiatives are different. They are within different organisations with different cultures and with different goals with different employees and traditions. These are things that need to be considered when planning and deploying software efficiently.

I hope that I can start a debate on how agile principles can or cannot be used to deliver software within DW/BI projects, and I hope that I get feedback and response from you. I will continuously discuss and solve(?) any questions or scenarios that I received from you.

Stay tuned!