Research on Data Mining

It’s time for some holidays at DMR. I will be back in the blogosphere in the beginning of August. In the meanwhile, here are some links that may interest you:

• Web Analysis, Behavioral Targeting and Advertising: if you wants to know anything about online advertising and BT
• 10 Red Hot BI Trends: and don’t be afraid, predictive analytics is among them (from KDnuggets)
• Data Mining: An Ill-defined Concept: a short and interesting article about the term data mining, from the statistician point of view

Hope to see you soon on Data Mining Research.

SAS tutorial - generation of a sample business datawarehouse scenario

This tutorial shows how to use SAS to implement ETL process which generates a star schema datawarehouse architecture.

We assume that you already have basic SAS/BASE knowledge and are familiar with SAS environment, assigning libraries, running SAS programs.

The aim of this tutorial is to generate a datawarehouse structure which would help monitor performance of a sample business scenario described here:
A manufacturing company Data Warehouse Business Scenario - requirements from a palm and tropical plants nursery which implements an analysis and reporting system to track sales, costs, forecasts and business performance management figures.

Tutorial overview:
- First step will be to read dimensions and populate sample dimensions data
- Then a fact table will be created.
- In the next step we will randomly generate transactions for the fact table with sales data for three years. To generate the numbers we will use SAS random number generators with uniform and random distributions.
- The final tasks will be to validate and extract generated data and feed the reporting application.

Please also be aware of the fact that SAS is very powerful and flexible system and the things we show in this tutorial can be done in many different ways. It is just one way to get the expected results.

SAS tutorial chapters

1. Load extracts into SAS - A couple of programs responsible for loading the dimensions extracts into SAS and an example on how to create dynamically additional dimension tables

2. Populate dimensions - Example of how to populate random dimensions in a fact table

3. Generate measures - Random generation of a given set of measures. The measures are generated randomly, however they apply business rules described in a business scenario

4. Sales fact table - In that lesson we create a fact table with sales figures designed in a star schema datawarehouse architecture. Additionally, we perform a statistical analysis of the newly generated data

5. Costs fact table - We randomly generate a fact table with costs. Costs are divided into fixed and variable costs and allocated on year and month detail level

6. SAS ETL Process - Run the whole process in a sequence which may be considered as a representation of ETL Process in SAS. The process could also be set up in SAS ETL Studio or SAS Warehouse Administrator

Header and trailer structured textfile processing in SAS - Example from the Data Warehousing Tutorial

Data Warehousing ETL tutorial

The ETL and Data Warehousing tutorial is organized into lessons representing various business intelligence scenarios, each of which describes a typical data warehousing challenge.
This guide might be considered as an ETL process and Data Warehousing knowledge base with a series of examples illustrating how to manage and implement the ETL process in a data warehouse environment.

The purpose of this tutorial is to outline and analyze the most widely encountered real life datawarehousing problems and challenges that need to be taken during the design and architecture phases of a successful data warehouse project deployment.

The DW tutorial shows how to feed data warehouses in organizations operating in a wide range of industries.
Each provided topic is thoroughly analyzed, discussed and a recommended approach is presented to help understand how to implement the ETL process.

Going through the sample implementations of the business scenarios is also a good way to compare BI and ETL tools and get to know the different approaches to designing the data integration process. This also gives an idea and helps identify strong and weak points of various ETL and data warehousing applications.

This tutorial shows how to use the following ETL and datawarehousing tools: Datastage, SAS, Pentaho, Cognos and Teradata.

Data Warehousing & ETL Tutorial lessons

• Sample design in Pentaho Data Integration" href="http://etl-tools.info/en/examples/generate-surrogate-key.htm">Surrogate key generation example which includes information on business keys and surrogate keys and shows how to design an ETL process to manage surrogate keys in a data warehouse environment. Sample design in Pentaho Data Integration
  
• Solution examples in Datastage, SAS and Pentaho Data Integration" href="http://etl-tools.info/en/examples/header-trailer-processing.htm">Header and trailer processing - considerations on processing files arranged in blocks consisting of a header record, body items and a trailer. This type of files usually come from mainframes, also it applies to EDI and EPIC files. Solution examples in Datastage, SAS and Pentaho Data Integration
  
• Sample loading in Teradata MultiLoad" href="http://etl-tools.info/en/examples/ftp-load-extract.htm">Loading customers - a data extract is placed on an FTP server. It is copied to an ETL server and loaded into the data warehouse. Sample loading in Teradata MultiLoad
  
• Sample Cognos implementation" href="http://etl-tools.info/en/examples/data-allocation.htm">Data allocation ETL process case study for allocating data. Sample Cognos implementation
  
• Sample Kettle implementation" href="http://etl-tools.info/en/examples/data-masking.htm">Data masking and scambling algorithms and ETL deployments. Sample Kettle implementation
  
• Sample design in Pentaho Kettle" href="http://etl-tools.info/en/examples/site-traffic-dw.htm">Site traffic analysis - a guide to creating a data warehouse with data marts for website traffic analysis and reporting. Sample design in Pentaho Kettle
  
• Sample outline in PDI" href="http://etl-tools.info/en/examples/data-quality.htm">Data Quality Tests - ETL process design aimed to test and cleanse data in a Data Warehouse. Sample outline in PDI
  
• XML ETL processing
  

Etl Tools Info portal

ETL-Tools.Info portal provides information about different business intelligence tools and datawarehousing solutions, with a main focus on ETL process and tools. On our pages you will find both general articles with high-level information on various Business Intelligence applications and architectures, as well as technical documents, with a low-level description of the presented solutions and detailed tutorials.
A great attention is paid to the Datastage ETL tool and we provide a number of Datastage examples, Datastage tutorials, best practices and resolved problems with real-life examples.
There is also a wide range of information on a rapidly growing Open Source Business Intelligence market (OSBI), with emphasis on applications from the Pentaho BI family, including a Pentaho tutorial.
We also provide a SAS Guide with tutorial, which illustrates the vision of SAS on Business Intelligence, Data Warehousing and ETL process.
We have recently added a new ETL case study (ETL course with examples) section which represents a set of business cases, each of which illustrates a typical data warehousing problem. We analyze the cases thoroughly and propose the most efficient and appropriate approach to solving that problems by showing sample ETL process designs and DW architectures.
Microsoft users may be very interested in exploring our Excel BI crosstabs section with FAQ and sample solutions.

What is Business Intelligence?

Business intelligence is a broad set of applications, technologies and knowledge for gathering and analyzing data for the purpose of helping users make better business decisions.
The main challenge of Business Intelligence is to gather and serve organized information regarding all relevant factors that drive the business and enable end-users to access that knowledge easily and efficiently and in effect maximize the success of an organization.

Business intelligence produces analysis and provides in depth knowledge about performance indicators such as company's customers, competitors, business counterparts, economic environment and internal operations to help making effective and good quality business decisions.

From a technical standpoint, the most important areas that Business Intelligence (BI) covers are:

DW - Data warehousing - architecture, modeling, managing, processing ETL process and data integration Reporting, Information visualization and Dashboards OLAP - Online Analytical Processing and multidimensional analysis Data cleansing and data quality management Performance management Data mining, statistical analysis, forecasting MIS - Management Information Systems CRM - Customer Relationship Management

Data Warehouse, Data Mart, Data Mining, and Decision Support Resources

Featured Resources

First Place Learning : Data Warehouse, Data Mart, Data Mining, and Decision Support

Additional Resources

KDNuggets : Data Mining and Knowledge Discovery Resource center ****

Alacrity, Inc. : Integrated Data Intelligence Software

Allen, Davis, and Associates : Data Warehousing Career Newsletter

AlphaBlox : Data Analysis Software

AltaPlan : OLAP Links

Aonix : Object Oriented Modeling Tool and Cleansing Software

Attar Software : Data Mining / Neural Nets

Bill Inmon : Leading Data Management and Data Warehouse Speaker and Writer

Brio Technologies : Brio Web Warehouse and Decision Support Suite

Bull : Data Warehousing Solutions

Business Intelligence : The OLAP Report -- Richard Creeth

Business Objects, Incorporated : WebIntelligence for enterprise decision support

Cognos, Incorporated : Data Warehousing Software Tool Suite

CDI : Creative Data Inc : Data Warehouse Consulting and Training -- good Data Warehouse Links

D2K, Incorporated : "Turning Data into Knowledge"

DataFlux : Data Quality and Integration Software

DataMirror : Data Integration, Data Protection, Data Audit Solutions

datawarehouseconsulting.com : Data Warehouse Consulting

datawarehousing.com : Data Warehousing Portal

Data Warehousing Institute : Conferences and whitepapers

Decision Point Applications : Packaged Data Warehouse Solutions

Decision Technology : DecisionCentric® Server

Decision Works : Data Warehouse Consulting and Education

Decision Works Ltd : TRACK Objects software and consulting

Dimensional Insight, Inc. : Reporting and analysis software

DM Review : Leading Data Management Industry Publication

Don Meyer & Associates : Data Warehousing Consultants

DW Soft : Data Warehouse Software and Service using Microsoft Repository

Epsilon Data Management : Databased Marketing Services and Training

Evolutionary Technologies, Inc. : ETI*EXTRACT(r) Tool Suite for Data Warehousing and Data Migration

FileTek : Software for managing massive amounts of atomic data

First Logic : Customer Data Management Software

Hyperion : ESSBASE - High Speed OLAP Processor

IBM : DataGuide

Informatica : The Data Mart Company

Information Builders : Data management software
Data Warehousing, Decision Support, Middleware, Data Access, ...

Intelligent Solutions, Inc. : Claudia Imhof / Data Warehousing and Data Modeling

IRI : CoSORT ETL Software

Kalido : Software for adaptive enterprise data warehousing and master data management

Kenan Systems Corporation : Market Analysis Software

Megaputer Intelligence : Data Mining and Warehousing

Micro Strategy : Relational OLAP (ROLAP) Software and Services

MiningCo : Data Mining plus excellent data management articles

Nautilus Systems, Inc. : Data Warehousing, Data Mining, and Data Visualization software

netcarve Technologies GmbH : Data Warehousing and Data Mining Solutions

NetScheme Solutions Inc.

Open Technologies, Inc. : Data Warehouse Consulting and Recruiting Specialists

OSMC : Data Warehousing Consultants

Paralogic, Inc. : Data Warehousing and Decision Support Consulting / Lexington, MA

Perl.Com : Perl is a scripting language useful for extracting and loading data

Pervasive : Data Integration

Pilot Software : Customer and Market Data Analysis Software

Poinpoint Solutions Inc. : Data Warehouseing solutions for the insurance industry

PLATINUM technology, inc. : Data management software and consulting

Princeton Softech : Database Management and Data Warehouse Software

Query Object Systems Corp : Business Solution Components

Ralph Kimball Associates : A pioneer and leader in the Data Warehouse field

Redbrick Systems : Multidimensional database software

Redbrick Whitepaper : Server Requirements

Retek : Data Warehousing for Retail Industry

Rocket Software : Business Intelligence

Rulequest Research : Data Mining Tools

saleslobby.com : Whitepaper - Building the Customer Data Warehouse

Salford Systems : CART software for tree-structure, non-parametric data analysis

SAS Institute : Data Warehouse and Data Mining Software

Seagate Software : Crystal Reports

Silvon Software : Supply Chain Data Warehousing

SolutionsIQ : Data Warehousing Solutions

Speedware Corp :Business Intelligence Software

Sybase, Inc. : Data Warehousing Database Software

Teleran : Data Warehousing and eCommerce Solutions

Teradata / NCR : Database machine

Software AG : Data modeling and data warehouse course outlines

Thinking Machines Corporation : Data mining software for loyalty management systems

Trillium Software : Data Cleansing and Data Reengineering

Universal Data Solutions, LLC : Len Silverston - Data Modeling and Data Warehouse - Coauthor of 'The Data Model Resource Book'

Datasets for Association Rule Mining

A normal transaction consists of a transaction-id and a list of items in every row or sentence. Sometimes, the items are represented as boolean values 0 if the item is not bought, or 1 if the item is bought. But the commonly used format for Market Basket data is that of numeric values for items without any other information:
1 3 5 9 11 20 31 45 49
3 7 11 12 15 20 43...
This format has to be converted in order to be used by ARMiner and ARtool, since those tools can only evaluate binary data. ARMiner and ARtool have a special converter for that purpose which have to be performed before analyzing the data. WEKA needs a special ASCII-Data format (*.arff) for data analysis containing information about the attributes and a boolean representation of the items. Since there is no unique format for input-data, it is impossible to evaluate the same dataset in one format with different tools. In this paper, we present a dataset generator that is able to generate datasets that are readable by ARMiner, ARtool,WEKA and other data mining tools. Additionally, the generator has the ability to produce large Market Basket datasets with timestamps to simulate transactions in both retail and e-commerce environments.