Table of Contents
Abstract
Introduction
Technologies
Data Warehouse
Examples
Characteristics
Platforms
Summary
|
|
2.0 Data Warehouse
Collecting the data to mine is a difficult process by itself. While
operational
data is often collected for business processes, once the immediate use of
the
data has passed, the data is often archived. Such data can now be
exploited for
data mining; however, a suitable system infrastructure requires high I/O
bandwidth for fast loading and parallelism to reduce the time for initial
loads
or updates.
Figure 1 illustrates the data sources and processes leading to the data
warehouse. The parallel streams indicate that cost-effective, scalable
parallelism is a critical technology for the data warehouse.
The data extraction process extracts useful subsets of data for mining.
Sampling may limit the size of the extracted data, and selection
predicates may
be applied to focus on applicable subsets of data. Aggregation may be done
if
summary statistics are useful.
Operational (or transaction) data is often not in the most suitable form
for
mining. The data needs to be cleaned, with incorrect fields being
corrected or
otherwise marked. This cleaning process can be quite sophisticated, with a
lot
of business knowledge being encoded in the cleaning rules. Fields that are
not
necessary for the analyses may be omitted in order to reduce storage
requirements and speed access for analysis. Some degree of normalization
may be
performed to reduce the amount of redundant data. Conversely, some
denormalization may be performed to speed up data access. Additional
fields may
be added using "overlays". For example, demographic overlays are commonly
used
in marketing research. Matching rules are needed for the overlays, and
these
rules are specific to the business domain.
|
|
