Table of Contents
Abstract
Introduction
Technologies
Data Warehouse
Examples
Characteristics
Platforms
Summary
|
|
5.0 Platforms for the Data Warehouse and Data Mining
Data mining technologies are characterized by intensive computations on
large
volumes of data. Significant processing power is critical, and parallelism
is a
key to enabling significant data mining. The systems should scale so that,
as
the demands for analysis grow, the system can be upgraded to provide the
necessary analysis in a timely and cost- effective fashion.
Clearly, a balanced system architecture that supports I/O, computation and
scaling in a cost-effective fashion is desirable. Systems used for
updating
operational data can be distinct from those used for the data warehouse,
so the
current enterprise investment in database and transaction platforms need
not be
wasted.
The requirements placed on a system for data mining are a superset of
those for
the data warehouse. Large main memories are necessary for good
performance.
Good memory management in the multiprocessor system is needed to ensure
that
thrashing at any level of the memory hierarchy does not result in
inadequate
performance.
The larger the volume of data that can be processed by data mining, the
greater
the confidence in the results. Hence, the highest capacity and performance
systems are of interest in this area. Of course, price (as always) is a
consideration.
There are two broad, alternative parallel architectures available now:
1. Massively parallel processing (MPP) systems
Examples are Teradata from AT&T and SP-2 from IBM.
2. Symmetric multiprocessing (SMP) systems
Examples are CHALLENGE servers from Silicon Graphics and servers from HP
and Sequent.
Until now, doubts about their ability to meet the demand of data
warehousing
and mining have kept SMP systems from widespread use for these purposes.
Today,
however, the high performance, symmetric multiprocessing CHALLENGE line of
servers from Silicon Graphics meets all these requirements. Unlike MPP
systems,
these servers are also excellent choices for general purpose use, thus
helping
to amortize the investment in system capacity.
The CHALLENGE servers provide the necessary system capabilities and can
scale
to meet increasing demands. The architecture and operating system are
tuned to
effectively support parallelism. Silicon Graphics has gained valuable
experience with the demands placed by data-intensive applications and has
tailored its systems to meet these demands.
The high-end CHALLENGE XLTM server can scale up to 36 processors (each 100
MIPS) with 16 gigabytes (GB) of main memory. It has 32 I/O channels where
each
I/O channel can support up to 20 MB per second. The server has sustained
more
than 500 MB per second of aggregate I/O bandwidth. Using RAID technology,
a
single server can support more than 3.2 terabytes (TB) of data storage.
Using 4
GB disk drives, a single server can have up to 2 TB of data storage, and
using
RAID technology, it can store more than 6 TB of data. Other CHALLENGE
servers,
such as the CHALLENGE L and CHALLENGE DM, are also suitable platforms for
data
mining.
The Oracle 7.1 RDBMS is a good choice as the data warehouse. The parallel
loading capabilities of Oracle 7.1 make it possible to populate and
refresh the
repository in a timely fashion. The parallel query option enables quick
retrieval of data from the warehouse for mining. The parallel query option
can
also be used for the complex queries that would be suggested by the data
mining
analysis. We have obtained a scaling factor of 11.5 with 16 CPUs running
complex queries.
Silicon Graphics has a close working relationship with Oracle Corporation
to
ensure that the system as a whole performs well. For example, a 200 GB
database
was installed at Silicon Graphics and the performance of loading, index
creation and ad hoc querying using Oracle 7.1 was studied and tuned.
Dramatic price/performance benefits can be gained by deploying the
CHALLENGE
servers for the data warehouse or data mining. As data points, Silicon
Graphics
CHALLENGE running Oracle 7 has demonstrated the best price/performance on
the
TPC-A and TPC-B benchmarks in the open systems market.
Silicon Graphics has developed similar relationships with other database
vendors, such as Sybase and Informix, and is committed to making the
CHALLENGE
servers the platforms of choice for data warehousing solutions based on
those
database systems.
|
|
