We have developed a general architecture for Web usage mining which is presented in [MJHS96] and [CMS97]. The WEBMINER is a system that implements parts of this general architecture. The architecture divides the Web usage mining process into two main parts. The first part includes the domain dependent processes of transforming the Web data into suitable transaction form. This includes preprocessing, transaction identification, and data integration components. The second part includes the largely domain independent application of generic data mining and pattern matching techniques (such as the discovery of association rule and sequential patterns) as part of the system's data mining engine. The overall architecture for the Web mining process is depicted in Figure 2.
Data cleaning is the first step performed in the Web usage mining process. Any of the cleaning techniques discussed in section 3.1.1 can be used to preprocess a given Web server log. Currently, the WEBMINER system uses the simplistic method of checking filename suffixes. Some low level data integration tasks may also be performed at this stage, such as combining multiple logs, incorporating referrer logs, etc.
After the data cleaning, the log entries must be partitioned into logical clusters using one or a series of transaction identification modules. The clean server log can be thought of in two ways; either as a single transaction of many page references, or a set of many transactions each consisting of a single page reference. The goal of transaction identification is to create meaningful clusters of references for each user. Therefore, the task of identifying transactions is one of either dividing a large transaction into multiple smaller ones or merging small transactions into fewer larger ones. This process can be extended into multiple steps of merge or divide in order to create transactions appropriate for a given data mining task. A transaction identification module can be defined as either a merge or a divide module. Both types of modules take a transaction list and possibly some parameters as input, and output a transaction list that has been operated on by the function in the module in the same format as the input. The requirement that the input and output transaction format match allows any number of modules to be combined in any order, as the data analyst sees fit. The WEBMINER system currently has reference length, maximal forward reference, and time window divide modules, and a time window merge module.
Access log data may not be the only source of data for the Web mining
process. User registration data, for example, is playing an increasingly
important role, particularly as more security and privacy conscious
client-side applications restrict server access to a variety of
information, such as the client user IDs. The data collected through user
registration must then be integrated with the access log data. There are
also known or discovered attributes of references pages that could be
integrated into a higher level database schema. Such attributes could
include page types, classification, usage frequency, page meta
information, and link structures. While WEBMINER currently does not
incorporate user registration data, various data integration issues are
being explored in the context of Web usage mining. For a study of data
integration in databases see [LHS
95]. Once the domain-dependent
data transformation phase is completed, the resulting transaction data
must be formatted to conform to the data model of the appropriate data
mining task. For instance, the format of the data for the association rule
discovery task may be different than the format necessary for mining
sequential patterns.
Finally, a query mechanism will allow the user (analyst) to provide more
control over the discovery process by specifying various constraints. The
emerging data mining tools and systems lead naturally to the demand for a
powerful data mining query language, on top of which many interactive and
flexible graphical user interfaces can be developed [HFW
96]. Some
guidelines for a good data mining language were proposed in
[HFW96], which among other things, highlighted the need for
specifying the exact data set and various thresholds in a query. Such a
query mechanism can provide user control over the data mining process and
allow the user to extract only relevant and useful rules. In WEBMINER, a
simple Query mechanism has been implemented by adding some primitives to
an SQL-like language. This allows the user to provide guidance to the
mining engine by specifying the patterns of interest.
As an example, consider a situation where the user is interested in the patterns which start with URL A, and contain B and C in that order, this pattern can be expressed as a regular expression A*B*C*. To see how this expression is used within a SQL-like query, suppose further that the analyst is interested in finding all such rules with a minimum support of 1 % and a minimum confidence of 90 %. Moreover, assume that the analyst is interested only in clients from the domain .edu, and only wants to consider data later than Jan 1, 1996. The query based on these parameters can be expressed as follows:
SELECT association-rules(A*B*C*)
FROM log.data
WHERE date >= 960101 AND domain = "edu" AND
support = 1.0 AND confidence = 90.0
This information from the query is used to reduce the scope, and thus the cost of the mining process. The development of a more general query mechanism along with appropriate Web-based user interfaces and visualization techniques such as those discussed in section 4, are planned in the future revisions of the WEBMINER system.
