Describe the steps and their purposes in knowledge discovery from databases

Describe the steps and their purposes in knowledge discovery from
databases.

The field of knowledge discovery in databases (KDD) is getting to be very
popular and has grown recently. The large amounts of data collected and
stored might contain some information, which could be useful, but it is not
obvious to recognise, nor trivial to obtain it. There is nothing new about
analysing data, but it is in the amount of data, where traditional methods
are becoming inefficient. There is no human capable to sift through such
amounts of data and even some existing algorithms are inefficient when
trying to solve this task. KDD systems incorporate techniques from large
variety of related fields to utilise their strengths in process of
discovering knowledge.

The main idea in KDD is to discover a high level knowledge (abstract
knowledge) from lower levels of relatively raw data, or to discover a
higher level of interpretation and abstraction than those previously known.
The steps involved are as follow:

- Task Discovery is one of first steps of KDD. Client has to state the
problem or goal, which often seems to be clear. Further investigation
is recommended such as to get acquainted with customer's organisation
after spending some time at the place and to sift through the raw data
(to understand its form, content, organisational role and sources of
data). Then the real goal of the discovery will be found.

- Data Discovery is complementary to step of task discovery. In the step
of data discovery, we have to decide whether quality of data is
satisfactory for the goal (what data does or does not cover).

- Data Cleaning is often necessary though it may happen that something
removed by cleaning can be indicator of some interesting domain
phenomenon (outlier or key data point). Analyst's background knowledge
is crucial in data cleaning provided by comparisons of multiple
sources. Other way is to clean data before loaded into database by
editing procedures. Recently, the data for KDD are coming from data
warehouses which contain data already cleaned on some way.

- Data Integration is a platform where multiple data sources may be
combined.

- Data Selection is where data relevant to the analysis tasks are
retrieved from the database.

- Model Development is an important phase of KDD that must precede
actual analysis of the data. Interaction with the data leads analysts
to formation of hypothesis (it is often based on experience and
background knowledge). Sub-processes of model development are:

- data segmentation (unsupervised learning techniques, for example
clustering);
- model selection (choosing the best type of model after exploring
several different types);
- parameter selection (parameters of chosen model).

- Data Mining is in general an ambition to understand why certain groups
of entities are behaving on the way they do, it is search for laws or
rules of such behaviour. As first should be analysed those parts where
such a groups are already identified. Sub-processes in data analysis
are:

- model specification - some formalism is used to denote specific model;
- model fitting - when necessary the specific parameters are determined
(in some cases the model is independent from data in other cases the
model has to be fitted to training data);
- evaluation - model is evaluated against the data;
- model refinement - model is refined in iterations according to the
evaluation results.


As mentioned above the model development and data mining are complementary
so it often leads to oscillation between those two steps.

Output Generation - output can be in various forms. The simplest form
is a report with analysis results. The other, more complicated forms,
are graphs or in some cases it is desirable to obtain action
descriptions which might be taken directly as outputs. Or there should
be a monitor as the output, which should trigger an alarm or action
under some certain condition. Output requirements might determine task
of designed KDD application. The knowledge is used and incorporated
into another system for further actions. Also, reports can be made
for presentation to the users.



Briefly discuss whether or not each of the following activities is a data
mining task.

Dividing the customers of a company according to their profitability

This is a data mining task as the customers have to be selected first
and then classified accordingly.

Computing the total sales of a company

Not data mining task since sales will be recorded in the sales
department and there need not be any data analysis to determine
whether its sales or not.

Sorting a student database based on student identification numbers

Yes, it is a data mining task as each cohort will have students and
each student a different ID number. So, selection and data mining
needs to be done.

Predicting the outcomes of tossing a fair pair of dice

No, this is not a data mining task as it is purely probabilistic
property.

Predicting the future stock price of a company using historical
records.

This is a data mining task as based on past information, the future
price of stock can be evaluated, that is, whether it is more probable
to decline or rise.