Data Mining notes: 7th semester.

CS 1435
Syllabus:
Structured: tables

Semi structured : tables, images

Unstructured: videos and stuff

Statistics: a branch of science dealing with collection of structured data oin huge amount.

Hypothesis: it is a study of data received. A statement and its outcome(See def.)

Null hypothesis: one variable doesn’t affect other.

------- hypothesis: one variable may affect other

Apart from AI, ML, stats, it also includes amths, dbms, etc

KDD- Knowledge discovering in db

Data cleaning – to remove the noise/ irrelevant data

uniform data in terms of key and data attributes

De-duplication of records – redundant data removed and at times replaced by key (according to
need of user)

Data selection – extract pattern and knowledge, domain experts and team member and managers
decide

Data transformation – modification of data so that it can be evaluated. Eg. Some may write
rs/inr/etc/…

Data descrtisation – numerical data deal

Missing data – like I wont give my data to salesperson of my income, or entry time maybe.

Clean – fill misiing val, identify outliers

Ignore the tuple – when a % of data missing, then only ignore

Filling missing data not always feasible

Use a global constant to fill by like unknown, or NA, or ?

Probable value using decision tree or Bayesian theory

Binning method – sort the data, partition them into bins (like equal width(distance) or equal
depth( frequency) bins), smooth by bin means.

Clustering – detect and remove the outliers

Combined computer and human inspection – detect suspicious method

Regression – use regression functions while feeding data

Equal width = W = (B-A)/N B – highest value, N is interval size

Equal depth - Divides the range into n intervals, contains approx. same no. of samples

Assignment – download 1 noisy data set. Perform binning for data smoothing, check which method
is good in terms of time

Bin data: 13, 15, 16, 16, 19, 20, 20, 21, 22, 22, 25, 25, 25, 25, 30, 33, 33, 35, 35, 35, 35, 36, 40, 45, 46,
52, 70, USE DS to place in the number of bins. (like bucket sort).

a)

Data cleaning overall:

 Discrepancy detection – instrumntatition, human, outdated, inconsistecy

Unique rule – each value must be different for a particular attribute, consecutive rule – there
can be no missing value in between min and max value, null rule – specifies
blanks/?/sp.char/….. not available.use a global const
 Tools – data scrubbing tool – useful in classification, data auditing tool, etl – transform
through a graphical user interface

Assignment 2: tool – explore them whether open source or commercial, salesforce.com explore all
tools. And classify as opensource or commercial. State one one line on each. Take dta from previous
assignment, and use any 2 tools. Observation – write features available, and then tell output and tell
which is better.

Data integration : convert same data to some standard. Like 1hr and 60min. unique. Cm to m to vice-
versa

Handling redundant data: chi-square test

Correlation test for nomina: A,B data tuples, o = observed, e=expected, n = no.of tuples. I from 1 to
c, j from 1 to c

Eg 3.1 hellen kember data mining book

Data transformation: smoothing (regression, clustering, binning)

Attribute construction(feature construction) – like clg level student db. Tc what is data problem and
acc. Select attributes

Aggregation – summarization. Eg. Monthly attencdnace, monthly income, calc monthly, then
calculate for year

Normalization – scale it down acc to size of data. Acceptance of paper

Discretization – place at some range level. Like 1st yr students age 17-18.

Concept hierarchy generation – dept, nit, silchar, assam, india. Instead we can write silchar, assam,
india.

Min-max normalization – v = observed vlue. A =attribute, min A = minimum value for a particular
attribute A. new_maxA max of what we want to normalize it to

Min-max norm protects the out of bound values. It can detect out of bound errors

Eg3.4

z-score normalization –

data reduction strategies: it should closely maintain integrity of original data. Wavelet: for
continuous data.

Parameter – paratmeter of data is stored not totally data….non-para – clustering/regression etc

used.

Lossy – lossless: data loss or not on getting back original data

CLASSIFICATION: to analyze the measurement of an object to identify the category to which it lies

Learning state: we describe state of predetermined classes

Classification – for classifying future or unknown object

Eg. If a customer is good to give a loan or not. So customer is safe or not

Decision tree induction: classifier in the form of tree.

Learning of decision tree from class level . so greedy strategy needed.

Id3: iterative determiser – decision tree algo

C4.5: successor of id3

Cart : classification and regression tree.

Built decision tree algo: to select branch : choose most suitable attribute, then create data structure

Entropy: summation(-pi log2 (pi) ===== summation pi log2 (p -i)) therefore a negative value

E(x) = -