You are on page 1of 21

Assignment: 1

Artificial Neural Network

Group members and Data Sets

CSC/14/51 – Nursery Data set (page 02-09)
CSC/14/05 - Thyroid Disease Dataset (page 10-16)
CSC/14/22- Wine Data set (page 17-20)

Performance Analysis of Different classifiers on WEKA

1|Page

Assignment: 1

Artificial Neural Network

Introduction
Gathered data sets are include valuable information and knowledge which is often hidden. Processing the huge data
and retrieving meaningful information from it is a difficult task. The aim of our work is to investigate the performance
of different classification methods using WEKA for different three dataset obtained from UCI data archive.
WEKA is an open source software which consists of a collection of machine learning algorithms for data mining tasks.
This assignment is to investigate the performance of different classification or clustering methods for a set of large
data set.

Materials and methods
We have used the popular, open-source data mining tool Weka (version 3.6.6) for this analysis. Three different data
sets have been used and the performance of a comprehensive set of classification algorithms (classifiers) has been
analyzed. The analysis has been performed on a Mac book pro with Intel® i5 CPU, 2.24 GHz Processor, OSX
Yosemite and 4.00 GB of RAM. The data sets have been chosen such that they differ in size, mainly in terms of the
number of attributes.
For this study the following
Data sets were used:
a) Nursery Database, which is developed to rank applications for nursery schools for providing certain facilities,
based on three factors.
 Occupation of parents and child's nursery
 Family structure and financial standing
 social and health picture of the family
Under this study there was 12960 samples (instances) were analyzed against eight attributes which are,
 parents
:
usual, pretentious,
great_pret
 has_nurs :
proper, less_proper,
improper,
critical, very_crit
 form
:
complete,
completed,
incomplete,
foster
 children
:
1,
2,
3,
more
 housing
:
convenient,
less_conv,
critical
 finance
:
convenient,
inconv
 social
:
non-prob,
slightly_prob, problematic and
 health
:
recommended, priority,
not_recom.
Classifiers were used:
A total of five classification procedures have been used for this performance comparative study. The
classifiers in Weka have been categorized into different groups such as Bayes, Functions, Lazy, Rules, Tree
based classifiers etc. The following sections explain a brief about each of these procedures/algorithms.
i.

ii.

iii.

Multilayer Perceptron: Multilayer Perceptron is a nonlinear classifier based on the Perceptron. A Multilayer
Perceptron (MLP) is a back propagation neural network with one or more layers between input and
output layer.
A Support Vector Machine (SVM): SVM is a discriminative classifier formally defined by a separating hyper
plane. In other words, given labeled training data (supervised learning), the algorithm outputs an optimal
hyper plane which categorizes new examples.
J48: The J48 algorithm is WEKA’s implementation of the C4.5 decision tree learner. The algorithm uses a
greedy technique to induce decision trees for classification and uses reduced-error pruning.

2|Page

Assignment: 1 Artificial Neural Network iv.1843 13. Step 3: Note the accuracy given by it and time required for execution.9475 0.4739 0.8567 0.0186 0.5261 2.7947 28.0186 0. Certain comparative studies were conducted and following factors were derived. k-NN is a type of instance based learning or lazy learning where the function is only approximated locally and all computation is deferred until classification.9483 0.6718 9. Step 2: Apply the classifier algorithm on the whole data set.5647 0.9761 0. v.3796 1.1512 31. Under this study I have used two types of test mode which are 10-fold cross-validation and percentage split 66%.62 97. the word Naive means strong.0514 2.5721 26.2706 40. Steps to apply classification techniques on data set and get result in Weka:      Step 1: Take the input dataset.0988 3.4353 2.077 0.9314 0.5617 2.7324 0 98.6204 0.0765 0.0951 5.1466 31.14 96.0525 2.185 47.0153 0.006 0.962 0.0854 0.1766 28.8618 0.03 90. assuming attributes as statistically independent.7298 0.6151 25.8152 Table 1 Results summary of 10 fold cross validation Classification Multilayer Perceptron Support Vector Machine J48 k-nearest neighbor Naive Bayesian Time taken seconds Correctly Classified Instances Incorrectly Classified Instances Kappa statistic Mean absolute error Root mean squared error Relative absolute error Root relative squared error 69.3282 0.6775 0 90. In this algorithm an object is classified by a majority vote of its neighbors.5491 0 97.7877 Table 2Results summary of 66% split 3|Page .5647 0.5218 5.0098 0.0514 2.03 97.0233 14.996 0.9636 0.962 0.56 99.23 97.0859 0. Naive Bayesian: Naive Bayesian classifier is developed on bayes conditional probability rule used for performing classification tasks. IBk: IBk is a k-nearest-neighbor classifier that uses the same distance metric.9063 0.4383 0.7299 0.9641 0. Classification Multilayer Perceptron Support Vector Machine J48 k-nearest neighbor Naive Bayesian Time taken seconds Correctly Classified Instances Incorrectly Classified Instances Kappa statistic Mean absolute error Root mean squared error Relative absolute error Root relative squared error 69.1767 28. Step 4: Repeat step 2 and 3 for different classification algorithms on different datasets.0234 47.006 0.28 97.4353 2.4821 3. We have used the 'Explorer' option of the Weka tool.3241 9.0014 0.9568 0.5179 0.1055 6. Step 5: Compare the different accuracy provided by the dataset with different classification algorithms and Identify the significant classification algorithm for particular dataset Results and Discussion The data sets have been submitted to a set of classification algorithms of Weka.9063 8.474 39.2701 0.6759 0. All attributes of the data set are considered as independent and strong of each other.1843 13.

02 0 Multilayer Support Perceptron Vector Machine J48 Cross validation 10 4|Page k-nearest neighbor Naive Bayesian 66% split 1. 66% split Capa Statistics Capa statistics coefficient is a statistical measure of inter-rater agreement or inter-annotator agreement for qualitative (categorical) items. the graph shows greater deviation from each test mode and cross validation gives better results under.01–0.04 0. The multilayer perception shows relatively low absolute error from others and J48 shows average error rate.41–0.95 0.but comparatively the cross validation method take more time than percentage split. The naïve Bayesian shows low performance in both test mode. It gives better results under the 66% split test mode.60 Moderate agreement 0.61–0. the 66% split take short time for SVM method of classification .06 0. From capa we can come to this conclusion. .8 0.08 0. Multilayer perception shows low absolute error under cross validation training mode.81–0.9 0. the multilayer perception classifier.40 Fair agreement 0.21– 0.75 Multilayer Support Perceptron Vector Machine J48 Cross validation 10 k-nearest neighbor Naive Bayesian 66% split The MAE measures the average magnitude of the errors in a set of five classes.1 0. If we consider the following graph the k-nearest neighbor classifier shows high mean absolute error than other classifier.80 Substantial agreement 0. Under multilayer perception. Incorrectly Classified Instances 12 10 8 6 4 2 0 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Cross validation 10 Naive Bayesian 102 100 98 96 94 92 90 88 86 84 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Cross validation 10 Naive Bayesian 66% split If we consider the incorrectly identified instances. When considering the two training modes there are no big deviation from each other except the multilayer perception.85 0.05 1 0.99 Almost perfect agreement Mean absolute error 0.       < 0 Less than chance agreement 0.Assignment: 1 Artificial Neural Network When considering time consuming for five classifiers under two testing sample methods.20 Slight agreement 0. All together all classifier shows relatively similar results except the naive Bayesian classifier. Time Taken in seconds 80 60 40 20 Correctly Classified Instances 0 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Cross validation 10 Naive Bayesian 66% split Correctly identified instances are showing better results under cross validation test mode. again the split validation shows poor performance than cross validation test mode.

considerably the multilayer perception classifier showing good results.Assignment: 1 Artificial Neural Network Root mean squared error /Relative absolute error /Root relative squared error for Cross validation Root mean squared error /Relative absolute error /Root relative squared error for 66% split 60 50 40 30 20 10 0 60 50 40 30 20 10 0 Multilayer Support Perceptron Vector Machine RMs J48 RAE k-nearest neighbor Multilayer Support Perceptron Vector Machine Naive Bayesian RMs RRSE J48 RAE k-nearest neighbor Naive Bayesian RRSE Above two graphs are showing comparison of different error parameters. it means lower error rate. The lower level showing good performance and higher percentage showing lower performance.0% train. but k-nearest and naïve Bayesian are showing high amount of error in determining the five classes. remainder test 100% 80% 60% 40% 20% 0% Time taken seconds Correctly Classified Instances Multilayer Perceptron Incorrectly Classified Instances Kappa statistic Support Vector Machine Mean absolute error J48 Root mean squared error k-nearest neighbor Relative absolute error Root relative squared error Naive Bayesian The above two graphs are showing the compared performance matrices of classifiers in percentage. The close look of these graphs are showing no significant changes between the parameters. Also if we consider training mode the 10 –fold cross validation is showing significant performance than 66% of split. 10-fold cross-validation 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Time taken seconds Correctly Classified Instances Multilayer Perceptron Incorrectly Classified Instances Kappa statistic Support Vector Machine Mean absolute error J48 Root mean squared error k-nearest neighbor Relative absolute error Root relative squared error Naive Bayesian split 66. This results proved that multilayer perception is the best classifier for the nursery dataset and naïve Bayesian is the lowest. 5|Page . Except others.

002 0 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor not_recom recommend priority spec_prior Naive Bayesian 0.4 0. But the class very recommended is showing significant different between classifiers and testing mode.8 0.01 0.004 0.006 0.04 0.02 0.08 0. TP Rate for % split TP Rate Cross Validation not_recom recommend priority spec_prior very_recom 1.Assignment: 1 Artificial Neural Network Precision of classifiers (66% split) 0.2 1 0.008 0.1 0.006 0.2 1 0.002 0 Multilayer Support Perceptron Vector Machine very_recom J48 k-nearest neighbor not_recom recommend priority spec_prior Naive Bayesian very_recom The above graphs are showing the precision comparison of five classifier against five classes that we have identified.04 0.6 0.12 0.02 0 0 Multilayer Support Perceptron Vector Machine 6|Page J48 k-nearest neighbor Naive Bayesian Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Naive Bayesian . But class not_recommeded shows high precision in both training mode.4 0.8 0.08 0.1 0. But ver_recommeded class show significant precision in 66% split.06 0.2 0 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Naive Bayesian k-nearest neighbor Naive Bayesian Considering the True positive rate the not recommended class is showing similar results under both testing mode and all types of classifier used same like us the priority and specific priority class. Considering above fact the cross validation again lead in performance.004 0.01 HUNDREDS HUNDREDS Precision of classifiers (Cross Validation) 0. Under cross validation training mode the class recommended shows zero precision among all classifier we have used.008 0. However we got good results in multilayer perception and J48 under cross validation testing mode.012 0.012 0.2 0 not_recom recommend priority spec_prior Multilayer Support Perceptron Vector Machine J48 very_recom 1. FP Rate for % split FP Rate Cross Validation not_recom recommend priority spec_prior very_recom 0.6 0.06 not_recom recommend priority spec_prior very_recom 0.

6 0.4 0.8 0.2 0 not_recom recommend priority spec_prior Multilayer Support Perceptron Vector Machine J48 very_recom 1.6 0.8 0.4 0.6 0.4 0.6 0.8 0. F-Measure for % split F.Assignment: 1 Artificial Neural Network False positive rate is high in 66% split rather than cross validation for priority and specific priority class when using naïve Bayesian classifier and this is less in SVM classifier.2 1 0.8 0.2 1 0.8 0. Recall for % split Recall Cross Validation not_recom recommend priority spec_prior very_recom 1.2 1 0. In overall view the multilayer perception shows good performance in classification.2 1 0.6 0. ROC Area for % split ROC Area Cross Validation not_recom recommend priority spec_prior very_recom 1.Measure Cross Validation not_recom recommend priority spec_prior very_recom 1.6 0.8 0.4 0.2 0 not_recom recommend priority spec_prior Multilayer Support Perceptron Vector Machine J48 very_recom 1.4 0.2 0 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Naive Bayesian k-nearest neighbor Naive Bayesian Study of F-measure does not affect significantly in both testing mode and different classifiers except the one class which is very-recommended.2 0 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Naive Bayesian Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Naive Bayesian When considering the ROC values the not recommended.2 0 not_recom recommend priority spec_prior very_recom 1.2 1 0.4 0.2 0 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Naive Bayesian k-nearest neighbor Naive Bayesian Recall measurement shows better performance under cross validation. 7|Page . However very recommended class shows greater difference between the classifiers. specific priority and priority classes are showing high performance than very recommended and recommended class. But under 66% percentage split we can see lots of differences in recall.2 1 0.

Most of the time multilayer perception gives the good performance.priority When seeing the above ROC curves of classes the recommended class shows poor performance for most of the classifiers.not recommended Class :.by running the model ROC curves were obtained for different classifiers of particular class.Very recommended Class: . The analysis of ROC time consuming process therefore I did only for cross validation mode.Assignment: 1 Artificial Neural Network ROC curve Analysis To analyze the ROC performance the above model was developed. May due to less amount of instances in that class. Class: .recommended Class: . 8|Page . (Depend on the data set).

J48. 9|Page . The best algorithm based on the nursery data is multilayer perception classifier with an accuracy of 99.7299% and the total time taken to build the model is at 69.k-nearest neighbor and naïve Bayesian. Also the performance of other classification methods are in decreasing order such as SVM.specific priority Under ROC analysis provides the good performance comparison of different classifiers. we have met our objective which is to evaluate and investigate five selected classification algorithms based on Weka.Assignment: 1 Artificial Neural Network Class: . By considering all aspects of performance parameter under two types of training method the multilayer perception significantly provide the more accurate results. When considering the time factor multilayer perception is more time consuming.56 seconds. Comparison of confusion matrix (cross validation vs 66% split for five classification) Multilayer perception svm J48 K-nearest neighbor Naïve Bayesian Conclusion As a conclusion. According to the time factor k-nearest neighbor and naïve Bayesian classifiers took less time but their accuracy is relatively lower than the multilayer perception.

Syndney.1336 20.2583 103. Classification Multilayer Perceptron Support Vector Machine J48 k-nearest neighbor Naive Bayesian Time taken seconds Correctly Classified Instances Incorrectly Classified Instances Kappa statistic Mean absolute error Root mean squared error Relative absolute error Root relative squared error 16. t. t.0585 0.0499 1.914 0 95. SVHC.62 94.5775 0. sick: f.2419 51. Certain comparative studies were conducted and following factors are derived.15 98.Assignment: 1 Artificial Neural Network b) Data sets used:  Thyroid disease dataset supplied by the Garavan Institute and J.7281 Table 3 Results summary of 66% split percentage 10 | P a g e .9501 0. query on thyroxine: f. other. STMW.4758 4. t.8149 0. TSH measured: f. t.7301 0. on thyroxine: f.1357 6. t. New South Wales Institute. t.1257 96.This date set used to identify those who has thyroids weather getting sick or negative. T4U: continuous. sex: M. t. I131 treatment: f. psych: f. hypopituitary: f.25 97. t. FTI measured: f. TBG measured: f.8643 0. TT4 measured: f. Under this study I have used two types of test mode which are 10-fold cross-validation and percentage split 66%. tumor: f.5385 0. query hyperthyroid: f.0319 0.01 93. t.3737 0. T3 measured: f. t. TSH: continuous. t. SVHD. referral source: WEST. Quinlan.2271 77. t.0234 0. T4U measured: f. We have used the 'Explorer' option of the Weka tool.7265 0. F. pregnant: f. t.088 0. t. TT4: continuous. TBG: continuous.8502 0 0. t. FTI: continuous. t. t.0411 0. t. Australia. goitre: f.  Under this study there was 3772 samples (instances) were analyzed against thirty attributes age: continuous. lithium: f.0456 0. Ross % .9548 90.5306 0. thyroid surgery: f. t. The data sets have been submitted to a set of classification algorithms of Weka.5242 0. T3: continuous.2699 2.1498 5.9566 63. t. query hypothyroid: f. SVI.2126 39.4604 56.1488 27. on antithyroid medication: f.

0146 0.8 0. Under J48 the graph shows greater deviation from each test mode.0888 0.2294 77. but when we use cross validation J48 giving better results.3689 81.6 0.4332 0.0336 0. So j48 will be better classifier for cross validation test.124 64.807 1.8943 0. in the both test Knearest neighbor classifier is faster since only taking 0 seconds.1953 33. though overall J48 classifier giving better results in both test methods.9447 0 96.1824 3.4 0.7572 0. Time Taken in seconds 20 15 10 5 Correctly Classified Instances 0 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Naive Bayesian 100 98 96 Cross validation 10 66% split 94 92 Correctly identified instances shows averagely better results in cross validation test.193 0. 90 88 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Cross validation 10 Incorrectly Classified Instances Naive Bayesian 66% split If we consider the incorrectly identified instances. Naive Bayes taking same time (0.0863 95. 8 6 4 2 0 Multilayer Support Perceptron Vector Machine J48 Cross validation 10 k-nearest neighbor Naive Bayesian 66% split Capa Statistics 1 0.3871 103.6465 0.2428 2. again the cross validation show poor performance than split validation test mode.8176 0.5249 0.27 93.21 97.01sec) on both test methods.Assignment: 1 Artificial Neural Network Classification Multilayer Perceptron Support Vector Machine J48 k-nearest neighbor Naive Bayesian Time taken seconds Correctly Classified Instances Incorrectly Classified Instances Kappa statistic Mean absolute error Root mean squared error Relative absolute error Root relative squared error 16.0384 0.0615 0.4648 0.1054 12.01 92.685 43.1553 29.7522 0.07 98.2 0 -0.248 53.6034 7.2 Multilayer Support Perceptron Vector Machine J48 Cross validation 10 11 | P a g e k-nearest neighbor Naive Bayesian 66% split .0005 0.1506 -0.3966 0.7703 0.6866 Table 4 Results summary of 10 fold validation When considering time consuming for five classifier under two test sample methods.8494 6.

If we consider the following graph the J48 classifier has lower mean absolute error than other classifier. Except others. 10-fold cross-validation 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Time taken seconds Correctly Classified Instances Incorrectly Classified Instances Multilayer Perceptron 12 | P a g e Kappa statistic Support Vector Machine Mean absolute error J48 Root mean squared error k-nearest neighbor Relative absolute error Naive Bayesian Root relative squared error . Mean absolute error 0.Assignment: 1 Artificial Neural Network The MAE measures the average magnitude of the errors in a set of five classes.06 0. considerably the J48 classifier showing good results since gives lower error rate. The Naise bayes shows relatively higher absolute error from others classifiers.Ovarall when use cross validation test method giving less error comparatively.1 0.04 0.08 0. but LIBSVM and Naise bayes show high amount of error in determining the five classes.02 0 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Cross validation 10 Naive Bayesian Root mean squared error /Relative absolute error /Root relative squared error for 66% split 66% split 120 100 80 60 40 20 0 Root mean squared error /Relative absolute error /Root relative squared error for Cross validation 120 100 80 60 40 20 0 Multilayer Support Perceptron Vector Machine RMs Multilayer Support Perceptron Vector Machine RMs J48 RAE k-nearest neighbor J48 RAE k-nearest neighbor Naive Bayesian RRSE Naive Bayesian RRSE Above two graphs is showing comparison of different error parameters.

This results proved that multilayer perception is the best classifier for the nursery dataset and naïve Bayesian is the lowest.868 0. 13 | P a g e .986 0.976 0.classified as 3523 18 | a = negative 27 204 | b = sick a b <-.classified as 3540 1 | a = negative 231 0 | b = sick a b <-.741 0.787 0.667 0.016 0.554 0.classified as 3497 44 | a = negative 60 171 | b = sick 1 1 0.95 0.968 0.987 0.992 0.213 0.878 0.993 0.963 0.48 0.5 0 0 0 0 0 0.94 0.Assignment: 1 Artificial Neural Network split 66.833 0.008 0. remainder test 100% 80% 60% 40% 20% 0% Time taken seconds Correctly Classified Instances Incorrectly Classified Instances Multilayer Perceptron Kappa statistic Support Vector Machine Mean absolute error J48 Root mean squared error k-nearest neighbor Relative absolute error Root relative squared error Naive Bayesian The above two graphs are showing the compared performance matrices in percentage.classified as 3484 57 | a = negative 87 144 | b = sick a b <-.99 0.986 0.739 0.573 0.825 0. TP Rate FP Rate Precision Recall FMeasure ROC Area Confusion Matrix 0.667 0.992 0.333 0.5 J48(negative) 0.06 0.993 0.984 0.48 0.787 0.classified as 3314 227 | a = negative 52 179 | b = sick Table 3 Results summary of 66% split percentage According to the above results we can conclude J48 has the good classification since it has both TP Rate and FP Rate higher when used percentage split test.213 0.45 0.941 1 0.984 0. Also if we consider training mode the 10 –fold cross validation showing significant performance than 66% of split. The close look of this graph showing no significant changes between the parameters.007 0. The lower level showing good performance and higher percentage showing lower performance.92 0.0% train.787 0.52 0.94 0.787 0.878 J48(Sick) 0.655 0.739 0.98 0.95 a b <-.97 0.92 Classification Multilayer Perceptron(negative) Multilayer Perceptron(Sick) Support Vector(negative) Machine Support Vector(Sick) Machine k-nearest neighbor(Negative) k-nearest neighbor(Sick) Naive Bayesian(Negative) Naive Bayesian(Sick) a b <-.

992 0.classified as 1197 10 | a = negative 25 50 | b = sick 1 1 0.936 0.016 0.994 0.968 0.951 a b <-.919 0.377 0.901 0.939 1 0.976 0.classified as 1198 9 | a = negative 16 59 | b = sick a b <-.classified as 1207 0 | a = negative 75 0 | b = sick a b <-.806 0.806 0.988 0.983 0.925 0.26 0.925 Classification Multilayer Perceptron(negative) Multilayer Perceptron(Sick) Support Vector(negative) Machine Support Vector(Sick) Machine k-nearest neighbor(Negative) k-nearest neighbor(Sick) Naive Bayesian(Negative) Naive Bayesian(Sick) a b <-.012 0.775 0.883 0.225 0.Assignment: 1 Artificial Neural Network TP Rate FP Rate Precision Recall FMeasure ROC Area Confusion Matrix 0.995 0.795 0.936 0.classified as 1135 72 | a = negative 16 59 | b = sick Table 4 Results summary of 10 fold split According to the above results we can conclude J48 has the good classification since it has higher TP Rate higher when used percentage cross validation 10 fold test.951 0.775 0.716 0.117 0.005 0.951 0.767 0.98 0.623 0.562 0.984 0.classified as 1188 19 | a = negative 39 36 | b = sick a b <-.74 0.883 0.623 0.988 0.951 J48(Sick) 0.5 J48(negative) 0. Since it has provided better performance on both cross validation and split percentage.985 0.74 0.985 0.984 0.96 0.667 0.5 0 0 0 0 0 0.995 0. ROC Curve 14 | P a g e .064 0.441 0. So finally according to all above classifiers J48 is the good classifier for the sick dataset.

Assignment: 1 Artificial Neural Network Fig1 ROC curve for J48 (cross validation-fold 10) Fig2 ROC curve for J48 (percentage split) 15 | P a g e .

1. J48 Naïve bayes Multi layer perception K-nearest neigbour LibSVM. 3. which means the performance of a classifier depend on number of instances.Assignment: 1 Artificial Neural Network Fig3 ROC curve for Naïve bayes (cross validation-fold 10) Fig4 ROC curve for Naïve based (percentage split) The above four ROC curve . it understood different classifier may give better performance for the different datasets. 5. 2.we can identify when we use J48 classifier with cross validation(fold 10) testing method for the above sick datasets the giving better smooth curve it shows the better classifier is J48 out of all the above five classifier. But anyhow in order to classify certain data we have to consider higher number of instances and higher number of attributes. 4. number of attributes. Conclusion Out of all above results in order to analyze the performance of a classifier though J48 classifier gave the better performance for the sick dataset. If we order the classifier according to the all above result it will be like following order(the lowest numer giving higher performance). 16 | P a g e . But finally to take the proper decision we have to run the same datasets through using different classifier and different testing mode such as different values of cross validation and appropriate percentage split (but 66% is the standard value).

9753 0.9506 0.3607 1.1798 0.0562 0.5694 Support Vector Machine 0.226 0.9753 0.5404 J48 0.0252 0.01 98.6355 25.8027 0.2259 0.06 98.9238 0.9574 0.1792 9.04 93.0413 0.6297 26.1294 4.54 57.5639 61.279 51.2787 0.0431 0.0247 0.2957 19.1172 5.9438 5.8852 13.191 2.0217 0.11 98.918 0.3956 0 95.3146 1.Assignment: 1 Artificial Neural Network c) Data set used: Relation: wine Instances: 178 Attributes: 14               Class Alcohol Malic_acid Ash Alcalinity_of_ash Magnesium Total_phenols Flavanoids Nonflavanoid_phenols Proanthocyanins Color_intensity Hue OD280/OD315_of_diluted_wines Proline Results and Discussion Classification Time taken seconds Correctly Classified Instances Incorrectly Classified Instances Kappa statistic Mean absolute error Root mean squared error Relative absolute error Root relative squared error Multilayer Perceptron 0.0874 0.809 0.9489 0.0486 0.7794 14.2788 50.9058 0.0713 2.74 96.1148 0.9371 27.3973 38.01 96.0865 0 94.0124 0.082 4.4678 59.6854 0.6176 k-nearest neighbor Naive Bayesian Table 5 Results summary of 10 fold cross validation Classification Time taken seconds Correctly Classified Instances Incorrectly Classified Instances Kappa statistic Mean absolute error Root mean squared error Relative absolute error Root relative squared error Multilayer Perceptron 0.3708 0.1821 9.7213 3.0058 Support Vector Machine 0.8844 J48 0 86.77 97.2046 0.9745 0.8202 6.6393 0.6292 3.926 0.2019 11.3607 1.6393 0.6393 37.8682 0.128 5.0723 43.8027 k-nearest neighbor Naive Bayesian Table 6 Results summary of 66% split 17 | P a g e .

6  0. The naïve Bayesian show low performance in both test mode.5 0. From capa we can come to this conclusion. .99 Almost perfect agreement 18 | P a g e 66% split 0. It gives better results under the 66% split test mode. 0. Capa Statistics Cross validation 10 1.81–0.61–0.Assignment: 1 Artificial Neural Network Time Taken in seconds When considering time consuming for five classifier under two test sample method the 66% split take short time for SVM method of classification . Under multilayer perception the graph show greater deviation from each test mode and cross validation gives better results under multilayer perception classifier.9 0.1 0 Correctly Classified Instances Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Naive Bayesian Cross validation 10 66% split 100 Cross validation 10 66% split 95 90 Correctly identified instances show better results under cross validation test mode.6 0.4 0 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Naive Bayesian .01–0.2 1 0.41–0. again the split validation show poor performance than cross validation test mode.but comparatively the cross validation method take more time than percentage split.3 0. 85 80 Multilayer Support Perceptron Vector Machine J48 k-nearest neighbor Naive Bayesian Incorrectly Classified Instances Cross validation 10 If we consider the incorrectly identified instances.8  < 0 Less than chance agreement 0.8 0.2 0.21– 0.4 0.2  0.20 Slight agreement 0. 66% split 14 12 10 8 6 4 2 0 Multilayer Perceptron Support Vector Machine J48 k-nearest neighbor Naive Bayesian Capa statistics coefficient is a statistical measure of inter-rater agreement or inter-annotator agreement for qualitative (categorical) items.7 0.40 Fair agreement  0.60 Moderate agreement  0.80 Substantial agreement  0. All together all classifier shows same conclusion except the naive Bayesian classifier.

If we consider the following graph the Support vector machine classifier shows high mean absolute error than other classifier.Assignment: 1 Artificial Neural Network Mean absolute error 0.2019 0.2957 0.0058 30 20 10 5. Except others.279 9.8027 2.1 0.0723 0.6393 0.15 0.3973 0.05 0 Multilayer Perceptron Support Vector Machine J48 k-nearest neighbor Naive Bayesian Root mean squared error /Relative absolute error /Root relative squared error for Cross validation RMs 70 RAE RRSE 59.4678 60 50 43.6176 25.5404 51.3956 57.2046 40 25. 66% split 0.6355 0.7794 0.5639 20 10 5.1172 11.9371 0.8844 50.0713 0 Multilayer Perceptron Support Vector Machine J48 k-nearest neighbor Naive Bayesian Above two graphs are showing comparison of different error parameters. The multilayer perception shows relatively low absolute error from others and J48 shows average error rate. Multilayer perception shows low absolute error under cross validation training mode.1294 0 Multilayer Perceptron Support Vector Machine J48 k-nearest neighbor Naive Bayesian Root mean squared error /Relative absolute error /Root relative squared error for 66% split RMs 70 RAE 61.1821 4.0865 38. When considering the two train modes there is no big deviation from each other except the multilayer perception. 19 | P a g e . but Support vector Machine and J48 show high amount of error in determining the five classes.1792 14.25 Cross validation 10 The MAE measures the average magnitude of the errors in a set of five classes.6297 0.54 60 RRSE 50 37.128 9.2788 0.5694 30 19.2 0.8682 40 27. considerably the multilayer perception classifier showing good results that means lower error rate.

This results proved that multilayer perception is the best classifier for the Wine dataset and naïve Bayesian is the lowest. So. Also if we consider training mode the 10 –fold cross validation showing significant performance than 66% of split. However. the results indicate that the performance of a classifier depends on the data set. then number of instances especially on the number of attributes used in the data set and one should not rely completely on a particular algorithm for their study.0% train. It should also be noted that classifiers of a particular group also did not perform with similar accuracies. Overall. remainder test 100% 80% 60% 40% 20% 0% Time taken seconds Correctly Classified Instances Incorrectly Classified Instances Multilayer Perceptron Kappa statistic Support Vector Machine Mean absolute error J48 Root mean squared error k-nearest neighbor Relative absolute error Root relative squared error Naive Bayesian The above two graphs are showing the compared performance matrices in percentage. The lower level showing good performance and higher percentage showing lower performance. This study focuses on finding the right algorithm for classification of data that works better on diverse data sets. it is observed that the accuracies of the tools vary depending on the data set used. The close look of this graph showing no significant changes between the parameters. 20 | P a g e . we recommend that users should try their data set on a set of classifiers and choose the best one. Final Conclusion Finally.Assignment: 1 Artificial Neural Network 10-fold Cross-Validation 100% 80% 60% 40% 20% 0% Time taken seconds Correctly Classified Instances Incorrectly Classified Instances Multilayer Perceptron Kappa statistic Support Vector Machine Mean absolute error J48 Root mean squared error k-nearest neighbor Relative absolute error Root relative squared error Naive Bayesian Split 66.

Gopala Krishna. Mohd Fauzi bin Othman and Thomas Moh Shan Yau “Comparison of Different Classification Techniques Using WEKA for Breast Cancer” IFMBE Proceedings Vol. 15. Vol 3. No. Bharath Kumar and Nagaraju Orsu “Performance Analysis and Evaluation of Different Data Mining Algorithms used for Cancer Classification”. Vol. September 2012 4. Issue 2. Samrat Singh and Vikesh Kumar ” Performance Analysis of Engineering Students for Recruitment Using Classification Data Mining Techniques” Samrat Singh et al . IJCSET .Volume 54– No. 2.2007. (IJARAI) International Journal of Advanced Research in Artificial Intelligence.February 2013 .5. 2.Assignment: 1 Artificial Neural Network References 1. Rohit Arora and Suman “Comparative Analysis of Classification Algorithms onDifferent Datasets using WEKA”. 3. 21 | P a g e . International Journal of Computer Applications (0975 – 8887). 2013. 31-37 .13.