6.
Experimentation
In this section, we describe the method of
evaluation of the proposed algorithm. Firstly, we
describe our dataset. We then calculate various
accuracy measures considering different
parameters as reference.
6.1 Description of dataset
This paper is about anomaly detection of user
behaviours. An ideal dataset should be obtained
from a practical system with concrete job functions.
But in fact, it is very sensitive for almost every
organization or company.
The performance of the algorithm was
analyzed by carrying out several experiments on a
credit card company dataset adhering to the TPC-C
benchmark[]. The TPC-C schema is composed of a
mixture of read only and read/write transactions
that replicate the activities found in complex OLTP
application environment. The database schema,
data population, transactions, and implementation
rules were designed to broadly represent modern
OLTP systems. We used two audit logs: one for
training the model and the second for testing it. The
training log comprised of normal user transactions
and testing log consisted of a mixture of normal as
well as malicious user transactions. Although there
are unusual records in real dataset, we also inject
some anomalies for detection. The injected
anomalies are set differently with the normal
behaviour pattern from several aspects. In totality,
about 20,000 transactions were used. In total,
about 99% of data was non-malicious while less
than 1% of data was malicious. Fig. 6(a) shows the
distribution of malicious and benign data in the
dataset used:
Fig 6(a) Frequency of data items and their access
frequency
The details of CDEs, DAEs and Normal data items
has already been given in Section 3 and examples
have been discussed in Section 5.
The access pattern data hereby shows that CDEs are
rarely accessed, that too only by a few user roles
and hence, protection of CDEs from malicious
access is of a greater significance as compared to
DAEs and Normal data elements.
6.2 Cluster Analysis
When the number of users/user roles exceeds a
given limit, it becomes exceedingly difficult for the
IDS to keep track of individual user access patterns
and hence detect anomaly. This is the reason that
clustering is a better and computationally efficient
solution for better performance of IDS. We prefer
Fuzzy clustering over hard clustering. Fuzzy
clustering (also referred to as soft clustering) is a
form of clustering in which each data point can
belong to more than one cluster. In non-fuzzy
clustering (also known as hard clustering), data is
divided into distinct clusters, where each data point
can only belong to exactly one cluster. In fuzzy
clustering, data points can potentially belong to
multiple clusters. Membership grades are assigned
to each of the data points(tags). These membership
grades indicate the degree to which data points
belong to each cluster. Thus, points on the edge of
a cluster, with lower membership grades, may be in
the cluster to a lesser degree than points in the
center of cluster. When we evaluate various
performance measures keeping the number of
clusters as a reference parameter, it is observed
that a particular count for clusters is the most
efficient in predicting results.
Fig 6(b) Variation of performance with number of
clusters
Fig 6(b) depicts variation in precision, recall, TNR,
accuracy with change in number of clusters. From
the graph, we can see that :-
 TNR does not vary with the number of clusters,
i.e. TNR is invariant.
 The precision is always greater than 0.94 and is
more or less constant.
 Recall reaches optimum value when number of
Fuzzy Clusters is greater than 3.
 Accuracy also reaches the optimum value
when number of clusters is greater than 3.
6.3 Distances and thresholds
In section 3.2, we have described Modified Jensen-
Shannon distance as a measure to calculate
distance between two user vectors of same length.
In probability theory and statistics, the Jensen–
Shannon divergence is a method of measuring the
similarity between two probability distributions. It
is also known as information radius (IRad) or total
divergence to the average. It is based on the
Kullback–Leibler divergence, with some notable
(and useful) differences, including that it is
symmetric and it is always a finite value. The square
root of the Jensen–Shannon divergence is a metric
often referred to as Jensen-Shannon distance. We
preferred to use modified Jenson-Shannon distance
to give weights to data attributes and avoid curse of
dimensionality. The variation of modified Jenson-
Shannon distance with Euclidean distance is shown
in the fig 6(g).
In section 3.3, we have defined modified Jaccard
distance to quantitatively measure the similarity
between two rules. The Jaccard index, also known
as Intersection over Union of the Jaccard similarity
coefficient, is a statistical measure used for
comparing the similarity and diversity of sample
sets. The Jaccard coefficient measures similarity
between finite sample sets, and is defined as the
size of the intersection divided by the size of the
union of the sample sets. The variation of modified
Jaccard index with Jaccard index is shown in fig 6(h).
The variation of precision, recall, TNR, accuracy
with the various thresholds, namely 𝛿1, 𝛿2, фUT , фLT
that were defined in section 3 is shown in the
following figures:
Fig 6(c) shows the variation of Precision, recall, TNR,
accuracy with 𝛿1. It can be observed from the graph
that Precision, TNR and Accuracy increase with the
increase in value of 𝛿1, while the value of Recall
decreases with increase in value of 𝛿1.
Fig 6(e) shows the variation of Precision, recall, TNR,
accuracy with 𝛿2. It can be observed from the graph
that the value of Precision, TNR and Accuracy starts
decreasing when the value of 𝛿2 increases beyond
a certain value. Recall, on the other hand, increases
for higher values of 𝛿2.
Fig 6(d) shows the variation of Precision, recall,
TNR, accuracy with фUT. It can be observed from the
graph that the value of Precision first decreases and
then exponentially increases with the increase in
value of фUT. An identical trend is followed by
Accuracy. Somewhat similar trend is followed by
TNR except that it does not decrease initially. On
the contrary, the value of Recall decreases with the
increase in value of фUT.
Fig 6(f) shows the variation of Precision, recall, TNR,
accuracy with фLT. It can be observed from the
graph that the values of all the parameters
fluctuate a little but remain more or less constant
with the increase in value of фLT.
With regards to the dataset we have used, following
inferences can be done from the graphs:
1. Value of 𝛿1 should be close to 0.65 for
optimum performance.
2. Value of 𝛿2 should be close to 0.55 for
optimum performance.
3. Value of фUT should be close to 0.59 for
optimum performance.
4. Value of фLT should be close to 0.2 for optimum
performance.
6.4 Comparison with related methods
Table 1 shows the performance measures used for
comparison of approaches. Using these
performance measures, we will compare our
approaches with other related works. Our various
approaches are:-
Approach 1. Our approach using modified Jenson-
Shanon distance and modified Jaccard index.
Approach 2. Using unmodified Jaccard index with
Jenson-Shanon distance.
Approach 3. Using Euclidean distance with
unmodified Jaccard index.
S.No. PERFORMANCE FORMULA
MEASURE
1 TNR TN
TN + FP
2 Precision TP
TP + FP
3 Accuracy TP + TN
TN + FP + TP + FN
4 F1 Score 2 ∗ Precision ∗ Recall
Precision + Recall
5 PPV TP
TP + FP
6 ACC TP + TN
TP + TN + FP + FN
7 NPV TN
TN + FN
8 FDR FP
FP + TP
9 FOR FN
TN + FN
10 BM TPR + TNR – 1
11 FPR FP
FP + TN
12 FNR FN
FN + TP
13 MK PPV + NPV – 1
14 MCC TP × TN − FP × FN
√(TP + FP)(TP + FN)(TN + FP)(TN + FN)
Table 1 ( Performance Measures)
In table 2 we have compared the three approaches If we compare Approach 1 with Approach 3, we
with each other. observe that:-
Sensitivity Approach 1 Approach 2 Approach 3  TNR and precision of Approach 1 is a lot better
Measures
than the TNR and precision for Approach 3
PPV 0.96 0.73 0.74
 .It has also got better accuracy as compared to
TPR 0.81 0.95 1.00
Approach 3.
ACC 0.89 0.80 0.83
 Approach 1 also has a much lower FPR and FDR
F1 Score 0.88 0.83 0.85 score as compared to Approach 3.
NPV 0.83 0.93 1.00  Amongst other performance measures, MK
FDR 0.04 0.27 0.26 and MCC values of Approach 1 are also slightly
FOR 0.17 0.07 0.00 better than that of Approach 3.
BM 0.77 0.60 0.65  Approach 3, on the other hand has got better
FPR 0.03 0.34 0.34 TPR, NPV and FOR measures as compared to
TNR 0.96 0.65 0.65 Approach 1. In fact, it has the best values for
FNR 0.19 0.05 0.00 these parameters in the entire table.
MK 0.79 0.66 0.74  Also, both Approach 1 and Approach 3 have got
MCC 0.78 0.63 0.70 somewhat similar F1 score.

Table 2 (Comparison of our approaches)
From the table, following observations can be
made:-
If we compare Approach 1 with Approach 2, we can
observe that:-
 TNR and FPR of Approach 1 is a lot better than
the TNR and precision for Approach 2.
 Approach 1 has also got better accuracy as
compared to Approach 2.
 Approach 1 has a much lower FPR and FDR
score as compared to Approach 2.
 Amongst other performance measures, MK
and MCC values of approach 1 are also better
than that of Approach 2.
 Approach 2, on the other hand has got better
TPR, NPV and FOR measures as compared to
Approach 1.
 Both Approach 1 and Approach 2 have got
somewhat similar F1 score.
In the measures like FPR and TNR where Approach
1 has good performnce, Approach 2 performs
rather poorly. However, in measures like TPR and
NPV, where Approach 2 performs better, Approach
1 also has good performance. For example, both
Approach 1 and Approach 2 have similar NPV scores
with Approach 2 performing slightly better As
Approach 1 performs far better than Approach 2 in
most of the measures, we can conclude that the
overall performance of Approach 1 is better than
Approach 2.
In the measures like TNR and precision,where
Approach 1 has one of the best score in the entire
table, Approach 3 performs rather poorly. Also,
Approach 3 lags far behind in measures like FPR and
FDR score. On the other hand, in the measures in
which Approach 3 performs better than Approach
1, Approach 1 is also performing quiet nicely. For
example, in case of NPV, both the approaches have
good scores, with Approach 3 performing better.
similar trends are observed in case of all other
measures except FNR, where Approach 3 has is far
superior. Considering all the above scenario, we can
say that the overall even though Approach 3 has the
best values for some performance measures, its
poor performance in other measures are clearly a
disadvantage due to which Approach 1 is better
than Approach 3.
Table 3 shows a comparison of our approaches with
various other related works. If we compare our
approach with other related approaches, we
observe that:-
 In comparison to HU Panda, our approach
works better with respect to all the
performance measures considered for the
purpose of comparison.
 In comparison to the work of Mostafa et al. our
approach performs better with respect to all
the performance measures that are considered
for comparison.
Sensitivity Approach Approach Approach HU Panda Hashemi Mostafa Mina Majumdar EliSa UP Rao et
Measures 1 2 3 et al. et al. Sohrabi et al. (2006) Bertino al.(2016)
et al. et al.
PPV 0.96 0.73 0.74 0.88 0.97 0.94 0.93 0.88 0.94 0.61

TPR 0.81 0.95 1.00 0.73 0.71 0.75 0.66 0.70 0.91 0.70

ACC 0.89 0.80 0.83 0.81 0.84 0.85 0.80 0.80 0.93 0.64

F1 Score 0.88 0.83 0.85 0.79 0.82 0.83 0.77 0.78 0.92 0.65

NPV 0.83 0.93 1.00 0.77 0.77 0.79 0.73 0.75 0.91 0.68

FDR 0.04 0.27 0.26 0.12 0.03 0.06 0.07 0.13 0.06 0.39

FOR 0.17 0.07 0.00 0.23 0.23 0.21 0.27 0.25 0.09 0.32

BM 0.77 0.60 0.65 0.63 0.69 0.70 0.60 0.60 0.85 0.35

FPR 0.03 0.34 0.34 0.10 0.02 0.05 0.05 0.10 0.06 0.45

TNR 0.96 0.65 0.65 0.90 0.98 0.95 0.94 0.90 0.94 0.65

FNR 0.19 0.05 0.00 0.28 0.29 0.25 0.35 0.30 0.09 0.30

MK 0.79 0.66 0.74 0.65 0.74 0.73 0.66 0.63 0.85 0.29

MCC 0.78 0.63 0.70 0.63 0.72 0.71 0.63 0.61 0.85 0.29

Table 3 (Comparison of our approaches with related works)

 In comparison to the work of Hashemi et al. measures as well, better TNR and precision
even though our approach scores just a little scores can easily cover up lower recall values.
less in measures like TNR and precision, it 7. Analysis and Conclusion
scores a lot better with respect to rest of the
In this paper we have tried to detect malicious
performance measures.
transactions keeping in mind that certain data
 If we consider the work of Mina Sohrabi et al.
elements hold more critical information. We also
our approach performs better with respect to
take into consideration user behaviour pattern in
all the performance measures that are present
this approach. A user regularly behaving as a
in the table.
normal user will be gradually improving his
 In comparison to the work of Majumdar et al.
suspicion score. We then analyse the approach
our approach performs better with respect to
w.r.t different parameters by conducting
all the performance measures that we have
experiments. Finally, we conclude that the
considered for the purpose of comparison.
approach works efficiently in determining the
 With comparison to the work of UP Rao et al.
nature of a transaction.
our approach performs better in context to all
the measures that are considered in the table
for comparison.
 In comparison to the work of Elisa Bertino, our
approach gives better TNR and precision
scores. It also gives comparatively better FDR
and FPR scores. In other measures, except TPR
and recall, both approaches have somewhat
similar score. Since our work is mostly related
to finding Critical Data Items in a dataset,
higher TNR and precision scores are more
desirable as compared to other performance
measures. Since our approach performs quiet
well with respect to other performance