Alshaikh 1 s2.0 S0167404823006041 Main

Journal Pre-proof
Exploring Perceptions of Decision-Makers and Specialists in

Defensive Machine Learning Cybersecurity Applications: The Need
for a Standardised Approach
Omar Alshaikh , Simon Parkinson , Saad Khan
PII: S0167-4048(23)00604-1
DOI: https://doi.org/10.1016/j.cose.2023.103694
Reference: COSE 103694
To appear in: Computers & Security
Received date: 19 September 2023

Revised date: 27 November 2023
Accepted date: 27 December 2023
Please cite this article as: Omar Alshaikh , Simon Parkinson , Saad Khan , Exploring Per-
ceptions of Decision-Makers and Specialists in Defensive Machine Learning Cybersecurity
Applications: The Need for a Standardised Approach, Computers & Security (2023), doi:
https://doi.org/10.1016/j.cose.2023.103694
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition
of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of
record. This version will undergo additional copyediting, typesetting and review before it is published
in its final form, but we are providing this version to give early visibility of the article. Please note that,
during the production process, errors may be discovered which could affect the content, and all legal
disclaimers that apply to the journal pertain.
© 2023 Published by Elsevier Ltd.

Exploring Perceptions of Decision-Makers and Specialists in
Defensive Machine Learning Cybersecurity Applications: The
Need for a Standardised Approach
Omar Alshaikha*[0000-0002-1350-8933], Simon Parkinsona [0000-0002-1747-
9914]
, Saad Khana [0000-0001-8613-8200]
a
Departement of Computer Science, University of Huddersfield, Huddersfield, HD1 3DH, UK
Machine learning (ML) utilisation has achieved a vast global impact. This is evident in the cybersecurity
sector, where ML has wide-ranging applications, such as identifying and blocking threats , uncovering unusual
software and user behaviour, and many others . However, the increase in successful cyberattacks demonstrates
that the effectiveness of ML in cybersecurity applications can be questioned . Although the attacks may be new,
ML is often adopted due to its ability to handle diverse and often unforeseen situations – a capability that is not
possible using traditional rule-based security mechanisms. As both the rate of attacks and adoption of ML
solutions are increasing, there is a need to determine whether ML-based security solutions are meeting the
expectations of businesses and whether businesses are genuinely aware of the ML capabilities and limitation s .
Moreover, current literature shows a significant variation in how ML solutions are evaluated in cybersecurity
applications, which might result in a poor understanding of ML capabilities. This paper explores the commo n
perceptions and observations of decision-makers and specialists using ML for cybersecurity regarding its
capabilities, implementation, evaluation, and communication. A semi-structured interview is conducted with
individuals in various managerial positions to perform this investigation . The finding of this study reveals a
pressing need for a standard to manifest ML capabilities. As significant variation in the understanding of
Machine Learning Cyber Security (MLCS) capabilities is observed, a standard could help better communicat e
MLCS capabilities. It is observed that external influences heavily impact ML adoption decisions, potentially
leading to misinterpretation of ML capabilities .
Keywords: Machine learning; Cybersecurity ; Capabilities; ML application; Perception; Cybercrime; Thematic analysis, Themes
E-mail address: Omar.alshaikh@hud.ac.uk

2 Omar Alshaikh et al. / Submitted to Computers and Security
1. Introduction
Modernisation, lifestyle changes, and technology dependency make humans more vulnerable to cyber-
attacks (Sarker et al., 2020). Cyber attackers are progressive and constantly attempting to establish new methods
to exploit vulnerabilities , resulting in successful attack outcomes (Dasgupta et al., 2020). In recent years, there
has been an alarming increase in the utilisation of sophisticated tools to initiate cyber-attacks, including
ransomware, phishing, and distributed denial-of-service (DDoS) attacks (Ali et al., 2023; Asiri et al., 2023;
Potamos et al., 2023). According to Petrosyan (2023) the estimated global cost incurred from cybercrime is
$8.15 trillion in 2023 and is expected to reach $13.82 by 2028. This represents a significant increase in cost by
11.34% each year between 2023 and 2028. The growing threat of cyberattacks highlights the need for strong
cybersecurity defence measures.
Cyberattacks are challenging to prevent; in addition to robust security systems, they also require continuous
monitoring and interpretation, sometimes beyond what is comprehensible by human analysts. ML has become
an essential tool in cybersecurity due to its ability to identify and detect potential threats in real-time with
minimal human intervention and resources (Parkinson et al., 2018b). For instance, ML algorithms can learn to
recognise patterns in large network traffic and system event datasets to identify anomalous behaviour and
provide indications of a cyber-attack (Sarker et al., 2020; Khan et al., 2018; Parkinson et al., 2018b). In addition,
ML can enhance existing security measures, such as intrusion detection systems and firewalls, by enabling them
to adapt to new threats quickly. According to Handa et al. (2019), ML-based approaches are increasingly used
in cybersecurity due to their higher accuracy and efficiency in detecting and mitigatin g threats. According to
recent surveys investigating the use of ML in cybersecurity, it is evident that most applications usually achieve
accuracy greater than 90% (Alshaikh et al., 2023; Franco, 2022). In ML, accuracy is defined as the fraction of
predictions that were correctly made (Shaukat et al., 2020; Parkinson et al., 2022).
Although a substantial amount of research studies demonstrate great promise in using ML to improve
security, its adoption is also associated with drawbacks. One major drawback is the potential of having a high
number of false positives and false negatives. For instance, due to issues in the training process, ML algorithms
may generate false alarms, flag legitimate activities as malicious, or miss real threats, resulting in security
breaches (Sun et al., 2018). Additionally, systems embedded with ML could be compromised and manipulated
by adversarial attacks, in which the attacker could exploit system vulnerabilities to misclassify benign activities
as malicious (Xue et al., 2020). This is widely referred to as the poisoning of ML systems (Tian et al., 2022).
Furthermore, relying on ML can create a false sense of security and lead to complacency in implementing other
essential security measures (McDaniel et al., 2016).
Previous work has demonstrated variations in evaluating MLCS applications (Alshaikh et al., 2023). More
organisations are choosing to adopt MLCS solutions to combat cyberattacks (Rawindaran et al., 2021), yet little
is known about how successful they are proving to be. Furthermore, there is a lack of understanding of how
decision-makers and specialists perceive the capabilities and limitations of MLCS solutions. Recent research
motivates our work as several studies have been conducted to understand the perceptions of experts in specific
areas of cybersecurity, such as understanding the perception of decision-makers when using attack graphs (Pirca
et al., 2023), user perception of authentication mechanisms for mobile banking (Kruzikova et al., 2022), and
user perception of facial recognition in the physical work (Seng et al., 2021), among others. This paper aims to
understand the decision-makers’ and specialists' perceptions of ML capabilities for defensive cybersecurity
applications. The contributions are made in the following three areas: (1) MLCS capabilities, (2) MLCS
implementation, and (3) MLCS evaluation and communication. The specific contributions of this article are:
• The acquisition of insight into the perceptions of decision-makers with MLCS applications is based
on an in-depth interview with 25 participants.
Omar Alshaikh / Computers and Security 3
• The establishment of three key analytical themes of capabilities, implementation, communicatio n

and evaluation, which all contain diversity in opinion and motivate the need for a standardised
approach.
This paper is structured into six sections: In Section 2, related work in the field of MLCS applications and
capabilities will be conducted to motivate the research, extract the knowledge gap, and reinforce the need for
this study. Section 3 provides the technique used for data collection, followed by data analysis. In Section 4,
the paper’s findings and contribution are demonstrated. In Section 5, the paper critically discusses the findings
and implications of real-life applications. Finally, in Section 6, the paper's conclusion is outlined, and future
study is presented.
2. Related Work
The increasing rate of cybercrime, the potential for financial losses, and the pressing need for data
confidentiality, integrity, and availability are the factors that make cybersecurity a critical concern for
individuals, organisations, and governments globally (Aftergood, 2017). ML has emerged as a valuable
cybersecurity tool, helping detect and respond autonomously to increasingly sophisticated cyber threats (Sarker
et al., 2020). ML aims to overcome the shortcomings of traditional manual and rule-based systems, which are
no longer flexible and adaptable to deal with the vast amounts of data and complex patterns involved in modern
cybersecurity systems (Thomas et al., 2020). This is why ML has been applied in several cybersecurity domains
(Ford & Siraj, 2014; Khan et al., 2018; Khan et al., 2022), such as intrusion detection, malware classification,
and vulnerability assessment.
Numerous research studies investigate the implementation of MLCS, attempting to create a level of
assurance of the ML application’s performance in cybersecurity. For example, Rawindaran (2021) investigated
the adoption of MLCS by small and medium-sized enterprises (SMEs) in developed countries by surveying 200
SMEs. The study concluded that 70% are utilising the traditional methods and acknowledge the benefit of ML;
however, they lack technical expertise, primarily due to budget constraints. The survey provides an overview
of SMEs’ adoption of MLCS. It is, however, limited in terms of sampling size for generalisation and providing
in-depth knowledge of participant perception. Another research study by Dasgupta et al. (2020) examined the
applications, limitations, and challenges of ML in cybersecurity by conducting a literature survey. The study
concluded several limitations in MLCS implementation, such as the lack of interpretability and transparency in
ML models and the susceptibility of ML models to adversarial attacks. However, the concentration of this study
is limited to secondary data and excludes participant perception of ML capabilities and evaluation.
In closely related and recent work, Rawindaran et al. (2022) performed a quantitative survey on 122
participants in Wales, UK. This follow-up study to their previous work attempts to understand SMEs' awareness
of intelligent cybersecurity solutions. An interesting aspect of the study is that 73% of respondents report being
knowledgeable on ML, yet 72% of respondents report the reason behind not using ML is ‘due to being unaware’.
Since they report to be knowledgeable on ML, it can be inferred that this vast majority are unaware of how to
implement it within their organisation and what the specific opportunities and benefits that it can deliver. This
finding is consistent with a systematic literature survey performed by Chidukwani et al. (2022). The work
identified a significant challenge hindering the adoption of cybersecurity solutions in which SMEs encountering
sever impact such as financial (the $1 billion) in Australian economy, losing of business opportunity, and exerts
a profound psychological impact on individual well-being. This challenge arises from conflicting, unclear, and
overwhelming information, making it difficult to interpret. This coupled with their other findings, such as a lack
of necessary knowledge, results in a problematic combination where cybersecurity ML solutions are poorly
understood and not fully utilised if even adopted. This challenge is by no means unique to the cybersecurity
sector. A recent literature survey by De Simone et al. (2023) identified a ‘lack of knowledge/skill’ and the
‘complexity of the solution’ as the main limitations preventing ML uptake and successful uses in manufacturing.
In cybersecurity, the level of skill and knowledge in digital literacy is assumed to be higher than in the
manufacturing area; however, there is an even greater burden on the requirement for new knowledge as new
attacks and defence mechanisms are continuously identified.
Several factors impact the performance of the ML application, such as biased and incomplete data, choice
of ML algorithm, etc. These factors have been extensively explored , as evidenced by the existing literature. ML
has immense potential in detecting and preventing cyber-attacks in real-time, but its capabilities are often
unrealistic and overestimated. So, the question is, how are the capability and efficacy of ML solutions perceived
in defensive MLCS solutions? It is essential to understand the practitioner's perspective on who is using
standalone ML products/services in tackling cyber threats. Therefore, in this paper, we have explored the
perceived effectiveness of MLCS by conducting semi-structured interviews. The participants are cybersecurity
experts and technology adoption decision-makers. The interview data collection technique and analysis are
presented in the following section. Table 1 provides a summary of the advantages and disadvantages of ML
utilisation.
T able 1 ML advantages and disadvantages
Category Points Reference

Advantages Autonomously detecting and responding to sophisticated Sarker et al., 2020
cyber threats.
Overcome the limitations of traditional manual and rule- Thomas et al., 2020
based systems, providing flexibility in dealing with vast data
and complex patterns in modern cybersecurity.
Minimises workload and accelerates task accomplishment. Dahiya et al., 2022
Disadvantages The lack of interpretability and transparency in ML models Dasgupta et al., 2020
makes them susceptible to adversarial attacks.
Being unaware of how to correctly implement ML may Rawindaran et al.,
result in missed opportunities and benefits. 2022
Lack of necessary knowledge leads to poor understanding Chidukwani et al.,
and underutilising of cybersecurity ML solutions. 2022
lack of skill and the 'complexity of the solution. De Simone et al., 2023
Biased and incomplete data and the choice of ML algorithm Miceli et al., 2022
impact ML application performance.
3. Methodol ogy
Various ML benefits that are presented in the literature may not match those implemented in practice. The
promising result generated during development and testing, the increasing cybercrime within the adopting
organisation, and the paradigm shift towards MLCS create the need to investigate further how decision-makers
and specialists perceive ML capabilities. The paper explores the perceptions and decision-making influencing
factors through user interviews. The interview methods and techniques used in this paper are presented in this
section.
3.1. Research approach
The inductive approach that follows standard practice in social science research (Azungah, 2018; May & Perry,
2022) was deemed the most appropriate due to the exploratory nature of the research question, which aims to
investigate the perception of decision-makers and the factors they perceive to influence the MLCS
implementation. The qualitative technique was selected due to its ability to facilitate the acquisition of
participants' insights, providing a profound understanding of topics (Bryman, 2016). This approach is well-
suited for research lacking prior knowledge, as it delves into human perception, thought, and perspective
(Walliman, 2021). Moreover, it proves particularly effective in capturing nuanced participant interpretations of
specific areas or topics (Clark et al., 2021). The semi-structured interviews were conducted as they provided a
platform for the participants to express their opinions and thoughts freely. Thematic analysis (Braun & Clarke,
2019) is then utilised to identify key interview themes. This approach was deemed appropriate to generate rich
and meaningful data, which is essential to achieve our goal. Appendix (A) contains the interview guide used
for all interviews.
3.2. Data collection
A total of 25 participants were interviewed between March and July 2022, with an average interview time
of 1 hour. The selection of 25 participants for this qualitative study aligns with the principles of saturation,
ensuring an in-depth exploration and a comprehensive understanding of the research topic (Hennink & Kaiser,
2022). The interviews consisted of four sections that focused on the participant ’s individual experience in
MLCS, interpreting general ML capabilities, interpreting MLCS capabilities, and understanding methods of
measuring MLCS performance. The interview questions were verified by an independent external expert and
tested through a pilot study. The funnelling technique was utilised to ensure the participants remained focused
throughout the interview (Kvale & Brinkmann, 2015). The interviews were conducted online through various
communication portals such as MS Teams, WhatsApp, and Cellcrypt. Social media platforms, including Twitter
and LinkedIn, were used to distribute interview invitations. To cater to participants’ language requirements, the
interviews were available in two languages (Arabic and English) and were translated into English by an expert
translator. Appendix (B) contains information on each of the participant’s background s. The selected sampling
was approached through phone calls, emails, and messages in social media channels. After participants were
accepted to participate, they were emailed the entire research plan, including the aim, objective, interview
questions, and consent form. The consent form required a signature con firming the acceptance to participate in
the research and highlighted data privacy, confidentiality, and anonymity of personal representation.
3.3. Data analysis
The selection of participants was based on seven criteria, which included whether they are employed in the
private and public sector, their managerial level, participants' experience, business type, business purpose and
interest, and whether their business had adopted any MLCS solution. A summary of the characteristics and
types is provided in Table 2, showing the diversity of our participants . The purposive and snowball sampling
techniques were employed to select participants who met the criteria and provided a holistic overview of the
research topic (Bryman, 2016). The data collected from the interviews were analysed using NVivo 12, which
is a Computer-Assisted Qualitative Data Analysis Software (CAQDAS). NVivo has been used to facilitate the
organisation and management of the data (Wong, 2008). The ability to identify and manage themes is essential
to this research. The participant’s name is hidden for confidentiality purposes and has been replaced with
numbers to facilitate refereeing and distinguishing of participants' viewpoints. All interviews were conducted
individually except interview 22, where two participants attended the same session together. Accordingly,
participants were named alphabetically to distinguish interference (22A, 22B).
T able 2. Considered criteria in the data sampling.
No. Criteria Type

1. Sectors ▪ Public ▪ Private
Managerial ▪ Entry-level position ▪ Higher management
2. ▪ Middle management
levels
Participant ▪ 1- 5 years ▪ More than ten years
3. ▪ 6 – 10 years
experience
▪ Banking & finance ▪ Entrepreneur
Business
4. ▪ Policing & security ▪ Information technology
types
▪ Education ▪ Medical sector
Beneficiaries’ ▪ Buyer ▪ Seller
5.
interest
6. ML adoption ▪ Adopting ML ▪ Not adopting ML
Common ▪ IT course certification ▪ Working in the field
7.
criteria ▪ IT degree holder ▪ Practising IT
▪ CEO ▪ Cybersecurity manager
▪ Cybersecurity consultant ▪ Head of AI association
▪ Chief IS officer ▪ IT automation manager
▪ Network security consultant ▪ Cybersecurity engineer
▪ Head of computer science ▪ Chief of technological solutions
8. Position/title ▪ Cybersecurity administrator ▪ Head of system development
▪ Deputy of CEO ▪ IS Manager
▪ Security manager ▪ Chairperson of IT department
▪ Cybersecurity policy and ▪ ML model creator
compliance analyst ▪ Cybersecurity solutions
▪ Data analyst consultant
4. Findings
In this section, the key findings of this paper are presented based on the analysis of the adopting individual’s
perceptions of MLCS capabilities, implementation, evaluation, and communication. We also present
commonalities and differences in the participants' perspectives, thereby gaining a diverse understanding. Figure
1 presents the research areas of interest relating to MLCS perceptions after analysing the data from interview
transcripts. The analysis revealed the following three primary areas and seventeen themes (illustrated in Figure
1):
(1) MLCS capabilities are where experts' perception is collected and analysed in terms of added value,
ability, and functionality in MLCS solutions.
(2) MLCS implementation demonstrates the perception of decision-makers and specialists regarding
MLCS. A specific focus is applied to the adoption, Critical Success Factors (CSFs), barriers, and
associated risks.
(3) Examines MLCS capabilities communication and evaluation from the decision-makers' and the
specialists’ point of view.
Figure 1 MLCS analytical themes identified in this investigation.
Now, we present a detailed discussion on the generated themes, along with participants’ perceptions of each
theme. We cross-reference the text with the participant by providing their participant number, matching the
table in Appendix B.
4.1. MLCS capabilities
The extracted perception of decision-makers and specialists on MLCS applications revealed seven themes,
and each will be demonstrated in the following seven subsections. The participant’s name is hidden for
confidentiality purposes and has been replaced with numbers to facilitate refereeing and distinguishing of
participants' viewpoints. Figure 2 presents an overview illustration of MLCS capabilities from the participants'
perspective. In the figure, the different perspective groups are presented.
Figure 2 MLCS capabilities perception, analytical themes, and description
4.1.1 Maturity and decision making

Participants considered various levels of maturity as one of the MLCS capabilities. The perspectives have
been divided into four categories. Table 3 shows the categorisation of participants based on ML maturity level
and decision-making. The concept of maturity relates to its ability to operate without human intervention, with
a fully mature system reliably autonomous. Partial maturity, on the other hand, involves ML being able to take
actions based on available data and analyse behaviour accordingly. However, human intervention is still
required to address any previously unseen situation. An immature system, meanwhile, is limited to repetitive
tasks such as providing recommendations or alerts for known threats or vulnerabilities, with all decisions and
unseen situations requiring human intervention. Lastly, the unknown category pertains to the participant's lack
of comprehension regarding ML functionality and maturity.
T able 3 Decision making & maturity level.
Maturity level Mature

Partially mature Immature Unknown
1, 3, 4, 6, 9, 10, 11, 12, 13, 14,
Participant opinion 2, 18, 21 22B 5, 7, 8,
15, 16, 17, 19, 20, 22A, 23, 24
Total 3 18 1 3
The first category believed MLCS is mature enough to perform independently, mainly in making or
supporting decision-makers. Two participants (4, 5) believed that the tool embedding ML should be able to
independently make decisions during the employee's abs ence based on its learning. Other participants (6, 10,
17, 21, 23) claimed that the maturity of MLCS depended on factors such as the availability of data, the maturity
of the model, and trends in technology where maturity is achievable in the presence of t hese factors. Finally,
one participant (18) provided specific detail and confidently stated that MLCS is mature enough to act to block,
alert, or filter anomalous behaviour from benign. Anomalous behaviour can be defined as data instances that
differ from others (Ahmed et al., 2016; Parkinson et al., 2018a). This can be seen in the participant’s response,
stating, “The next-generation firewalls have a lot of ML capabilities; it can block and filter email to detect
anomalies and warn you of abnormal behaviour, so we rely on it a lot in our work”. The specifics of their
response are explainable due to their technology-focused engineering role.
The second category of participants supported that MLCS is partially matured and requires human
intervention. Four participants (2, 6, 11, 24) illustrated that ML capability and maturity depend on the
application use case and the size of available resources, which may sometimes replace manual human tasks.
Others (3, 20) stated that the maturity of MLCS in making decisions should be monitored, and humans should
act as a supervisor of the decisions made by the machine. Participant 3 stated: “I think I will give it more than
70 percent, yet I am not sure how mature it is” This is to prevent the algorithm from making decisions which
could have an adverse impact. For this reason, understanding and interpret ing how the decisions are made is
essential (Burkart, 2021). Several participants (6, 9, 10, 11, 13, 16) claimed that MLCS could decide to a certain
extent, but it cannot control itself, which requires human intervention. A participant (22A) observed MLCS
capabilities as a trained function that will interact simply with the available data and will not be able to address
new and previously unseen data.
Participants in the third category had a limited understanding of MLCS maturity level. Three participants (5,
7, 8) illustrated a lack of knowledge of MLCS maturity level in making decisions with an optimistic pe rspective
on MLCS as a worldwide trend, as stated by participant 8: “I am not sure about ML maturity, I do not have
deep knowledge about this matter”.
The last category of participants claimed that MLCS is immature. One participant (22B) supported this
argument, stating that MLCS solutions interact with behaviour previously labelled malicious or after an attack
compromises a system. This participant also stated that ML would have limited capability in a large organisation
with thousands of employees, as it would become too difficult to identify patterns of interest. Further, they
expressed that if MLCS solutions were to learn knowledge that could be translated beyond the organisation,
there is potential to lose intellectual property, as illustrated by participant (22B): “So, you cannot get the
machine to go with an organisation with seven thousand or a hundred employees. Actually, I am going to learn
exactly what this employee does day in and day out; we'll then block him from the network because, on one day
a year, he copies some files, and he emails them somewhere”.
The insights gathered from interviews revealed that 11.1% of participants characterised MLCS as mature,
while 66.67% assessed it as partially mature, 3.7% as immature, and 13.5% refrained from expressing an
opinion. The diversity of perspectives within the community of cybersecurity professionals underscores the
significance of elucidating the potential strengths and weaknesses of MLCS solutions to enhance
comprehension and informed decision-making.
4.1.2 Consciousness
It has been established that participants hold divergent views on whether ML can exhibit consciousness when
dealing with threats and interacting with values in the datasets. Seven participants believe ML can possess some
level of consciousness or acquire it in the future with adequate training and time , as stated by participant (3):
“Definitely, yes, I do believe that strongly, and I think attempts are there, and what we have talked about for
that technology to be scary is all also there”. Further, participant (6) responded, “The consciousness, I think it
will get to at very later stages, something known as artificial general intelligence which will come and the
scientists believe in 2050”. However, all remaining participants believe ML lacks consciousness due to its
limited ability to learn from data and reflect on situations , as stated by participant (20): “Well, I do not think
machine learning has a consciousness; machine learning leads to the applications which can develop
consciousness”. Three participants firmly reject the possibility of ML becoming conscious, considering it
merely a tool for mathematical calculations based on data sets. Two participants question the need to imbue ML
with consciousness, citing a lack of evidence for its added value. Two participants either expressed scepticism
or advocated for more knowledge on the subject. Finally, the remaining participants view ML as a learning tool
that lacks the consciousness and other human attributes that differentiate humans from machines. The various
viewpoints of participants can be concluded as 31.82% of participants believe ML can exhibit consciousness,
while over half of participants 77.27% lean toward the idea that ML lacks consciousness, with various reasons
provided.
4.1.3 Predicting and detection of threats

The participants’ perspective of ML capabilities in threat prediction and detection is analysed in this section.
The opinions of the participants varied. For instance, participants oppose ML as an efficient tool for predicting
and detecting threats and limit its capabilities to a reactive tool that facilitates the detection process, excludin g
its ability to predict, as stated by participant (22B), “We do not have the machines warning us up front that it
might happen; it goes back to a point about kids ’ learning process falling over: you cut your knee and learn
from that experience”. On the other hand, proponents argue that ML could be developed to provide proactive
measures that enhance the detection and prediction of threats , such as participant (2) who responded, “Machine
learning is able to see the attack before it happened”. The upcoming paragraphs examine the similarities and
differences in the participants' points of view.
Over half the participants expressed that ML functions based on supplied information could predict the next
step, providing sufficient data is available. One participant (24) defines prediction as anticipating future actions
resulting from both a true and false prediction while being optimistic regarding the generated result of the ML
model. However, some participants (1, 3, 9, 17, 19, and 22B) were reticent about ML being capable of predicting
when operating in complex scenarios. Additionally, participants (8, 10, 14, and 24) added that ML could predict
a problem; however, further verification and validation by humans are needed to ensure the accuracy of the
prediction.
For ML capabilities in threat detection, participants (1, 20, and 22B) consider ML's capabilities on the
reactive side as it aids in analysing behaviour in the recovery stage and mitigating the consequences. They are
opponents of ML being an efficient tool for detecting threats. In contrast, several participants claimed that ML
could provide a proactive measurement in detecting the attack in the early stages or before execution, which is
faster than manual detection by providing real-time notification. Furthermore, one participant (13) claimed ML
facilitates detection and allows the end-user to feed information to the broader community regarding new
attacks.
Proponents of ML capabilities in threat detection illustrated that ML could accelerate the detection process,
minimise damages, and prevent the distribution of anomalies in the network. However, two participants (22A
and 22B) had limited ML capabilities to detect unknown threats not previously identified. These threats could
be part of a zero-day attack. One participant (9) has a mediocrity position by stating that a domain expert's
assistance is required for ML to classify the data and categorise the threats. Participants' perceptions can be
concluded that over 50% claim that ML could predict the next step, with 27.24% of participants reticent about
ML capabilities being predictive, while 13.64% claimed it requires further verification.
4.1.4 Creativity
This subsection aims to understand participants' perceptions of ML capabilities in dealing with new
challenges and threats. Most participants provided their views on ML capabilities , stating that it could exceed
conducting repetitive tasks, while ten participants had a different opinion on the subject.
According to the thirteen participants, ML capabilities go beyond repetitive tasks, and they can do tasks
exceeding human capabilities in specific areas. For instance, participant (1) described, “Now, they are trying to
reach the human brain's level. The human brain is not used to make repetitive tasks. Repetitive task is not even
the current situation anymore”. For example, one participant (5) stated that ML capabilities could interact
creatively with the given task. In contrast, two participants (6, 22B) claimed that ML exceeds human capabilities
and depends on resources to produce more innovative and reliable products/services. Furthermore, one
participant (7) indicated that the rapid development in the ML field cou ld exceed expectations. According to
one participant (21), ML could be creative and applied in critical sectors , such as the medical industry.
On the other hand, others expressed that ML can only perform routine and repetitive tasks. According to one
participant (2), humans are the machine's controller, and the machine lacks the skills and competencies obtained
by humans: “I can completely depend on machine learning to do the weekly and monthly scans internally and
externally to take the new exploit that some unknown hackers invent. Then, based on the report, you can use
your human skills. So, today, we use 70% of the ML in our daily job, and 30% depends on our skills” . Moreover,
participants (7, 18) stated that ML maturity in cybersecurity is insufficient to make decisions, and human
intervention is required to block, restrict, or allow any recommended actions .
Several participants (8, 9, 12, 19, 21) indicated that although ML is currently used for repetitive tasks, it will
gradually become faster and more efficient in the future. Additionally, two participants (9, 22B) expressed that
the manual methods are inadequate to cope with the rapid changes in cybersecurity, and the use of ML is
necessary to increase capability in detecting and responding to threats. Furthermore, three participants (1, 15,
22B) claimed that ML is partially creative as it eliminates specific jobs that require repetitive tasks. However,
ML is still limited in that it requires human intervention. In brief, participants' perceptions of ML creativity are
varied, with 52% viewing it as creative, 12% as limited to repetitive tasks, and 36% as partially creative.
4.1.5 Automation
Upon analysing participants' responses, it became clear that seven participants (9, 10, 11, 17, 18, 19, 22A)
cannot fully differentiate between ML and automation as they consider ML part of the IT system providing
routine/pre-defined functionality, for instance, participant (15) explained that “ML does help in automating a
lot of things coming around, in the matters of IT security”. However, nine participants (4, 5, 6, 12, 17, 18, 20,
21, 24) understand the difference between the two and recognise ML’s potential in behaviour analysis and
identifying patterns; for instance, participant (4) claimed: “Machine learning in cybersecurity is more than
automation; it is a comprehensive tool for alert or detection. Once is being used in the field to detect threats or
attacks”. Additionally, two participants (16, 22A) stated that organisations are moving towards entire ML
system automation due to a global shortage of individuals with cybersecurity expertise. The use of intelligent
tools embedded with ML is seen by many as a potential solution to this expertise gap.
Two participants (11, 22B) pointed out gaps in MLCS capabilities and the need for human intervention .
Further, they stated that adopting ML depends on the organisation's orientation towards balancing business
continuity and security. For example, ML may block application s and classify new users as malicious if
implemented at a fully automated level. In summary, 44% of participants demonstrate an understanding of the
distinctions between ML functionality and automation procedures, while 56% possess limited knowledge and
exhibit scepticism towards ML.
4.1.6 Behaviour analysis
This section presents the participants’ perceptions of ML capabilities in distinguishing between benign and
malicious behaviours. Most participants believe that behaviour analysis is a crucial feature that gives ML
advantages over other techniques. However, some participants fear behavioural analysis could be exploited to
understand organisational behaviour and launch an attack.
Except for participant (7), all participants believe ML is widely utilised and beneficial to monitor and detect
benign and malicious behaviours. For example, four participants (9, 12, 13, 19) from the banking sector state
that ML helps monitor customer profiles and analyse their transaction history. As stated by participant (9), "We
must identify and manage any unusual activities within a large volume of financial transactions . So, to achieve
that, you need ML; you need several things, such as increasing the speed of detection, lowering the cost of
manual monitoring, having a proper cyber analysis, or monitoring the increased number of cyber threats”. Other
participants (11, 16, 22A) anticipated ML prevalence in behaviour analysis for all business processes soon.
Conversely, one participant (4) expressed concerns about the limitations of ML in detecting maliciou s
behaviour. Furthermore, participants (14, 22A) claimed that the model could lack accuracy in detecting
abnormal behaviour, depending on the working hours and types of tasks. Additionally, two participants (11, 18)
were sceptical about ML's ability to recognise suspicious behaviour. Lastly, many participants (2, 3, 4, 6, 8, 9,
17, 18) believe ML could be misused for monitoring and analysing behaviour to initiate attacks and exploit
security vulnerabilities. In conclusion, 96% of participants recognise ML's prominent role in behavioural
analysis, with 12% anticipating its future prevalence. However, 16% express concerns about potential accuracy
reduction and 32% perceive it as potentially misused and a double-edged sword, highlighting its benefits and
risks in this context.
4.1.7 Human intervention and ML dependency
The participants’ perspectives on MLCS capability regarding ML dependency and human intervention are
categorised into two groups. Participants from the first group believed that ML could reduce human workload ,
which adds value to the business . However, the decisions taken by ML might not always be explainable and
require human intervention for validation purposes . Participants in this group also believed that news articles
about machines taking over or replacing humans altogether are speculative and full of hyperbole. Participant
(11) stated, “In detecting email spam, ML will help detect malicious emails. So, you will be almost 100%
protected if you do an automatic response. But many of your internal applications will be detected and discarded
as an attack or spam as they are written by employees differently with typing mistakes or different writin g
styles; that is why both should work together”. Furthermore, as stated by participant (24) “It helps the physician
make a decision, it would not, I do not think any soon replace any physician work or stuff like that”
The second group of participants argued that human beings have several limitations and constraints in
conducting tasks, and ML capabilities are better than human capabilities. It also aids in overcoming human error
and processing big data. Three participants believe ML development is replacing humans' role and control over
jobs. For instance, participant (10) stated, “ML can be reliable completely to replace a human and take over”.
In summary, 64% see human-ML collaboration as complementary, with 32% recognising ML's potential to
surpass human capabilities and reduce errors.
4.2. ML cybersecurity implementation
Table 4 shows participants' MLCS implementation status in which some have implemented, intended to
implement, and do not intend to implement ML in their businesses. Most participants have already implemented
ML, whereas some are in the process despite initial reluctance. Only one participant has not yet adopted ML
due to the complexity of policies and procedures for new technology adoption.
T able 4 Implementation status in participant entities
Implementati on Not implemented/ Not implemented/

Implemented
status have the intention no intention
1, 2, 3, 4, 6, 9, 10, 11,
Participants 12, 13, 14, 15, 16, 17, 5, 8, 20, 22A, 22B 7
18, 19, 21, 23, 24
Total 19 5 1
The generated beneficiary's perception of MLCS implementation revealed six themes, which are discussed
individually in the following subsections. Figure 3 illustrates MLCS implementation from the beneficiaries ’
perspective.
Figure 3 MLCS implementation perception, analytical themes and description

4.2.1 Critical success factors
Several CSFs were highlighted by participants that could facilitate the implementation of MLCS. Participants
emphasised the importance of the reliability of ML models (3, 6, 10, 13, 14, 17, 24) and their ability to detect
or predict a threat (4, 9, 11, 19). In addition, 10 participants believe that ML could reduce operational costs and
time, providing new business opportunities, increasing revenue, as stated by participants (14, 19, 22A), and
optimising resource utilisation (20, 22A). Accessibility and simplicity were also considered crucial factors for
adoption by seven participants, as well as the creative potential of ML, which is not limited to conducting
repetitive or tedious tasks as seen by 10 participants, and its compatibility with various business requirements
(1, 6, 17, 23). The participants (1, 5, 8, 9, 21, 24) also emphasised the importance of the team working on
developing ML products and services with a mixture of skills and abilities in strategic and data analytics.
4.2.2 Barriers
The theme explores the challenges that hinder the implementation of MLCS from the participants'
perspective. Three participants (3, 7, 18) argue that the development of ML technology requires counter-
measures due to the high level of development. Meanwhile, participants (1, 5, 7, 14) claim that measuring the
effectiveness of performance reasoning with the rapid development of technology is a critical challenge in
implementing ML technology. The absence of MLCS capability understanding and knowledge in the ML field
is also a significant barrier to adoption.
ML is associated with task-completion limitations that might limit the adoption or implementation of the
technology, such as detecting threats in the non -support language (2, 13, 15, 17). Furthermore, participants
pointed out that each ML type has different requirements and constraints, and language barriers could limit the
efficiency of ML. Most participants indicated that there is a lack of skilled and qualified human resources
worldwide; the human factor, the data cycle, and restrictions associated with accessing the knowledge base are
other significant barriers to adopting ML technology in cybersecurity (1, 5, 6, 11, 12, 18, 22B, 23, 24). In
addition, implementing ML is challenging and requires sufficient time and resources to learn and train (12, 14,
16, 22A). Furthermore, participants (4, 18, 20) claimed that cloud adoption policy might hinder organisations
from adopting MLCS due to the risk of data leakage. In contrast, one participant (9) proposed using a cloud
platform because of their data storage capacity and scalable processing capabilities . The participant also
acknowledged that allocating a budget for this in difficult financial economic times is challenging.
Finally, top management support is a crucial aspect of ML implementation. However, several participants
commented that top management lacks knowledge of true ML capabilities and is mainly impacted by media
momentum. Cybersecurity is not always perceived as valuable by the top management. It receives less attention
in project planning and design stages and is only called for in the execution stage, as illustrated by participants
(1, 6, 8, 9, 14).
4.2.3 Data processing
Here, we summarise the participants’ perspectives regarding the role of ML in processing data and its impact
on business. Firstly, a group of participants highlighted that implementing ML, if done correctly, can
significantly reduce manual labour and provide more time for creativity, as stated by participant (2): “Frankly
speaking, we close a whole department in our organisation because of the ML and AI. So, the whole job is
carried out by (RPA) Robotic Application Automation”. Three participants also stressed the importance of data
anonymity in cybersecurity and the need for organisations to adopt standards and policies to ease the
implementation of MLCS. On the other hand, one participant (9) considered the lack of labelled data as an
obstacle to implementing ML.
Secondly, many participants conditioned the efficiency of MLCS solutions on input training data. One
participant (18) emphasised the planning phase as critical to the success of the ML implementatio n .
Furthermore, one participant (24) alluded that ML could aid organisations in processing large amounts of data,
leading to a better understanding of business progress. Three participants (10, 12, 24) noted that no standard
approach is available for dividing data into training, testing and validation sets, as stated by participant (12),
“There is no standard to follow as it is case-based, and we have to make our choice”.
Lastly, many participants indicated that products and services that incorporate ML do not allow the end-user
to give input/suggestions in the development process. Participants (10, 12, 19) claimed that in their experience ,
ML is commonly utilised in classifying users and imposing limited access based on age restrictions to increase
privacy. However, despite understanding the theoretical aspects, several participants expressed a lack of
experience in the practical implementation of ML. Additionally, one participant (16), representing the security
sector, confirmed that they lack knowledge of the process of the ML model as the ir employer handles the
planning, data collection, and project recommendation phases , and they only conduct the execution phase as
stated by (16) “We have the (NCSC) National Cyber Security Center, they are helping us by providing the latest
ML technologies to have a good security application. So, basically, what did they do? They plan it; they collect
the data, which usually is our limitations in our organisation, then study what kind of projects in the wor ld
market, and then provide us with the recommended projects to work on. Then, the execution part is based on
the organisation itself”.
4.2.4 Adaptation
In this theme, the discussion centres around the participants' perception of the ML model best suited for
cybersecurity use and its adaptability to technology challenges and business requirements. Participant (22A)
emphasises ML's constant learning ability, which makes it an essential tool in developing services. Two
participants (4, 6) believe that ML's ability to adapt to change quickly and updates makes it a beneficial tool for
cybersecurity applications.
There is a discrepancy in the participants' views regarding the ML types best suited for cybersecurity.
The first group supports utilising supervised learning, which could provide more control of the data and model
and avoid data being poisoned or misconfigured ; as stated by participant (4), “Unsupervised learning is not
advised in cybersecurity because they are prone to machine learning poisoning ” and participant (23) who stated,
“I know only one ML type used in most cybersecurity solutions, which is supervised learning” . In contrast,
the second group argues that unsupervised learning is more practical in real-life situations as they criticised
supervised learning in coping with new viruses or attacking techniques . The third group suggests a hybrid
methodology combining different ML learning techniques to overcome individual techniques' limitations ,
indicating difficulties in model updating as stated by participant (17): “We can have a hybrid approach, where
you label a few entries and then train the algorithm to learn to label other entries” . Finally, the fourth group
lacks knowledge about the different ML types and their functionalities and performance measurements, makin g
adopting the required technology correctly challenging , as stated by participant (5): “I do not know very much
about that, unfortunately”.
Despite having detailed experience in many fields, most participants possess at least intermediate-lev el
knowledge of cybersecurity. However, some participants lack working knowledge of ML's components, details,
features, and types, which could lead to misleading adoption of the technology or setting unrealistic
expectations. This could be due to the skills shortage in the discipline and many employees being relatively new
to the discipline. Nonetheless, some participants confidently discussed the differences in the ML model usage
and learning technique. In brief, 20% of participants claimed supervised learning is adopted, 16% mentioned
unsupervised learning, 8% said a mixed approach, and 48% expressed a lack of knowledge about different M L
types.
4.2.5 ML application
From the participants' perspective, ML applications have the potential to make a significant impact in various
fields. Based on the analysis, participants' orientations toward ML implementation can be categorised into four
main areas.
The first category is the security and defence field. Several participants stated that ML is successfully used
in vulnerability assessment, spam filtering, intrusion prevention systems, and firewalls. Furthermore, ML has
strong potential for further development and deployment in cybersecurity and defence, particularly in countries
with significant funding for such projects (2, 3, 6, 7, 8, 18, 20, 23). Participants informed that ML has
applications in many cybersecurity-related areas, such as facial recognition, Auto Motor Vehicle Recognition
(AMPR), identifying criminal behaviour (1, 15, 16), and Digital Forensics and Incident Response (4, 22A).
Many participants reported that ML is being used in Network Operation Centers (NOC) and Security Op eration
Centres (SOC) for analysis tasks (4, 9, 11, 14, 15, 16, 18, 19, 22A, 23). Organisations heavily rely on ML to
identify email threats by analysing initial messages, considering keywords, urgent requests, and transactions
involving the sending or receiving of funds (2, 3, 9). In contrast, one participant (20) claimed that the utilisation
of ML in security applications is currently restricted and emphasised the need for an expert model to provide
recommendations and assess the readiness of infrastructu re and security measures. Other participants also
discussed that various ML techniques are used in cyberattacks, including zero-day exploits, web crawlers ,
Denial of Service (DoS) attacks, and polymorphic viruses. Several participants stated that ML has the potential
for dual benefits, enhancing security and testing security measures. For example, as stated by participant (13),
“This tool can work on both sides and now it depends on the human factor itself to decide either way; this can
be used in a good or bad way”.
The second category is where participants believe ML implementation is valuable in business, banking, and
finance. For example, participants mentioned ML being used to detect money laundering transactions and fraud,
reducing time and errors in manual transaction checks (6, 16, 19). In addition, reinforcement learning techniques
are applied in stock market trading, including Bitcoin and cryptocurrency . This is evident in participant (12)
response, “It has lots of implementations, for example, the stock market of Bitcoin or the Cryptocurrency”.
Collaborative ML and decentralised systems are also promising areas for ML development in creating services
and microservices (4).
Several participants highlighted the education sector with potential ML implementation. ML is seen as vital
for development in this sector, particularly in mitigating the consequences of the pandemic , as stated by
participant (7): “When COVID-19 started, people wondered how we would study from home. Now, after two
years, studying from home has become like a piece of cake because of the use of ML technology ”. For instance,
ML can be embedded in university dashboards to provide analytical reports on student performance, results,
and market requirements (20).
For the fourth category, the healthcare sector has also drawn attention to ML implementation, with several
participants highlighting its potential. ML applications in cancer diagnosis and disease-causing genetic analysis
were mentioned by participants, such as (5), “It could be used to diagnose and review certain diseases, such as
cancer diagnostic”. However, some workers in the medical field express a lack of trust in ML applications,
fearing replacement or a lack of understanding of ML model components (24). Nevertheless, ML is seen as a
tool that can provide predictive measurements in the medical sector (21). Participants perspective tends towards
security applications (58%), education (21%), healthcare (24%), and banking and finance (15%).
4.2.6 Adoption
There are four aspects to consider in MLCS adoption: motivation, reluctance, conditions, and impact. The
first aspect involves participants' motivation and encouragement for MLCS adoption. Five participants (2, 11,
12, 16, 19) stated that ML is necessary to increase organisation security. Many participants claimed that
adopting ML is influenced by an organisation's reputation in the market , and several participants adopted ML
technology based on their experiences and best practices (15, 16, 18, 22B). Participant (15) responded,
“Anything that comes has to be evaluated based on best practices, seeing whatever is in the market, comparing
it to another product, seeking professional views, and the organisation's reputation” . Digital transformation is a
significant factor driving the adoption of ML in the business process (4, 9, 15, 18, 19, 20, 23). Participants (2,
7, 18) stated that marketing and advertising are misleading about the true capabilities of ML.
The second aspect is where participants expressed reluctance to adopt ML technology for various reasons.
Some participants do not understand ML capabilities, while others fear misusing the technology (24). Other
participants claim that adopting ML will not provide a safe environment (22B), and unethical practices in data
utilisation are also a huge concern (8, 9, 12). In addition, the limitations of ML in detecting and responding to
new threats are a significant challenge (11). Additionally, participants (21) illustrated that adversaries'
endeavour to exploit security protection is a concern , stating, “Because, as the adversaries are using AI and ML,
then you would have to do better”. Two participants (11, 18) pointed out that ML adoption in large organisations
is limited to senior management for behaviour analysis and that entry-level positions may suffer from attacks
such as phishing as they might lack experience in how to use the systems .
From the third aspect, several participants conditioned ML adoption on various factors. Participants (7, 8,
15, 21, 22B) cited obstacles, such as policy and procedure in the security field, the need for a standard for ML
model selection, and geopolitical motivations for withholding certain ML capabilities . Participant (21) stated,
“It is basically evolving ML in geopolitics right now, which has another important element these days. So, that
definitely impacts technologies that are accessible, processors, and what is accessible or not”. Other participants
(7, 8, 9, 12, 14, 15, 16, 19) were conditioning MLCS adoption of Return on Investment (ROI), while some (6,
14, 17, 21) emphasised customisation of ML models to be compatible with various requirements . Participant
(22B) suggested short-term contracts to evaluate the technology's value. At the same time, participants (3, 7, 9,
16) believed there were fewer obstacles for MLCS to flourish, especially with the existence of computer
powering, cloud systems, and AI algorithms.
Finally, participants also discussed the impact of adopting ML on organisations. Participants (4, 12, 15, 19,
20, 22A, 23) stated that adopting ML could increase organisational efficiency by reducing workload. However,
one participant (11) claimed that ML adoption provides a false sense of security and only assists with limit ed
capability in detecting and responding to new threats, stating that “It gives them use of mind, but at the same
time, it is a false sense of security”. Two participants (7, 24) stated that no preview examples were observed in
either the defence or attack side. Participants (10, 11, 16, 18, 23) stated that ML is more suited to defence than
attack. Many participants claimed that ML capability is a double-edged sword that could impact an
organisation's assets and reputation, which necessitates a predetermined measure to enable optimum
implementation and reduce the associated risk.
4.3. Evaluating and communicating ML capabilities
The generated beneficiary's perception of MLCS in evaluating and communicating MLCS capabilities revealed
four themes, which are presented and discussed individually below. Figure 4 illustrates MLCS implementatio n
from the beneficiaries’ perspective.
Figure 4 MLCS evaluating and communicating capabilities perception, analytical themes and description
4.3.1 Communication
This analysis presents the challenges around communicating MLCS capabilities from the participants'
perceptions, along with their suggestions on how to resolve them. Several challenges were discussed from the
participant's perspective. Four participants (3, 4, 11, 18) noted that ML providers might selectively choose
performance measurements to maintain a competitive edge. Three participants (4, 18, 23) highlighted that ML
terminology is being abused to promote products/services , as stated by participant (4): “Today, ML,
unfortunately, is being abused in the cybersecurity field. The reason here is people or enterprises think that once
they acquire new technology, including ML, it will work as they are studying or reading about it online.” At the
same time, other participants (5, 6, 9, 11, 14, 15) claimed that vendors tend to hide model drawbacks and attempt
to convince buyers, focusing merely on the strength of the software, where they strive to get vendors to compete
to reveal the true capabilities. For example, participant (9) responded, “I play mind games with those guys. So,
I let them compete with each other’s. So, I invite two or three vendors to provide their sugges ted solutions, and
I tell them that I am looking at all these solutions; so, give me a comparison from your side” . Six participants
(3, 4, 11, 14, 16, 23) highlighted the lack of understanding of fundamental ML capabilities that could lead to
setting unmet aims, creating a false sense of security and difficulties in identifying true ML capabilities in a
product or service. Two participants (14, 23) emphasised the need to communicate the added value of ML to
end-users; highlighting that entities are influenced by vendor reputation promoting security applications rather
than evaluating the actual capabilities.
Participants (6, 12, 13, 15, 16, 21, 24) claimed that an organisation's dependency on external experts
regarding data confidentiality and trust is challenging. However, other participants (13, 15, 17, 19) claimed the
experts provided sufficient information for decision-making. While four participants (4, 13, 14, 20) viewed ML
as an embedded feature, others (18, 23) considered it a standalone tool. Vendors presenting accuracy was seen
as an influential factor in purchasing decisions . For example, as stated by a participant (19), “It depends on
vendor to vendor; they may have a very good team. So, the outcome of false positives is meagre, but certain
vendors still need some work or tweaks to enhance their ML embedded processes ”. Participant (6) stated that
the government's readiness for adopting MLCS still needs to be discussed and tested. Participant (24) says that
methods of information delivery should be distinguished depending on the audience. Another participant shared
their experience of unsuccessful implementation of MLCS and urged every user to perform extensive testing
before ML use.
Methods and techniques to improve communicating MLCS capabilities between ML beneficiaries were
indicated by participants in several aspects. Several participants suggested reducing dependence on external
expertise and training frontline staff to improve communication. Participant (6) suggested measuring
government readiness to implement MLCS and proposing initiating guidance for implementation and
evaluation. Participant (10) suggested that the producer or provider should provide performance measurement.
Two participants (20, 24) proposed testing the model and metrics on the organisation's system to demonstrate
capability. Many participants recommended a Proof of Concept (POC) product demonstration to verify the
seller’s claims. This can be seen in responses, such as that by participant (15): “I always say that it is a matter
of seeing is believing that is one thing; you would not spend lots of money until you test the solutions ”. In
contrast, participants (3, 6) doubt the entities' readiness to understand and evaluate the provided claims because
of the lack of ML knowledge.
Several participants recommended methods for improving communication and understanding of ML
capabilities, such as simulation models, ML assurance entities, NDAs or assurance clauses, regulatory policies,
threat modelling matrices, seeking consultation, training, and software assurance.
4.3.2 Associated risk
Organisations are implementing ML techniques in cybersecurity for different purposes, but there are debates
surrounding the risks associated with adopting ML. Through analysing participant perceptions, four types of
opinions were identified.
First, four participants (4, 6, 17, 19) consider ML an assistive tool to mitigate risk in business processes.
Second, a participant (14) considered ML integration as losing control over the business as well as taking
unexplainable actions. Third, six participants (1, 2, 3, 7, 11, 16) observed a lack of knowledge surrounding
MLCS implementation and proposed solutions for spreading awareness and understanding of risks associated
with MLCS implementation. For example, participant (18) “Yes, we need to work on the awareness of the
people at the moment of the associated risk”. Finally, other participants (10, 12, 14, 16, 18, 24) consider ML
implementation greatly benefitting businesses; however, MLCS should be designed to minimise the associated
risks. Further, two participants (15, 21) consider MLCS implementation dependent on the risk level and the
domain in which it is implemented. Five participants (1, 5, 8, 13, 23) either did not consider or were optimistic
about the risk associated with ML implementation.
Participants (7, 11, 16, 18, 22A, 22B) proposed solutions such as implementing a training scheme, creating
a risk assessment policy, and launching competitions to mitigate the risks associated with ML implementatio n ,
as stated by participant (7) “Unfortunately, many vendors do not have enough training for their products. I have
seen it many times, and they will sell XDR, but they have no training for the XDR. They will give seminars,
like a few days' training, but they still need to gain proficient training. We need to work on having enough
professionals with a proper training scheme”. In addition, they suggest that spreading awareness is necessary to
consider MLCS as an assisting tool to sustain security and not entirely dependent on it, as it migh t misclassify
benign to malicious behaviours and provide a false sense of security. In conclusion, participants hold varied
views on ML and risk: 34.48% optimistically highlight benefits and risk mitigation, while 41.38% adopt a
pessimistic stance, associating it with risks and limited knowledge. The remaining 24.14% display varied views,
influenced by subjectivity, neutrality, or neglecting associated risks.
4.3.3 Accuracy
The participants' understanding and perception of accuracy and accuracy-based performance measurements
in evaluating MLCS models are explored in this theme. Interestingly, three participants (1, 7, 16) found accuracy
challenging to quantify in cybersecurity, while a large group defined it as the percentage of true or false
outcomes in classification. Furthermore, accuracy was considered as continuous self-learning (6), achieving
effectiveness and efficiency (8), or being influenced by data availability and human intervention (10, 23). The
definition of accuracy ranged from risk probability (15) to understanding customer requirements (18). However,
participants acknowledged the difficulty of measuring accuracy and the need for updated metrics as the
environment evolves (1, 22A, 22B).
Numerous participants highlighted accuracy as a primary evaluation criterion for the model (13 participants).
Accuracy played a role in decision-making for eight participants, while others considered the model's outcome
result in the primary judgment factor and used POC demonstration for model justification (2, 5, 6, 7, 8, 9, 10,
12, 15, 16). Risk tolerance and accepted accuracy percentages varied among participants, with some accepting
compromises and manual measurements (1, 3, 11, 19), while others co nsidered high-risk levels unacceptable
below 98 or 99% (4, 14, 21). There was recognition by 10 participants that no product or service could achieve
100% accuracy, as stated by (14): “Some generate 100 % accuracy in testing. It is impossible to reach 100% of
accuracy. So, if you find out that, you should know something is wrong.”, and a few risks might be accepted
for competitive advantages. However, there were differing opinions, with some claiming that 100% accuracy
is achievable. Participant (5) stated, “They managed to do it with 100% accuracy because they tried and tried
again across different algorithms ”, and others deemed it unattainable and misleading (12). Moreover,
participants highlighted the lack of knowledge among cybersecurity professionals in assessing ML accuracy
and performance (6, 22B).
Eight participants viewed accuracy as performance measurement in cybersecurity as subjective and
dependent on the domain and risk tolerance. One participant suggested using precision instead of accuracy and
cautioned against relying solely on accuracy as a measure of trust (17). Participant (10) indicated that factors
such as data size, training, and testing affected accuracy results. Concerns were raised by participants (12, 19,
20, 21) about misleading practices to inflate accuracy results. An acceptable accuracy amount depends on the
cost factor (24). In conclusion, 52% of participants consider accuracy as the primary evaluation criterion of the
model, while 48% emphasise the need to verify generated results, such as POC.
4.3.4 Performance measurement
Implementations of MLCS have revealed a lack of understanding and consensus among participants
regarding the various aspects of capability, implication, and performance measurement. Two participants (3, 7)
emphasised assessing whether the ML model satisfies organisational requirements as an evaluation standard . In
contrast, others (2, 23) focused on performance metrics such as false positives, true positives, Key Performance
Risk (KPR) and Key Performance Indicator (KRI). Two participants (5, 15) likewise highlighted the trial-and -
error evaluation strategy and using POC approaches to minimise costs and achieve optimal solutions.
Additionally, participants (1, 5) emphasised the significance of tangible outcomes as indicators and the value
of personal experiences in validating ML.
Performance measurement in ML was found to vary based on factors such as detection rate, purpose, and
objective needs for protection. The evaluation of successful implementation was based on the outcomes and the
achievement rate according to participants (1, 4, 8, 11, 19, 21). Furthermore, evaluating ML performance also
involved metrics provided by vendors after project completion or based on preventive tools participants (1, 7,
19, 22A, 22B). Decision-making with minimal human intervention was considered an indicator of MLCS
performance by one participant (10).
Participants proposed different techniques individually (12, 15, 18, 22B) to evaluate ML model efficiency ,
including training and testing phases, accuracy measurements, vulnerability assessments, market research, and
seeking advice from trusted individuals. According to participant (12), accuracy is commonly touted as a
benchmark for measurement, “The accuracy is widely utilised and has been advertised in general”. In contrast,
from a personal viewpoint, it is imperative to emphasise the significance of promoting the F1 Score, which
integrates precision and recall s cores, particularly for models tailored to address real-world scenarios and
imbalanced datasets.
Moreover, some participants (6, 10, 17, 20, 24) emphasised the subjectivity of evaluating ML models based
on the specific aims, issues, domains, and target tas ks. Participant (1) claimed that performance measurement
could not be identified unless an attack was encountered. Additional knowledge and understanding of
cybersecurity performance measurements were identified as requirements by nine participants. The lack of
consensus on the evaluation process was acknowledged, and participants stressed the need for renewable
evaluation methods.
In summary, evaluating MLCS performance remains a complex and subjective process, with participants
expressing different pers pectives on capability assessment and performance measurement. Additional
knowledge, practical experience, and renewable evaluation approaches were suggested to improv e
understanding and establish more standardised methods. Table 5 summarises participants’ adopted performance
metrics to evaluate the MLCS model.
T able 5 Performance measurement techniques
Participants Performance measurement technique

17 + 6 Subjective to the usage and types
8 Lack of performance measurement understanding
2, 13, 19 False positive, false negative
3, 9 KPI, false positive
4, 11, 14, 16,
Detection rate (accurate & faulty)
19, 21
5 Personal inference and experience, trial, and error
1, 7, 19, 22A,
Successful or failed outcome
22B
10 Minimal human intervention
12 Model training and testing result, cost function, F1 score
15 Vulnerability assessments and monitoring tools.
18 Technical, market, and social research
20 Confusion matrix, F1, precision and accuracy
23 Accuracy, cost, false positive, time
22A, 22B Peers experience
24 Accuracy, recall, precision
5. Discussion
The adoption of MLCS has become increasingly prevalent in recent years, with organisations recognising
its potential to enhance their security measures. ML offers advanced capabilities in threat detection, anomaly
detection, and pattern recognition, among others, thereby strengthening cyber defence systems . However, the
successful implementation of MLCS requires careful consideration of various factors, including policy and
practice implications, knowledge and awareness, and the identification of potential limitations and
consequences. This work aims to investigate the perception of decision-makers and the factors they perceive to
influence. In this section, we discuss and justify why the study has highlighted the need for a standardised
approach before providing valuable insights into the adoption and implications of MLCS through our interview
analysis and published literature. This section is divided into three subsections: (1) the need for a standardised
approach is presented and discussed, (2) the implications of ML adoption in cybersecurity, highlighting the
need for policies and standards ; (3) the implications on knowledge and awareness ; and (4) potential limitation s
for future research.
5.1 The need for a standardised approach

The exploration of perceptions investigated in this article has identified three key central areas (capability,
implementation, and communication) concerning how decision-makers and practitioners perceive MLCS
applications. Although the analysis identified three key areas, each has a large set of themes. This is due to the
diversity in practitioner perspectives, which results from interviewing participants with a wide range of
expertise in the MLCS sector. This is evidenced in Table 2, which presents the different criteria created by the
study’s participants. The number of themes and expert criteria evidence that there is a large range of knowledge
and expertise in MLCS, meaning that it is unlikely that a single decision-maker or practitioner will be an expert
across the whole MLCS area. However, decision-makers and practitioners will likely need to know how they
can access appropriate expertise and knowledge in specific areas on demand. This is also likely to increase as
both ML and defensive cyber security technologies evolve, with more solutions being developed and deployed.
For example, there has recently been a surge in applications utilising large language models for adversarial
detection (Yinka-Banjo et al., 2020), and as yet, there is no standardisation or consensus on how their
performance is evaluated and communicated. Although the focus of this article is on machine learning
applications, the findings are translatable to the application of other forms of AI. The rapid development and
application of LLMs demonstrate the speed at which new AI technologies can be developed and adopted in
cybersecurity, which could exacerbate the key challenges identified in this article. This is where the use of a
standardised approach, involving the development and deployment of an appropriate standard would be
advantageous. This would ensure that all can understand MLCS capability, implementation, and communicatio n
while referring to a traceable and verifiable standard. Historically, this is something common with the
development and deployment of new technologies in the Computing discipline where there are multip le
stakeholders. For example, standards in software development and deployment (Pfleeger et al., 1994),
communication (Espina et al., 2014), and more recently, cloud computing (Moravcik et al., 2018) and artificial
intelligence. Taddeo et al, 2019 discuss the role of standards for making AI in cybersecurity reliable and
trustworthy. They also discuss that such standards can increase the possibility of adversarial attacks as defensive
solutions become more capable. The article concludes that standards are needed to focus on building reliable
AI cybersecurity solutions. However, it could be stated that ‘reliability’ in cybersecurity applications in a
dynamic environment is going to be challenging, and that a standard aiming to improve communication and
understanding of a solution can better understand its capabilities and consequences when a successful attack
does occur. The work presented in this article can steer the direction of future standards to help better understand
and communicate MLCS defensive solutions, along with promoting robust development practices.
5.2 Implications of policy and practice
The prevalence of ML in cybersecurity has drawn significant attention to the need for policy and standard
requirements in managing ML performance and ensuring better explanations of ML outcomes. In addition,
numerous research studies have highlighted the potential implications of MLCS and the challenges associated
with its deployment and execution. For example, a survey (Wazid et al., 2022) discussed the convergence of
cybersecurity and ML, its advantages, challenges, and a comprehensive comparative study of techniques for
different attack categories. Moreover, Chan et al. (2019) emphasised the importance of policy frameworks that
address ethical and legal concerns surrounding ML in cybersecurity. These frameworks can establish guidelines
for developing and deploying ML models, ensuring they adhere to fairness, transparency, and accountability
principles. Additionally, Apruzzese et al. (2023) sought to enhance the understanding of the role of ML in
cybersecurity by emphasising its benefits and discussing the challenges that impact its practical implementation
in real-world cybersecurity systems.
In addition to ethical considerations, policy and standard requirements are essential for effectively
controlling and managing MLCS researchers have identified the need for policies that provide clear guidance
on implementing and evaluating ML systems. For example, the European Commission (2021) argued that
policies should address issues related to data privacy, model explainability, and adversarial attacks on the ML
model. These policies aid in establishing best practices for ML deployment, ensuring the te chnology is utilised
securely and responsibly. Moreover, implementing policies and standards can promote collaboration between
industry and academia (Biddle et al., 2011), fostering the development of robust ML algorithms and frameworks
that enhance cybersecurity practices and culture (Da Veiga et al., 2020; Culot et al., 2019). Several government
initiatives provide a road map for ML implementation, such as UK NCSC guidance for assessing ML intelligent
techniques for cybersecurity (National Cybersecurity Centre, 2019). Furthermore, several practical
implementations of the Risk Management Framework (RMF) were introduced to mitigate risks. According to
Rodriguez (2021), organisations often adopt the RMF to mitigate risks, seeking a standardised approach to
respond proactively or reactively. The National Institute of Standards and Technology (NIST) has established
the RMF, offering standardised documentation that assists entities in identifying system risks and utilising
security controls to mitigate potential associated risks (Kohnke et al., 2017). Furthermore, ISO 31000:20 1 8,
recognised globally, serves as a prominent standard, providing practical guidance on principles and methods
for risk management (Rampini et al., 2019). However, Rampini et al. (2019) argue that the critical success
factors of implementing the RMF are debatable and warrant further investigation, mainly through quantitative
research on a larger scale.
Adopting and implementing policies and standards that provide clear guidance for MLCS is crucial due to
the current lack of communication standards for effectively conveying ML capabilities. The absence of such
standards poses several challenges in the field. Firstly, with clear guidelines, organisations will be able to
understand the performance and risks of MLCS systems clearly. This is something observed in other areas of
cyber security, such as when setting information security policies (Höne et al., 2002). This lack of
communication standards can lead to misunderstandings and unrealistic expectations, potentially undermining
the overall effectiveness of MLCS systems (Diesch et al., 2020). In addition, standardised communicatio n
frameworks can facilitate effective collaboration among stakeholders, such as cybersecurity professionals,
policymakers, and end-users (Guggenmos et al., 2022). This alignment of expectations and increased
transparency can enhance trust and confidence in MLCS systems, leading to broad er adoption and more
effective cybersecurity operations.
5.3 Implications for knowledge and awareness
The development of MLCS has brought numerous benefits, E.g., enhanced threat detection and response
capabilities, improved efficiency in analysing large volumes of data, and the ability to detect complex patterns
and anomalies (Chan et al., 2019). These advancements can potentially revolutionise cybersecurity practices
and mitigate emerging threats effectively. Research studies have highlighted the positive impact of MLCS on
cybersecurity. For example, Khan et al. (2022) found that ML-based intrusion detection systems demonstrated
higher accuracy rates compared to traditional rule-based methods. Additionally, ML algorithms have shown
promise in detecting previously unknown and sophisticated cyber threats, enhancing the overall resilience of
cybersecurity systems (Wazid et al., 2022).
The successful adoption and utilis ation of MLCS necessitate developing knowledge and spreading awareness
among beneficiaries. Industry certifications and training programs equip cybersecurity professionals with the
skills to utilise ML effectively. Zhang et al. (2021) emphasise the necessity of developing training schemes to
enhance entities' understanding of potential risks and their consequences. According to Khando et al. (2021),
raising awareness can be achieved by adopting best practices and implementing training courses to acknowledge
vulnerability management and apply business safeguards. However, Khan & AlShare (2019) argue that
customising training content may be challenging due to the rapid and continuous changes in the fields of ML
and cybersecurity. Additionally, other studies attempt to establish different approaches to overcome possible
shortages in knowledge. For instance, Monczka et al. (2021) presented a technique known as “competitive
bidding” or “competitive procurement” that involves soliciting bids or proposals from multiple vendors or
suppliers to secure the best terms, pricing, and overall value for a particular product or service. However, it is
argued that the traditional approach may be susceptible to delays (Karimi et al., 2020) and supplier collusion
(Jones & Kovacic, 2019), potentially conflicting with the required speed, prompt response, and handling
necessary in cybersecurity maturity (Kumar et al., 2023). In contrast, Harju et al. (2023) discuss the utilisation
of advanced technology in the procurement process and claim that it adds uncertainty due to cyberattacks and
risks associated with its implementation. Without adequate knowledge and awareness, one could misinterpret
ML outputs, make erroneous decisions, increase vulnerabilities, fail to integrate MLCS into existing
infrastructure properly, and undermine the full potential of ML to tackle cyber threats. There is also a risk of
perpetuating biases and unfairness in MLCS, which can have adverse societal impacts.
5.4 Limitations and future research:
It is essential to acknowledge the limitations of our study and discuss the potential for future research. The
qualitative nature adopted in this research limits the generalisability of the findings. Quantitative studies could
be employed to validate and expand the results obtained through qualitative data collection . Additionally, we
could focus on establishing policies and standards for communicating MLCS capabilities, followed by
experimental studies to test their effectiveness in real-world scenarios. The geographical sampling in this study
is constrained to specific nations due to the necessity of employing data collection methodologies that mandate
direct engagement with the sample population for interview purposes. Expanding the geographical scope of the
sample could significantly enhance the study’s robustness and facilitate more comprehensive result validation.
The discussion highlights the Importance of policy and practice implications, knowledge and awareness, and
the identification of limitations in the adoption of MLCS. Establishing policies and standards can guide
organisations in effectively leveraging MLCS capabilities while addressing potential risks. Furthermore,
enhancing knowledge and awareness through training programs and certifications can he lp stakeholders
maximise the benefits of ML in cybersecurity. Although this research has shed light on the subject and provided
an in-depth understanding, further research is needed to overcome the limitations and explore additional
avenues for ensuring the secure and effective implementation of MLCS.
6. Conclusion and Key finding
This section presents a summary of the key findings and offers a brief recap to underscore the research aim
and achievement of objectives in the conclusion subsection.
6.1 Key findings
In this section, the pivotal findings of the paper are explained concerning each area of interest:
A. MLCS Capabilities:
1.
Participant perceptions of MLCS capabilities varied based on experience and observation, lacking a
consensus.
2. Model maturity levels were diverse, with a notable acceptance of partially mature models.
3. 32% believed ML could exhibit consciousness, while 77% disagreed.
4. Half of the participants perceived MLCS as capable of predicting, while others viewed it
as reactive or requiring verification.
5. Approximately 52% recognised the model's ability to handle new threats, with 36% considering it
partially creative.
6. Participants, comprising 56%, expressed limited knowledge and scepticism about ML's self-
developing capabilities.
7. ML was considered critical in behaviour analysis, with reluctance about accuracy and potential
misuse.
8. A significant 64% saw human-ML collaboration as complementary, while 32% believed ML
exceeded human capabilities, posing potential threats.
B. MLCS Implementation:
1. Nineteen participants implemented MLCS applications, emphasising the need for standardisation (5
intending).
2. Top management support (7 participants) and team collaboration (6 participants) were vital for reliable
MLCS application implementation.
3. Unreasonable expectations, often influenced by media or organisational reputation, motivated
implementation.
4. Nine participants highlighted the significance of data, citing its shortage as a hindrance to ML model
efficiency.
5. Supervised learning was the most recognised ML type (48% lacked knowledge of ML types,
potentially leading to misguided expectations).
6. Security emerged as the top potential field for ML utilisation (58% agreement).
7. Entity reluctance was attributed to a lack of knowledge about MLCS application capabilities.
8. Achieving ROI was identified as a critical condition for MLCS applications.
C. MLCS Communicati on and Evaluation:
1. Up to 36% of participants claim vendors misuse of ML terminology and hide model drawbacks to
attract customers.
2. Lack of internal skills prompts hiring external experts, posing data leakage threats.
3. Model simulations and POC are vital for providing practical evidence of communicated capabilities.
4. Participants' perceptions are split: 34.48% are optimistic about ML benefit s, and 41.38% are
pessimistic about associated risks.
5. Evaluation criteria vary: 52% prioritise accuracy, while 48% emphasise the need to verify results.
6. PM selection is a critical case, potentially exploited by vendors lacking ML knowledge .
6.2 conclusion
This paper provides new insight into the perceptions of MLCS and emphasises the need for a standardised
approach. Our analysis revealed that the implementation of MLCS is undergoing significant shifts. However,
there is a lack of consistent understanding of the capabilities and consequences of ML. Additionally,
organisations have various knowledge levels regarding MLCS capabilities, leading to varied and inconsistent
approaches in their implementation and achieved aims.
Through qualitative interviews with 24 participants, this study investigated three main aspects of MLCS:
capabilities, implementation, and communication and evaluation of abilities. The re are apparent discrepancies
in understanding the actual value added by ML and the performance measurements utilised in MLCS. These
discrepancies highlight the need for establishing policies and standardisations that govern MLCS capabilities.
This would bridge the current gaps in knowledge and understanding by providing systematic and well-tested
tools and techniques , ensuring that ML is utilised to its full potential. MLCS users need to develop and share
the standard approaches for different applications to enable the successful integration of ML in cybersecurity
and enhance the overall security posture of the digital landscape.
References
Aftergood, S. (2017). Cybersecurity: The cold war online. Nature, 547(7661), 30 -31.
Ahmed, M., Mahmood, A. N., & Hu, J. (2016). A survey of network anomaly detection techniques. Journal of Network and Computer
Applications, 60, 19-31.
Ali, T . E., Chong, Y. W., & Manickam, S. (2023). Machine Learning Techniques to Detect a dDoS Attack in SDN: A Systematic
Review. Applied Sciences, 13(5), 3183.
Alshaikh, O., Parkinson, S., & Khan, S. (2023, February). On the Variability in the Application and Measurement of Supervised
Machine Learning in Cyber Security. In Ubiquitous Security: Second International Conference, UbiSec 2022, Zhangjiajie, China,
December 28–31, 2022, Revised Selected Papers (pp. 545–555). Singapore: Springer Nature Singapore.
Apruzzese, G., Laskov, P., Montes de Oca, E., Mallouli, W., Brdalo Rapa, L., Grammatopoulos, A. V., & Di Franco, F. (2023). T he
role of machine learning in cybersecurity. Digital Threats: Research and Practice, 4(1), 1-38.
Asiri, S., Xiao, Y., Alzahrani, S., Li, S., & Li, T . (2023). A Survey of Intelligent Detection Designs of HT ML URL Phishing
Attacks. IEEE Access.
Azungah, T . (2018). Qualitative research: deductive and inductive approaches to data analysis. Qualitative research journal, 18(4),
383–400.
Biddle, B., Curci, F. X., Haslach, T. F., & Marchant, G. E. (2011). The expanding role and importance of standards in the information
and communications technology industry. Jurimetrics, 52, 177.
Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health, 11(4),
589–597.
Bryman, A. (2016). Social research methods ( 5th ed.). Oxford University Press.
Burkart, N., & Huber, M. F. (2021). A survey on the explainability of supervised machine learning. Journal of Artificial Intelligence
Research, 70, 245-317.
Chan, L., Morgan, I., Simon, H., Alshabanat, F., Ober, D., Gentry, J., ... & Cao, R. (2019, June). Survey of AI in cybersecur ity for
information technology management. In 2019 IEEE Technology & engineering management conference (TEMSCON) (pp. 1–8).
IEEE.
Chidukwani, A., Zander, S., & Koutsakis, P. (2022). A Survey on the Cyber Security of Small-to-Medium Businesses: Challenges,
Research Focus and Recommendations. IEEE Access, 10, 85701-85719.
Clark, T ., Foster, L., Bryman, A., & Sloan, L. (2021). Bryman's social research methods. Oxford university press.Culot, G., Fattori,
F., Podrecca, M., & Sartor, M. (2019). Addressing industry 4.0 cybersecurity challenges. IEEE Engineering Management Review,
47(3), 79-86.
Da Veiga, A., Astakhova, L. V., Botha, A., & Herselman, M. (2020). Defining organisational information security culture—
Perspectives from academia and industry. Computers & Security, 92, 101713.
Dahiya, N., Gupta, S., & Singh, S. (2022). A review paper on machine learning applications, advantages, and techniques. ECS
Transactions, 107(1), 6137.
Dasgupta, D., Akhtar, Z., & Sen, S. (2020). Machine learning in cybersecurity: a comprehensive survey. The Journal of Defense
Modeling and Simulation, 19(1), 57–106.
De Simone, V., Di Pasquale, V., & Miranda, S. (2023). An overview on the use of AI/ML in Manufacturing MSMEs: solved issues,
limits, and challenges. Procedia Computer Science, 217, 1820-1829.
Diesch, R., Pfaff, M., & Krcmar, H. (2020). A comprehensive model of information security factors for decision-makers. Computers
& Security, 92, 101747.
Espina, J., Falck, T ., Panousopoulou, A., Schmitt, L., Mülhens, O., & Yang, G. Z. (2014). Network topologies, communication
protocols, and standards. In Body sensor networks (pp. 189-236). London: Springer London.
European Commission. (2021, April 21). Laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and
amending certain Union legislative acts. Brussels. Retrieved from https://eur-lex.europa.eu/legal-
content/EN/TXT/?uri=celex%3A52021PC0206
Ford, V., & Siraj, A. (2014, October). Applications of machine learning in cyber security. In Proceedings of the 27th international
conference on computer applications in industry and engineering (Vol. 118). Kota Kinabalu, Malaysia: IEEE Xplore.
Franco, M. F., Sula, E., Huertas, A., Scheid, E. J., Granville, L. Z., & Stiller, B. (2022, June). SecRiskAI: A machine learning-based
approach for cybersecurity risk prediction in businesses. In 2022 IEEE 2 4th Conference on Business Informatics (CBI) (Vol. 1, pp. 1-
10). IEEE.
Guggenmos, F., Häckel, B., Ollig, P., & Stahl, B. (2022). Security first, security by design, or security pragmatism–strategic roles of
IT security in digitalization projects. Computers & Security, 118, 102747.
Handa, A., Sharma, A., & Shukla, S. K. (2019). Machine learning in cybersecurity: A r eview. Wiley Interdisciplinary Reviews: Data
Mining and Knowledge Discovery, 9(4), e1306.
Harju, A., Hallikas, J., Immonen, M., & Lintukangas, K. (2023). The impact of procurement digitalization on supply chain resilience:
empirical evidence from Finland. Supply Chain Management: An International Journal, 28(7), 62-76.
Hennink, M., & Kaiser, B. N. (2022). Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Social
science & medicine, 292, 114523.
Höne, K., & Eloff, J. H. P. (2002). Information security policy—what do international information security standards say?. Computers
& security, 21(5), 402-409.
Jones, A., & Kovacic, W. E. (2019). Fighting Supplier Collusion in Public Procurement: Some Proposals for Strengthening
Competition Law Enforcement. Competition Policy International, Antitrust Chronicle, April.
Karimi, S., Zhakfar, Z., & Sarwary, M. I. (2020). Study of Excessive Bureaucracy in Construction Projects–Causes of Low Level of
Competition and Lengthy Tendering Process: A Case Study of Afghanistan. Int. J. Eng. Adv. Technol, 10(1), 66-73.
Khan, H. U., & AlShare, K. A. (2019). Violators versus non -violators of information security measures in organizations—A study of
distinguishing factors. Journal of Organizational Computing and Electronic Commerce, 29(1), 4-23.
Khan, M. N. R., Ara, J., Yesmin, S., & Abedin, M. Z. (2022). Machine learning approaches in cybersecurity. In Data Intelligence and
Cognitive Informatics: Proceedings of ICDICI 2021 (pp. 345-357). Singapore: Springer Nature Singapore.
Khan, S., & Parkinson, S. (2018). Eliciting and utilising knowledge for security event log analysis: An association rule mini ng and
automated planning approach. Expert Systems with Applications, pp. 113, 116 –127.
Khan, S., & Parkinson, S. (2018). Review into state of the art of vulnerability assessment using artificial intelligence. Guide to
Vulnerability Analysis for Computer Networks and Systems: An Artificial Intelligence Approach, pp. 3 –32.
Khando, K., Gao, S., Islam, S. M., & Salman, A. (2021). Enhancing employees information security awareness in private and public
organisations: A systematic literature review. Computers & security, 106, 102267.
Kohnke, A., Sigler, K., & Shoemaker, D. (2017). Implementing cybersecurity: A guide to the national institute of standards and
technology risk management framework. CRC Press.
Kruzikova, A., Knapova, L., Smahel, D., Dedkova, L., & Matyas, V. (2022). Usable and secure? User perception of four authentication
methods for mobile banking. Computers & Security, 115, 102603.
Kumar, S., Gupta, U., Singh, A. K., & Singh, A. K. (2023). Artificial Intelligence: Revolutionizing cyber security in the Digital
Era. Journal of Computers, Mechanical and Management, 2(3), 31-42.
Kvale, S., & Brinkmann, S. (2015). InterViews: Learning the craft of qualitative research interviewing ( 3rd ed.). Sage.
May, T ., & Perry, B. (2022). Social research: Issues, methods and process. McGraw-Hill Education (UK).
McDaniel, P., Knightly, E., & Li, B. (2016, August). Keynotes. In 2016 2 5th International Conference on Computer Communication
and Networks (ICCCN) (pp. i-iii). IEEE.
Miceli, M., Posada, J., & Yang, T . (2022). Studying up machine learning data: Why talk about bias when we mean
power?. Proceedings of the ACM on Human-Computer Interaction, 6(GROUP), 1-14.
Monczka, R. M., Handfield, R. B., Giunipero, L. C., & Patterson, J. L. (2021). Purchasing & supply chain management. Cengage
Learning.
Moravcik, M., Segec, P., & Kontsek, M. (2018, November). Overview of cloud computing standards. In 2018 1 6th International
Conference on Emerging eLearning Technologies and Applications (ICETA) (pp. 395-402). IEEE.
National Cyber Security Centre [NCSC]. (2019, April 18). Intelligent Security T ools: Assessing Intelligent Tools for Cyber Security.
https://www.ncsc.gov.uk/collection/intelligent-security-tools.
Parkinson, S., & Khan, S. (2018b). Identifying irregularities in security event logs through an object-based Chi-squared test of
independence. Journal of information security and applications, pp. 40, 52 –62.
Parkinson, S., Crampton, A., & Hill, R. (Eds.). (2018). Guide to vulnerability analysis for computer networks and systems: an artificial
intelligence approach. Springer.
Parkinson, S., Khan, S., Crampton, A., Xu, Q., Xie, W., Liu, N., & Dakin, K. (2021). Password policy characteristics and keystroke
biometric authentication. IET Biometrics, 10(2), 163-178.
Parkinson, S., Vallati, M., Crampton, A., & Sohrabi, S. (2018 a). GraphBAD: A general technique for anomaly detection in security
information and event management. Concurrency and Computation: Practice and Experience, 30(16), e4433.
Petrosyan, A. (2023, September 15). Annual cost of cybercrime worldwide 2017-2028. Statista. Global cybercrime estimated cost
2028 | Statista.
Pfleeger, S. L., Fenton, N., & Page, S. (1994). Evaluating software engineering standards. Computer, 27(9), 71 -79.
Pirca, A. M., & Lallie, H. S. (2023). An empirical evaluation of the effectiveness of attack graphs and MITRE ATT&CK matrices in
aiding cyber attack perception amongst decision-makers. Computers & Security, 130, 103254.
Potamos, G., T heodoulou, S., Stavrou, E., & Stavrou, S. (2023, March). Building Maritime Cybersecurity Capacity Against
Ransomware Attacks. In Proceedings of the International Conference on Cybersecurity, Situational Awareness and Social Media:
Cyber Science 2022; 20–21 June; Wales (pp. 87-101). Singapore: Springer Nature Singapore.
Rampini, G. H. S., T akia, H., & Berssaneti, F. T . (2019). Critical success factors of risk management with the advent of ISO 31000
2018-Descriptive and content analyzes. Procedia Manufacturing, 39, 894-903.
Rawindaran, N., Jayal, A., & Prakash, E. (2021). Machine learning cybersecurity adoption in small and medium enterprises in
developed countries. Computers, 10(11), 150.
Rawindaran, N., Jayal, A., & Prakash, E. (2022). Exploration of the Impact of Cybersec urity Awareness on Small and Medium
Enterprises (SMEs) in Wales Using Intelligent Software to Combat Cybercrime. Computers, 11(12), 174.
Rodriguez, J. F. (2021). Risk Management Framework on Platform Information Technology Systems: An Exploratory Qualitative
Inquiry of the Naval Sea Systems Command (Doctoral dissertation, Capella University).
Sarker, I. H., Kayes, A. S. M., Badsha, S., Alqahtani, H., Watters, P., & Ng, A. (2020). Cybersecurity data science: an overview from
machine learning perspective. Journal of Big data, pp. 7, 1–29.
Seng, S., Al-Ameen, M. N., & Wright, M. (2021). A first look into users’ perceptions of facial recognition in the physical world.
Computers & Security, p. 105, 102227.
Shaukat, K., Luo, S., Varadharajan, V., Hameed, I. A., & Xu, M. (2020). A survey on machine learning techniques for cyber security
in the last decade. IEEE Access, 8, 222310-222354.
Shaukat, K., Luo, S., Varadharajan, V., Hameed, I. A., & Xu, M. (2020). A survey on machine learning techniques for cyber sec urity
in the last decade. IEEE access, 8, 222310-222354.
Sun, N., Zhang, J., Rimba, P., Gao, S., Zhang, L. Y., & Xiang, Y. (2018). Data -driven cybersecurity incident prediction: A
survey. IEEE communications surveys & tutorials, 21(2), 1744-1772.
T addeo, M., McCutcheon, T ., & Floridi, L. (2019). T rusting artificial intelligence in cybersecurity is a double-edged sword. Nature
Machine Intelligence, 1(12), 557-560.Thomas, T., P. Vijayaraghavan, A., Emmanuel, S., T homas, T., P. Vijayaraghavan, A., &
Emmanuel, S. (2020). Machine learning and cybersecurity. Machine Learning Approaches in Cyber Security Analytics, pp. 37–47.
T ian, Z., Cui, L., Liang, J., & Yu, S. (2022). A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine
Learning. ACM Computing Surveys, 55(8), 1-35.
Usama, M., Qadir, J., Raza, A., Arif, H., Yau, K. L. A., Elkhatib, Y., & Al-Fuqaha, A. (2019). Unsupervised machine learning for
networking: T echniques, applications and research challenges. IEEE Access, 7, 65579-65615.
Walliman, N. (2021). Research methods: The basics. Routledge.
Wazid, M., Das, A. K., Chamola, V., & Park, Y. (2022). Uniting cyber security and machine learning: Advantages, challenges and
future research. ICT Express.
Wong, L.P. (2008). “Data analysis in qualitative research: a brief guide to using NVivo”, Malaysian Family Physician, Vol. 3 No. 1,
pp. 14–20.
Xue, M., Yuan, C., Wu, H., Zhang, Y., & Liu, W. (2020). Machine learning security: Threats, countermeasures, and evaluations. IEEE
Access, 8, 74720-74742.
Yinka-Banjo, C., & Ugot, O. A. (2020). A review of generative adversarial networks and its application in cybersecurity. Artificial
Intelligence Review, 53, 1721-1736.
Zhang, Z., He, W., Li, W., & Abdous, M. H. (2021). Cybersecurity awareness training programs: a cost–benefit analysis
framework. Industrial Management & Data Systems, 121(3), 613-636.
Appendi x A: Interview Questions

A. Indivi dual Experience
1. What is your current role?
2. Could you tell me a little about your career path?
3. Please tell me about your experience with working with ML/AI technologies .
4. Please tell me about your experience with working in cybersecurity.
B. Interpreting General Machine Learning capability

5. From your point of view, what are ML capabilities?
6. What are the different ML types?
7. Why do you think we continue to see growth in the use of ML technologies?
8. Do you and your organisation use or plan to use any ML techniques?
9. What kind of change/impact can ML technology bring to an organisation? And why?
10. How do you think ML technologies will develop in the future?
11. How can ML technologies be applied in the future?
C. Interpreting Cybersecurity Machine Learning Capability

12. What is the current scope to apply ML to cybersecurity?
13. What specific application examples have you seen in defence of an organisation?
14. What specific application examples have you seen in attacking an organisation?
15. Overall, do you think ML is best situated to be used in defence or attack? And why?
16. Have you noticed any trends in the use of ML in cybersecurity?
17. Have you or any of your collaborators implemented any ML in security applications? If yes, can you
talk about that experience? If not, why?
18. What is the process for developing and applying ML techniques for security applications?
19. What are the barriers and success factors of ML implementation?
D. Understandi ng interpretations of how it is measured.

20. How are ML techniques evaluated in cybersecurity applications?
21. How do you define accuracy in general and particularly in cybersecurity?
22. When you are trying to conduct evaluation processes, do you have different ways of evaluating
depending on the type of ML used and the type of security application? And why?
23. What different types of evaluation measurements do you use/exist and why/what for?
24. If/when purchasing or commissioning an ML product, are performance metrics provided by the
technology provider and do you validate them?
25. In the purchasing or selling process, do you think the available information is enough to make an
informed decision? And why
26. What do you think can be done to help communicate ML cybersecurity application capabilities?
E. Miscellanies (additional info)

27. Do you have anything to add that we have not covered?
Appendi x B: Participant’s Overview Background

Part. Business
Working sector Position Field of interest Country
No. Type
1. IT CEO IS, Cybersecurity & AI Bahrain Private
2. Banking sector Cybersecurity manager IS & Cybersecurity Kuwait Private
3. Banking sector Cybersecurity manager IS & Cybersecurity Bahrain Private
AI & cybersecurity
4. Business sector Cybersecurity consultant UAE Private
solution
IT/IS solution
5. Cybersecurity manager IS & Cybersecurity UK Private
provider
Education –
6. Head of AI association AI Bahrain Public
Security
Education -
7. Head of IT IS & IT Bahrain Public
Security
Education -
8. Trainer AI & Cybersecurity Bahrain Public
Security
9. Banking sector Chief IS officer IS & Cybersecurity Bahrain Private
Cybersecurity &
10. Business sector IT Automation Manager Bahrain Private
Automation
Cybersecurity solutions
11. IT IS & Cybersecurity Bahrain Private
consultant
12. Finance sector Data analyst AI, DL & ML Bahrain Public
13. Business sector Network security consultant IS & Cybersecurity KSA Private
14. IT Cybersecurity engineer Cybersecurity solution KSA Private
Security - Chief of Division of
15. IS & Cybersecurity Bahrain Public
Policing Technological Solutions
Security -
16. Head of System Development IS & Cybersecurity Bahrain Public
Policing
Education &
17. Lecturer AI and cybersecurity UK Private
Academic
Security - IS & Cybersecurity
18. Cybersecurity administrator Bahrain Public
Policing solution
19. Banking sector IS Manager IS & Cybersecurity Bahrain Private
Education &
20. Chairperson of IT department AI and cybersecurity Bahrain Private
academic
21. IT Deputy of CEO IS, Cybersecurity & AI KSA Private
22A Head of IT
Security IS & Cybersecurity UK Public
22B Security Manager
Cybersecurity policy and
23. Business sector Cybersecurity solution KSA Private
compliance analyst
24. Medical sector ML model creator ML USA Private
Biographical Sketch
Omar Alshaikh is a dedicated police officer at the Ministry of Interior in Bahrain and currently pursuing a PhD
at the University of Huddersfield. With a passion for artificial intelligence and leadership, he actively seeks to
integrate cutting-edge technology into police sciences to enhance pedagogical methods and address evolving
security challenges. Omar's academic journey includes the achievement of an MSc in Operations Management,
equipping him with essential knowledge in strategy, planning, and managing operations . Furthermore, he has
earned the prestigious distinction of being a Fellow of the Higher Education Academy in the United Kingdom.
Omar's commitment to the field extends beyond his professional duties as he conducts research and publishes
papers focusing on the intersection of artificial intelligence and cybersecurity, making valuable contributions
to this dynamic domain.
Simon Parkinson is Professor of Cyber Security and a member of the UK Government’s Cyber Security
Advisory Board (GCAB), and he has been researching topics on the interface between AI and Cyber Security,
with a focus in transport. He has led a variety of research and knowledge exchange projects, with funding
coming from EPSRC, EU Commission, DSTL, amongst others. He has authored numerous papers on these
topics as well as other cross -discipline applications of artificial intelligence, where he has a special interest in
Automated Planning.
Saad Khan is a Senior Lecturer in the School of Computing and Engineering at the University of Huddersfield
and part of the Centre for Cyber Security. Khan’s research interest involves developing and utilising artificial
intelligence and machine learning techniques for cyber security in various domains, such as SIEM systems,
vulnerability, and anomaly detection, learning domain knowledge, mitigation planning and access control.
Credit Author Statement

Omar Alshaikh: Conceptualisation, Methodology, Investigation, Data curation, Writing-
Original draft preparation. Simon Parkinson: Project administration, Supervision, Writing -
Review & Editing, Resources. Saad Khan: Formal analysis, Validation, Writing - Review
& Editing.
Declaration of interests
☒ The authors declare that they have no known competing financial interests or personal
relationships that could have appeared to influence the work reported in this paper.
☐ The authors declare the following financial interests/personal relationships which may be
considered as potential competing interests:

Alshaikh 1 s2.0 S0167404823006041 Main

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Alshaikh 1 s2.0 S0167404823006041 Main

Uploaded by

Copyright:

Available Formats

Journal Pre-proof

Exploring Perceptions of Decision-Makers and Specialists in

Omar Alshaikh , Simon Parkinson , Saad Khan

To appear in: Computers & Security

Received date: 19 September 2023

© 2023 Published by Elsevier Ltd.

E-mail address: Omar.alshaikh@hud.ac.uk

• The establishment of three key analytical themes of capabilities, implementation, communicatio n

T able 1 ML advantages and disadvantages

Category Points Reference

3.1. Research approach

3.2. Data collection

3.3. Data analysis

T able 2. Considered criteria in the data sampling.

No. Criteria Type

Figure 1 MLCS analytical themes identified in this investigation.

4.1. MLCS capabilities

Figure 2 MLCS capabilities perception, analytical themes, and description

4.1.1 Maturity and decision making

T able 3 Decision making & maturity level.

Maturity level Mature

4.1.3 Predicting and detection of threats

4.1.6 Behaviour analysis

4.1.7 Human intervention and ML dependency

surpass human capabilities and reduce errors.

4.2. ML cybersecurity implementation

T able 4 Implementation status in participant entities

Implementati on Not implemented/ Not implemented/

Figure 3 MLCS implementation perception, analytical themes and description

4.2.1 Critical success factors

4.2.3 Data processing

4.3. Evaluating and communicating ML capabilities

4.3.2 Associated risk

4.3.4 Performance measurement

T able 5 Performance measurement techniques

Participants Performance measurement technique

5.1 The need for a standardised approach

5.2 Implications of policy and practice

5.3 Implications for knowledge and awareness

5.4 Limitations and future research:

6. Conclusion and Key finding

6.1 Key findings

Appendi x A: Interview Questions

B. Interpreting General Machine Learning capability

C. Interpreting Cybersecurity Machine Learning Capability

D. Understandi ng interpretations of how it is measured.

E. Miscellanies (additional info)

Appendi x B: Participant’s Overview Background

Credit Author Statement

You might also like