Behaviour & Information Technology

ISSN: 0144-929X (Print) 1362-3001 (Online) Journal homepage: https://www.tandfonline.com/loi/tbit20

User preference of cyber security awareness

delivery methods

Jemal Abawajy

To cite this article: Jemal Abawajy (2014) User preference of cyber security
awareness delivery methods, Behaviour & Information Technology, 33:3, 237-248, DOI:
10.1080/0144929X.2012.708787

To link to this article: https://doi.org/10.1080/0144929X.2012.708787

Published online: 01 Aug 2012.

Submit your article to this journal

Article views: 6217

View related articles

View Crossmark data

Citing articles: 98 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=tbit20
Behaviour & Information Technology, 2014
Vol. 33, No. 3, 237–248, http://dx.doi.org/10.1080/0144929X.2012.708787

User preference of cyber security awareness delivery methods

Jemal Abawajy*
School of Information Technology, Deakin University, Waurn Ponds, 3220 Australia
(Received 10 February 2012; final version received 21 June 2012)

Operating systems and programmes are more protected these days and attackers have shifted their attention to
human elements to break into the organisation’s information systems. As the number and frequency of cyber-attacks
designed to take advantage of unsuspecting personnel are increasing, the significance of the human factor in
information security management cannot be understated. In order to counter cyber-attacks designed to exploit
human factors in information security chain, information security awareness with an objective to reduce information
security risks that occur due to human related vulnerabilities is paramount. This paper discusses and evaluates the
effects of various information security awareness delivery methods used in improving end-users’ information security
awareness and behaviour. There are a wide range of information security awareness delivery methods such as web-
based training materials, contextual training and embedded training. In spite of efforts to increase information
security awareness, research is scant regarding effective information security awareness delivery methods. To this
end, this study focuses on determining the security awareness delivery method that is most successful in providing
information security awareness and which delivery method is preferred by users. We conducted information security
awareness using text-based, game-based and video-based delivery methods with the aim of determining user
preferences. Our study suggests that a combined delivery methods are better than individual security awareness
delivery method.
Keywords: information security; human factor; awareness delivery methods; social engineering; information security
programme

1. Introduction factor. A substantial proportion of security breaches

Organisations, irrespective of size or form, have
become increasingly dependent on information tech-
nology to maintain competitive edge and improve
services. At the same time, these organisations are
facing information security risks from a vast array of
threats that range from spam emails to well organised
and complicated attacks such as malwares (malicious
software) that could steal, corrupt or destroy data as
well as ruin or render systems unusable. The outcomes
of information security breaches can range from less
damaging such as productivity loss to the worst case
that could lead to loss of sensitive information that
might lead to huge financial liabilities and losses,
adversely affecting the reputation of the organisation.
In some cases, the organisations may never recover
from the incident at all or it may take quite a long time
to return to a normal pre-incident levels.
Attackers often choose the path of least resistance
which is mainly the unintentional vulnerabilities
created by the human factors. As a result, cyber
security threats that exploit human behaviour are
constantly evolving. In many information security
violations, people tend to be the major contributing
originate from inside the organisation mainly due to
the users’ ignorance or careless behaviours such as
sharing passwords and opening unknown e-mails and
attachments. These activities might potentially open up
the organisation to deliberate threats from hackers and
jeopardise the assets of the organisation. The mistake
that enabled a hacker to infiltrate Google’s internal
systems earlier this year is an example of how an
employee can undermine all the security counter-
measures that might be in place in an organisation
(http://www.theregister.co.uk/2010/01/14/google_china_
attack_analysis/). The attack was successful because an
employee clicked on a link in a phishing email that
subsequently directed the employee to a website set up
by the attackers, which then downloaded malware onto
the employee’s computer.
Although significant information security risks
resulting from human factors are still of primary
concern, organisations have and continue to invest in
technology-based information security solutions (e.g.
firewalls, antivirus software, and intrusion detection
system) to defend organisational assets. Certainly,
technology-based information security solutions are

*Email: jemal@deakin.edu.au

Ó 2012 Taylor & Francis

238 J. Abawajy
very important part of information security pro-
grammes. However, the state-of-art technology-based
security solutions alone cannot provide the overall
security necessary to defend organisational assets from
the wide variety of threats. As many aspects of current
information security systems rely on the users and their
decisions, even state-of-art information security sys-
tems can provide a false sense of strong protection
against all attacks. As the information security
intrusions and losses have escalated, so too have the
number of information security laws and regulations.
Moreover, the ever-growing body of information
security laws and regulations has made information
security a critical issue requiring the organisations to
act decisively to improve information security. Thus,
organisations need to develop and promote an effective
information security programmes to counter these
threats.
The number of layers of technological defences can
be as strong as possible, but information security is
only as strong as its weakest link. Managing the
human side of the information security just as carefully
as the technical side, therefore, is paramount to
decreasing information security threats due to hu-
man-related vulnerabilities and improving an organi-
sation’s overall IT security posture. As organisations
become more reliant on technology to achieve their
business objectives, it is critical to continuously
enhance the security awareness culture in organisations
and transform this culture into actual security con-
scious behaviours. There is ample evidence that, with
the right security awareness, people can become an
organisation’s strongest defence against security
threats (Kumaraguru et al. 2007). Therefore, organisa-
tions must combat user awareness vulnerabilities with
effective information security awareness programmes
in order to achieve an adequate security posture.
Information security awareness programme with the
objectives to create a security-conscious environment
by removing vulnerabilities associated with human
behaviours is paramount for securing organisational
assets.
In this paper, we evaluate the various channels (e.g.
screen savers, interactive videos, emails/posters, news-
letters, class room training sessions, social media)
through which security awareness can be delivered.
Although it is important for a security awareness
programme to ensure that the appropriate topics are
covered, the critical success factor for a security
awareness programme is the delivery methods (Shaw
et al. 2009). There are many forms of information
security awareness delivery methods ranging from the
distribution of messages (e.g. via pamphlets, e-mails,
intranet pages, screen savers, posters, mouse pads and
pens) to formal presentations (e.g. lunch meetings, and
training courses) to improve users’ information secur-
ity performance. There have been some researches on
the efficiency of various information security delivery
methods. However, research is scant regarding effec-
tive delivery method of information security aware-
ness. Also, a side-by-side comparison of different
awareness delivery methods is lacking (Cone et al.
2007). Moreover, there is very little work that looked
at the security awareness delivery methods from the
recipients’ point of view. To address these gaps, this
study focuses on determining what delivery method is
preferred by users and most successful in providing
information security awareness.
The rest of the paper is organised as follows: In
Section 2, we present the background information
regarding information security programmes and the
review of the information security awareness techni-
ques. We also discuss social engineering attacks with
emphases on phishing since we focus on information
security attacks that exploit user awareness vulner-
abilities with particular emphases on phishing attacks.
In Section 3, a classification of information security
delivery methods is presented. In Section 4, the
methodology used in this paper is described. Also, a
description on how each delivery method was applied
and evaluated is discussed. The results of the study and
the discussion are presented in Section 5 and Section 6
respectively. The conclusion is presented in Section 7.
2. Background
As information security becomes a central concern
across organisations, it is essential that sound informa-
tion security programme is implemented to ensure
proper protection of the organisational assets. In this
section, we present the background information
regarding information security programmes and the
review of the current awareness techniques. We also
discuss social engineering attacks with emphases on
phishing since we focus on information security attacks
that exploit user awareness vulnerabilities with parti-
cular emphases on phishing attacks.
2.1. Information security awareness programme
Information security awareness can be defined as the
level of comprehension that users have about the
importance of information security best practices.
Generally, employees in any organisation have varying
levels of security awareness. They are also increasingly
engaging in dangerous online activities such as social
networking, blogging and instant messaging with a
considerable number of them unaware of their
exposure to security risks while doing so (Shaw et al.
2009). Employee behaviour is the primary source of
 Behaviour & Information Technology
costly data breaches. According to a recent survey of
IT security practitioners by Ponemon Institute, a
privacy and information management research firm,
employee negligence or maliciousness is the main cause
of many data breaches (Ponemon Institute 2012). The
survey indicates that 78% of respondents reported that
their organisations have experienced a data breach as a
result of negligent or malicious employees or other
insiders. Information security breaches can lead to
both direct cost (e.g. loss of intellectual property) and
indirect costs (e.g. loses of reputation and potential
loss in market share).
There is ample evidence that security awareness
training is the most cost-effective form of security
control (Albrechtsen and Hovden 2010). For example,
Dodge (2007) conducted experiment on the level of
awareness of a phishing attack after training. The
results show that the training was effective as the
amount of students falling victim to the phishing scam
dropped. These results show that awareness levels can
be increased through interactive content. This also
indicates that the medium through which awareness
material is provided also plays a significant role.
Similarly, Eminagaoglu et al. (2010) showed that
awareness campaigns can have a positive effect on
reducing security threats. In their study, the results
showed that weak password usage was significantly
decreased and users continually improved their aware-
ness and complied with policies after under-going a
security awareness training course. As information
security awareness reduces both the number and the
extent of information security breaches, this translates
in reduction of both the direct and indirect costs that
result in information security breaches.
Another major deriver to perform security aware-
ness training is that information security threats from
organised crime are on the rise. Similarly, customers
are more sensitive to security issues than in the past.
Another reason is to comply with many regulations
that mandate user awareness training programmes to
be a critical part of the administrative foundation of a
secure information processing environment. For ex-
ample, the Payment Card Industry Data Security
Standard (PCI-DSS) mandates information security
awareness training for any business or organisation
that accepts credit cards or processes cardholder
information. Information security awareness is also a
critical element in increasing the level of staff aware-
ness and compliance with the information security
policies, practices and relevant guidance related to
their duties in the organisation.
Increasing the information security awareness of
users is an important part of a holistic approach to
information security management. In order to decrease
information security threats caused by human-related
239
vulnerabilities, an increased concentration on informa-
tion security awareness is necessary. The main reason
to provide security awareness is to modify employees’
behaviour and attitude toward information security
management. Generally, an information security
awareness programmes are intended to create organi-
sational-wide security-minded cultures so that people
work in a more secure manner and protect the assets of
their organisation. Information security awareness is
about establishing, promoting and maintaining good
security habits as a critical element of an effective
information security management. The fundamental
goal of information security awareness is to create a
change in attitudes to bring about an overall change in
the organisational culture. The cultural change is the
realisation that information security is critical because
a security failure has potentially adverse consequences
for everyone. Also, information security awareness
aims to increase users understanding of how to follow
responsible computing practices and why it is neces-
sary. This increased staff awareness should translate in
the reduction of the likelihood of accidental breaches
and increase the probability of suspicious activities
being recognised and reported.
Knapp et al. (2007) has identified a positive
correlation between user information security aware-
ness programme and perceived security effectiveness.
The success of information security programme
ultimately depends on establishing a security positive
environment, where the end-users understand and take
part in the behaviours that are expected of them. The
ideal information security programme is to make all
employees of an organisation to share responsibilities
for the security of information and information
systems accessible to them. In order to combat user-
based vulnerabilities and improve the security aware-
ness levels of users, a robust awareness programme is
required to ensure that people understand their
information security responsibilities, organisational
policies, and how to properly use and protect the
organisational assets. Also, to keep the users abreast
and refreshed, any awareness programme must be
ongoing and be an integral part of the organisational
culture. Providing continuing security awareness is
relatively cost-effective given the fact that a large data
breach caused by one staff could cost an organisation
millions in resources and diminish reputation. More-
over, as information security awareness is a dynamic
process, any awareness should be continually mea-
sured and managed to keep current of changes in risk
profiles (Kruger and Kearney 2006).
Although the enormity of the problem associated
with effective user security awareness is evident (Cone
et al. 2007), it is often an ignored element of an
organisation’s security programme. Since information
240 J. Abawajy
security is both a human and a technological problem,
it is just as important to invest in the human-related
vulnerabilities such as user awareness and operator
carelessness (Abawajy et al. 2008).
2.2. Social engineering attacks
Social engineering is an important problem to address,
because it specifically targets the ‘people link’ that
information security officers are trying to strengthen.
In this paper, we will focus on information security
attacks that exploit user awareness vulnerabilities with
particular emphases on phishing attacks. Phishing is an
increasingly sophisticated attack in which cyber
criminals use spoofed emails and fake web sites to
deceit people into giving up personal information
(Islam and Abawajy 2012). Phishing attacks exploit the
fact that users tend to trust email messages and web
sites based on cues that actually provide little or no
meaningful trust information. They tend to target the
most common activities (email and web) that the
majority of users spend substantial times on. Also,
phishers are increasingly setting their sights on social
networking sites such as LinkedIn, Myspace, Face-
book and Twitter. Some people may be aware of
phishing, but they do not associate that awareness to
their own vulnerability or to strategies for identifying
phishing attacks (Downs et al. 2006). The study also
suggests that while people can protect themselves from
familiar risks, they tend to have difficulties generalising
what they know to unfamiliar risks.
Lack of awareness of the dangers of reverse social
engineering attacks can result in an unsuspecting
employee disclosing company confidential information
or render sophisticated information security technolo-
gies useless. Although automated systems can be used
to identify some fraudulent email and web sites, these
systems are not completely accurate in detecting
phishing attacks (Sheng et al. 2007). For example,
Valentine (2006) explained how he was able to walk
into a retail chain store and obtained the keys to the
store’s server room without even being asked to
provide any type of identification. Technology-based
solution alone could not provide bullet-proof line of
defence against attacks that exploit human-related
vulnerabilities such as phishing. Therefore, it is
necessary for organisations to have the tools to protect
people from such attacks as well as methods for raising
the people’s awareness not to fall for phishing attacks.
Proper information security awareness is an effec-
tive way to defend against social engineering attacks.
Continuous information security awareness is required
to sustain a desirable level of information security
awareness (Abawajy and Taihoonn 2010). Also, the
security awareness programme should be developed
with the aim of inculcating truly positive work habit to
better protect organisational assets and prevent in-
formation security incidents. The key to success in
awareness is keeping the messages relevant and
consistent and at the same time make the delivery
mechanisms interesting to everyone (Kruger and
Kearney 2006). A major challenge with security
awareness programmes is the lack of a fully developed
methodology to deliver them (Valentine 2006). Re-
search on the delivery method of security awareness
training concluded that the delivery of security
awareness information is just as important as ensuring
that the information is relevant and consistent (Shaw
et al. 2009).
3. Information security awareness delivery methods
The main goal of a security awareness programme is to
raise employee information security awareness in an
organisation. As with any programme, the success of
information security awareness programme will rely
heavily on how the awareness information is delivered.
There are many information security awareness
delivery models. These models are capable of raising
the awareness of employees about a wide array of
cyber security problems that ranges from spam and
phishing to well organised attacks intended to corrupt
or disable systems. In this section, we will review the
various information security awareness delivery meth-
ods commonly used to provide a context for subse-
quent discussion of our work.
3.1. Conventional delivery methods
The conventional methods of delivering information
security awareness encompass electronic resources
and paper resources. Paper-based security awareness
delivery methods include leaflets and posters with
attention grabbing slogans on relevant topics
(e.g. reminding users that passwords are not to be
shared) and newsletters (news clipping, memos and
newsletters).
Posters are commonly used to announce only one
topic at any given time. They are commonly displayed
in areas where people gather such as cafeterias and
meeting rooms. They are used to highlight time-
sensitive issues and remind people of very specific
actions that they can take to improve the organisa-
tion’s security posture. Posters are also commonly used
to serve as periodic information security reinforce-
ment. One major concern with posters is that message
may be overlooked.
Newsletter is a periodic (e.g. monthly or quarterly)
information security awareness channel. Newsletters
can be both in electronic or print format. One of the
 Behaviour & Information Technology
primary purposes of the newsletters is to reinforce
information security awareness programme. An ad-
vantage of using newsletters is that they can convey a
number of messages at the same time as compared to a
poster (Wilson and Hash 2003). However, newsletters
may only appeal to a select target group and requires
proper distribution mechanism. Also, there is no
guarantee to know if the staff have read the newsletter
and they have comprehended the contents of the
newsletter.
3.2. Instructor-led delivery methods
A variety of formal presentations (e.g. brown-bag
seminars and class room style workshops) facilitated
by local or external information security experts is used
to raise information security awareness of employees.
These approaches are typically top-down and aim at
having an impact on the individual level through an
expert-based approach (i.e. instructor-led) directed at a
large population. One of the advantages of instructor-
led delivery method is that the instructor is able to
perceive nonverbal student cues, modify instructional
methods accordingly, and provide timely answers to
student questions.
Although often people express interest for class-
room-based delivery method, it has its own drawbacks.
It is fairly expensive and provides a ‘static solution for
a fluid problem’ Valentine (2006). As many users find it
to be boring and ineffective (Leach 2003), the success
of this approach depends upon the ability of the
instructor to engage the audience. It also tends to fail
because it is based on rote and does not require users
to think about and apply information security con-
cepts (Cone et al. 2007). One way to address these
shortcomings is to use sharing experiences and knowl-
edge between the employees of an organisation
facilitated by participation, collective dialogues and
considerations, and group-work processes (Albrecht-
sen and Hovden 2010). This group-based sharing of
experiences and knowledge between employees and
information security professionals is important for the
information security work as it is likely to create
common insight among employees. However, this
approach assumes that the participants are knowl-
edgeable in the subject being discussed.
3.3. Online delivery methods
There are many different forms of online security
awareness delivery models. Online delivery methods
include e-mail broadcasting, online synchronous and
asynchronous discussion, information uploading,
blogging, animation, and multimedia. These delivery
methods are generally well suited for supporting
241
multimodal teaching methods over different geogra-
phical areas.
A number of organisations have developed blogs
(i.e., online materials) for teaching people to identify
phishing web sites accurately. eBay’s tutorial on
spoofed emails (http://pages.ebay.com/education/
spooftutorial/), Microsoft’s Security tutorial on Phish-
ing, the Phishing E-card from the US Federal Trade
Commission (http://www.ftc.gov/bcp/edu/pubs/consu-
mer/alerts/alt127.htm), and a URL tutorial from the
MySecureCyberspace port. The effectiveness of some
of these online materials was examined and it was
found that, while these materials could be improved,
they are surprisingly effective when users actually read
them (Kumaraguru et al. 2007).
Another online delivery method is the Web-based
computer security awareness training (WBT), which
offers user-friendly and flexible models that enable
users to enhance security awareness at their own pace.
It also provides the organisation with the ability to
train users to an enterprise-wide standard. Security
alert messages (e.g. screen savers, pre-logon messages,
email messages) are an alternative way for raising
information security awareness. Email is considered as
another key security awareness tool for delivering
awareness material (Wilson and Hash 2003).
Emails are used to highlight time-sensitive issues
and remind people of very specific actions that they
can take to improve the organisation’s security
posture. One of the strengths of email is that it has
the flexibility to convey one message or many. Email
campaigns can also be directed at targeted audiences.
For example, the American Express warned their
customers of a phishing scheme that presents itself as
a security measure by the company and asks for social
security number, mother’s maiden name and date of
birth. Although emails are cost effective means of
awareness message delivery method, it may be under-
mined due to volume of emails and spam. Also, it does
not reach those without computers.
Screensavers bring information security message
directly to the employees’ desktop and their daily work
environment. The advantage of the screensavers is that
everyone is guaranteed to see them at least once, which
make them an ideal channel for conveying essential
security awareness messages in a minute or less.
A range of mobile learning platforms that provide
anytime and anywhere information security awareness
to geographically diverse audiences have emerged.
Social media is an example of using emerging
technologies and new media for information security
awareness. The popularity of social networks makes it
an ideal tool through which awareness can be created
on existing and emerging security threats. One of the
advantages of the mobile learning based on social
242 J. Abawajy
media is that you can monitor if the recipients like the
message or not.
As noted in (Valentine 2006), one of the major
challenges with creating and implementing online
security awareness programmes is the lack of a fully
developed methodology to deliver them and to
measure the true effectiveness of the programme. For
example, the disadvantage of WBT-based delivery
method includes (Cone et al. 2007): (i) users attempt
to complete the sessions with minimal time or thought;
(ii) becomes monotonous; (iii) fails to challenge the
user and (iv) provides no dialogue for further elabora-
tion. This could lead to lack of self-motivation or
feelings of isolation that can hinder success in the
delivery of the awareness. To address some of these
shortcomings, the WBT-based delivery method de-
signers may make the content engaging by including
graphics, assessments and animations. Another issue
with WBT-based approach is that it can be expensive
to create training programmes.
3.4. Game-based delivery methods
Game-based delivery methods may offer an effective
alternative to, or supplement for, more traditional
modes of awareness. Online games combine graphics,
play and training concepts to create compelling
training experiences. The benefit of game-based
awareness delivery method is that it can challenge,
motivate and engage the participants. Game-based
delivery methods are highly interactive and can be used
to support organisational security awareness objectives
while engaging typical users.
Several game-based security awareness delivery
methods have been used currently. CyberCIEGE
(Cone et al. 2007) is a game-based information security
awareness delivery method. Fung et al. (2008) per-
formed a comparative analysis of game-based and
traditional classroom delivery methods. One group of
eight students was given the opportunity to play with
the CyberCIEGE game and the other group of eight
attended a short training session of one hour in a
traditional classroom setting. Results from the study
indicated that 75% of the students taking classroom
training shown significant improvement while only
60% for the students in the game groups. The authors
explain the performance differences between the
groups to be due to the language and culture factors
as the classroom training was conducted in Thai even
though the material is in English. The game players
had to interface with only English language.
Another game-based awareness training system is
the Anti-Phishing Phil that aims to teach users how to
distinguish phishing URLs from legitimate once,
where to look for cues in web browsers, and how to
use search engines to find legitimate sites. A study that
compares the Anti-Phishing Phil with other types of
training demonstrates that participants who played the
game performed better in identifying phishing websites
than participants who completed two other types of
training (Sheng et al. 2007). Liberty Mutual created
some simple Flash-based game applications for em-
ployees to play voluntarily (Rudis 2012). Players win
the games by making correct security choices.
Although the games were voluntary, about 25% of
Liberty Mutual employees played each game at least
once (Rudis 2012).
3.5. Video-based delivery methods
Educational videos play an important role as part of
information security awareness programme. There is
no need for a classroom trainer or have staff who
cannot be reached through e-learning courseware. One
of the major issues raised with conventional delivery
methods is the problem of holding a trainees attention
sufficiently long to impart a message particularly when
the topic is regarded by the trainees as potentially
mundane. Online video is a medium that provides
visual and audio learning for participants. Learners
can study independently and learn at their own pace
and only what they need to know. Learners can also
start and stop the training as their schedule permits
because it is not time-dependant. Also, the interactive
feature makes the online video more effective than
non-interactive techniques, but it is more expensive.
Moreover, the videos can be watched and re-watched
as needed, making the online training videos a flexible
and effective training choice.
3.6. Simulation-based delivery methods
Simulation-based information security awareness de-
livery methods have also been receiving some atten-
tions. In a simulation-based delivery method, users are
sent simulated phishing emails to test users’ vulner-
ability to phishing attacks and then follow-up with
training (Jagatic et al. 2007, Spagat 2009). At the end
of the study, users are given materials that inform them
about phishing attacks and subsequent phishing emails
were used to assess progresses in phishing detection
abilities of the users. A similar approach, called
embedded training that teaches users about phishing
during their regular use of email is described in
(Kumaraguru et al. 2007). A study that compared
simulation-based and pamphlet-based delivery models
concluded that users who were sent the simulated
phishing emails and follow-up notification were better
able to avoid subsequent phishing attacks than those
who were given a pamphlet containing information on
 Behaviour & Information Technology
how to combat phishing (NYSO; New York State
Office of Cyber Security & Critical Infrastructure
Coordination 2005).
4. Methodology
The aim of this study is not to generalise, but to
interpret some users’ experiences of information
security delivery methods. This is an exploratory study
which seeks to find new insights, as there is no large
and firm theoretical background on the subject of
information security awareness delivery methods. By
comparing three different delivery models through
conducting experiments, data were collected and
subjected to analyses. The design and analysis of this
study draws on methodological experiences from those
used in Sheng et al. (2007). This research method
enabled us to manipulate different delivery methods
and to assess their influence on the effectiveness of
learning security awareness concepts and skills.
4.1. Sample
A total of 60 voluntary participants were involved in
this study. Since our research is qualitative in nature,
the size of the sample does not have to be large
(Saunders et al. 2003). The participants were chosen in
such a way that the demographic is a representative of
the real settings. Each respondent completed pre-
knowledge information questionnaires. Over 70% of
the participants are reported as currently working
either in a full time or a part time job. All participants
reported to have a private personal computer and use
computers at work and/or at home. About 20% of the
participants had received some form of formal security
training in the past while about 60% of the partici-
pants indicated that they enjoy playing games at least
when playing for solely entertainment purposes. This
demographic is a representative of the real settings as
IT users tend to possess varying and unequal levels of
security awareness.
4.2. Selecting awareness delivery methods
Enhancing information security awareness levels of
general users often depends on the delivery method
used. As discussed previously, there are many different
information security awareness delivery methods. In
selecting the delivery methods, we made conscious
decisions to keep the time the subjects spend on the
activity to a minimum. Also, the materials should be
easy to comprehend by non-technical users while at the
same time not to cause undue frustration with
technical users. Most importantly, we decided that
the material to be short, to the point and easily
243
accessible via a web browser. For the game-based
model, we looked at a product that is easy to play and
built on solid learning science principles such that it
ensures high level of retention of content while playing
the game. For text-based awareness material, we
focused on materials that were short, to the point
and easily accessible via a web browser.
Each delivery method covered the same security
awareness topics and effort was made to make each
session as comparable as possible across the three
models. The selected training materials point out cues
for a phishing email and outlines simple actions that
users can take to protect themselves. For text-based
information security awareness delivery model, we
used a short web article that details phishing and
anti-phishing techniques including the use of illegiti-
mate emails (http://websearch.about.com/od/dailyweb
searchtips/qt/dnt0810.htm). For video-based informa-
tion security awareness delivery model, we used the
‘How to Avoid Phishing’ (http://www.5min.com/
Video/How-to-Avoid-Phishing-12254) video. The
Federal Trade Commission has released three 60-
second videos to help alert consumers to phishing
scams.
For the game-based information security aware-
ness delivery method, we examined three different
systems namely Master of Security (http://www.kon
gregate.com/games/gmentat/master-of-security), Anti-
Phishing Phil (http://wombatsecurity.com/antiphish-
ing_phil/index.html) and CyberCIEGE (Cone et al.
2007). We finally decided to go with the Anti-
Phishing Phil software. The motivation for choosing
the Anti-Phishing Phil software is that it is very easy
to use with almost zero learning time. Phil is a fish
character that swims through the ocean searching for
worms. Each worm is associated with a real URL
(representing legitimate websites) or a fake URL
(representing phishing websites). Phil’s job is to eat all
the worms (which have legitimate URLs) and reject
all the bait (which have phishing URLs) before
running out of time. The player can move Phil
around the screen and when Phil moves near a worm,
the URL that is associated with the worm is
displayed. The users learn to analyse URLs and
determine whether they are fraudulent or not. The
game is split into four rounds, each of which is two
minutes long. Upon completing a round, a feedback
screen is displayed, showing which choices you got
right or wrong, and why your choices were right or
wrong.
4.3. Data collection procedure
The aim of this study is to assess the effectiveness of
various information security awareness delivery
244 J. Abawajy
models as well as which delivery method is preferred by
the subjects. The study communicated a range of
information security message in various delivery
formats focusing on a phishing attack. The reason
for focusing on phishing attack is that it is a single
topic that has a wealth of information for both
technical and non-technical users. Moreover, it is one
of the core techniques for exploiting the human side of
information security that is constantly overlooked.
There is high level of user unfamiliarity with common
phishing attacks suggesting educating users about
online safety practices. People are a direct target for
the phishing scammers to get the private and sensitive
information. It is very difficult for the people to detect
phishing websites and emails as they often look
legitimate. Information security awareness is the
most important factor in mitigating phishing. By
helping users develop secure habits, information
security awareness programme leads to a secure work
environment and possibly halt security-related da-
mages before it starts.
We run three information security awareness
sessions for phishing attacks using text-based, game-
based and video-based security awareness delivery
methods. These sessions inform people how to avoid
falling for phishing attacks by providing them with
information that explains what phishing is, the
potential danger or the increasingly varied techniques
used by those who wish to exploit users such as
fraudulent URLs embedded in emails.
We randomly assigned the participants to one of
the three sessions – text-based, game-based and video-
based information security awareness. This allowed us
to evenly assign 10 participants to each experimental
group. For example, one group started with the game-
based awareness followed by the text-based delivery
model, watch a short video and finally play the game
again. Note that at any given time we have one group
that only receives game-based awareness session, one
group that only receives text-based awareness session
and one group that receives video-based awareness
session.
After experiencing each different awareness deliv-
ery method, we collected data to see if the awareness
method improved the knowledge of the participants
about phishing. Specifically, after each session, sub-
jects have to answer about 10 questions that assess
their ability to distinguish legitimate URLs from
fraudulent ones, suspicious links in email messages,
etc. For example, we tested the participants’ ability to
identify phishing web sites from a set of real and
phishing web sites at the end of a session. This helps
the subjects to reflect on the knowledge they have
gained from the session they just completed. Two sets
of questions were used to minimise the effect of
memorising the answers from the first set of questions
and depending on his/her memory to answer the
second set. A sample email that was used in the post-
test s shown below:
Reply-To: Director@cba.com.au
To: BAlen@gmail.com
Dear valued Commonwealth Bank client,
We has discovered an unauthorised withdrawal of large
sum from your account. Please click on the ‘‘My
Account’’ and go to the section of your NetBank home
page to login and verify and update your account details
immediately.
If you do not update your profile, your Netbank account
will be deactivated and deleted.
Yours sincerely,
Managing Director
Commonwealth Bank of Australia, AFSL and
Australian credit licence 234945
We selected 30 websites and 30 emails to test our
participants’ ability to identify phishing emails and web
sites after each session. Similar approach was used in
(Sheng et al. 2007). In our case, the test also included
information about the varied techniques used by those
who wish to exploit users and how the participants
could protect themselves from phishing attacks. Half of
the emails and websites we selected were phishing
emails and websites. The websites included popular
brands, legitimate web sites from popular financial
institutions and online merchants as well as random
websites. We divided the emails and the websites into
three groups (G1, G2 and G3). Each group has five
phishing email/websites and five legitimate email/sites.
We randomised the order, in which the three groups
were presented so that 20 of the participants saw G1,
and the next 20 of the participants saw G2 and the
remaining 20 subjects saw G3. Participants are scored
based on how well they can identify which emails/
websites are legitimate and which are not. We measured
false positives and false negatives before and after
training. A false positive occurs when a legitimate
email/website is wrongly judged as a phishing email/
website. A false negative occurs when a phishing email/
website is mistakenly judged to be legitimate.
5. Results
5.1. Delivery methods relative performance
Figure 1 presents the relative performance of the three
groups in the post-test compared to the pre-test cases.
The participants were scored based on how well they
can identify which emails are legitimate and which are
not. We used the scoring scheme employed in the Anti-
Phishing Phil tool. Participants are rewarded with
 Behaviour & Information Technology
100 points if they correctly identify good URLs or
correctly reject a bad one. They are slightly penalised
for rejecting a good answer (false positive) by losing 10
points. They are penalised with 100 points deduction if
they accept phishing attempts and is caught by
phishers (false negative). In this experiment, the total
mark that a participant can achieve is 1000. We then
normalised the score as follows:
achieve
score ¼
10
where ‘achieve’ is the accumulated total score. As
shown in Figure 1, awareness rates increased in all
three groups significantly from the pre-test values.
Prior to undertaking the security awareness training,
only half of the participants had an idea of what
phishing is. In pre-awareness training, we found that
users are generally aware of phishing threats. However,
their knowledge on the identification and prevention of
phishing was very poor. After the awareness activities,
awareness increased with about 30% of the partici-
pants changing their original answer. For all delivery
methods, about 5% of the participants still lacked
knowledge on the identification and prevention of
phishing.
The game-based delivery method has significantly
lower false positives than the other delivery methods
for questions pertaining to websites. That is to say,
participants were able to establish website authenticity
more quickly after playing the game as compared to
the other delivery methods. This is not surprising that
the game focuses on teaching users how to distinguish
legitimate links from fraudulent ones and provide users
with immediate opportunities to practice this proce-
dure multiple times.
The subjects in the video-based group did well in
email-born phishing-related questions. This is mainly
due to visual presentation of email-born phishing
Figure 1. Relative performance of the delivery methods.
245
examples and discussions. For text-based and video-
based delivery methods, many of the participants did
reflect the new-found knowledge that phishing at-
tempts could also be sent via email as well as
illegitimate URLs. This shows that while the awareness
rate from the text-based delivery model did not
increase as much as the other models, a broadening
in the knowledge of anti-phishing techniques was
certainly evident through the text-based and video-
based delivery methods.
5.1. Measuring changes in participants
Ideally, an awareness programme must influence
behaviour changes that deliver measurable benefits.
In order to see how much the text-based and the video-
based awareness delivery methods increased the know-
ledge of the participants about phishing, we admini-
strated the game-based awareness model one last time.
This would also separate personal thought of those
participants from what they had actually learnt.
Figure 2 shows the outcome of the experiment
before and after the text-based and video-based
awareness activities. Note that the game-based delivery
method consists of four rounds and each round focuses
on a different class of phishing URLs. Also, each
round is timed to limit how long one can take to
consider the URLs. In each round of the game, four
good worms and four phishing worms are randomly
selected from the 20 URLs in the data file for that
round. Players have to correctly recognise at least six
out of eight URLs within two minutes to move on to
the next round. From the graph, it is clear that both
the video-based and the text-based delivery methods
improved the participants’ knowledge about phishing
attacks. The results for each round improved approxi-
mately by 50% on the second attempt of the game.
Whilst these statistics present alarming support for
video and text-based security awareness training, one
Figure 2. Impact of conventional methods of security
awareness training.
246 J. Abawajy
cannot overlook the possibility that participants learnt
by their mistakes after the first attempt and self-cor-
rected their answers on the second attempt to suit.
5.2. Delivery method preferences
At the end of the study, we solicited response from the
participants as to their preferred awareness delivery
method. As shown in Figure 3, close to 8% of the
participants were undecided on their preferred informa-
tion security awareness delivery method. Surprisingly,
the game-based delivery model only polled fewer than
5%. This was despite 60% of the respondents having
admitted to enjoying video games, a statistic that
showed most were familiar and comfortable with games
at least when playing for solely entertainment purposes.
In contrast, over 50% of the participants prefer
video-based delivery model while text-based method
polled over 33%. The text-based and video-based
delivery methods have fared better mainly due to the
clear, concise, and easier to follow information. The
preference of video-based delivery method as com-
pared to the other two delivery models may be due to
lack of interest when presented with a document to
read or simply skimming through the document. Also,
the preference of a video as information security
awareness delivery method may be due to being able to
better understand the ideas and concepts taught when
it is given in both a verbal and visual form. Moreover,
the text-based delivery model has a predetermined
structure set by the author and, according to Bloom’s
taxonomy; the subjects are more focused to remember
than to actually understand (i.e. ‘knowledge’). This
may explain why the text-based delivery model is fared
less to video-based model.
6. Discussion
People are in many cases the last line of defence against
threats such as malicious code, disgruntled employees,
and malicious third parties. Human vulnerabilities
Figure 3. Delivery method preference.
have been a significant source of information security
risks and the result of breaches in information security
leads to billions of dollars annually in individual and
corporate losses and even to crimes (Workman 2007).
Therefore, the significance of the human factors in
information security management cannot be under-
stated. Unfortunately, human element gets less atten-
tion compared to other elements within an
organisation. Some organisations also fall short in
realising that the breakdown of IT security merely not
caused by technology failure itself but in many cases
caused by human element. However, the significance of
people’s failure to take precautions against informa-
tion security threats have been largely ignored (Work-
man 2007).
It is imperative to minimise Human vulnerabilities
in order to improve organisational security posture.
The development of information security policies,
standards, procedures, and guidelines is only the
beginning of an effective information security pro-
gramme. Since the most important organisational asset
is the personnel, it is imperative that all employees
must have certain level of information security aware-
ness. It is critical for organisations to ensure that not
only should the average employee be aware about the
threats facing the organisation but also understand
what needs to be done in case he/she notices a potential
issue. Information security conscious employees could
play a significant role in minimising information
security risks and protecting the organisation’s critical
assets and valuable intellectual property. By expanding
the base of employees who are more aware of
information security risks, organisations can reduce
the level of information security risks due to the
human vulnerabilities. Without some level of employee
information security awareness, the information secur-
ity posture of an organisation will be incomplete. To
this end, information security awareness programmes
must be designed with intention of creating organisa-
tional-wide security-minded cultures so that people
work in a more secure manner and protect the assets of
their organisation.
The information security delivery methods are the
critical prerequisites to improved security awareness
levels. Kumaraguru et al. (2007) found that generally
existing online anti-phishing training materials tend to
make users more cautious about opening and acting
upon email, but do not teach people how to determine
whether a website or email is fraudulent. In contrast,
our results show that the various delivery methods
examined in this paper appear to have been mostly
successful in aiding the participants with a clearer
understanding of what phishing is and how to best
minimise its dangers. As shown in Figure 1, video-
based group performed roughly as well as the text-
 Behaviour & Information Technology
based training material in terms of website based
related questions, and better on email based questions.
The results also suggest that the existing online
training materials are surprisingly effective in helping
users identify phishing websites when users actually
read the training materials. The game-based delivery
method was able to provide knowledge as to what to
look for in the URL’s. In contrast, the text-based and
the video-based awareness methods added the knowl-
edge that emails are the main carrier of phishing
attacks. This indicates that these three delivery models
should be used together to get the maximum benefits
out of the information security awareness programme.
It also seems that some people may be aware of
phishing, but they do not associate that awareness to
their own vulnerability or to strategies for identifying
phishing attacks. This concurs with the findings in
Downs et al. (2006).
Even though organisations have started providing
decent information security awareness programmes,
finding the appropriate ways to increase information
security awareness is an open research issue and this
paper attempts to help towards this direction. It is of
course impossible to meet every individual’s personal
needs, although, it is substantiated that people learn
and understand in several different ways of which some
tend to be more efficient. Also, our informal experi-
ences show that some users simply will not expend any
effort to learn even the most basic mechanics of a
game. For these users, interactive training methods will
not be effective if they require anything more involved
that the repeated clicking of a mouse or pressing of an
enter key. This implies that, although a one-size-fits-all
information security awareness strategy may be easy
on the organisation, it will not be effective in achieving
the information security goals of the organisations. In
addition, the risk areas could change as the informa-
tion risk profile changes. Thus, the delivery mechanism
should also change as the risk areas change. Therefore,
the key to success in awareness is keeping the messages
relevant and consistent, while varying the delivery
mechanisms, to keep everyone interested.
The study also suggests that information security
awareness level can be increased and sustained through
multiple delivery methods. All three delivery methods,
if used wisely, can help in driving an organisation’s
security awareness programme effectively. The conclu-
sion from this study suggests that game-based and
video-based delivery models can be a suitable non-
linear information security delivery methods when
learning to understand IT security and therefore also a
suitable alternative/complement to conventional linear
instruction. This conclusion concurs with that of Shaw
et al. (2009) which contend that security awareness
information must be disseminated in different ways
247
(e.g. newsletter, video, seminar and lecture) so that
users receive many different messages (Shaw et al.
2009).
Since information security awareness delivery
method research is rather scarce, this study has
provided insights on the effectiveness and the influence
of various information security awareness delivery
methods on the end-users. While automated detection
systems should be used as the first line of defence
against phishing attacks, raising user awareness offers
a complementary approach to help people better
recognise fraudulent emails and websites. Thus,
effective management of information security requires
a combination of technical and non-technical controls
to manage risks to organisational assets. Although a
great deal of effort has been expanded on trying to
solve the phishing problem through automated detec-
tion and prevention system, little research has been
done in the area of training users to recognise and
avoid phishing attacks (Kumaraguru et al. 2010). We
believe that the study results and recommendations can
be applicable to every organisation or company that is
operating large amounts of information, and has a
high level of computer usage among their employees.
Our results suggest information security awareness
training is a powerful means of empowering people
with knowledge on focused topics. After completing
the training almost all of the participants had the
correct idea about phishing and the dangers it poses
both to the individual and to an organisation.
7. Conclusion
In this paper, we studied two information security
awareness programme-related issues: (i) what delivery
approaches are effective in raising information security
awareness level of people; and (ii) which delivery
method is preferred by users. We looked at three
information security awareness delivery methods (i.e.
text-based, game-based and video-based) in terms of
their effectiveness in raising information security
awareness and user preferences of the delivery method.
Our results suggest information security awareness
training is a powerful means of empowering people
with knowledge on focused topics. After completing
the training almost all of the participants had the
correct idea about phishing and the dangers it poses
both to the individual and to an organisation.
Although the results suggest that video presentation
to be the most preferred security awareness training
delivery method, the training provided through the use
of the various delivery methods used appear to have
been mostly successful in aiding the participants with a
clearer understanding of what phishing is and how to
best minimise its dangers.
248 J. Abawajy
Further studies have been planned in the future.
We plan to repeat this experiment by enlarging the
population size, catch possible side effects during the
learning experience such as satisfaction, gender and
age as well as educational level differences, and
efficiency in terms of time-consumption versus ac-
quired knowledge. Although many of the concepts
included in cyber security awareness training are
universal, such training often must be tailored to
address the policies and requirements of a particular
organisation. Therefore, one way to further our study
is to look into how different companies have different
needs, but the delivery model does not take into
consideration this matter – thus generalising the
applicability of the model to all of the companies of
different nature and sizes.
Acknowledgements
This work is partially supported by Securing Cyperspace
Faculty Research Cluster. The help of Maliha Omar is
greatly appreciated. This work greatly benefited from the
anonymous reviewers comments as well.
References
Abawajy, J. and Tai-hoon, K., 2010. Performance analysis of
cyber security awareness delivery methods. In: Security
technology, disaster recovery and business continuity.
Berlin, Germany: Spring-Verlag, 142–148.
Abawajy, J.H., Thatcher, K., and Tai-hoon, K., 2008.
Investigation of stakeholders’ commitment to informa-
tion security awareness programs. In: Proceedings of the
2nd international conference on information security and
assurance (ISA 2008). Los Alamitos, CA: IEEE Com-
puter Society, 472–476.
Albrechtsen, E. and Hovden, J., 2010. Improving informa-
tion security awareness and behaviour through dialogue,
participation and collective reflection. An Intervention
Study, Computer and Security, 29 (4), 432–445.
Cone, B.D., et al., 2007. A video game for cyber security
training and awareness. Computers & Security, 26 (1), 63–
72.
Dodge, R.C., 2007. Phishing for user security awareness.
Computers & Security, 26, 73–80.
Downs, J., Holbrook, M., and Cranor, L., 2006. Decision
strategies and susceptibility to phishing. In: Proceedings
of the Second Symposium on Usable Privacy and Security
(SOUPS’06). New York, NY: ACM Press, 79–90.
Eminagaoglu, M., Ucar, E., and Eren, S., 2010. The positive
outcomes of information security awareness training in
companies – a case study. Information Security Technical
Report, 4, 1–7.
Fung, C.C., et al., 2008. Raising information security
awareness in digital ecosystem with games – a pilot
study in Thailand. In: 2nd IEEE international conference
on digital ecosystems and technologies (IEEE DEST
2008). Los Alamitos, CA: IEEE Press, 375–380.
Islam, R. and Abawajy, J., 2012. A multi-tier phishing detec-
tion and filtering approach. Journal of Network and
Computer Applications. DOI: 10.1016/j.jnca.2012.05.009.
Jagatic, T., et al., 2007. Social phishing. Communications of
the ACM, 50 (10), 94–100.
Knapp, K.J., et al., 2007. Information security effectiveness:
conceptualization and validation of a theory. Interna-
tional Journal of Information Security and Privacy, 1 (2),
37–60.
Kruger, H.A. and Kearney, W.D., 2006. A prototype for
assessing information security awareness. Computers &
Security, 25 (4), 289–296.
Kumaraguru, P., et al., 2010. Teaching johnny not to fall for
phish. ACM Transactions on Internet Technology, 10 (2),
1–31.
Kumaraguru, P., et al., 2007. Protecting people from
phishing: the design and evaluation of an embedded
training email system. In: Proceedings of the computer
human interaction (CHI 2007). New York, NY: ACM
Press, 905–914.
Leach, J., 2003. Improving user security behaviour. Compu-
ters & Security, 22 (8), 685–692.
New York State Office of Cyber Security & Critical
Infrastructure Coordination, 2005. Gone phishing: a
briefing on the anti-phishing exercise initiative for New
York State Government. Aggregate exercise results for
public release.
Ponemon Institute, 2012. The human factor in data protection
[online]. Ponemon Institute. Available from: http://www.
trendmicro.com/cloud-content/us/pdfs/security-intelligen
ce/reports/rpt_trend-micro_ponemon-survey–2012.pdf
[Accessed 9 June 2012].
Rudis, B., 2012. Achievement unlocked: designing a compel-
ling security awareness program. In: Security conference
and training, 17–19 April, Boston.
Saunders, M.K., Lewis, P., and Thornhill, A., 2003.
Research methods for business students. 4th ed. Spain:
Financial Time Press.
Shaw, R.S., et al., 2009. The impact of information richness
on information security awareness training effectiveness.
Computers & Education, 52 (1), 92–100.
Sheng, S., et al., 2007. Anti-phishing phil: the design and
evaluation of a game that teaches people not to fall for
phish. In: Proceeding on symposium on usable privacy and
security (SOUPS’07). New York, NY: ACM Press, 88–
99.
Spagat, E., 2009. Justice Department Hoaxes Employees.
News Article [online]. Available from: http://news.yahoo.
com/s/ap/20090129/ [Accessed 8 June 2012].
Valentine, J.A., 2006. Enhancing the employee security
awareness model. Computer Fraud & Security, 6, 17–19.
Wilson, M. and Hash, J., 2003. Building an information
technology security awareness and training program [on-
line]. National Institute of Standards and Technology.
http://www.itl.nist.gov/lab/bulletns/bltnoct03.htm [Ac-
cessed 8 June 2012].
Workman, M., 2007. Gaining access with social engineering:
an empirical study of the threat. Information Systems
Security, 16 (6), 315–331.