IEEE TRANSACTIONS ON MULTIMEDIA, VOL.

23, 2021 2575

Robust CAPTCHAs Towards Malicious OCR

Jiaming Zhang , Jitao Sang , Kaiyuan Xu , Shangxi Wu , Xian Zhao , Yanfeng Sun , Yongli Hu ,
and Jian Yu

Abstract—Turing test was originally proposed to examine

whether machine’s behavior is indistinguishable from a human.
The most popular and practical Turing test is CAPTCHA,
which is to discriminate algorithm from human by offering
recognition-alike questions. The recent development of deep
learning has significantly advanced the capability of algorithm
in solving CAPTCHA questions, forcing CAPTCHA designers
to increase question complexity. Instead of designing questions
difficult for both algorithm and human, this study attempts to
employ the limitations of algorithm to design robust CAPTCHA
questions easily solvable to human. Specifically, our data analysis
observes that human and algorithm demonstrates different
vulnerability to visual distortions: adversarial perturbation is
significantly annoying to algorithm yet friendly to human. We are Fig. 1. Increasing content complexity of CAPTCHAs.
motivated to employ adversarially perturbed images for robust
CAPTCHA design in the context of character-based questions.
Four modules of multi-target attack, ensemble adversarial training,
image preprocessing differentiable approximation, and expectation that is indistinguishable from a human, and later developed into
are proposed to address the characteristics of character-based a form of reverse Turing test with more practical goal of distin-
CAPTCHA cracking. Qualitative and quantitative experimental guishing between computer and human. Among reverse Tur-
results demonstrate the effectiveness of the proposed solution. We ing tests, CAPTCHA (Completely Automated Public Turing
hope this study can lead to the discussions around adversarial test to tell Computers and Humans Apart) turns out the most
attack/defense in CAPTCHA design and also inspire the future
attempts in employing algorithm limitation for practical usage. well-known one used in anti-spam systems to prevent abuse use
of automated programs [2].
Index Terms—Adversarial example, CAPTCHA, OCR.
Most early CAPTCHAs, like the reCAPTCHA [3] which
I. INTRODUCTION assists in the digitization of Google books, belong to the tra-
ditional character-based scheme involving with only numbers
LAN Turing first proposed the Turing Test question “Can
A machines think like human?” [1] Turing test was initially
designed to examine machine’s exhibited intelligent behavior
and English characters. With the fast progress of machine learn-
ing especially deep learning algorithms, simple character-based
CAPTCHAs fail to distinguish between algorithm and hu-
man [4]–[6]. CAPTCHA designers were therefore forced to in-
Manuscript received December 20, 2019; revised May 5, 2020 and June 15, crease the complexity of content to be recognized. As shown
2020; accepted July 16, 2020. Date of publication August 4, 2020; date of current
version August 24, 2021. This work was supported in part by the National Key R, in Fig. 1, while the extremely complex CAPTCHAs reduce the
and D Program of China under Grant 2018AAA0100604, and in part by the Na- risks to be cracked by algorithms, it also heavily increases the
tional Natural Science Foundation of China under Grants 61632004, 61832002, burden of human recognition. It is noteworthy that the effective-
61672518, U19B2039, 61632006, 61772048, 61672071, and U1811463, and
in part by the Beijing Talents Project (2017A24), and in part by the Beijing ness of simply increasing content complexity is based on the
Outstanding Young Scientists Projects (BJJWZYJH01201910005018). The as- assumption that human has consistently superior recognition
sociate editor coordinating the review of this manuscript and approving it for capability than algorithm. The last few years have witnessed
publication was Dr. Vasileios Mezaris. (Corresponding author: Yongli Hu.)
Jiaming Zhang and Jitao Sang are with the School of Computer and In- human-level AI in tasks like image recognition [7], speech pro-
formation Technology & Beijing Key Laboratory of Traffic Data Analy- cessing [8] and even reading comprehension [9]. It is easy to
sis and Mining, Beijing Jiaotong University, Beijing 100044, China, and
also with the Peng Cheng Laboratory, Shenzhen 518055, China (e-mail:
imagine that with the further development of algorithms, con-
lanzhang1107@gmail.com; jtsang@bjtu.edu.cn). tinuously increasing content complexity will reach such a critical
Kaiyuan Xu, Shangxi Wu, Xian Zhao, and Jian Yu are with the point that algorithm can recognize yet human cannot recognize.
School of Computer and Information Technology & Beijing Key Lab
of Traffic Data Analysis and Mining, Beijing Jiaotong University, Bei-
Let’s review the initial goal of CAPTCHA: to discriminate
jing 100044, China (e-mail: 15281106@bjtu.edu.cn; kirinng0709@gmail.com; human from algorithm by designing tasks unsolvable to al-
lavender.zxshane@gmail.com; jianyu@bjtu.edu.cn). gorithms. Therefore, the straightforward solution is to employ
Yanfeng Sun and Yongli Hu are with the Beijing Key Laboratory of Mul-
timedia and Intelligent Software Technology & Beijing Artificial Intelligence
the limitations of algorithms to facilitate CAPTCHA design.
Institute, Faculty of Information Technology, Beijing University of Technology, While algorithms have advanced their performance in many
Beijing 100124, China (e-mail: yfsun@bjut.edu.cn; huyongli@bjut.edu.cn). perspectives including visual/vocal recognition accuracy, they
Color versions of one or more of the figures in this article are available online
at https://ieeexplore.ieee.org.
remain some notorious limitations with regards to human. Re-
Digital Object Identifier 10.1109/TMM.2020.3013376 searchers and practitioners already employed such limitations to
1520-9210 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
 2576
design new form of CAPTCHAs, e.g., developing cognitive [10]
and sequentially related [11] questions to challenge algorithm’s
lack of commonsense knowledge and poor contextual reasoning
ability.
Following this spirit, we are interested to explore the possi-
bility of improving the robustness of CAPTCHA towards al-
gorithm cracking without changing the traditional character-
based scheme. In other words, is it possible to design character
CAPTCHAs only friendly to human instead of simply increas-
ing content complexity? The key lies in finding the algorithm
limitation compatible to the scheme of character image. One
candidate is the vulnerability to visual distortions. We have con-
ducted data analysis and observed that human and algorithm
exhibit different vulnerability to visual distortions (the obser-
vations are detailed in Section III). This inspires us to exploit
those distortions friendly to human but obstructing algorithm to
pollute the original character CAPTCHA. Specifically, adversar-
ial perturbation [12] exactly meets this requirement: adversarial
attack1 and CAPTCHA share the common intention that human
is imperceptible to but algorithm is significantly affected by the
same distortion. The notorious characteristic of adversarial per-
turbation for visual understanding turns out to be the desired one
for CAPTCHA design.
Inspired by this, we employ adversarial perturbation to de-
sign robust character-based CAPTCHA in this study. Current
state-of-the-art cracking solution views CAPTCHA OCR (Op-
tical Character Recognition) as a sequential recognition prob-
lem [13]–[17]. To remove the potential distortions, further im-
age preprocessing operations are typically added before OCR.
Correspondingly in this study, we propose to simultaneously at-
tack multiple targets to address the sequential recognition issue
(Section IV-A), differentiably approximate image preprocessing
operations (Section IV-C) and stochastic image transformation
(Section IV-D) in the adversarial example generation process to
cancel out their potential influence. Moreover, since we have no
knowledge about the detailed algorithm the cracking solution
used (e.g., neural network structure), the generated adversarial
examples are expected to be resistant to unknown OCR algo-
rithms in the black-box cracking. This study resorts this issue
to ensemble adversarial training by generating adversarial ex-
amples effective towards multiple algorithms (Section IV-B). In
summary, the contributions of this study are two-fold:
r We have discovered the different vulnerability between hu-
man and algorithm on visual distortions. Based on the ob-
servations, adversarial perturbation is employed to improve
the robustness of character-based CAPTCHA.
r Corresponding to the characteristics of typical OCR crack-
ing solutions, we proposed a novel methodology address-
ing issues including sequential recognition, indifferen-
tiable image preprocessing, stochastic image transforma-
tion and black-box cracking.
1 Adversarial attack refers to the process of adding small but specially crafted
perturbation to generate adversarial examples misleading algorithm. To avoid
confusion with the process of attacking CAPTCHA, in this study, we use “adver-
sarial attack” to indicate the generation of adversarially distorted CAPTCHAs
and use “CAPTCHA crack” to indicate the attempt of passing CAPTCHA with
algorithms.
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 23, 2021
II. RELATED WORK
A. Character-Based CAPTCHAs
In online services, character-based CAPTCHAs are the most
popular protection to deter character recognition programs.
Since the initial goal of CAPTCHA, friendly design and security
of CAPTCHAs have been studied. A fundamental requirement
of CAPTCHAs necessitates that be designed to be easy for hu-
mans but difficult for computers. In traditional CAPTCHA de-
sign, the trade-off between usability and security is difficult to
balance. The three traditional designs are most common: back-
ground confusion, using lines and collapsing [18]. But there
are some studies that use auto-generated methods to synthe-
sis CAPTCHA images, e.g., using GANs, instead of manual
design [19]. These auto methods, which are applied to both
character-based CAPTCHA and image-based CAPTCHA, are
novel approaches for generating CAPTCHAs, but they still at-
tempt to increase content complexity of CAPTCHAs.
To overcome the limitations of traditional character-based
CAPTCHAs, other designs have been proposed, e.g., 3D-based
CAPTCHAs, Animated CAPTCHAs [18]. 3D approaches to
CAPTCHA design involve the rendering of 3D models to an
image [20], [21]. However, it has been demonstrated that this
approach is easy to attacks [22], [23]. Animated CAPTCHAs
attempt to incorporate a time dimension into the design. The
addition of a time dimension is assumed to increase the secu-
rity of the resulting CAPTCHA. Nevertheless, techniques that
can successfully attack the CAPTCHAs design have been de-
veloped [24].
The last few years have witnessed deep learning plays an
important role in the field of artificial intelligence. The recogni-
tion rate of character-based CAPTCHAs increases year by year.
George et al. proposed a hierarchical model called the Recursive
Cortical Network (RCN) that incorporates neuroscience insights
in a structured probabilistic generative model framework, which
significantly improved the recognition rate [25]. To remove the
interference in the background, Ye et al. proposed the GAN-
based approach for automatically transforming training data
and constructing solvers for character-based CAPTCHAs [26].
The convolutional neural network shows a powerful perfor-
mance in the recognition of various characters, including Chi-
nese characters [27]. But the convolutional neural network has
low recognition accuracy in confusion class. To solve this prob-
lem, Chen et al. proposed a novel method of selective learn-
ing confusion class for character-based CAPTCHAs recogni-
tion [28]. As the complexity of character-based CAPTCHAs in-
creases, the methods based on combining convolutional neural
network and recurrent neural network achieve state-of-the-art
performance [13]–[17]. In this paper, we employ the archi-
tecture consists of convolution neural network (CNN) layers
and long short-term memory (LSTM) as the default OCR algo-
rithm. We also test our CAPTCHAs on the latest method [17] in
Section V-D, which is an attention-based model that also con-
sists of CNN layers and LSTM.
B. Adversarial Example
While deep learning has achieved great performance, it also
has some security problems. Recent work has discovered that the
 ZHANG et al.: ROBUST CAPTCHAS TOWARDS MALICIOUS OCR
existing machine learning models, not just deep neural networks,
are vulnerable to adversarial example [12]. Given a trained clas-
sifier F with model parameters W , a valid input x and with cor-
responding ground truth prediction y, i.e., y = F (x) with model
parameters W . It is often to get a similar input x is close to x ac-
cording to some distance metric d(x, x ), and cause y = F (x )
with model parameters W . An example x with this property
is known as a untargeted adversarial example. A more power-
ful but difficult example called targeted adversarial example is
more than a misclassification example, i.e., t is a target label and
t = y, t = F (x ) with model parameters W .
Prior work that considers adversarial examples can be gener-
ally classified into two categories: white-box attack and black-
box attack. White-box attack has full knowledge of the trained
classifier F including the model architecture and model param-
eters W . Black-box attack has no or limited knowledge of the
trained classifier F . Black-box setting is apparently harder than
white-box setting for attackers because of the leaked gradient
information. It seems that black-box attack is impossible, but
adversarial examples that affect one model can often affect an-
other model, which is called transferability [29]. In the paper,
we rely on the transferability and deploy ensemble-based ap-
proaches to generate adversarial CAPTCHAs.
Szegedy et al. [12] first pointed out adversarial examples and
proposed a box-constrained LBFGS method to find adversar-
ial examples. To decrease expensive computation, Goodfellow
et al. [30] proposed the fast gradient sign method (FGSM) to gen-
erate adversarial examples by performing a single gradient step.
Kurakin et al. [31] extended this method to an iterative version,
and find out that adversarial examples can influence physical
world. Dong et al. [32] further extended the fast gradient sign
method family by proposing momentum-based iterative algo-
rithms. In addition, there are some more powerful methods called
optimization-based attack methods. Deepfool [33] is an attack
technique optimized for the L2 distance metric. This method
is based on the assumption that the decision boundary is partly
linear, then the distance and direction of the data points to the
decision boundary can be calculated approximately. C&W [34]
is another targeted optimization-based method. It achieves its
goal by increasing the probability of target label.
To defend against adversarial examples, several adversarial
defensive methods have been proposed, which has been an ac-
tive field of AI research. Referring to [35], we generally di-
vide adversarial defensive methods into two categories. Atha-
lye et al. [35] identify gradient masking, or called obfuscated
gradients, which leads to a false sense of security in defenses
against adversarial examples. The authors addressed that the rea-
son why many adversarial defenses can defend against adver-
sarial examples is that the fast and optimization-based methods
cannot succeed without useful gradient information. The most
common gradient masking methods include input transforma-
tion and stochastic gradients. Input transformation techniques,
e.g., image cropping and image binarization, cause the gradients
to be non-existent or incorrect. In this paper, image binarization
will definitely result in non-differentiable if gradient masking is
not overcome. Some adversarial defense methods cause the net-
work itself is randomized or the input is randomly transformed.
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
2577
These methods based on stochastic gradients lead adversarial
attack methods using a single sample of the randomness to in-
correctly estimate the true gradient. Goodfellow et al. [30] first
proposed adversarial training method, adversarial examples are
regarded as training samples to fit the model until these samples
are classified correctly. The idea is effective and general for all
types of adversarial attacks. This makes the network more robust
against the adversarial examples, but cost expensive computa-
tion, especially at a large scale, e.g., the ImageNet [36] scale.
In general, the existing defensive methods cannot completely
eliminate adversarial attacks.
Many researchers have found that adversarial example can be
applied in other tasks, such as semantic segmentation [37], face
detection [38], and even speech recognition [39] and transla-
tion [40]. The majority of the published papers have focused on
how to eliminate the impact of adversarial examples in applica-
tion. Li et al. [41] evaluated adversarial examples among differ-
ent detection services, such as violence, politician, and pornog-
raphy detection. Ling et al. [42] proposed a uniform platform for
comprehensive evaluation on adversarial attacks and defenses in
application, which can benefit future adversarial examples re-
search. In contrast, studies on employing adversarial examples
against the malicious algorithm are relatively limited. Osadchy
et al. [43] employed adversarial examples to design CAPTCHAs
and analyzed security and good usability of CAPTCHAs. But
they only considered these data types like MNIST and Ima-
geNet instead of CAPTCHA data types. Zhang et al. [44] stud-
ied the robustness of adversarial examples on different types of
CAPTCHA and gave the suggestions that how to improve the se-
curity of CAPTCHA using adversarial examples. Shi et al. [45]
improved the effectiveness of the adversarial example by us-
ing the Fourier transform to generate CAPTCHA images in
the frequency domain. However, they only considered gener-
ating adversarial examples on CNN systems, which is essen-
tially the adversarial attack algorithm based on the classification
task. In contrast, the current state-of-the-art CAPTCHA crack-
ing system consists of feature extraction module and sequential
recognition module (CNN + LSTM). Shi et al. [45] deployed
character-based adversarial CAPTCHAs on a large-scale online
platform and tested the proposed CAPTCHAs on convolutional
recurrent neural networks [46]. However, they ignored experi-
ments and discussions on adversarial defense technologies, such
as image binarization and adversarial training. In Section V-D,
we compare our method with ACs [45] to prove that considering
the sequential recognition is essential. In Section V-B, we show
the necessity of considering image preprocessing.
III. DATA ANALYSIS
To justify the feasibility of employing algorithm limitations
for CAPTCHA design and motivate our detailed solution, this
section conducts data analysis to answer two questions: (1)
Whether human and algorithm have different vulnerability to
visual distortion? (2) What characteristics to consider when em-
ploying distortions to design robust CAPTCHA?
Text-based CAPTCHA is the most widely deployed scheme
requiring subjects to recognize characters from 0-9 and A-Z.
 2578
Fig. 2. Human v.s. algorithm vulnerability analysis results on Gaussian and adversarial distortions.
Due to its simplicity, character-based CAPTCHA is very ef-
fective to examine the robustness towards cracking algorithm
as well as friendliness to human. Therefore, this study em-
ploys character-based CAPTCHA as the example scheme to
conduct data analysis, develop solution and implement experi-
ments. Specifically, during data analysis, we assume that each
CAPTCHA question is constituted by single character in an
RGB image with unique resolution of 48 × 64 px. The char-
acter font is fixed as DroidSansMono. The remainder of the sec-
tion will report the observations regarding human and character
recognition performance in different scenarios.
A. Vulnerability Analysis to Visual Distortion
This subsection designs character recognition competition be-
tween human and algorithm to analyze their vulnerability to vi-
sual distortions. We employed two types of visual distortions:
(1) Gaussian white noise is one usual distortion to generate
CAPTCHAs. In this study, the added one-time Gaussian white
noise follows normal distribution with mean μ̃ = 0, variance
σ̃ = 0.01 and constant power spectral density. (2) Adversarial
perturbation has been recognized as imperceptible to human but
significantly confusing algorithm. We employ the widely used
F GSM [30] to add adversarial perturbation, where one-time
perturbation is constituted with step size of 0.02. To examine
the change of recognition performance with increasing distor-
tion difficulty, we added 8 levels of distortions onto the origi-
nal character images accumulatively: each level corresponds to
5 one-time Gaussian white noises and adversarial perturbations
respectively. Examples for derived distorted CAPTCHA images
in different levels are illustrated in Fig. 2(a) and (b).
Regarding the human side, we recruited 77 master work-
ers from Amazon Mechanical Turk (MTurk). Each subject was
asked to recognize 450 character CAPTCHAs with Gaussian
and adversarial distortions in different levels respectively. Re-
garding the algorithm side, we employed the state-of-the-art
OCR (Optical Character Recognition) algorithm, which is the
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 23, 2021
segmentation-based approach for OCR works by segmenting a
text line image into individual character images and then rec-
ognizing the characters [47]. The resultant average recognition
accuracies for Gaussian and adversarially distorted CAPTCHAs
are shown in Fig. 2(c) and (d). We can see that, for Gaussian dis-
torted CAPTCHAs, human’s recognition accuracy consistently
declines as the distortion level increases, indicating that Gaus-
sian white noise tends to undermine human’s vision. On the con-
trary, the examined OCR algorithm demonstrates good immu-
nity to Gaussian white noise, possibly due to the noise removal
effect by multiple convolutional layers [48]. It is easy to imagine
that if we design CAPTCHA by adding Gaussian white noise,
as the noise level increases, the resultant CAPTCHAs will crit-
ically confuse humans instead of obstructing the cracking OCR
algorithms.
For adversarially distorted CAPTCHAs, we observed quite
opposite recognition results. Fig. 2(d) shows that humans are
more robust to the adversarial perturbations, while OCR algo-
rithm is highly vulnerable as the adversarial distortion increases.
This is not surprising since adversarial perturbation is specially
crafted to change the algorithm decision under the condition of
not confusing human. This characteristic of adversarial perturba-
tion demonstrates one important limitation of algorithm regards
to human ability, which perfectly satisfies the requirement of ro-
bust CAPTCHA: algorithm tends to fail, while human remains
successful. Therefore, we are motivated to employ adversarial
examples to design robust CAPTCHA to distinguish between
algorithm and human.
B. Characteristics Affecting Robust CAPTCHA Design
The previous subsection observes that adversarial pertur-
bation is effective to mislead state-of-the-art OCR algorithm,
which shows its potential to be employed to design robust
CAPTCHA. However, typical CAPTCHA cracking solution in-
volves beyond OCR, e.g., image preprocessing operations like
 ZHANG et al.: ROBUST CAPTCHAS TOWARDS MALICIOUS OCR
Fig. 3. The affection of image preprocessing: (a) distorted characters before
(top row) and after (bottom row) image binarization and (b) recognition accuracy
on adversarially distorted characters.
binarization and Gaussian filtering will be applied to remove dis-
tortions before issuing to the OCR module. Fig. 3(a) illustrates
the adversarially distorted CAPTCHA images before and after
binarization preprocessing. It is easy to conceive that the effec-
tiveness of adversarial perturbation will be critically affected by
image preprocessing operations.
We further quantified this affection by analyzing the OCR
performance on the same adversarially distorted CAPTCHA
images from previous subsection. The recognition accuracies
on the CAPTCHAs before and after binarization preprocessing
are plotted and compared in Fig. 3(b). It is shown that after
removing most distortions via image binarization, OCR algo-
rithm demonstrates basically stable performance in recognizing
CAPTCHAs with different levels of adversarial perturbation.
This tells us that standard adversarial perturbation is insufficient
to obstruct the cracking method. It is necessary to design the
robust CAPTCHA solution considering the characteristics (like
preprocessing operations) of CAPTCHA cracking method.
IV. METHODOLOGY
As shown on the left of Fig. 4, typical cracking of character-
based CAPTCHA consists of two stages as image preprocessing
and OCR. The above data analysis has demonstrated that im-
age preprocessing has the effect of distortion removal, making
it not possible to straightforwardly employ adversarial pertur-
bation for robust CAPTCHA design. In addition to the image
preprocessing stage, the OCR stage also possesses character-
istics obstructing CAPTCHA: (1) sequential recognition, dis-
abling the traditional single character-oriented adversarial per-
turbation; and (2) black-box crack, making it ineffective to attack
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
2579
one specific OCR model. To address the above characteristics
of CAPTCHA cracking, our proposed CAPTCHA generation
framework consists of three modules: multi-target attack, en-
semble adversarial training, and image preprocessing differen-
tiable approximation. The proposed framework and its relation
to CAPTCHA cracking are illustrated on the right of Fig. 4.
A. Multi-Target Attack Towards Sequential Recognition
Typical CAPTCHAs usually contain more than one character
for recognition, e.g. the example CAPTCHAs contain 4 charac-
ters. Therefore, state-of-the-art CAPTCHA cracking solutions
are forced to address a sequential character recognition prob-
lem at the OCR stage [47]. Specifically, OCR stage consists of
three sub-modules as feature extraction, sequential recognition,
and output decoding. Feature extraction is basically realized by
a convolutional neural network to encoding the input image as
neural feature. Sequential recognition is typically realized by a
recurrent neural network to process the issued image neural fea-
ture and output multiple tokens including characters (0-9, A-Z)
and blank token ∅.2 Output decoding serves to transform the se-
quential tokens into final character recognition results, by merg-
ing sequentially duplicated tokens and removing blank ∅ tokens.
For example, the original token sequence “aa∅b∅∅ccc∅dd” will
be transformed to “abcd”.
While CAPTCHA cracking views OCR as a sequential recog-
nition problem, standard adversarial perturbation is designed to
attack single target. In this study, we propose to attack multiple
targets corresponding to the multiple tokens derived from OCR
sequential recognition. The generated adversarial CAPTCHA
image is expected to simultaneously misclassify all the char-
acter tokens. For specific token sequence t, all the characters
appearing in t constitute the original set Θ, while the remain-
ing characters from (0-9, A-Z) constitute the adversary set Θ̄.
Denoting the raw image as x and the corresponding adversary
image as x , the multi-target attack is formulated as the following
optimization problem:

min d(x, x 
) + λ · [max F (x )θj i − F (x )θθ̄i ]+ (1)
x
j=θ̄i i
θi ∈Θ
where d(·, ·) is distance function to minimize the modification
from x to x3 , λ is the weight parameter balancing between
the image modification and the misclassification confidence in
the second term. Within the second term, θi is the character
appearing in the original set Θ, θ̄i is its one-to-one mapping
character in the adversary set Θ̄, F (x )θi denotes the output of
the second-to-last layer, the logits, corresponding to token θi af-
ter sequential recognition, F (x )θj i denotes its j dimension, and
[f ]+ is the positive part function denoting max(f, 0). Note that
the one-to-one mapping from θi to θ̄i can be either random or
2 For typical 4 character-based CAPTCHAs, recurrent neural network usu-
ally outputs 12-token sequence to improve tolerance for segmentation and align-
ment [47].
3 Alternative choices for the distance function are allowed. In our experiment,
we use L2 distance.
 2580
Fig. 4. The proposed robust CAPTCHA designing framework. The left represents the process of CAPTCHA cracking, including sequential recognition, fea-
ture extraction, image binarization (Gaussian filtering) and stochastic transformation. The right represents our solution of CAPTCHA generation, including the
corresponding multi-target attack, ensemble adversarial training, differentiable approximation and expectation, respectively.
fixed. Random one-to-one mapping leads to targeted adversar-
ial attack, and fixed mapping leads to non-targeted adversarial
attack.4
When the original set Θ contains only one character, the
multi-target attack reduces to single-target attack as the stan-
dard adversarial perturbation. In fact, according to the mecha-
nism of output decoding in CAPTCHA cracking, we only need
to misclassify any one of the character tokens to invalid the fi-
nal recognition result. The above equation in Eqn. (1) provides a
general case of attacking flexible numbers of character tokens. In
practice, the number of attacked characters is one important pa-
rameter to control the model performance. More attacked char-
acters guarantee higher success rate to resist crack, yet leading
to more derived distortions and human recognition burden. The
quantitative influence of attacked character number on the im-
age distortion level and algorithm recognition rate is discussed
in Section V-C.
B. Ensemble Adversarial Training Towards Black-Box Crack
As mentioned in Section I, CAPTCHA cracking may em-
ploy multiple OCR algorithms for character recognition. At the
stage of designing CAPTCHA, it is impractical to target one
specific OCR algorithm, which requires to design adversarial
CAPTCHA images that are effective to as many OCR algo-
rithms as possible. Fortunately, it is recognized that adversarial
perturbation is transferable between models: if an adversarial
image remains effective for multiple models, it is more likely to
transfer to other models as well [29]. Inspired by this, in order
to improve the resistance to unknown cracking models, we pro-
pose to generate adversarial images simultaneously misleading
multiple models.
4 The reported experimental results in Section V are based on random one-
to-one mapping.
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 23, 2021
Specifically, given K white-box OCR models with their cor-
responding the output of the second-to-last layer as J1 , . . ., JK ,
we re-formulate the objective function in Eq. (1) by replacing
F (x ) with F̃ (x ) defined as follows:
K

F̃ (x ) = αk Jk (x ) (2)
k=1

where αk is the ensemble weight with K k=1 αk = 1. In most
cases, αk = 1/K except that one model is more important than
others. Among the three sub-modules of OCR stage, feature
extraction has the most model choices (e.g. various CNN struc-
tures as GoogLeNet [49], ResNet [50]) which can be easily im-
plemented into different CAPTCHA cracking solutions. There-
fore, this study addresses the black-box cracking issue by attack-
ing multiple feature extraction models. Specifically, the training
data and basic structure of Ji (x ) and F (x ) are identical ex-
cept for the different CNN structures in the feature extraction
sub-module. On the number of CNN structures, the larger the
value of K, the stronger the generalization capability of the
derived adversarial CAPTCHA images. However, an excessive
K value will lead to high computational complexity and trivial
weight αk to underemphasize single model. Referring to previ-
ous studies on ensemble adversarial attack [51], 3 ∼ 5 models
achieve a good balance between transferability and practicality.
In this study, we select K = 3 and evenly set αk = 1/3. The
experimental results in [51] show that under the same training
set, the adversarial examples can achieve stronger transferability
when the network structure is more similar, and it is reasonable
to choose the model with large structure difference to employ
ensemble adversarial training. The performance of employing
ensemble adversarial training to resist different OCRs is reported
in Section V-D.
 ZHANG et al.: ROBUST CAPTCHAS TOWARDS MALICIOUS OCR
C. Differentiable Approximation Towards
Image Preprocessing
The data observations in Section III-B demonstrate the dis-
tortion removal consequences from binarization operation, re-
quiring us to consider the affection of image preprocessing in
adversarial image generation. To address this, we regard image
preprocessing operation as part of the entire end-to-end solu-
tion so that we can generate corresponding adversarial images
effectively to mislead the whole cracking solution.
According to the usability to be incorporated into the end-to-
end solution, image preprocessing operations can be roughly
divided into two categories as either differentiable or non-
differentiable. For each category, we select one representative
operation to address in this study, i.e., Gaussian filtering and
image binarization. Regarding the differentiable Gaussian fil-
 1 − x
2
tering operation, g(x )= √2πσ , we can readily incorporate it
e 2σ2
into the OCR model (Eq. (1), Eq. (2)) by replacing the input
image x with the preprocessing image g(x ). Both forward and
backward propagation are conducted on the replaced function
F (g(x )), leading to the generated adversarial images expected
to eliminate the affection from Gaussian filtering.
Regarding the non-differentiable image binarization, we can-
not straightforwardly incorporate it into the objective function.
Instead, we find a differentiable approximation s(x ) to image
binarization and incorporate the approximated function into the
end-to-end solution. In this study, s(x ) is defined as follows:
1 tion such as color change, image translation, random rotation,
s(x ) = x −τ
1 + e− ω
where τ denotes the threshold of image binarization, ω denotes
the degree of lateral expansion of the curve. Note that to guaran-
tee that the generated adversarial images are resistant to im-
age binarization, we only employ the approximated s(x ) at
the backward propagation stage to update the generated image,
while the forward propagation still use the actual x to calculate
∇x F (x).
To simultaneously resist to the affections from Gaussian fil-
tering and image binarization, we concatenate s(·) and g(·) in
the final objective function. Therefore, the overall optimization
problem incorporating the three proposed modules is as follows:
 +
 the end-to-end solution.
  θi  θi
min
d(x, x ) + λ · max F̃ (φ(x ))j − F̃ (φ(x )))θ̄
x j=θ̄i
θi ∈Θ
(4)
where F̃ (·) denotes the ensemble of multiple OCR models de-
fined in Eq. (2), and φ(x ) = s(g(x )) denotes the approximated
image preprocessing operations defined in Eq. (3).
D. Expectation Towards Stochastic Image Transformation
The above three subsections are more than enough for gen-
eral CAPTCHAs generation towards OCR cracking solution.
However, a potential cracker could use a number of transforma-
tions to make the adversarial perturbations meaningless, e.g.,
the cracker could slightly rotate the image, doing so entirely
bypasses general adversarial examples. Prior work has shown
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
2581
that these adversarial examples fail to remain effective under
image transformations, and the gradient of the expected value
can increase adversarial even under various image transforma-
tions [52]. To integrate image preprocessing issues with potential
transformations, we compute expectation of stochastic image
transformation including different angles rotation. The expecta-
tion allows us for the construction of adversarial examples that
remain adversarial over a chosen transformation distribution T .
More concretely, we use a chosen distribution T of transforma-
tion functions t taking an input x controlled by the adversary
to the true input t(x) perceived by the OCR rather than op-
timizing the objective function of a single example. We then
re-formulate the second term in Eq. (4) by replacing x, x with
t(x), t(x ) defined as follows:

Et∼T [max F̃ (φ(t(x )))θj i − F̃ (φ(t(x ))))θθ̄i ]+ (5)
θi ∈Θ
j=θ̄i i
Furthermore, rather than simply taking the d(·, ·) to constrain
the solution space, we instead aim to constrain the expected
effective distance between the adversarial and original inputs.
The first term in Eq. (4) is replaced by d(t(x), t(x )) defined as
follows:
Et∼T [d(t(x), t(x ))] (6)
In practice, the distribution T can model perceptual transforma-
(3) or addition of noise. These transformations amount to a set of
the random linear combinations, which are more thoroughly de-
scribed in Section V-E. Then we can approximate the gradient of
the expected value through sampling transformations indepen-
dently at each gradient descent step in optimizing the objective
function and differentiating through the transformation. Given
its ability to generate robust adversarial CAPTCHA images, we
use the gradient of the expected value to directly eliminate the
affection from stochastic transformation for differentiable im-
age transformation. But regarding some non-differentiable im-
age transformation, we cannot straightforwardly differentiate
through the transformation. Instead, we can use the same cate-
gory as mentioned in Section IV-C that finding a differentiable
approximation and incorporate the approximated function into
i
V. EXPERIMENTS
We examined CAPTCHA images with 4 characters for ex-
periments. The CAPTCHAs are RGB images with resolution
of 192 × 64 px. Regarding the cracking method, we consid-
ered image binarization and Gaussian filtering (kernel size:
3 × 3, σ = 0.8) at the image preprocessing stage. The OCR
stage is instantiated with CNN structures for feature extraction
and LSTM+softmax for sequential recognition. Regarding our
proposed CAPTCHA generation method, image binarization is
approximated with τ = 0.8, ω = 0.05, and 4 CNN structures are
employed for ensemble adversarial training. All experiments are
conducted on Nvidia GTX 1080Ti GPU with 11 G memory.
 2582
Fig. 5. Example images (top row) and theri attention maps (bottom row). From
left to right, we show the original image, the image with Gaussian white noise,
the adversarial generated by our method and the adversarial image generated by
our method but without considering image preprocessing.
A. Qualitative Attention Analysis
Visual attention has been widely used to explain which region
of image contributes much to the model decision [53]. In this
study, we extracted the attention map using Grad-CAM [54] to
understand the change of recognition performance under differ-
ent visual distortions.
The first and second columns of Fig. 5 visualize the atten-
tion map of the raw image and the image with Gaussian white
noise. It can be found that Gaussian white noise brings trivial
attention change from the original image. Both attention maps
keep to the region where characters exist. This well explains the
data observations in Section III-A that algorithm is generally
robust to Gaussian white noise. We also visualized the attention
map of the CAPTCHA images generated from our proposed
method on the third column of Fig. 5. It is shown that the atten-
tion maps deviate much from the original image and focus on
unrelated regions where there exist no characters. This justifies
our motivation to employ adversarial perturbation to mislead the
algorithm prediction result, and demonstrates the effectiveness
of our proposed CAPTCHA design method.
To further validate the necessity of considering image pre-
processing in robust CAPTCHA design, the attention maps for
the images generated from our method but without considering
image preprocessing are shown in the fourth column of Fig. 5
for comparison. It is easy to conceive that without considering
image preprocessing, the generated images fail to deviate the
attention from the character regions. This is consistent with the
fact that image preprocessing has the effect of weakening or
eliminating adversarial perturbation.
B. Quantitative Performance Comparison
To compare the performance of the proposed robust
CAPTCHA (rCAPTCHA) designing method, we report the
recognition accuracies of state-of-the-art cracking solution un-
der the following settings:
r Raw: the original CAPTCHA images without adding ad-
versarial perturbations;
r rCAPTCHA_parallel: the proposed solution to generated
adversarial images, expect that the sub-module of OCR is
replaced by 4 parallel recognition networks (each realized
by one fully-connected layer) to address one character’s
recognition;
r rCAPTCHA_w/o preprocessing: the proposed solution to
generated adversarial images, but without considering the
image preprocessing stages;
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 23, 2021
Fig. 6. Example CAPTCHAs with different complexity levels (from top to
bottom: easy, medium, hard). Each row from left to right shows the differ-
ent settings of Raw, rCAPTCHA_parallel, rCAPTCHA_w/o preprocessing and
rCAPTCHA.
r rCAPTCHA: the proposed solution to generated adversar-
ial images, considering both sequential recognition and im-
age preprocessing operations.
The state-of-the-art cracking solution is trained over 20,000
CAPTCHA images with batch size 128. To examine the appli-
cation scope of the proposed CAPTCHA generation methods,
we conducted experiments on the CAPTCHAs with three lev-
els of complexities: easy, medium, hard. Fig. 6 shows exam-
ples of different complexity levels of CAPTCHAs in the above
four settings. For each of the settings, we selected/generated
500 CAPTCHA images for testing, and summarize the de-
rived average recognition accuracy in Table I. Experimental
observations include: (1) By adding adversarial perturbations,
the right 3 columns consistently obtain lower accuracies than
the first column, showing the usability of employing adversar-
ial perturbations in resisting CAPTCHA cracking. (2) With-
out considering the sequential recognition or image preprocess-
ing characteristics, the resisting effect of rCAPTCHA_parallel
and rCAPTCHA_w/o preprocessing is not as obvious as that
of rCAPCHA. This validates the necessity of multi-target at-
tack and differentiable approximation modules. (3) Regarding
CAPTCHAs with different complexities, we observed consis-
tent phenomenon among the four settings, demonstrating the
wide application scope of the proposed CAPTCHA generation
method.
The notable decrease in algorithm recognition accuracies
shows the effectiveness of employing adversarial perturbation
to mislead cracking solution. To facilitate the correlation under-
standing between misleading cracking solution and friendly to
human recognition, we also provide the human recognition ac-
curacy for each experimental setting in Table I. Similar to the
data analysis, we have recruited 164 workers from MTurk to
recognize 4 character-based CAPTCHA images. The reported
accuracies are averaged over 1,200 CAPTCHAs. By comparing
different rows, it is shown that the increasing content complex-
ity brings slight decrease of algorithm recognition accuracy but
causes huge trouble to human recognition. Among different set-
ting columns, while the algorithm recognition accuracy fluctu-
ates a lot, the human recognition performance basically remains
stable, validating the different distortion vulnerability between
human and algorithm. In summary, regarding CAPTCHA im-
ages with different complexities, the proposed CAPTCHA gen-
eration method succeeds to invalid the cracking algorithm with-
out increasing human recognition burden.
 ZHANG et al.: ROBUST CAPTCHAS TOWARDS MALICIOUS OCR 2583

TABLE I
THE RECOGNITION OF DIFFERENT COMPLEXITY LEVELS OF CAPTCHAS IN THE DIFFERENT SETTINGS. THE RESULTS OF ALGORITHMS ARE
OBTAINED AFTER GAUSSIAN FILTERING AND IMAGE BINARIZATION

Fig. 7. The influence of λ on derived image distortion and cracking recognition Fig. 8. The influence of |Θ| on derived image distortion and cracking recog-
accuracy. nition accuracy.

C. Parameter Influence Analysis

The proposed robust CAPTCHA generation method mainly
involves with two parameters: the weight parameter λ in Eq. (1)
and the number of attacked characters |Θ|. As introduced in
the methodology, the weight parameter λ controls the relative
importance between the visual distortion and misclassification
confidence. We adjusted λ within range of [10,30] with step of 1
and examined its influence on the derived image distortion and
cracking recognition accuracy. The image distortion is measured
as the sum of squared pixel-wise difference between the original
and adversarial images. The averaged distortion and recognition
accuracy with the change of λ are drawn in Fig. 7. It is shown
that as λ increases, more image distortion is observed in the
derived CAPTCHA images and cracking method tends to fail
in recognizing the generated CAPTCHAs. This is consistent
with the definition of λ in Eq. (1). In practical applications, to
prevent annoying human subjects in recognizing the generated
CAPTCHAs, an appropriate λ is selected with moderate image
distortion and guaranteed cracking resistant performance. Our
experimental results reported in Section V-B are based on λ =
20.
Regarding the number of attacked characters, we set |Θ| to
{1, 2, 3, 4} respectively and examined the corresponding aver-
aged image distortion and algorithm recognition accuracy in
Fig. 8. As shown in Fig. 8(a), with |Θ| increases, more image
distortion is needed to misclassify the characters. In Fig. 9 we
show example CAPTCHAs generated by rCAPTCHA with dif-
ferent levels of image distortions. Combing with Fig. 8(a), it is
demonstrated that using the proposed rCAPTCHA method to at-
tack even all 4 characters, the derived CAPTCHAs are generally
friendly to human and not bringing extra recognition burden. As
shown in Fig. 8(b), the increase of |Θ| enhances the confidence

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
 2584
TABLE II
TRANSFERABILITY OF ADVERSARIAL IMAGES GENERATED BETWEEN PAIRS OF MODELS. THE ELEMENT (i, j) REPRESENTS THE ACCURACY OF
THE ADVERSARIAL IMAGES GENERATED FOR MODEL i (ROW) TESTED OVER MODEL j (COLUMN)
Fig. 9. Example CAPTCHAs with different image distortions: from left to
right shows images with distortions of 100, 200, 300, and 400.
to mislead the cracking algorithm and obtains consistently lower
recognition accuracy. With the introduction of multi-attack to-
wards sequential recognition, the proposed rCAPTCHA method
possess the flexibility to attack arbitrary number of characters.
In our experiments, to guarantee the resistance capability, we
fixed |Θ| = 4.
D. Robustness Towards Different OCRs
To compare the generalization and transferability of our pro-
posed rCAPTCHA method and ACs [45], we implemented dif-
ferent cracking methods and examined their recognition accu-
racy on the generated CAPTCHAs. For generating CAPTCHAs
of our method, we respectively trained 3 OCR models with dif-
ferent CNN structures, which are denoted as 4ConvNet, mini-
ResNet and mini-DenseNet. For generating CAPTCHAs of [45],
we trained the same OCR as above, except that the sub-module
(LSTM) of OCR is replaced by 4 parallel recognition networks
(fully-connected layer). For testing CAPTCHAs, we trained 2
OCR models. One is the same OCR with CNN structure of
mini-GoogLeNet, the other is mini-GoogLeNet w/ attention [17],
which also uses mini-GoogLeNet but adopts the attention mech-
anism in LSTM. 4ConvNet uses four convolutional layers for
feature extraction. The LSTM input required a fixed-size fea-
ture vector, so we modified the native network. mini-XNets
are employed due to the quicker convergence times and low
resolution of CAPTCHA images: mini-ResNet consists of five
ResBlocks and two convolutional layers, mini-DenseNet con-
sists of four DenseBlocks with four convolutional layers, and
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 23, 2021
mini-GoogLeNet consists of two Inception modules with six
convolutional layers.
Three models of 4ConvNet, mini-ResNet, mini-DenseNet are
selected as white-boxs, with the remaining mini-GoogLeNet
model and mini-GoogLeNet w/ attention as the black-boxs. The
black-box models are regard as the potential OCR cracking to
simulate the alternative cracking choices in real-world appli-
cations. Averaged over 100 tested CAPTCHAs, Table II sum-
marizes the black-box cracking recognition accuracy under dif-
ferent training-testing pairs. For example, the value of 91% at
element (1,1) represents the recognition accuracy of original
CAPTCHA images on 4ConvNet. The value of 0% at element
(8,1) indicates the recognition accuracy trained with ensembled
3 white-box models and tested on 4ConvNet. Lower accuracy
value means superior resistant performance to cracking solu-
tions and better transferability of the method.
In the top half of Table II (ACs), the adversarial CAPTCHAs
without considering the sequential recognition obtain higher av-
erage accuracies than rCAPTCHA. The accuracies of ACs are
no lower than 60% when image preprocessing and image trans-
formation are not involved. It is expected that the employing
image preprocessing and image transformation can increase the
accuracy, and 60% means that these adversarial CAPTCHAs are
almost recognized by the OCR. This demonstrates when exclud-
ing controlled variables, considering the sequential recognition
problem is more important than adopting Fourier transform dur-
ing the generation of adversarial CAPTCHAs. In the bottom
half of Table II (rCAPTCHA), we can observe that the adversar-
ial images generated with one model perform well on their own
models but generally perform poorly on other models. However,
if we generate the CAPTCHA images with ensemble training of
3 models, the testing recognition accuracies for all 5 models
are no higher than 7%. This demonstrates the transferability of
the proposed rCAPTCHA method in employing ensemble train-
ing towards black-box cracking. Specially, the value of 7% at
element (8, 4) demonstrates our method can perform well on
GoogLeNet w/ attention (GoogLeNet w/). The reason is that the
network structure of the current state-of-the-art OCR model is
similar (CNN + LSTM). We also generate the adversarial images
on GoogLeNet w/ attention (the last row), which validates the
 ZHANG et al.: ROBUST CAPTCHAS TOWARDS MALICIOUS OCR
TABLE III
DISTRIBUTION OF TRANSFORMATIONS
TABLE IV
THE RECOGNITION OF RAW IMAGES AND ADVERSARIAL IMAGES. THE RESULT
ARE OBTAINED AFTER STOCHASTIC TRANSFORMATION
generalization of our proposed mechanism. It is expected with
more models implemented in ensemble training, the resistant
performance towards arbitrary black-box cracking methods will
be guaranteed. In practical applications, we can carefully select
white-box models with typically different structures to improve
the generalization and transferability to specific models.
E. Robustness Towards Stochastic Image Transformation
To justify the performance of our method for stochastic im-
age transformation, we implemented a set of transformations on
different complexity levels of CAPTCHAs and examined their
recognition accuracies on the generated CAPTCHAs. Specif-
ically, we considered the distribution of transformations that
includes rotation, color change, rescale, and translation of the
image. We selected/generated the 1,000 images for each of the
settings, randomly chose an image transformation for each im-
age, and examined whether our method is robust over the chosen
distribution. For each adversarial example, we evaluated over
100 stochastic transformations sampled from the distribution at
evaluation time. We give the specific parameters we chose of
stochastic transformations in Table III, where each parameter is
sampled uniformly at random from the specified range. These
stochastic transformations include color change, image transla-
tion, rotation and rescale.
Table IV summarizes the results. Experimental observations
include: (1) By applying stochastic transformations, the sec-
ond row consistently obtains lower accuracies than the first
row, showing the usability of employing expectation towards
stochastic image transformation. (2) After stochastic transfor-
mations, the accuracies of raw images become lower than
before transformations. The decrease in algorithm recognition
accuracies shows that stochastic transformation does hurt per-
formance of normal image, which is caused by the loss of image
information and the shortcoming of neural network, e.g., lack
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
2585
Fig. 10. Example CAPTCHAs with different complexity levels (from top to
bottom: easy, medium, hard). Each row from left to right shows the different
settings of image transformation.
of rotation-invariance. The increase from 0.0%, 0.0%, 4.0% to
1.3%, 5.4%, 9.0% means that stochastic transformation partially
plays a role in eliminating effectiveness of adversarial examples.
Because the stochastic transformation is a kind of adversarial
defense technology, more specific discussions about adversarial
defense are reported in Section V-F. The reason that algorithm
recognition accuracy increases along with complexity levels of
CAPTCHAs is the noises associated with hard CAPTCHA im-
ages mix with the adversarial perturbation and thus adversarial
perturbation will be weakened as the image gets more complex.
By comparing the results in Table I, the larger the “approxima-
tion” component is, the more we can detect this trend.
Fig. 10 shows example CAPTCHAs of different complexity
levels after stochastic transformations, combing with images of
non-transformations, which observes that these images are still
adversarial after different image transformations. This demon-
strates that our approach is effective in generating robust adver-
sarial examples.
Nevertheless, when the number of transformations the cracker
can choose is too large, expectation will occasionally fail to gen-
erate an effective adversarial example. In this study, the number
we choose is 4 and the scope of transformations is also reason-
able. Another case occurs when we replace non-differentiable
image preprocessing with differentiable approximation. Be-
cause the essence of expectation is also an “approximation,”
when the “double approximation” occurs, the effectiveness of
adversarial example will be weakened.
F. Discussion on Adversarial Defense
The image non-differentiable preprocessing and stochastic
image transformation both are aimed at weakening the ef-
fectiveness of adversarial example, which are called adver-
sarial defenses. However, the existence of robust adversarial
CAPTCHAs implies that defenses based on stochastic trans-
forming input are not secure: adversarial examples generated by
using differentiable approximation and expectation can circum-
vent these defenses. Prior work has shown that most of adver-
sarial defenses are based on obfuscated gradients, in this study:
(1) Image non-differentiable preprocessing is a kind of shattered
gradients, which are nonexistent or incorrect gradients caused
either intentionally through non-differentiable operations or un-
intentionally through numerical instability; (2) Stochastic image
transformation is a kind of stochastic gradients, which depend
on test-time randomness [35]. But if the obfuscated gradient
 2586
Fig. 11. The recognition accuracy of relax adversarial training.
information can be approximated, it can only provide a false
sense of security.
There is another kind of adversarial defense technology called
adversarial training, which is not dependent on obfuscated gra-
dient. Adversarial training solves a min-max game through a
conceptually simple process: train on adversarial examples un-
til the model learns to classify them correctly [30]. To further
validate the adversarial defense, we study the adversarial train-
ing approach of [55] in this subsection. For this scenario, we
generated/selected 500 adversarial CAPTCHA images for test-
ing. Then we started to fine-tune the model with the 50,000
steps. However, due to the complexity of OCR model, com-
pared with the common CNN model, standard adversarial train-
ing does not show the effectiveness as usual. After training with
50,000 steps, the accuracy of OCR is still 0%. So we relax the
constraints of standard adversarial training to examine whether
the idea of adversarial training will work. Then we fine-tune the
model on the same 500 adversarial images as testset. The re-
sults are shown in Fig. 11, from combined results we make the
following observations. As training data increases, adversarial
training significantly improves the accuracy of the OCR model.
But its shortcomings also obvious that the time cost is gradu-
ally increasing. Moreover, if a cracker wants to use adversarial
training, he is supposed to have access to the training dataset
we use and the parameter in algorithm we choose, e.g., distance
function to minimize the modification and so on.
We discussed adversarial defense technology on adversarial
CAPTCHAs we proposed. The defense technologies based on
obfuscated gradients cannot hinder the type of CAPTCHAs. The
adversarial training based on non-obfuscated gradients is still
effective but limited to practicality.
VI. CONCLUSION
This study designs robust character-based CAPTCHAs to
resist cracking algorithms by employing their unrobustness
to adversarial perturbation. We have conducted data analysis
and observed human and algorithm’s different vulnerabilities
to visual distortions. Based on the observation, robust
CAPTCHA (rCAPTCHA) generation framework is introduced
with three modules of multi-target attack, ensemble adversarial
training, differentiable approximation to image preprocessing,
and expectation to stochastic image transformation. Qualitative
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 23, 2021
and quantitative experimental results demonstrate the effective-
ness of generated CAPTCHAs in resisting cracking algorithms.
We ascribe the main contribution not as proposing a specific
CAPTCHA system, but as introducing the idea of exploiting
algorithm unrobustness to increase the robustness of automated
system towards cracking. Character-based CAPTCHA, which is
most friendly and effective to human, to validate this idea with
the simplest scheme. This idea is expected to easily adopt to
generate robust image-based and other CAPTCHAs.
It is noted that similar to the game competition between ad-
versarial attack and defense, with more CAPTCHA designers
employing adversarial attack to resist cracking, future cracking
solutions are expected to employ adversarial defense techniques
for self-enhancement. We hope this study could draw attention
of future CAPTCHA designing on the competition between ad-
versarial attack and defense. Moreover, with the development
of deep learning and other AI algorithms, we are confronted
with critical security-related problems when algorithms are ma-
liciously utilized towards human. In this case, it is necessary to
get aware of the limitations of current algorithms and appropri-
ately employ them to resist the abuse use of algorithms.
REFERENCES
[1] A. M. Turing, “Computing machinery and intelligence-am turing,” Mind,
vol. 59, no. 236, pp. 433–434, 1950.
[2] L. Von Ahn, M. Blum and J. Langford, “Telling humans and computer-
sapart automatically,” Commun. ACM, vol. 47, no. 2, pp. 56–60, 2004.
[3] L. Von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum, “Re-
captcha: Human-based character recognition via web security measures,”
Science, vol. 321, no. 5895, pp. 1465–1468, 2008.
[4] A. A. Chandavale, A. M. Sapkal, and R. M. Jalnekar, “Algorithm to break
visual captcha,” in Proc. Int. Conf. Emerg. Trends Eng. Technol., 2009, pp.
258–262.
[5] G. Mori and J. Malik, “Recognizing objects in adversarial clutter: Breaking
a visual captcha,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern
Recognit., 2003, vol. 1, pp. 1–8.
[6] S. Sivakorn, I. Polakis, and A. D. Keromytis, “I am robot: (Deep) learn-
ing to break semantic image captchas,” in Proc. IEEE Eur. Symp. Secur.
Privacy, 2016, pp. 388–403.
[7] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpass-
ing human-level performance on imagenet classification,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1026–1034.
[8] J. F. Gemmeke et al., “Audio set: An ontology and human-labeled dataset
for audio events,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process.,
2017, pp. 776–780.
[9] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ ques-
tions for machine comprehension of text,” 2016, arXiv:1606.05250.
[10] M. R. Ogiela, N. Krzyworzeka, and L. Ogiela, “Application of knowledge-
based cognitive captcha in cloud of things security,” Concurrency Com-
put.: Pract. Experience, vol. 30, no. 21, pp. 1–11, 2018.
[11] D. Geman, S. Geman, N. Hallonquist, and L. Younes, “Visual turing test
for computer vision systems,” Proc. Nat. Acad. Sci., vol. 112, no. 12,
pp. 3618–3623, 2015.
[12] C. Szegedy et al., “Intriguing properties of neural networks,” in Proc. Int.
Conf. Learn. Representations, 2014, pp. 1–10.
[13] Q. Liu, L. Wang, and Q. Huo, “A study on effects of implicit and explicit
language model information for dblstm-ctc based handwriting recogni-
tion,” in Proc. Int. Conf. Document Anal. Recognit., 2015, pp. 461–465.
[14] T. M. Breuel, “High performance text recognition using a hybrid
convolutional-LSTM implementation,” in Proc. IAPR Int. Conf. Docu-
ment Anal. Recognit., 2017, vol. 1, pp. 11–16.
[15] M. Jenckel, S. S. Bukhari, and A. Dengel, “Transcription free LSTM OCR
model evaluation,” in Proc. Int. Conf. Frontiers Handwriting Recognit.,
2018, pp. 122–126.
[16] H.-R. Shin, J.-S. Park, and J.-K. Song, “OCR for drawing images us-
ing bidirectional LSTM with CTC,” in Proc. IEEE Student Conf. Electric
Mach. Syst., 2019, pp. 1–4.
 ZHANG et al.: ROBUST CAPTCHAS TOWARDS MALICIOUS OCR
[17] Y. Zi, H. Gao, Z. Cheng, and Y. Liu, “An end-to-end attack on text
CAPTCHAs,” IEEE Trans. Inf. Forensics Secur., vol. 15, pp. 753–766,
2020.
[18] Y.-W. Chow, W. Susilo, and P. Thorncharoensri, “CAPTCHA design and
security issues,” in Proc. Adv. Cyber Secur.: Principles, Techn., Appl.,
2019, pp. 69–92.
[19] H. Kwon, Y. Kim, H. Yoon, and D. Choi, “CAPTCHA image generation
systems using generative adversarial networks,” IEICE Trans. Inf. Syst.,
vol. 101, no. 2, pp. 543–546, 2018.
[20] M. E. Hoque, D. J. Russomanno, and M. Yeasin, “2D CAPTCHAs from
3D models,” in Proc. IEEE SoutheastCon, 2006, pp. 165–170.
[21] M. Imsamai and S. Phimoltares, “3d captcha: A next generation of the-
captcha,” in Proc. Int. Conf. Inf. Sci. Appl., 2010, pp. 1–8.
[22] V. D. Nguyen, Y.-W. Chow, and W. Susilo, “On the security of text-based
3D captchas,” Comput. Secur., vol. 45, pp. 84–99, 2014.
[23] Q. Ye, Y. Chen, and B. Zhu, “The robustness of a new 3D captcha,” in
Proc. IAPR Int. Workshop Document Anal. Syst., 2014, pp. 319–323.
[24] V. D. Nguyen, Y.-W. Chow, and W. Susilo, “Breaking an animated
CAPTCHA scheme,” in Proc. Int. Conf. Appl. Cryptography Netw. Se-
cur., 2012, pp. 12–29.
[25] D. Georgeet al., “A generative vision model that trains with high data
efficiency and breaks text-based captchas,” Science, vol. 358, no. 6368,
pp. 2612–2621, 2017.
[26] G. Ye et al., “Yet another text CAPTCHA solver: A generative adversarial
network based approach,” in Proc. ACM SIGSAC Conf. Comput. Commun.
Secur., 2018, pp. 332–348.
[27] Y. Lv, F. Cai, D. Lin, and D. Cao, “Chinese character CAPTCHA recog-
nition based on convolution neural network,” in Proc. IEEE Congr. Evol.
Comput., 2016, pp. 4854–4859.
[28] J. Chen, X. Luo, Y. Liu, J. Wang, and Y. Ma, “Selective learning con-
fusion class for text-based captcha recognition,” IEEE Access, vol. 7,
pp. 22246–22259, 2019.
[29] W. Zhou et al., “Transferable adversarial perturbations,” in Proc. Eur. Conf.
Comput. Vision, 2018, pp. 452–467.
[30] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harness-
ing adversarial examples,” in Proc. Int. Conf. Learn. Representations,
2015, pp. 1–10.
[31] A. Kurakin, I. J. Goodfellow, and S. Bengio, “Adversarial examples in
the physical world,” in Proc. Int. Conf. Learn. Representation Workshop,
2017.
[32] Y. Dong et al., “Boosting adversarial attacks with momentum,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 9185–9193.
[33] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: A simple
and accurate method to fool deep neural networks,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit., 2016, pp. 2574–2582.
[34] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural
networks,” in Proc. IEEE Symp. Secur. Privacy, 2017, pp. 39–57.
[35] A. Athalye, N. Carlini, and D. A. Wagner, “Obfuscated gradients give a
false sense of security: Circumventing defenses to adversarial examples,”
in Proc. Int. Conf. Mach. Learn., 2018, pp. 274–283.
[36] J. Deng et al., “Imagenet: A large-scale hierarchical image database,” in
Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 248–255.
[37] C. Xie et al., “Adversarial examples for semantic segmentation and object
detection,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 1369–1378.
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on September 13,2022 at 15:14:47 UTC from IEEE Xplore. Restrictions apply.
2587
[38] M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, “Accessorize to
a crime: Real and stealthy attacks on state-of-the-art face recognition,” in
Proc. ACM Sigsac Conf. Comput. Commun. Secur., 2016, pp. 1528–1540.
[39] N. Carlini and D. Wagner, “Audio adversarial examples: Targeted attacks
on speech-to-text,” in Proc. IEEE Secur. Privacy Workshops, 2018, pp.
1–7.
[40] J. Li, S. Ji, T. Du, B. Li, and T. Wang, “Textbugger: Generating adversarial
text against real-world applications,” in Proc. Annu. Netw. Distrib. Syst.
Secur. Symp., 2019, pp. 1–15.
[41] X. Li, K. Yu, S. Ji, Y. Wang, C. Wu, and H. Xue et al., “Fighting against
deep-fake: Patch&pair convolutional neural networks,” in Proc. Compan-
ion Web Conf., 2020, pp. 88–89.
[42] X. Ling et al., “Deepsec: A uniform platform for security analysis of deep
learning model,” in Proc. IEEE Symp. Secur. Privacy, 2019, pp. 673–690.
[43] M. Osadchy, J. Hernandez-Castro, S. Gibson, O. Dunkelman, and D.
Pérez-Cabo, “No bot expects the deepCAPTCHA! introducing immutable
adversarial examples, with applications to CAPTCHA generation,” IEEE
Transactions. on Information. Forensics. and Security., vol. 12, no. 11,
pp. 2640–2653, Nov. 2017.
[44] Y. Zhang, H. Gao, G. Pei, S. Kang, and X. Zhou, “Effect of adversarial
examples on the robustness of CAPTCHA,” in Proc. Int. Conf. Cyber-
Enabled Distrib. Comput. Knowl. Discovery, 2018, pp. 1–109.
[45] C. Shi et al., “Text CAPTCHA is dead? a large scale deployment and
empirical study,” in Proc. ACM Conf. Comput. Commun. Secur., 2020,
pp. 1–16.
[46] B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural network
for image-based sequence recognition and its application to scene text
recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 11,
pp. 2298–2304, Nov. 2017.
[47] T. M. Breuel, A. Ul-Hasan, M. A. Al-Azawi, and F. Shafait, “High-
performance OCR for printed english and fraktur using LSTM networks,”
in Proc. 12th Int. Conf. Document Anal. Recognit., 2013, pp. 683–687.
[48] F. Liao et al., “Defense against adversarial attacks using high-level rep-
resentation guided denoiser,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., 2018, pp. 1778–1787.
[49] C. Szegedy et al., “Going deeper with convolutions,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit., 2015, pp. 1–9.
[50] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016,
pp. 770–778.
[51] Y. Liu, X. Chen, C. Liu, and D. Song, “Delving into transferable adversarial
examples and black-box attacks,” in Proc. Int. Conf. Learn. Representa-
tions, 2017, pp. 1–10.
[52] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok, “Synthesizing ro-
bust adversarial examples,” in Proc. Int. Conf. Mach. Learn., 2018,
pp. 284–293.
[53] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional
networks,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 818–833.
[54] R. R. Selvaraju et al., “Grad-CAM: Visual explanations from deep net-
works via gradient-based localization,” in Proc. IEEE Int. Conf. Comput.
Vis., 2017, pp. 618–626.
[55] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards
deep learning models resistant to adversarial attacks,” in Proc. Int. Conf.
Learn. Representations, 2018, pp. 1–10.