International Journal on Document Analysis and Recognition (IJDAR)

https://doi.org/10.1007/s10032-023-00446-7

ORIGINAL PAPER

A multifaceted evaluation of representation of graphemes for

practically effective Bangla OCR
Koushik Roy2 · Md Sazzad Hossain1 · Pritom Kumar Saha1 · Shadman Rohan2 · Imranul Ashrafi2 ·
Ifty Mohammad Rezwan2 · Fuad Rahman1 · B. M. Mainul Hossain3 · Ahmedul Kabir3 ·
Nabeel Mohammed2

Received: 4 August 2022 / Revised: 18 March 2023 / Accepted: 10 June 2023

© The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2023

Abstract
Bangla Optical Character Recognition (OCR) poses a unique challenge due to the presence of hundreds of diverse conjunct
characters formed by the combination of two or more letters. In this paper, we propose two novel grapheme representation
methods that improve the recognition of these conjunct characters and the overall performance of OCR in Bangla. We have
utilized the popular Convolutional Recurrent Neural Network architecture and implemented our grapheme representation
strategies to design the final labels of the model. Due to the absence of a large-scale Bangla word-level printed dataset, we
created a synthetically generated Bangla corpus containing 2 million samples that are representative and sufficiently varied
in terms of fonts, domain, and vocabulary size to train our Bangla OCR model. To test the various aspects of our model,
we have also created 6 test protocols. Finally, to establish the generalizability of our grapheme representation methods, we
have performed training and testing on external handwriting datasets. Experimental results proved the effectiveness of our
novel approach. Furthermore, our synthetically generated training dataset and the test protocols are made available to serve
as benchmarks for future Bangla OCR research.

Keywords Bangla OCR · Word-level OCR · Synthetic Bangla OCR dataset · OCR benchmark · Neural networks

1 Introduction
Koushik Roy, Md Sazzad Hossain, Pritom Kumar Saha have contributed
equally to this work. Optical character recognition, or OCR, can be considered
a complicated and diverse problem. Many approaches have
B Fuad Rahman been taken to address this complex problem. These solutions
fuad@apurbatech.com contain both rule-based deterministic and heuristic solutions.
Koushik Roy Although rule-based solutions can be considered for their
koushik.roy@northsouth.edu interpretability, it has proven difficult to capture all the appli-
Md Sazzad Hossain cable rules, thus reducing the model’s adaptability. This has
sazzad_hossain@apurbatech.com led to the rise in heuristic method-based solutions, which
Pritom Kumar Saha stem mostly from statistics and machine learning. Earlier
pritom_saha@apurbatech.com solutions contained a mixture of rule and popular statistical
Shadman Rohan
shadman.rohan@northsouth.edu
Imranul Ashrafi Nabeel Mohammed
imranul.ashrafi@northsouth.edu nabeel.mohammed@northsouth.edu
Ifty Mohammad Rezwan 1 Apurba Technologies, Dhaka, Bangladesh
mohammad.rezwan@northsouth.edu
2 Apurba-NSU R&D Lab, North South University, Dhaka
B. M. Mainul Hossain 1229, Bangladesh
mainul@iit.du.ac.bd
3 Institute of Information Technology, University of Dhaka,
Ahmedul Kabir Dhaka, Bangladesh
kabir@iit.du.ac.bd

123

machine learning-based solutions such as Hidden Markov
Models [1], Bayesian Models [2]. The recent availability of
large datasets and deep learning models have brought sig-
nificant improvements [3, 4]. These deep learning models
have been found to learn more useful features and outper-
form their statistical and rule-based counterparts. During
the recent emergence of deep learning in OCR, most solu-
tions have been geared towards accessible languages such
as English [5, 6] or Chinese [7] consisting of a plethora of
resources. Languages such as English tend to have a limited
number of characters and do not have any special conjunct
character to themselves. On the other hand, languages such
as Bangla, as addressed in this paper tend to have a consid-
erably larger set of characters. In addition, these characters
have some special characteristics that make it difficult for
OCR. An instance of such is the usage of modifiers where
one character is conjoined with another character to form a
new grapheme. Another special attribute of Bangla emerges
with the conjunct characters. A certain variant form of the
character may appear before or subsequent (or both) to the
base character, mainly discerning a certain form of sound and
the word itself. Such linguistic flexibility of Bangla adds a
new dimension of difficulty to the OCR problem.
There are multiple approaches to optical character recog-
nition, which include the character-based approach and the
word-based approach. Character-based approach requires the
document to be first segmented into lines, then words, and
finally characters. Each level of segmentation adds a new
level of complexity to the problem. This is specifically true
for the Bangla language due to the morphological flexibil-
ity mentioned before. The word-level approach removes the
last layer of complexity and preserves the conjuncts as much
as possible. Moreover, there are two different approaches to
word-level approach in OCR. The first approach, commonly
known as the Fixed method [8, 9], tends to perform OCR
within a fixed set of words from a dictionary. The second
approach, known as the Dynamic method [10, 11], takes into
account the characters in words and performs OCR based on
the dictionary of characters or graphemes it uses while being
trained on a diverse set of words. Finally, when it performs the
recognition, the usage of the dynamic architecture enables the
model to potentially detect words that might not be present
in the dictionary it was trained on. The latter approach is
used in this paper. At the time of conducting our research,
no diverse, correct, large-scale dataset for Bangla word-level
OCR was present. Moreover, the datasets that existed con-
formed primarily for character level recognition and, in some
cases, contained numerous errors.
The research conducted in this paper is a part of building
a word-level printed text recognition service required for a
commercial project. Following this, we propose a pipeline
to create efficient, diverse large-scale synthetic datasets with
word-level annotations. The popular CRNN [10] word level
123
K. Roy et al.
OCR model was used as the baseline, which was further
adapted as per the specifications of grapheme representa-
tion methods in the final layer. The models with the various
grapheme representation methods score over a number of test
sets that also include real, non-synthetic samples. To summa-
rize, the main contributions of this paper are threefold:
1. We present a fully synthetic corpus for Bangla word-level
OCR consisting of 2 million (2,074,992 to be exact) sam-
ples covering a vocabulary of 691,664 words.
2. Six test sets we created are used to evaluate the OCR mod-
els trained on the synthetic corpus on aspects such as real
and synthetic seen and unseen words, fonts, formatting,
etc.
3. We propose two novel grapheme representation methods
suitable for the morphological richness of an intricate lan-
guage like Bangla, relevant to determine the final design
of our models. To confirm the generalizability of the pro-
posed methods, we performed training and testing on
external publicly available Bangla handwriting datasets,
which are completely different from our synthetic data
and test protocols.
The datasets generated and used during the experiments
are available at figshare.1
The experiments reproducible code is publicly available
at GitHub.2
The remainder of this paper is organized as follows, Sect. 2
consists of the review of the literature relevant to this paper.
Next, in Sect. 3, we describe the methodology to generate
our synthetic training set and further outline the test pro-
tocols. In Sect. 4, we elaborate on the methods we used to
perform the experiments, our proposed grapheme represen-
tation methods, and the evaluation metrics. In Sect. 5, we
report all the metrics achieved by our models using different
grapheme representation methods and provide an in-depth
analysis. Finally, we provide our concluding observations in
Sect. 7.
2 Related works
Despite being a challenging task, OCR has seen a lot of appli-
cations in real life. The problem’s challenges can be relevant
to the difficulty of understanding the target language. Other
challenges include predicting document content of different
levels. These can include paragraphs, sentences, words and
characters. Concentrating on these levels of challenges of
OCR, in addition to the inherent complexity of the Bangla
1
https://doi.org/10.6084/m9.figshare.20186825.
2 https://github.com/apurba-nsu-rnd-lab/bangla-ocr-crnn.
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR
language, multiple efforts have been made towards amelio-
rating the challenges [12]. OCR pipelines are commonly
distributed into two different components: the first is the Text
Detection module [13–16] responsible for segmenting and
localizing each sentence, word, and character, and the second
is the Text Recognition module [5, 6, 8–11] which leverages
the segmented word or character images for recognition. One
other variation of OCR models uses an end-to-end strategy
to perform both text detection and recognition in a single
pipeline [6, 17–20]. As this paper looks into the recogni-
tion problem, we limit the scope of the review to only Text
Recognition problems, more specifically, towards word-level
recognition.
In recent years, due to the prevalence of Neural Networks,
OCR models have achieved outstanding performances [10,
21, 22]. However, for Bangla, the development of word-level
OCR has been slower when compared with the growth of
similar OCR pipelines in other languages. We discuss the rel-
evant works for this literature survey in two parts. Firstly, we
discuss the state-of-the-art OCR models in other languages
and finally review the pre-existing OCR models for Bangla.
2.1 OCR in other languages
The advancement of deep neural networks in the last decade
kept pushing the ceiling of OCR research and there have been
a lot of influential works. The OCR challenge can be divided
into two key domains. One is scene text and printed word
recognition and the other is handwriting recognition.
2.1.1 Scene text and printed words recognition
One of the earlier approaches phrased the text recognition
problem as a classification problem [9]. Each word was
assigned a specific label and a deep Convolutional Neural
Network (CNN) was trained with an English vocabulary of
90,000 words. However, the major drawback of the model
appears when used in languages with a higher character set.
Consequently, the model is unable to handle words out of
the designated vocabulary. For instance, when experimenting
with Chinese, a language with a very large number of char-
acters making it is extremely difficult for a deep CNN model
to learn such a large number of patterns. Thus, a CNN model
phrased as an object detector would fail to generalize in mul-
tiple scenarios and cannot be used for a better-performing
OCR model.
A dynamic word-based recognition was proposed by Shi
et al. [10]. Prior to this, almost all deep learning models
for word-level recognition focused on prediction from a
fixed vocabulary which as aforementioned has its own draw-
backs. Shi et al. proposed a novel neural network architecture
that does not require any pre-determined word vocabulary.
This architecture included three chief modules. These mod-
ules are known as feature extractors, sequence modeling
and transcription layer. The feature extraction module is a
CNN-based feature extractor and is loosely based on the well-
known VGG16 [3] architecture. Using a sequence model
implemented with a Bidirectional Long Short-Term Mem-
ory (LSTM) [23] based encoder–decoder architecture, the
dependency on various lengths of texts was omitted. The tran-
scription layer was responsible for predicting the sequence
labels for each timestep in the model using their maximum
probability scores and determining the respective label. This
model is popularly known as the Convolutional Recurrent
Neural Network (CRNN). The paper reports a Recognition
accuracy of 89.6% for English on the ICDAR 2013 [24]
dataset.
Baek et al. [25] showed that using attention [26] based
prediction layer instead of Connectionist Temporal Classi-
fication (CTC) [27] yielded 1.7% and 2.6% better accuracy
on their experiments and even before them, using attention
instead of CTC had become a popular method of achieving
state-of-the-art performance. ASTER [28] uses a rectification
network and a recognition network where the rectification
network transforms the images to deal with perspective and
curved texts without any human annotations. The recogni-
tion network uses attention LSTM to decode the prediction.
FAN [29] addresses the alignment issue between feature
areas and targets which they called the attention drift present
in attention-based encoder–decoder models. SCATTER [30]
improved the architecture proposed by Baek et al. [25] by
introducing a selective decoder that operates on both visual
features from the CNN layer and contextual features from
the Bidirectional Long Short-Term Memory (BiLSTM) layer
harnessing a two-step 1D attention mechanism. The method
recognized cursive texts and texts on complex backgrounds
better than Baek et al. [25].
To combat the shortcomings of Recurrent Neural Network
(RNN) based models, Yu et al. [31] proposed an end-to-
end framework known as the Semantic Reasoning Network.
The paper introduces a Global Semantic Reasoning Module
that mainly flows the semantic information in a “multi-way
parallel” manner. This sub-network can essentially learn the
words or characters simultaneously, making it more robust.
The introduction of this network eliminates the need for
a time-step-based sequential learner, which can at times
pass erroneous information resulting in the accumulation of
unexpected semantic information. To capture the semantic
information, the authors used Transformer blocks [26], where
the input is a feature extracted from the Parallel Visual Atten-
tion Module. The paper reports a recognition rate of 95.5%
on the ICDAR 2013 dataset [24] and also achieved a recog-
nition rate of 82.7% on ICDAR 2015 dataset [32].
Transformer-based architectures [33–37] have been widely
used recently for their better learning and generalization
capability. Aberdam et al. [38] proposed a Multimodal
123

Semi-Supervised contrastive learning-based method utiliz-
ing a visual representation learning algorithm for scene text
recognition. Self-supervised contrastive learning and masked
image modeling are utilized by [39] to learn discrimination
and generalization for text recognition. Chu et al. [40] pro-
posed an Iterative Language Modeling Module (IterLM) to
enhance the scene text recognition further. A novel single
visual model is introduced by Du et al. [41], where recog-
nition is done through simple linear prediction. Zheng et al.
[42] proposed and regularization-based method to reduce the
domain discrepancy between real and synthetic data.
2.1.2 Handwriting recognition
It is difficult to replicate the success of scene text and printed
word recognition for handwriting recognition. Handwritings
are incredibly more varied than printed words and even the
human recognition accuracy of handwriting can be lower
than printed word recognition. However, the deep learning
architectures used for printed word recognition are also used
in recognizing handwriting. Chammas et al. [43] used the
CRNN architecture and applied Temporal Dropout to image-
level and internal network representation during fine-tuning
and reported better results than the CRNN baseline on Span-
ish and German benchmark datasets. The CRNN baseline
has also been used in [44], where the authors proposed a
novel loss function called SoftCTC and demonstrated state-
of-the-art results on the handwriting recognition benchmark
datasets. Yousef et al. [45] and Maillette de Buy Wenniger
et al. [46] have also achieved good performance by uti-
lizing CTC loss-based architecture. Encode–decoder-based
sequence-to-sequence models such as AttentionHTR [47]
has also seen their use in handwriting recognition. Resource
scarcity is another issue that faces handwriting recognition
and thus Fogel et al. have proposed the ScrabbleGAN [48] to
generate varying-length handwritten texts. Many researchers
have also worked on the resource efficiency of handwriting
recognition [43, 45, 46, 49].
2.2 OCR in Bangla language
Although a significant amount of research has been done
on OCR, the improvements in word-level OCR for Bangla
have been slower in comparison with other languages. The
major hindrance to this dormant growth of performance is
due to the unavailability of any standard dataset for a word-
level OCR. We combat this problem by presenting a Bangla
word-level OCR corpus that can be used to train and evaluate
a word-level text recognition model. Character recognition
of the Bangla language is challenging as most characters are
cursive in nature, and there are no well-defined strokes [50].
In the pre-deep learning phase, researchers tried to improve
the performance of the character-level OCR by combining
123
K. Roy et al.
structural analysis, and algorithmic analysis and using the
template and feature matching techniques [51–59]. However,
since the advent of deep learning, feed-forward neural net-
works have become the preferred method in OCR studies
[60–66].
Due to the absence of an open-source printed word dataset,
research on Bangla OCR has mostly advanced in handwriting
recognition, specifically character-based classification mod-
els. On segmented character recognition tasks, many studies
[67–71] have used deep learning to achieve good perfor-
mance. Sharif et al. proposed a hybrid HOG-CNN model
[72] to classify Bangla compound characters and achieved
92.50% accuracy on the CMATERdb3.1.3.3 isolated char-
acters dataset. Hasan et al. [73] proposed a Deep CNN with
a Bidirectional Long Short-Term Memory model to predict
Bangla compound characters. They experimented with their
proposed model on the same CMATERdb3.1.3.3 dataset and
achieved a new milestone recognition accuracy of 98.50%.
Paul et al. [74] proposed a BiLSTM-CTC-based model
trained on 47,720 text lines. Their dataset contained 4,72,167
words, consisting of 28,67,659 characters, and was classi-
fied into 166 unique labels. The test set comprised 61,582
words containing 3,69,931 characters. During training, they
ignored the peephole connections of the BiLSTM to reduce
the CTC loss, and the weights of the model were initialized by
the Xavier Initializer [75], which helped the loss function to
converge fast. On the test set, they achieved character recog-
nition rate and word recognition rate of 99.32% and 96.65%
respectively which outperforms Google’s Tesseract 4.0 [21]
character and word recognition accuracy of 91.79, 76.31,
and Google Drive OCR 98.54, 92.86%, respectively. How-
ever, despite the reported great performance, the dataset is not
available for comparative study. Recently, two Bangla hand-
written word datasets BN-HTRd [76] and BanglaWriting
[77] have been published, but no baseline word recognition
model or results have been presented in the papers.
3 Proposed datasets
Neural network-based solutions require a dataset with suf-
ficient size and variation to perform well based on current
training methods. Although optical character recognition is a
well-defined problem with satisfactory models, the require-
ment of language-specific datasets at scale, in this case, a
Bangla word-level dataset, still remains. We are aware that
the process of deriving annotated data is a sufficiently lengthy
process and requires a large number of resources. To solve
our problem, we propose a synthetic word dataset. Despite
steps taken to produce synthetically generated Bangla char-
acter dataset [78], no existing large synthetic word dataset
prior to ours exists to our knowledge. In addition to our syn-
thetic training dataset, we have also created six unique test
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR
Fig. 1 An overview of our synthetic data generation pipeline
sets to evaluate the models trained on the proposed synthetic
dataset. In this section, we discuss the procedure of creating
our proposed training and testing sets.
3.1 Method of synthetic data generation
We generated word images with varying fonts and font sizes
from a list of Bangla words present in Unicode format. Fig-
ure 1 shows the flow of the synthetic data generation process.
In order to achieve good representation, we require a list of
words that cover the graphemes and grapheme combinations
that can frequently appear in the Bangla language within the
current context. In addition to the graphemes, special charac-
ters also need to be brought into consideration. The word list
mostly reflects these requirements. After the construction of
the word list, we need to determine multiple fonts that are an
adequate indicator of the font styles present in everyday use.
In the following subsections, we describe our word list
generation process and further outline our font selection cri-
terion.
3.1.1 Word list generation
In order to prepare our synthetic training set, we curated a
list of words for the data generation pipeline, and for this, we
needed a huge dictionary of unique words. Some published
datasets have a good number of Bangla textual data [79–82]
among which we have chosen the Dakshina Dataset [79] due
to its vast number of unique words and categories. We have
also used the SUMono dataset [80] which is later used to
generate one of our test protocols which is further discussed
in Sect. 3.2.
For our list, we extracted 691,664 unique words from
the Dakshina Dataset [79] after discarding words contain-
ing non-Bangla characters. In essence, the final wordlist
only contains characters from the Bengali Unicode Block
3 and punctuations. This dataset contains various articles
which consist of distinct categories such as Natural Science,
3 https://www.unicode.org/charts/PDF/U0980.pdf.
Fig. 2 Samples of the word “Rahasyaghana” written in Bangla that
translates to “Mysterious” in English represented in four of the twenty-
two fonts we used for data generation
Social Science, Computer and IT, Literature, Mass Media
and Blogs, History, Mythology, etc., including words that
are complex in terms of spelling, expansive in their length,
and consisting special characters as well.
3.1.2 Fonts selection
Font variation allows us to encode different font styles within
the word images. Additionally, this aids our model to be
robust to font changes. Through careful inspection of a
number of real-world Bangla documents and consulting var-
ious government organizations of Bangladesh, we limited
the selection to twenty-four different fonts including bold
variants of three of the said twenty-four fonts. Although
hundreds of different Bangla fonts exist, only the fonts fre-
quently used in computer-composed documents, including
the default fonts of the major operating systems, were con-
sidered. The fonts also needed to be free to use and readily
available online for download to be selected. These fonts
were thus assumed to capture a variety of the most popu-
lar fonts that tend to appear in printed documents. Samples
of four fonts out of the twenty-four are shown in Fig. 2. We
believe that this procedure of font selection for image gen-
eration would capture the necessary variation required in the
styling of the word images.
3.1.3 Image generation
A modified version of the TextRecognitionDataGenerator, 4 a
popular open-source synthetic data generation tool, was used
for generating the word images from our generated word list.
To recognize texts with varying font sizes, we generated the
dataset using eight different font sizes. These sizes were 8,
10, 12, 18, 24, 32, 48 and 64. Words images were generated
by adding small padding around the text.
3.2 Test sets creation
We created six test sets representing various sources and
types of printed documents to analyze the performance and
effectiveness of our models trained on a synthetically gener-
ated training dataset. The segmentation pipeline used over the
documents is an integral part of the pre-processing step used
for segmenting words to accumulate the hybrid and real test
4 https://github.com/Belval/TextRecognitionDataGenerator.
123

Fig. 3 Synthetically generated image samples from Protocol I of the
test sets
sets for evaluating the word-level OCR. For segmentation, we
used the Apurba Segmentation pipeline [12], which depends
on three significant steps to extract words from a document
effectively. To be clear, we used this framework, and it does
not constitute a contribution to this paper. Choosing this par-
ticular segmentation pipeline is not influential to our research
as words segmented by the segmentation pipeline were later
double-checked by a human before compiling the word-level
test datasets.
Instead of one big test set, the six different protocols were
created to rigorously test the OCR model and bring robust-
ness to the model. The following sections provide details on
why each protocol was created and the steps taken to create
these six test protocols in form of datasets, each serving a par-
ticular purpose. These sections also contain image samples
that provide intricate descriptions of different partitions of
the test data. Additionally, the annotations of the real-world
images were performed by the authors. In order to improve
the quality of the annotation, each annotation was later veri-
fied by more than one person.
3.2.1 Protocol I
Evaluation aspect In order to evaluate the performance of
our models on words it has not encountered in the training
set, we establish the first test protocol. This protocol consists
of a large amount of unique synthetic words to test the per-
formance of our model on a large scale which is difficult to
do with real data due to resource constraints of creating a
sizeable real dataset.
Creation method For this protocol, from a list of around
hundred and fifty thousand unique words taken from the
SUMono dataset [80], we curated a list of 75,138 unique
words which were not used to generate the prior defined 2
million training dataset or the validation dataset. We then
used the synthetic data generation pipeline to generate text
123
K. Roy et al.
Fig. 4 Synthetically generated image samples from Protocol II of the
test sets
images of the prior eight variant sizes along with five new
fonts as the test set. These fonts were not used to generate
the training set or the validation set. Samples of word images
for this protocol is shown in Fig. 3.
3.2.2 Protocol II
Evaluation aspect: This is a hybrid test set that was cre-
ated by producing two documents, each containing a hundred
words. These documents were printed, scanned, and finally
processed by the Apurba segmentation pipeline. This proto-
col has been created to evaluate text recognition performance
on clean, printed and scanned documents. Upon segmenta-
tion, the authors annotated each word sample in this protocol.
This is cleaner in content than many other real-world exam-
ples, as observed from the samples in Fig. 4.
Creation method For this dataset, we used two fonts, “Lohit”
and “Kalpurush”, with fixed sizes, where the “Lohit” font was
not used to generate the 2 million training set. The words were
also randomly taken and had overlapped.
3.2.3 Protocol III
Evaluation aspect For this variation, a diverse set of articles
were randomly selected from a collection of newspaper arti-
cles available online. This protocol was created to test the
robustness of the model on styled texts, such as bold, under-
line and italic texts.
Creation method The fonts used to generate this pro-
tocol were “SiyamRupali” and “Kalpurush” which were
also present in the training set and “Lalsalu”, “Lohit” and
“BenSen” which were not present in the training set. To intro-
duce further variations in the data, the fonts were in variable
sizes, and we replicated some of the samples by generat-
ing images in Bold, Italic, and Underlined font variations.
Figure 5 shows the variations introduced in this protocol.
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR
Fig. 5 Synthetically generated image samples from Protocol III of the
test sets
Fig. 6 Real-world image samples from Protocol IV of the test sets
We printed out the documents and scanned them again to
introduce real-world foreground and background noises [83].
Consequently, we used the Apurba segmentation pipeline
explained in the earlier section to convert the document into
single images of words. This eventually culminated in a total
of 3056 test samples for this protocol.
3.2.4 Protocol IV
Evaluation aspect In this test protocol, we introduced three
different types of documents. These documents include type-
set pages, printed books and old binarized books. This
protocol evaluates the recognition model’s performance on
documents from different domains.
Creation method The documents were scanned and fur-
ther segmented using the Apurba segmentation pipeline.
The resulting test protocol comprises 1105 real-world word
image samples. Among them, 456 word images are from
typeset documents, 305 images are extracted from printed
books and 344 words are retrieved from old binarized books.
In Fig. 6, we demonstrate segmented word samples from the
three categories of documents collected for this test set.
3.2.5 Protocol V
Evaluation aspect The fifth test protocol has six variations
of real-world noise across nine different documents. More-
Fig. 7 Real-world image samples from Protocol V of the test sets
Fig. 8 Real-world image samples of words with at least one conjunct
character from Protocol VI of the test sets
over, this protocol contains single-character images, unlike
the prior test sets. It is also known that single-character-
based word recognition provides a depth in the challenge
for word-level recognition models [10]. This protocol con-
sists of multiple real-world challenging cases with varying
noises, as shown in Fig. 7.
Creation method This test set is based on documents col-
lected from various government organizations in Bangladesh.
Upon segmenting the documents, we retrieved 1931 real-
world images. This test set has further complications, which
come with the noise that the documents carry after the scan-
ning process.
3.2.6 Protocol VI
Evaluation aspect In addition to having real-world noises,
all the words in this protocol consist of at least one conjunct
character. We created this protocol to test the performance
of our three grapheme representation methods on conjunct
characters. Samples are shown in Fig. 8.
Creation method Our final test protocol is based on vari-
ous books of stories and poems, religious books, scientific
textbooks, etc. After running multiple pages from different
documents through the segmentation pipeline, the extracted
123

word images that meet the criteria of this protocol were hand-
picked.
In Table 1, we provide a summary of the six test protocols.
4 Methodology
In this section, we discuss the methods we used for con-
ducting the experiments on our aforementioned datasets and
describe each component of our model in detail. Initially, we
provide an overview of our OCR pipeline and further elab-
orate our baseline architecture which is modified with the
grapheme representation strategies. Next, we describe our
proposed grapheme Representation methods which define
the final labels for the models and finally, elaborate on the
evaluation metrics we used to measure the performance of
our models.
4.1 Overview
For our OCR architecture, we mainly have four components—
Grapheme Extractor, Image Feature Extractor, Sequence
Modeling and Prediction. First, we used different proposed
grapheme representation methods for encoding the charac-
ters. Then we used a feature extractor to extract feature
vectors that represent the whole input image. After that, we
used sequence modeling to capture the contextual informa-
tion of the characters. Finally, we used alignment-free CTC
approach to decode the predictions.
4.2 OCR baseline
To establish our baseline for the OCR task, we leverage the
Scene Text Recognition model by Shi et. al. [10] which,
as aforementioned, is composed of a feature extractor com-
ponent followed by a sequence learner and CTC prediction
layer. Almost all the State of the Art word level OCR models
have a ResNet [4], VGG [3] or RCNN [84] based feature
extractor and some sort of sequence modeling such as BiL-
STM but differ in the prediction stage. Some have CTC-based
prediction and others have attention-based sequence pre-
diction and the attention-based models tend to have better
accuracy but are bigger in size [25]. We have also reviewed
many other OCR models [25, 28–31] which includes various
Feature Extractors and Attention or CTC-based prediction.
However, most of them did not mention the number of train-
able parameters or model size but the ResNet and the RCNN
have more parameters than the modified VGG extractor used
in CRNN and attention-based predictions are also slower than
the CTC. CRNN architecture is the most lightweight among
all the research we have reviewed and thus we have found it
the most practical one to be used in production and chose it as
our baseline. Also, as our research does not bring any novelty
123
K. Roy et al.
Fig. 9 Overview of the CRNN architecture
in model architecture and only contributes to the final pre-
diction layer, we have kept the model architecture the same
throughout the experiments.
The feature extractor of the CRNN is tasked to produce
sequential features from local regions of the input. Each local
region, also known as a receptive field, is a fixed area over
which convolutions, pooling, and non-linearity functions are
applied to make them translation invariant. These deep fea-
tures, shown in Fig. 9 as component b and c are robust to any
form of variables in each local region. Next, the sequence
of features extracted from the receptive field is further con-
joined for the sequence learning task. The objective of the
sequence learner (component e) is to capture semantic cues
using the contextual features (component d) extracted from
the input sample (component a).
The sequence learning component is comprised of recur-
rent layers which use the sequential features reshaped from
the convolutional receptive fields to predict a set of prob-
abilities for each local region using an encoding scheme
(component f ). The inclusion of a sequence learner is
significant for the prediction task because it is easier to pre-
dict characters sequentially, knowing the a priori characters,
rather than determining each character independently. We
employed a bi-directional LSTM as our sequence learner.
Decoding the prediction (component g) from an RNN-
based architecture is complex due to its redundant output
from a close timestamp. Also, the length of the ground truth
and prediction can be different, and their accurate align-
ment may not be achievable. For example, in nature, Bangla
is visually complex and very difficult to predict using an
alignment-based solution. Moreover, the alignment infor-
mation is very expensive to annotate. In Fig. 10 (left), the
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR

Table 1 Summary of the test protocols including their respective data type, font information, sources, and the samples count
Proto Type Font Sources Samples Tests

1 Synthetic Fonts not in training Online articles 75,138 Performance on large scale
data
2 Hybrid 1 out of 2 fonts in training Real documents 199 Performance on clean
scanned text
3 Hybrid 2 out of 5 fonts in training. Contains Online articles 3056 Performance on stylized
bold, italic, underline text
4 Real Unknown Typeset, printed, and 1105 Performance on images
binarized books from different domains
5 Real Unknown Government documents 1931 Performance on real-world
noise
6 Real Unknown Textbooks, story, poems 114 Performance on conjunct
and religious books, etc characters

Table 2 Diacritic forms of some of the vowels

Fig. 10 Example of a Bangla word using alignment based approach

(left) and alignment-free approach (right)

first row is the predicted character block, and the second

row shows a breakdown of characters in those blocks. This
demonstrates that a word can be broken down into different
levels of alignment.
We use an alignment-free approach for the learning pro-
cess by employing a Connectionist Temporal Classification
(CTC) layer. This would eliminate the labor of labeling the
positions of each character as we make predictions based
on each frame. Leveraging the CTC loss allows performing
alignment-free training as it optimizes by summing over all
in equation 1 would be computationally expensive, this is
possible probabilities for all the frames. However, a caveat for
calculated using the dynamic programming technique.
this algorithm is that it repeats the same character over mul-
tiple frames as the CNN feature for each character is spread
over a region. To combat this, the CTC algorithm introduces
4.3 Proposed grapheme representations
a special blank character to denote the start and end of a new
character. The objective of the CTC loss function is shown
Since our word recognition model performs a sequence clas-
in equation 1,
sification task, we need to map the graphemes in a word to
a finite set of labels. There are several challenges regard-
ing the representation of graphemes in Bangla. One of the
 
T
significant problems that we address in this paper is the rep-
p(Y |X ) = − ln( pt (xt |X )) (1) resentation of the graphemes, as this is a critical issue and
A S A,B t=1
not at all straightforward. Although the Bangla alphabet has
a finite set of vowels and consonants, the letters often form
where, p(Y |X ) represents the probability of the output, Y conjunct characters. In the Unicode standard, the consonant
given X , the input CNN feature. pt denotes the conditional conjuncts do not have any dedicated Unicode. The conjunct
probability of a single alignment xt at a single time-step given characters are formed by writing two or more vowels or con-
the CNN feature X and further combined over the set of sonants together. In Bangla, vowels are used in both their
all time steps. The final summation refers to marginalizing original and diacritic forms, as seen in Table 2 combined
over all sets of valid alignments. As computing the method with a consonant.

123

Table 3 A possible representation of a grapheme in multiple combina-
tions illuminating the difficulty of prediction
Consonants can also form clusters with one or two other
consonants connected by a ‘hoshonto’, an example of which
is can be seen in Table 3. In Table 3, we show an exam-
ple of a grapheme extracted from a word. In all traditional
sequential recognition models, Bangla Unicode text are used
as labels for words. In that way, the grapheme in Table 3 will
have three labels to represent it, shown on the upper right
cell. However, by observing Table 3 we realize that as we
are working with image-level data, it may be harder for the
model to understand that the grapheme on the left column
is composed of three different characters. While the model
can certainly learn it, treating graphemes distinctly based on
their visual presentation may be easier. Instead of using three
different labels to represent this grapheme, we can combine
the unicodes and use them as a single label for the grapheme,
shown on the lower right cell.
In our experiments, we break down words into three dif-
ferent representations. As mentioned above, the graphemes
may also involve conjuncts or modifiers, which are essen-
tially special structural variants of characters or combined
characters. The takeaway is that if they are visually different,
they too can be considered a separate class in the vocabulary.
This adds a new level of complexity because if we try to
add the possibility of all combinations of consonant clusters
along with the vowel diacritics, this will lead the number of
classes in vocabulary to be in the thousands. On the other
hand, if we separate the graphemes naively like in other lan-
guages, namely English and Chinese, where the characters
are split to their root form, we will end up with a small set
of characters which is a poor representation of the consonant
clusters and diacritics, and therefore the recognition accu-
racy may fall. So we try to reach an optimum trade-off. In
Table 4 Samples of our three proposed grapheme representation of Bangla words
123
K. Roy et al.
Table 4, we provide examples of representing characters in
different ways and further discuss the proposed representa-
tion approach.
In the first Naive Separation method, the graphemes are
separated at the root level and broken into their smallest form,
similar to the legacy implementation of character extraction
from unicode strings. Due to this, the modifiers are also sep-
arated, and consonant clusters are broken down. Thus, each
character is treated as a class. In Bangla, a consonant cluster
comprises two to three possible consonants by a ‘hoshonto’
character previously shown in Table 3. In the first row of Pos-
sible Representations in Table 3, we show that the grapheme
can be represented with multiple characters, however, joined
by the ‘hoshonto’ character in between.
In the second Vowel Diacritics Separation (VDS) method,
the vowel diacritics are separated, and the consonant clusters
are kept in their own form, which is further demonstrated in
Table 4.
Finally, the last All Diacritics Separation (ADS) method
for grapheme representations is based on the vowel diacritics
separation method. But now, atop vowel diacritics separa-
tion, we also separate a few Brahmi-Sanskrit diacritics: jofola
(grapheme created by ‘Hoshonto + Jo’), rofola (grapheme
created by ‘Hoshonto + Ro’) and ref (grapheme created by
‘Ro + Hoshonto’). Like the vowel diacritics, these diacrit-
ics are used along with almost every consonant and create
consonant clusters. But jofola, rofola and ref remain visually
identical in every consonant cluster. It can be shown that these
diacritics only appear along with the base consonant clusters
or characters and produce unique clusters. From Table 4, for
the first and the third example of the separation method, we
can find that due to the presence of the ‘Ro + Hoshonto’
grapheme, it is followed by newer graphemes such as ‘Ro
+ Hoshonto + Bo’ and ‘Ro + Hoshonto + To’ as shown in
Table 5. We can also show that some graphemes may also
coexist in other graphemes where they do not appear visually.
Since these diacritics themselves are not unique but may pro-
duce thousands of unique clusters, we separate them from the
clusters and only keep the unique clusters formed by com-
bining different consonants. This ensures that the consonant
clusters without any diacritics are kept in their original forms
while the clusters with vowel diacritics and/or the Brahmi-
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR
Table 5 Composition of multiple conjunct-based characters and rein-
force the examples in Table 4, showing how one grapheme may co-exist
in other graphemes in their Conjunct form
Sanskrit diacritics are broken down. Thus we get a list of
classes that is neither too low to represent the language nor
too complex.
Both our novel Vowel Diacritics Separation (VDS) and
All Diacritics Separation (ADS) do not break down com-
pound characters, even if it is a combination of more than
two consonants. The algorithm to extract labels in the VDS
and ADS method has been provided in “Appendix B”. Our
experiments showed that VDS and ADS-based methods per-
formed better than the legacy Naive method on consonant
conjuncts or consonant clusters. The experimental setup and
results regarding the grapheme representation methods are
described in the following section.
4.4 Evaluation metrics
We perform our evaluation on six different test sets with
varied attributes. For performance comparison, we use three
metrics. Firstly, we used the Word Recognition Rate (WRR)
where we calculate the percentage of words that exactly
match the ground truth (Levenshtein Distance of 0).
We also use the Sum of Normalized Edit Distance (NED)
of each word as another metric that was used in the ICDAR
competition [32] for method evaluation. Edit distance or Lev-
enshtein distance [85] as defined in Eq. 2 is a measure of
the number of steps required for an incorrectly predicted
sequence to transform to a correct one.
⎧
⎪
⎪max(i, j) if min(i, j) = 0
⎪
⎪ ⎧
⎨
⎨ Leva,b (i − 1, j) + 1
⎪ 5.1 Experimental setup
Leva,b (i, j) =
⎪
⎪min Leva,b (i, j − 1) + 1 otherwise.
⎪
⎪ ⎪
⎩
⎩ Leva,b (i − 1, j − 1) + 1(ai =b j )
Equation 2 denotes the Levenshtein distance between two
strings a and b, where i and j are the positions for the
two respective strings. The first part of this equation where
Leva,b (i, j) = max(i, j), if min(i, j) = 0 determines the
number of insertions or deletions required to turn an empty
string a into b or vice-versa. The later part of the equation
is a recursive definition where the three equations determine
the number of insertions, deletions and substitutions respec-
tively.
edit_distance( pr ed, gt)
NED = (3)
max(| pr ed|, |gt|)
Equation 3 is used to calculate the Normalized Edit Dis-
tance (NED), where pr ed and gt are the prediction and
ground truth strings respectively. | pr ed| and |gt| denotes
the length of those strings. The sum of the NED of all words
is used as a metric.
N
i=1 E Di
C R R = 1 − N (4)
i=1 max(| pr edi |, |gti |)
Our final metric is the Character Recognition Rate (CRR)
as defined in Eq. 4 which denotes the percentage of char-
acters correctly recognized. Edit Distance (ED) counts the
number of characters that were not correctly recognized by
aligning pr ed and gt. By dividing the total number of incor-
rectly predicted characters by the total number of characters
which is calculated by taking the summation of the length of
each longest string among the prediction pr ed and ground
truth string gt, we get the percentage of incorrectly recog-
nized characters in a word. Finally, by subtracting from 1,
we find the percentage of correctly recognized characters or
the Character Recognition Rate (CRR).
5 Experiments
In this section, we elaborate on the details of the experi-
ments we conducted and also present their results. Firstly,
we describe the experimental setup used for all of our exper-
iments. Next, we show the results of the variants of our model
on each of our aforementioned test sets while trained on the
synthetic training set. Then we train and test on real hand-
writing data to test our grapheme representation schemes on
different real datasets as well as their applicability in Optical
Handwriting Recognition (OHR). Finally, we provide some
qualitative analysis based on the results shown in the prior
subsections to determine the shortcomings of the data.
(2)
We used our synthetically generated dataset comprised of
2 million image samples for training our model. During
training, we applied the Gauss Noise, Pixel Dropout for salt-
and-pepper noise, Elastic Transform for a pencilic effect,
Random Brightness and Contrast, Motion Blur, etc. These
noises were introduced to mimic the realistic noises found
in real data and Perspective and Rotation transformations
were also added. The Albumentations [86] library was used
to perform the augmentations. The model is initialized with
123
 K. Roy et al.

Kaiming initialization [87] and initially trained with a learn- Table 6 Labels for all the variants of our models
ing rate of 5e−5 . Cosine Annealing with Warm Restarts [88] Configuration Classes Total params
learning rate scheduler was used where the first restart occurs
after 15 epochs and the minimum learning rate was set to CRNN-Naive 115 8.76M
1e−6 . We trained the models for 30 epochs and optimized CRNN-VDS 591 9.01M
them using the Adam Optimizer. We used 128 × 32 sized CRNN-ADS 262 8.84M
input, from which a 29-frame feature sequence has been Table 7 Performance of our models with two feature extractors using
generated. This is different from the input size of 100 × 32 our three proposed grapheme representation methods on the six test
mentioned in [10]. We have increased the width of the input protocols
image because Bangla words tend to be longer in length and a Protocol Model WRR (%) Total NED CRR (%)
bigger input size lets the model recognize the visual compli-
cations of the conjunct characters better. Initial testing with I CRNN-Naive 93.43 1029.3625 98.28
these longer images gave us better accuracy on all the tests CRNN-VDS 92.11 1340.5018 97.88
and thus we have shown results using this size in this paper. CRNN-ADS 93.08 1131.4137 98.15
To test the robustness of our grapheme representation II CRNN-Naive 81.91 18.9000 95.42
methods on real data, we chose two OHR datasets, BN-HTRd CRNN-VDS 85.93 14.8988 95.83
[76] and BanglaWriting [77]. Handwriting datasets were cho- CRNN-ADS 80.90 24.1583 93.88
sen due to the lack of proper Bangla Scene Text or Printed III CRNN-Naive 62.11 373.5325 90.34
word datasets for which we created our synthetic dataset in CRNN-VDS 64.01 372.2867 90.36
the first place. However, choosing these datasets also enabled CRNN-ADS 63.02 376.7820 90.15
us to test our grapheme representation methods in OHR. Dur- IV CRNN-Naive 43.71 269.0556 80.66
ing training on the real handwriting datasets, a learning rate of CRNN-VDS 48.69 239.9502 82.31
1e−3 was used and on the BanglaWriting dataset the models CRNN-ADS 42.81 272.4529 79.80
were trained for 60 epochs. All other configurations remained V CRNN-Naive 87.52 150.5297 97.17
the same as the models trained on the synthetic dataset. Ignor- CRNN-VDS 87.99 135.8027 97.10
ing some of the data lost to mislabeling, we were able to CRNN-ADS 86.85 148.4270 96.93
extract 108,061 words from BN-HTRd and 21,221 words VI CRNN-Naive 78.07 7.1286 94.25
from BanglaWriting. BanglaWriting provides both raw and CRNN-VDS 78.95 7.1302 94.25
noise-reduced versions and we have chosen raw images as
CRNN-ADS 81.58 6.1917 95.33
they are more natural and difficult to recognize. As no prior
train-test split was provided with either of the datasets, we Bold indicates that the model has the best result on the protocol among
the three models
randomly split each of the datasets and used 90% for training
and 5% each for validation and testing. We have performed the test results in the following subsections. Each model
both intra-dataset and inter-dataset testing. Intra-dataset test- configuration denotes one of the grapheme representation
ing is performed on the 5% test split but inter-dataset testing strategies paired with the CRNN architecture. For exam-
was performed on the entire dataset, which means the model ple, CRNN-Naive represents the CRNN architecture with the
trained on BN-HTRd was tested on the entire BanglaWriting Naive representation method. For reporting results for each
dataset and vice-versa. All our experiments were conducted model on all of our test sets, we use the WRR, Total NED,
on a machine with an NVIDIA V100 GPU using the Pytorch and CRR as indicators for evaluating the performance of the
framework [89] where we used a mini-batch size of 256. We models.
used the weights of the best validation epoch during testing
and provided results on the test sets mentioned above. 5.2.1 Results of training on the synthetic dataset and
testing on our six test protocols
5.2 Results
We report the results of the training experiments performed
In this section, we report the results of our models in two dif- on the synthetic dataset in Table 7.
ferent scenarios. Firstly, the models trained on the synthetic As previously mentioned, the purpose of Protocol I is to
dataset were tested on the six test protocols we described determine how the models perform on unseen synthetic data,
earlier. Then we trained on real handwriting datasets and that is, words that the model has not encountered during train-
reported intra and inter-dataset results on those real datasets. ing. For the first protocol, from Table 7, we can observe that
There are three different configurations that were trained the model with the Naive method outperformed our proposed
and tested, Table 6 lists all the different model configura- VDS and ADS grapheme representation methods, but from
tions explored. The labels in Table 6 are also used to present the table, it is clear that the margin is very low. Protocol I is

123
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR

also the only protocol that is a completely synthetic dataset Table 8 Training and testing schemes on handwriting dataset
while the rest of the protocols are either hybrid or real. Trained on Tested on Scheme
Protocol II mainly comprises synthetically generated
words that were later printed and scanned and thus it is con- BN-HTRd BN-HTRd INTRA1
sidered a hybrid dataset. In Table 7, for the second protocol, BanglaWriting (Raw) BanglaWriting (Raw) INTRA2
it can be observed that the CRNN-VDS model achieved the BN-HTRd BanglaWriting (Raw) INTER1
highest WRR and CRR of 85.93% and 95.83% respectively. BanglaWriting (Raw) BN-HTRd INTER2
In addition to that, due to the low Total NED, we consider
our architecture, CRNN-VDS as the best performing model
Table 9 Performance of our three proposed grapheme representation
for the second protocol. methods on real handwriting datasets
Protocol III is an extension of the previously reported pro-
Scheme Model WRR (%) Total NED CRR (%)
tocol. This test set consists of multiple varieties of fonts and
font sizes with bold, italic and underlined samples as well. INTRA1 CRNN-Naive 79.48 831.5276 89.33
All of these data were generated at the document level and CRNN-VDS 82.91 757.5683 90.27
later segmented for evaluation. From the results of Protocol CRNN-ADS 83.69 712.8907 90.85
III in Table 7, we can clearly observe that our CRNN-VDS INTRA2 CRNN-Naive 69.40 116.3189 87.47
model achieves the highest 64.01% WRR and 90.36% CRR CRNN-VDS 69.30 126.9041 86.39
among all the other models and the result is reassured by the CRNN-ADS 72.50 108.6174 88.27
low Total NED. INTER1 CRNN-Naive 60.58 3647.4147 82.02
The purpose of Protocol IV was to further evaluate the CRNN-VDS 61.96 3692.2762 81.75
model with real samples comprised of typeset documents,
CRNN-ADS 62.94 3516.3509 82.93
and printed and old binarized books. Upon segmentation of
INTER2 CRNN-Naive 50.42 23897.9807 76.64
the documents, we retrieved around eleven hundred word
CRNN-VDS 49.47 26075.8238 74.13
samples on which we perform the testing while the model
CRNN-ADS 52.14 23,325.9911 76.99
trained was on the synthetic 2 million dataset. From the
results of Protocol IV in Table 7, we can observe that our Bold indicates that the model has the best result on the scheme among
the three models
CRNN-VDS model displayed the best results.
Our fifth test set, Protocol V, mainly consists of documents
from various government organizations in Bangladesh. Addi- chosen BN-HTRd [76] and BanglaWriting [77] and demon-
tionally, it also consisted of certain variations of real-world strate our novelty on these datasets. As these are published
noises. From Table 7, we can observe that the CRNN-VDS and fairly recent datasets and due to the scarcity of a good
model attained the best WRR of 87.99% and consequently Bangla real or synthetic printed text or scene text dataset,
achieved a CRR of 97.10% which is slightly lower than the these datasets were chosen to test our novel representation
model with the naive representation method. Then again, the methods. We have divided our training and testing on these
lowest Total NED among the three models makes CRNN- datasets into four schemes as presented in Table 8. The first
VDS the better model. two schemes are intra-dataset training and testing and in the
Protocol VI consists of real-world data where each word last two schemes we tested the model trained on BN-HTRd on
comprises at least one conjunct character. It was specifi- the entire BanglaWriting dataset and vice-versa. The results
cally designed to test our grapheme representation schemes. of these experiments are shown in Table 9.
Here we can observe the consistency of the CRNN-VDS From the results of Table 9 we can observe the consistency
model which achieved a better WRR than the X1 model. The of the CRNN-ADS model with the ADS grapheme represen-
CRNN-ADS model with the ADS grapheme representation tation method on the real handwriting datasets which has
was the best-performing model with a WRR and CRR of surpassed the baseline naive extraction method in terms of
81.58% and 95.33% accompanied by the lowest Total NED. all the metrices.
This shows the CRNN-ADS model’s competitive perfor-
mance on real data alongside CRNN-VDS and its superiority
on words with conjunct characters.

5.2.2 Results of training and testing on real handwriting 6 Discussion

datasets
In this section, we take a deeper look into our grapheme
In order to establish that our proposed grapheme represen- representation methods and analyze them from different per-
tation methods also work with other real datasets, we have spectives.

123

Fig. 11 Percentage of conjuncts mispredicted by each of the models
on the printed words test protocols
6.1 Analysis on conjunct characters
The evaluation metrices used so far were the WRR which
represents absolute matches and CRR and Total NED which
highly depend on the calculation of edit distance. To calculate
the edit distance between two words, the words are broken
down into single characters. Bangla consonant clusters are
formed with two or more consonants with a ‘hoshonto’ char-
acter in between each of the consonants and thus, the edit
distance does not reflect how well these consonant clusters
are being predicted. For this, we have performed a separate
analysis to calculate the percentage of mispredicted conso-
nant clusters.
Although the CRNN-Naive model did better in terms of
our evaluation metrics on Protocol I as shown in Table 7,
from Fig. 11, we can see that the CRNN-Naive model had
a higher error rate than CRNN-VDS and CRNN-ADS. This
trend was consistent on all the protocols, as the novel charac-
ter representation methods performed better on the consonant
clusters. Protocols IV, V and VI consist of real data, and
protocol V is mainly comprised of simple words and single-
character words, there are not many complex consonant
clusters there to predict and thus the error rate is low for
all the models. However, on protocols IV and VI, we can
really see the difference. On protocol IV, the error rate of
the CRNN-Naive model is 30.18% whereas the CRNN-VDS
and CRNN-ADS models have considerably lower error rates
of 21.52% and 21.2% respectively. Protocol VI, specially
built for testing such cases, also demonstrates a visible dif-
ference between the novel representation methods and the
naive method. Here the CRNN-VDS and CRNN-ADS mod-
els have an error rate of 10.74% and 11.57% respectively, and
the CRNN-Naive model mispredicted 17.36% of the conso-
nant clusters.
Some word samples with conjunct characters have been
shown in Table 10. We can see that the CRNN-Naive model
has a high edit distance on the samples, while the CRNN-
VDS and CRNN-ADS models have predicted the words
perfectly or with a low edit distance.
Figure 12 also exhibits a similar trend to Fig. 11 as here
too the CRNN-VDS, and CRNN-ADS models with the novel
123
K. Roy et al.
Fig. 12 Percentage of conjuncts mispredicted by each model on the
handwriting training and testing schemes
grapheme representation methods are performing better than
the CRNN-Naive model with the naive method. In all the
schemes, there is a noticeable difference. In the first scheme,
there is almost a 10 percent difference between the best-
performing CRNN-VDS model and the CRNN-Naive model
with the naive representation. On the last three schemes, the
margins were even greater.
Finally, by observing the performance of the models on
Bangla consonant clusters in Figs. 11 and 12 as well as by
the results of the models in Sects. 5.2.1 and 5.2.2, we can
conclude that the VDS grapheme representation is fitting for
the Bangla OCR problem. The model has achieved better
scores on the test protocols as well as the real handwriting
datasets and also recognized the consonant clusters better.
ADS grapheme representation also showed better perfor-
mance than the naive method on the real handwriting datasets
and did better at recognizing consonant clusters. Also, the
high WRR and CRR achieved on our test protocols indicate
that the models trained on the synthetic dataset were perform-
ing well on the hybrid and real test sets. As the CRNN-VDS
model with VDS grapheme representation performed better
than the rest, we chose it to further test its generalizabil-
ity over three disparate real-world test sets reported in the
“Appendix A” and found that the model was performing
exceptionally well on an unseen real-world printed dataset.
6.2 Analysis on short and long words
In Bangla text, as conjunct characters can be broken down
into two or more characters, the length of Bangla words
tends to be longer than in other scripts, such as Latin. This
is why it is important for a Bangla OCR system to be able
to predict longer words effectively. However, the prediction
sequence length is usually a fixed length (29 in our case) and
thus, it is difficult to predict a long sequence using the naive
method. This is where our CRNN-VDS and CRNN-ADS
models shine. We have conducted experiments on various
word lengths: 1–4, 5–8, 9–12 and 13+ to test the models’
performance at different text lengths. As different represen-
tation methods can provide different numbers of labels for
the same text, we have considered the number of Unicode
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR

Table 10 Example words with

at least one conjunct where
CRNN-VDS and CRNN-ADS
models outperform the
CRNN-Naive model

Table 11 WRR of the models at

Protocol Model # samples Length 1–4 Length 5–8 Length 9–12 Length 13 and above
different word lengths (in terms
of the number of Unicode I CRNN-Naive 96.71 95.69 92.54 84.61
characters) on the six test
protocols CRNN-VDS 96.92 94.58 90.45 83.87
CRNN-ADS 97.21 95.54 91.66 84.54
Samples 4224 37,255 25,048 8610
II CRNN-Naive 71.25 87.50 100.00 0.00
CRNN-VDS 78.75 89.42 100.00 0.00
CRNN-ADS 67.50 88.46 100.00 0.00
Samples 80 104 15 0
III CRNN-Naive 37.43 61.49 71.55 63.46
CRNN-VDS 37.15 63.78 72.80 68.44
CRNN-ADS 35.75 63.29 71.23 67.11
Samples 358 1441 956 301
IV CRNN-Naive 30.68 47.89 55.93 52.27
CRNN-VDS 37.17 52.48 57.63 63.64
CRNN-ADS 28.61 46.61 55.93 61.36
Samples 339 545 177 44
V CRNN-Naive 80.16 93.17 89.68 100.00
CRNN-VDS 82.52 92.67 88.79 97.50
CRNN-ADS 79.11 93.30 88.20 100.00
Samples 761 791 339 40
VI CRNN-Naive 75.00 74.67 96.55 33.33
CRNN-VDS 75.00 76.00 93.10 50.00
CRNN-ADS 100.00 76.00 100.00 50.00
Samples 4 75 29 6
Bold indicates that the model has the best result on the protocol among the three models

characters in a text to be the word length to keep the cal-
culation consistent across all the extraction methods. From
Table 11, we can observe that our novel methods perform
better than the naive method on both short and long words
on all real and hybrid test protocols.
6.3 Limitations and error cases
The novel representation methods do not perform exception-
ally well on the large-scale synthetic testing protocol I. The
synthetic testing protocol I data are visually similar to the syn-
thetic training dataset. Some of the samples of the training
samples with synthetically added noise are shown in Fig. 13.
Due to the visual similarity, it is the easiest protocol for our
recognition models and in this protocol, the margin is very
slim between the models.
The CRNN-VDS method performs the best on real-world
samples when it has been trained on a large enough dataset.
However, as the number of classes for the CRNN-VDS is
more than five times higher than the baseline CRNN-VDS
model, it needs a lot of data to achieve good accuracy and dis-
tinguish each class properly. This has been made clear from
the results on the handwriting datasets, where the training
samples were low and the CRNN-ADS model overperformed

123

Fig. 13 Samples of synthetic words with synthetic noise used to train
the models
the CRNN-VDS model on all the testing schemes. The high
number of classes also increases the total trainable param-
eters of the model as shown in Table 6 which increases
model complexity and inference time. However, the increase
in complexity and parameter size is not too large and thus
CRNN-VDS model remains a viable solution in the avail-
ability of large datasets. When the training set is smaller, our
novel CRNN-ADS model is the go-to architecture which gets
a better result on real-world handwriting datasets as shown
in Table 9.
One weakness of the CRNN-VDS and the CRNN-ADS
models is falsely assuming a single character as a conso-
nant cluster occasionally. This type of misprediction greatly
affects the recognition metrics, such as the NED. As the
edit distance is calculated by aligning the prediction and
ground truth strings and counting the number of Unicode
characters not correctly recognized, mispredicting a single
character as a consonant cluster significantly increases the
edit distance as a single consonant cluster can have three or
more characters in it. Table 12 shows some sample cases
where CRNN-Naive predicted simple words perfectly. Yet,
CRNN-VDS or CRNN-ADS mispredicted a single character
as a consonant cluster and greatly increased the edit dis-
tance. Despite having this error case, our novel CRNN-VDS
and CRNN-ADS models are performing better on real-world
printed word testing protocols and handwriting datasets.
Table 12 Examples where
CRNN-VDS or CRNN-ADS
fails with a high edit distance
due to conjunct prediction
instead of single character
123
K. Roy et al.
7 Conclusion
We investigated the challenges in Bangla word-level OCR
constituted by conjunct characters present in Bangla script
and proposed two novel grapheme representation strategies,
Vowel Diacritics separation (VDS) and All Diacritics separa-
tion (ADS). Due to the scarcity of word image data in Bangla
OCR, we created a synthetic corpus containing 2 million
samples with Bangla words having multiple variations in the
design and size of the fonts, covering a range of domains. To
evaluate the models trained on our synthetic corpus, we also
proposed six different testing sets, each designed to deter-
mine a particular observation on a unique problem of Bangla
OCR.
Both our novel separation methods have shown supe-
rior conjunct recognition accuracy on all the test sets and
the external datasets. On external handwriting datasets, the
best-performing novel method has surpassed the conjunct
recognition rate of the baseline by a margin of 10% or more.
Overall OCR results on the testing protocols demonstrate that
our novel VDS separation method outperformed the baseline
naive separation on every hybrid and real test dataset. Results
on external handwriting datasets demonstrated the superior-
ity of the ADS separation method. However, results on the
test protocols III and IV have demonstrated that our model
struggles with noisy italic, bold, and underlined data present
in protocol III and real typeset and binarized data present
in protocol IV compared to other protocols. By establish-
ing our research as a baseline, future work can be done by
utilizing our released synthetic datasets and test protocols
to improve upon the shortcomings of our method and the
metrics reported in this paper can be used for comparison in
future research.
Acknowledgements This research was supported in part by the
Enhancement of Bangla Language in ICT through Research & Devel-
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR
opment (EBLICT) Project, under the Ministry of ICT, the Government
of Bangladesh.
Declarations
Conflict of interest The authors have no competing interests to declare
that are relevant to the content of this article.
Appendix A: Results on an external testing
partition
To evaluate the generalizability of the models in accordance
with the training data, in this section, we conducted a testing
experiment on extrinsic data. The data we used is primarily
subjected to be used for an OCR project by the Government
of Bangladesh. Due to the confidentiality of the assignment,
we cannot publish these external testing sets. However, we
report the results in Table 13 to stress the performance of
the models trained on completely synthetic data. We tested
on entirely unseen data that were not seen by the model in
training or validation and were accumulated from domains
unknown to the model.
The word-level data used for this testing has been mainly
comprised of three types of documents. Firstly, the Computer
Composed test set consists of data written using computers.
Examples of Computer Composed documents can be govern-
ment notices, letters, etc. Next, we use the Letterpress test set
containing data published from press-based printing, where
some common examples are books and posters. The third
testing set is the Typewriter dataset which consists of data
composed using a typewriter device mostly used for legal
documents in Bangladesh.
In Fig. 14, we show examples of the data we used for
testing. We can observe the natural noise that exists in the
samples, making it more challenging for the model to predict
correctly. This is mainly because natural noises are difficult
to recreate synthetically. In Table 13, we report the results
of the evaluation. For this testing, we pick our CRNN-VDS
model with VDS Character Representation mainly due to its
competitive performance in other reported test sets with real
data (Protocol II-V).
From Table 13, we can observe that the CRNN-VDS
model has achieved a WRR of 79.03% on the computer-
Table 13 Performance of the X2 model when tested on external testing
sets
Test set Model WRR (%) CRR (%)
Computer Composed CRNN-VDS 79.03 93.04
Letterpress CRNN-VDS 57.86 83.61
Typewriter CRNN-VDS 28.05 70.10
Fig. 14 Samples of words from Computer Composed, Letterpress and
Typewriter data used for extrinsic testing
composed dataset with around five hundred thousand data
which we believe is a significant accomplishment consider-
ing that the model has not seen data of this domain and being
trained on completely synthetic data. The success in recogni-
tion is due to our synthetic data generation process. We have
meticulously selected almost all the open-source and popu-
lar Bangla fonts and generated images of different lengths.
During the training process, we have also added noises that
reflect some of the real-world noise such as blur, salt-and-
pepper, etc. Figure 13 shows some of the input images with
added noise used during training. These images capture the
visual diversity present in real-world computer-composed
documents and thus are able to train the model well enough
to obtain high accuracy on real-world unseen test sets.
The CRNN-VDS model has also shown inadequate per-
formances on the Letterpress and Typewriter testing sets,
achieving a WRR of 57.86% and 28.05% respectively. The
sub-standard performance is mainly due to the introduction
of unique noises in these datasets that are, in some cases,
even different in terms of the color schemes of the back-
ground. Also, the synthetic data the model was trained on
mostly mimicked computer-composed data and not letter-
press or typewriter data as the texts from those domains
usually have different fonts, paper textures, and noises unique
to those domains. Finally, due to this evaluation, we can make
estimations of the performance of the synthetically trained
CRNN-VDS model in production circumstances.
Appendix B: Algorithm for the extraction
methods
In traditional sequential models, Unicode texts are broken
down into single characters and each character is used as a
label. Our Novel VDS and ADS methods aim to keep the
consonant clusters unbroken by keeping all the participating
consonants of a cluster together as its label instead of using
multiple labels to represent it. VDS and ADS are rule-based
methods that are used to extract labels from a text for training
or testing purposes for the CRNN-VDS and CRNN-ADS
architectures. Algorithms 1–2 represents the VDS method
and 3–4 represents the ADS method.
123
 K. Roy et al.

Algorithm 1 VDS Grapheme Extraction (Part 1)

Require: Necessary variables:
1: wor d  unicode text from which grapheme labels will be extracted.
2: consonants  list of consonant characters
3: f or msCluster  a hash table where the keys are consonant characters and values are lists of consonants that form a consonant cluster with
that character.
4: f or msT ri ppleCluster  a hash table where the keys are consonant clusters of two consonants and values are lists of consonants that form a
consonant cluster of three consonants with that cluster.
5: hoshonto, jo, jo f ola  variables that hold the Unicode for these
6: char s ⇐ []  stores the extracted labels of wor d
7: i ⇐ 0  loop counter
8: ad just ⇐ 0
9: while i < length(wor d) do
10: if i + 1 < length(wor d) & wor d[i + 1] == hoshonto then
11: if i + 2 < length(wor d) & wor d[i + 2] == jo then
12: if wor d[i] in consonants then
13: char s.append(wor d[i − ad just : i + 3])
14: else
15: char s.append(wor d[i − ad just : i + 1])
16: char s.append( jo f ola)
17: end if
18: ad just ⇐ 0
19: i ⇐i +3
20: else if i + 2 < length(wor d) & ad just = 0 & wor d[i − ad just : i + 1] in f or msT ri pleCluster
21: & wor d[i + 2] in f or msT ri pleCluster [wor d[i − ad just : i + 1]] then
22: if i + 3 < length(wor d) & wor d[i + 3] == hoshonto then
23: ad just ⇐ ad just + 2
24: i ⇐i +2
25: else
26: char s.append(wor d[i − ad just : i + 3])
27: ad just ⇐ 0
28: i ⇐i +3
29: end if
30: else if i + 2 < length(wor d) & ad just == 0 & wor d[i] in f or msCluster
31: & wor d[i + 2] in f or msCluster [wor d[i]] then
32: if i + 3 < length(wor d) & wor d[i + 3] == hoshonto then
33: ad just ⇐ ad just + 2
34: i ⇐i +2
35: else
36: char s.append(wor d[i − ad just : i + 3])
37: ad just ⇐ 0
38: i ⇐i +3
39: end if
40: else
41: char s.append(wor d[i − ad just : i + 1])
42: char s.append(hoshonto)
43: ad just ⇐ 0
44: i ⇐i +2
45: end if

Algorithm 2 VDS Grapheme Extraction (Part 2)

46: else
47: char s.append(wor d[i : i + 1])
48: i ⇐i +1
49: end if
50: end while

123
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR

Algorithm 3 ADS Grapheme Extraction (Part 1)

Require: Necessary variables:
wor d  unicode text from which grapheme labels will be extracted. consonants  list of consonant characters
f or msCluster  a hash table where the keys are consonant characters and values are lists of consonants that form a consonant cluster with
that character.
3: f or msT ri ppleCluster  a hash table where the keys are consonant clusters of two consonants and values are lists of consonants that form a
consonant cluster of three consonants with that cluster.
hoshonto, jo, r o, r e f , jo f ola, r o f ola  variables that hold the Unicode for these
char s ⇐ []
6: i ⇐ 0
ad just ⇐ 0
while i < length(wor d) do
9: if i + 1 < length(wor d) & wor d[i + 1] == hoshonto then
if wor d[i] is r o then
char s.append(r e f )
12: ad just ⇐ 0
i ⇐i +2
else if i + 2 < length(wor d) & wor d[i + 2] == jo then
15: char s.append(wor d[i − ad just : i + 1])
char s.append( jo f ola)
ad just ⇐ 0
18: i ⇐i +3
else if i + 2 < length(wor d) & wor d[i + 2] == r o then
char s.append(wor d[i − ad just : i + 1])
21: char s.append(r o f ola)
if i + 3 < length(wor d) & wor d[i + 3] == hoshonto & i + 4 < length(wor d)
& wor d[i + 4] == jo then
24: char s.append( jo f ola)
i ⇐i +5
else
27: i ⇐i +3
end if
ad just ⇐ 0
30: else if i + 2 < length(wor d) & ad just = 0 & wor d[i − ad just : i + 1] in f or msT ri pleCluster
& wor d[i + 2] in f or msT ri pleCluster [wor d[i − ad just : i + 1]] then
if i + 3 < length(wor d) & wor d[i + 3] == hoshonto then
33: ad just ⇐ ad just + 2
i ⇐i +2

123
 K. Roy et al.

Algorithm 4 ADS Grapheme Extraction (Part 2)

else
36: char s.append(wor d[i − ad just : i + 3])
ad just ⇐ 0
i ⇐i +3
39: end if
else if i + 2 < length(wor d) & ad just == 0 & wor d[i] in f or msCluster
& wor d[i + 2] in f or msCluster [wor d[i]] then
42: if i + 3 < length(wor d) & wor d[i + 3] == hoshonto then
ad just ⇐ ad just + 2
i ⇐i +2
45: else
char s.append(wor d[i − ad just : i + 3])
ad just ⇐ 0
48: i ⇐i +3
end if
else
51: char s.append(wor d[i − ad just : i + 1])
char s.append(hoshonto)
ad just ⇐ 0
54: i ⇐i +2
end if
else
57: char s.append(wor d[i : i + 1])
i ⇐i +1
end if
60: end while

References Proceedings of the AAAI Conference on Artificial Intelligence,

1. Rabiner, L., Juang, B.: An introduction to hidden Markov models.
IEEE ASSP Mag. 3(1), 4–16 (1986)
2. Congdon, P.: Bayesian Statistical Modelling. Wiley Series in
Probability and Statistics, Wiley (2006). https://doi.org/10.1002/
9780470035948
3. Simonyan, K., Zisserman, A.: Very deep convolutional networks
for large-scale image recognition. arXiv preprint arXiv:1409.1556
(2014)
4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image
recognition. In: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pp. 770–778 (2016)
5. Almazán, J., Gordo, A., Fornés, A., Valveny, E.: Word spotting and
recognition with embedded attributes. IEEE Trans. Pattern Anal.
Mach. Intell. 36(12), 2552–2566 (2014)
6. Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Synthetic
data and artificial neural networks for natural scene text recogni-
tion. arXiv preprint arXiv:1406.2227 (2014)
7. Feng, X., Yao, H., Zhang, S.: Focal CTC loss for Chinese optical
character recognition on unbalanced datasets. Complexity 2019
(2019)
8. Kang, L., Riba, P., Rusiñol, M., Fornés, A., Villegas, M.: Pay
attention to what you read: non-recurrent handwritten text-line
recognition. arXiv preprint arXiv:2005.13044 (2020)
9. Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Reading
text in the wild with convolutional neural networks. Int. J. Com-
put. Vis. 116(1), 1–20 (2016). https://doi.org/10.1007/s11263-
015-0823-z
10. Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network
for image-based sequence recognition and its application to scene
text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11),
2298–2304 (2016)
11. Hu, W., Cai, X., Hou, J., Yi, S., Lin, Z.: GTC: Guided training
of CTC towards efficient and accurate scene text recognition. In:
vol. 34, pp. 11005–11012 (2020)
12. Rifat, M.J.R., Banik, M., Hasan, N., Nahar, J., Rahman, F.: A novel
machine annotated balanced Bangla OCR corpus. In: Singh, S.K.,
Roy, P., Raman, B., Nagabhushan, P. (eds.) Comput. Vis. Image
Process., pp. 149–160. Springer, Singapore (2021)
13. Anthimopoulos, M., Gatos, B., Pratikakis, I.: Detection of artificial
and scene text in images and video frames. Pattern Anal. Appl.
16(3), 431–446 (2013)
14. Chen, H., Tsai, S.S., Schroth, G., Chen, D.M., Grzeszczuk, R.,
Girod, B.: Robust text detection in natural images with edge-
enhanced maximally stable extremal regions. In: 2011 18th IEEE
International Conference on Image Processing, pp. 2609–2612
(2011). IEEE
15. Epshtein, B., Ofek, E., Wexler, Y.: Detecting text in natural scenes
with stroke width transform. In: 2010 IEEE Computer Society
Conference on Computer Vision and Pattern Recognition, pp.
2963–2970 (2010). IEEE
16. Huang, W., Qiao, Y., Tang, X.: Robust scene text detection with
convolution neural network induced MSER trees. In: European
Conference on Computer Vision, pp. 497–511 (2014). Springer
17. Alsharif, O., Pineau, J.: End-to-end text recognition with hybrid
hmm maxout models. arXiv preprint arXiv:1310.1811 (2013)
18. Gordo, A.: Supervised mid-level features for word image repre-
sentation. In: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pp. 2956–2964 (2015)
19. Neumann, L., Matas, J.: A method for text localization and recog-
nition in real-world images. In: Asian Conference on Computer
Vision, pp. 770–783 (2010). Springer
20. Mishra, A., Alahari, K., Jawahar, C.: Image retrieval using tex-
tual cues. In: Proceedings of the IEEE International Conference on
Computer Vision, pp. 3040–3047 (2013)
21. Smith, R.: An overview of the Tesseract OCR engine. In: Ninth
International Conference on Document Analysis and Recognition
(ICDAR 2007), vol. 2, pp. 629–633 (2007). IEEE

123
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR
22. Shi, B., Wang, X., Lyu, P., Yao, C., Bai, X.: Robust scene text
recognition with automatic rectification. In: Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition,
pp. 4168–4176 (2016)
23. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural
Comput. 9(8), 1735–1780 (1997)
24. Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., i Bigorda, L.G.,
Mestre, S.R., Mas, J., Mota, D.F., Almazan, J.A., De Las Heras,
L.P.: ICDAR 2013 robust reading competition. In: 2013 12th Inter-
national Conference on Document Analysis and Recognition, pp.
1484–1493 (2013). IEEE
25. Baek, J., Kim, G., Lee, J., Park, S., Han, D., Yun, S., Oh, S.J., Lee,
H.: What is wrong with scene text recognition model comparisons?
Dataset and model analysis. In: Proceedings of the IEEE/CVF
International Conference on Computer Vision (ICCV) (2019)
26. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,
Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need.
arXiv preprint arXiv:1706.03762 (2017)
27. Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connec-
tionist temporal classification: labelling unsegmented sequence
data with recurrent neural networks. In: Proceedings of the 23rd
International Conference on Machine Learning, pp. 369–376
(2006)
28. Shi, B., Yang, M., Wang, X., Lyu, P., Yao, C., Bai, X.: ASTER: an
attentional scene text recognizer with flexible rectification. IEEE
Trans. Pattern Anal. Mach. Intell. 41(9), 2035–2048 (2019). https://
doi.org/10.1109/TPAMI.2018.2848939
29. Cheng, Z., Bai, F., Xu, Y., Zheng, G., Pu, S., Zhou, S.: Focusing
attention: towards accurate text recognition in natural images. In:
2017 IEEE International Conference on Computer Vision (ICCV),
pp. 5086–5094 (2017). https://doi.org/10.1109/ICCV.2017.543
30. Litman, R., Anschel, O., Tsiper, S., Litman, R., Mazor, S.,
Manmatha, R.: Scatter: selective context attentional scene text rec-
ognizer. In: 2020 IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), pp. 11959–11969 (2020). https://doi.
org/10.1109/CVPR42600.2020.01198
31. Yu, D., Li, X., Zhang, C., Liu, T., Han, J., Liu, J., Ding, E.: Towards
accurate scene text recognition with semantic reasoning networks.
In: Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp. 12113–12122 (2020)
32. Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bag-
danov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar,
V.R., Lu, S., et al.: ICDAR 2015 competition on robust reading.
In: 2015 13th International Conference on Document Analysis and
Recognition (ICDAR), pp. 1156–1160 (2015). IEEE
33. Feng, X., Yao, H., Qi, Y., Zhang, J., Zhang, S.: Scene text recog-
nition via transformer. arXiv preprint arXiv:2003.08077 (2020)
34. Atienza, R.: Vision transformer for fast and efficient scene text
recognition. In: Document Analysis and Recognition–ICDAR
2021: 16th International Conference, Lausanne, Switzerland,
September 5–10, 2021, Proceedings, Part I, vol. 16, pp. 319–334
(2021). Springer
35. Wu, J., Peng, Y., Zhang, S., Qi, W., Zhang, J.: Masked vision-
language transformers for scene text recognition. arXiv preprint
arXiv:2211.04785 (2022)
36. Wang, P., Da, C., Yao, C.: Multi-granularity prediction for scene
text recognition. In: Computer Vision—ECCV 2022: 17th Euro-
pean Conference, Tel Aviv, Israel, October 23–27, 2022, Proceed-
ings, Part XXVIII, pp. 339–355 (2022). Springer
37. Xie, X., Fu, L., Zhang, Z., Wang, Z., Bai, X.: Toward understand-
ing wordart: corner-guided transformer for scene text recognition.
In: Computer Vision–ECCV 2022: 17th European Conference, Tel
Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII, pp.
303–321 (2022). Springer
38. Aberdam, A., Ganz, R., Mazor, S., Litman, R.: Multimodal
semi-supervised learning for text recognition. arXiv preprint
arXiv:2205.03873 (2022)
39. Yang, M., Liao, M., Lu, P., Wang, J., Zhu, S., Luo, H., Tian, Q.,
Bai, X.: Reading and writing: discriminative and generative mod-
eling for self-supervised text recognition. In: Proceedings of the
30th ACM International Conference on Multimedia, pp. 4214–
4223 (2022)
40. Chu, X., Wang, Y.: IterVM: iterative vision modeling module for
scene text recognition. In: 2022 26th International Conference on
Pattern Recognition (ICPR), pp. 1393–1399 (2022). IEEE
41. Du, Y., Chen, Z., Jia, C., Yin, X., Zheng, T., Li, C., Du, Y., Jiang,
Y.-G.: Svtr: scene text recognition with a single visual model. arXiv
preprint arXiv:2205.00159 (2022)
42. Zheng, C., Li, H., Rhee, S.-M., Han, S., Han, J.-J., Wang, P.: Push-
ing the performance limit of scene text recognizer without human
annotation. In: Proceedings of the IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition, pp. 14116–14125 (2022)
43. Chammas, E., Mokbel, C., Likforman-Sulem, L.: Handwriting
recognition of historical documents with few labeled data. In: 2018
13th IAPR International Workshop on Document Analysis Systems
(DAS), pp. 43–48 (2018). IEEE
44. Kišš, M., Hradiš, M., Beneš, K., Buchal, P., Kula, M.: SoftCTC—
Semi-Supervised Learning for Text Recognition using Soft Pseudo-
labels. arXiv (2022). arXiv:2212.02135
45. Yousef, M., Hussain, K.F., Mohammed, U.S.: Accurate, data-
efficient, unconstrained text recognition with convolutional neural
networks. Pattern Recogn. 108, 107482 (2020). https://doi.org/10.
1016/j.patcog.2020.107482
46. Maillette de Buy Wenniger, G., Schomaker, L., Way, A.: No
padding please: efficient neural handwriting recognition. In: 2019
International Conference on Document Analysis and Recognition
(ICDAR), pp. 355–362 (2019). https://doi.org/10.1109/ICDAR.
2019.00064
47. Kass, D., Vats, E.: AttentionHTR: handwritten text recognition
based on attention encoder–decoder networks. In: Uchida, S., Bar-
ney, E., Eglin, V. (eds.) Document Analysis Systems, pp. 507–522.
Springer, Cham (2022)
48. Fogel, S., Averbuch-Elor, H., Cohen, S., Mazor, S., Litman, R.:
Scrabblegan: Semi-supervised varying length handwritten text gen-
eration. In: 2020 IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), pp. 4323–4332 (2020). https://doi.
org/10.1109/CVPR42600.2020.00438
49. Souibgui, M.A., Fornés, A., Kessentini, Y., Megyesi, B.: Few shots
are all you need: a progressive learning approach for low resource
handwritten text recognition. Pattern Recogn. Lett. 160, 43–49
(2022). https://doi.org/10.1016/j.patrec.2022.06.003
50. Rahman, A., Kaykobad, M.: A complete Bengali OCR: a novel
hybrid approach to handwritten Bengali character recognition. J.
Comput. Inf. Technol. 6(4), 395–413 (1998)
51. Pal, U., Chaudhuri, B.B.: OCR in Bangla: an Indo-Bangladeshi lan-
guage. In: Proceedings of the 12th IAPR International Conference
on Pattern Recognition, Vol. 3—Conference C: Signal Processing
(Cat. No.94CH3440-5), vol. 2, pp. 269–2732 (1994). https://doi.
org/10.1109/ICPR.1994.576917
52. Sattar, M., Rahman, S.: An experimental investigation on Bangla
character recognition system. Bangladesh Comput. Soc. J. 4(1),
1–4 (1989)
53. Rahman, A.F.R., Fairhurst, M.: Multi-prototype classification:
improved modelling of the variability of handwritten data using
statistical clustering algorithms. Electron. Lett. 33(14), 1208–1210
(1997)
54. Pal, U.: On the development of an optical character recognition
(OCR) system for printed Bangla script. PhD thesis, Indian Statis-
tical Institute, Calcutta (1997)
123

55. Chaudhuri, B., Pal, U.: A complete printed Bangla OCR system.
Pattern Recogn. 31(5), 531–549 (1998)
56. Rahman, A.F.R., Fairhurst, M.C.: A new hybrid approach in com-
bining multiple experts to recognise handwritten numerals. Pattern
Recogn. Lett. 18(8), 781–790 (1997)
57. Rahman, A.F.R., Rahman, R., Fairhurst, M.C.: Recognition of
handwritten Bengali characters: a novel multistage approach. Pat-
tern Recogn. 35(5), 997–1006 (2002)
58. Mahmud, J.U., Raihan, M.F., Rahman, C.M.: A complete OCR
system for continuous Bengali characters. In: TENCON 2003. Con-
ference on Convergent Technologies for Asia-Pacific Region, vol.
4, pp. 1372–1376 (2003). IEEE
59. Kamruzzaman, J., Aziz, S.: Improved machine recognition for
Bangla characters. In: International Conference on Electrical and
Computer Engineering 2004, pp. 557–560 (2004). ICECE 2004
Conference Secretariat, Bangladesh of Engineering and Technol-
ogy
60. Alam, M.M., Kashem, M.A.: A complete Bangla OCR system for
printed characters. JCIT 1(01), 30–35 (2010)
61. Ahmed, S., Kashem, M.A.: Enhancing the character segmentation
accuracy of Bangla OCR using BPNN. Int. J. Sci. Res. (IJSR) ISSN
(Online), 2319–7064 (2013)
62. Chowdhury, A.A., Ahmed, E., Ahmed, S., Hossain, S., Rahman,
C.M.: Optical character recognition of Bangla characters using neu-
ral network: a better approach. In: 2nd ICEE (2002)
63. Ahmed, S., Sakib, A.N., Ishtiaque Mahmud, M., Belali, H., Rah-
man, S.: The anatomy of Bangla OCR system for printed texts using
back propagation neural network. Glob. J. Comput. Sci. Technol.
(2012)
64. Afroge, S., Ahmed, B., Hossain, A.: Bangla optical character
recognition through segmentation using curvature distance and
multilayer perceptron algorithm. In: 2017 International Conference
on Electrical, Computer and Communication Engineering (ECCE),
pp. 253–257 (2017). IEEE
65. Hossain, S.A., Tabassum, T.: Neural net based complete character
recognition scheme for Bangla printed text books. In: 16th Interna-
tional Conference on Computer and Information Technology, pp.
71–75 (2014). IEEE
66. Pramanik, R., Bag, S.: Shape decomposition-based handwritten
compound character recognition for Bangla OCR. J. Vis. Commun.
Image Represent. 50, 123–134 (2018)
67. Ghosh, R., Vamshi, C., Kumar, P.: RNN based online handwritten
word recognition in Devanagari and Bengali scripts using horizon-
tal zoning. Pattern Recogn. 92, 203–218 (2019)
68. Purkaystha, B., Datta, T., Islam, M.S.: Bengali handwritten charac-
ter recognition using deep convolutional neural network. In: 2017
20th International Conference of Computer and Information Tech-
nology (ICCIT), pp. 1–5 (2017). IEEE
69. Islam, M.S., Rahman, M.M., Rahman, M.H., Rivolta, M.W.,
Aktaruzzaman, M.: Ratnet: a deep learning model for Ben-
gali handwritten characters recognition. Multimed. Tools Appl.
81, 10631–10651 (2022). https://doi.org/10.1007/s11042-022-
12070-4
70. Maity, S., Dey, A., Chowdhury, A., Banerjee, A.: Handwritten Ben-
gali character recognition using deep convolution neural network.
In: Bhattacharjee, A., Borgohain, S.K., Soni, B., Verma, G., Gao,
X.-Z. (eds.) Machine Learning, Image Processing, Network Secu-
rity and Data Sciences, pp. 84–92. Springer, Singapore (2020)
71. Roy, A.: AKHCRNet: Bengali Handwritten Character Recognition
Using Deep Learning (2020)
72. Sharif, S., Mohammed, N., Momen, S., Mansoor, N.: Classification
of Bangla compound characters using a HOG-CNN hybrid model.
In: Proceedings of the International Conference on Computing and
Communication Systems, pp. 403–411 (2018). Springer
123
K. Roy et al.
73. Hasan, M.J., Wahid, M.F., Alom, M.S.: Bangla compound charac-
ter recognition by combining deep convolutional neural network
with bidirectional long short-term memory. In: 2019 4th Interna-
tional Conference on Electrical Information and Communication
Technology (EICT), pp. 1–4 (2019). IEEE
74. Paul, D., Chaudhuri, B.B.: A BLSTM network for printed Bengali
OCR system with high accuracy. arXiv preprint arXiv:1908.08674
(2019)
75. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep
feedforward neural networks. In: Proceedings of the Thirteenth
International Conference on Artificial Intelligence and Statistics,
pp. 249–256 (2010). JMLR Workshop and Conference Proceedings
76. Rahman, M.A., Tabassum, N., Paul, M., Pal, R., Islam, M.K.:
BN-HTRd: A Benchmark Dataset for Document Level Offline
Bangla Handwritten Text Recognition (HTR) and Line Segmenta-
tion. arXiv (2022). https://doi.org/10.48550/ARXIV.2206.08977.
https://arxiv.org/abs/2206.08977
77. Mridha, M.F., Ohi, A.Q., Ali, M.A., Emon, M.I., Kabir, M.M.:
Banglawriting: a multi-purpose offline Bangla handwriting dataset.
Data Brief. 34, 106633 (2021). https://doi.org/10.1016/j.dib.2020.
106633
78. Banik, M., Rifat, M.J.R., Nahar, J., Hasan, N., Rahman, F.: Okkhor:
a synthetic corpus of Bangla printed characters. In: Arai, K.,
Kapoor, S., Bhatia, R. (eds.) Proceedings of the Future Technolo-
gies Conference (FTC) 2020, vol. 1, pp. 693–711. Springer, Cham
(2021)
79. Roark, B., Wolf-Sonkin, L., Kirov, C., Mielke, S.J., Johny, C.,
Demirsahin, I., Hall, K.: Processing South Asian languages written
in the Latin script: the Dakshina dataset. In: Proceedings of the 12th
Language Resources and Evaluation Conference, pp. 2413–2423.
European Language Resources Association, Marseille, France
(2020). https://aclanthology.org/2020.lrec-1.294
80. Al Mumin, M.A., Shoeb, A.A.M., Selim, M.R., Iqbal, M.Z.:
Sumono: a representative modern Bengali corpus. SUST J. Sci.
Technol. 21(1), 78–86 (2014)
81. Biswas, E.: Bangla Largest Newspaper Dataset. Kaggle (2021).
https://doi.org/10.34740/KAGGLE/DSV/1857507. https://www.
kaggle.com/dsv/1857507
82. Ahmed, M.F., Mahmud, Z., Biash, Z.T., Ryen, A.A.N., Hossain,
A., Ashraf, F.B.: Bangla Online Comments Dataset. Mendeley
Data (2021). https://doi.org/10.17632/9xjx8twk8p.1. https://data.
mendeley.com/datasets/9xjx8twk8p/1
83. Farahmand, A., Sarrafzadeh, H., Shanbehzadeh, J.: Document
image noises and removal methods (2013)
84. Lee, C.-Y., Osindero, S.: Recursive recurrent nets with attention
modeling for OCR in the wild. In: 2016 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 2231–2239
(2016). https://doi.org/10.1109/CVPR.2016.245
85. Wagner, R.A., Fischer, M.J.: The string-to-string correction prob-
lem. J. ACM (JACM) 21(1), 168–173 (1974)
86. Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A.,
Druzhinin, M., Kalinin, A.A.: Albumentations: fast and flexi-
ble image augmentations. Information (2020). https://doi.org/10.
3390/info11020125
87. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: sur-
passing human-level performance on imagenet classification. In:
2015 IEEE International Conference on Computer Vision (ICCV),
pp. 1026–1034. IEEE Computer Society, Los Alamitos, CA, USA
(2015). https://doi.org/10.1109/ICCV.2015.123
88. Loshchilov, I., Hutter, F.: SGDR: Stochastic Gradient Descent with
Warm Restarts (2017)
A multifaceted evaluation of representation of graphemes for practically effective Bangla OCR
89. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.,
Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga,
L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Rai-
son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang,
L., Bai, J., Chintala, S.: Pytorch: an imperative style, high-
performance deep learning library. In: Advances in Neural Infor-
mation Processing Systems 32, pp. 8024–8035. Curran Asso-
ciates, Inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-
an-imperative-style-high-performance-deep-learning-library.pdf
Publisher’s Note Springer Nature remains neutral with regard to juris-
dictional claims in published maps and institutional affiliations.
Springer Nature or its licensor (e.g. a society or other partner) holds
exclusive rights to this article under a publishing agreement with the
author(s) or other rightsholder(s); author self-archiving of the accepted
manuscript version of this article is solely governed by the terms of such
publishing agreement and applicable law.
123