Proceedings of the SMART–2019, IEEE Conference ID: 46866

8th International Conference on System Modeling & Advancement in Research Trends, 22nd–23rd November, 2019
College of Computing Sciences & Information Technology, Teerthanker Mahaveer University, Moradabad, India

Text Extraction through Video Lip Reading

Using Deep Learning

S.M. Mazharul Hoque Chowdhury1, Mushfiqur Rahman2, Marzan Tasnim Oyshi3 and Md. Arid Hasan4
Department of Computer Science & Engineering,
1

Daffodil International University, Dhaka, Bangladesh

2
Department of Computer Science & Engineering,
Daffodil International University, Dhaka, Bangladesh
3
Department of Computer Science & Engineering,
Daffodil International University, Dhaka, Bangladesh
4
Department of Computer Science & Engineering,
Daffodil International University, Dhaka, Bangladesh
E-mail: 1mazharul2213@diu.edu.bd, 2mushfiqur.cse@diu.edu.bd, 3marzan@diu.edu.bd,
4
arid15-5332@diu.edu.bd

Abstract— Automated text extraction from video data
through lip reading can overcome the language barrier and
open the door of opportunities in terms of security, connectivity
and physical challenges. The conversion is possible by
analyzing facial expression using deep learning method. But
this conversion is a challenging task due to the varieties of
pronunciation and accents of the same word causing different
countenance. In this research, a method of converting video
data to text data through lip reading has been proposed. The
proposed method includes test dataset, image frame analysis
and having text output from identified words. In the proposed
technique, the test dataset will be organized by combining all
the possible facial expressions of different words.
Keywords: Automated, Audio, Conversion, Frame,
Identification, Training Data, Video, Sequence, Word
I. Introduction
Technologies are developing frequently all over
the world. People are getting introduced with incipient
contrivances and technological solutions every single day.
Cameras are one of the mostly used contrivance by integrating
incipient features and exhibiting their capabilities in
different sectors. Now it is possible to detect face, ascertain
emotion through countenance, object identification and
many more. Even medical imaging made it facile to detect
diseases, encephalon damage, bone fracture, etc. On the
other hand, keenly intellective computer systems now can
convert audio data into text data. It can be habituated to
identify suspicious conversation and avail the physically
challenged people who are struggling with hearing. Audio
less video data is one of the very commonly available data.
But retrieving audio data from audio less video data is a
very arduous task. But if we can identify and analyze the
countenances of video data, we will be able to retrieve text
data from audio less video data. This will help to reduce
the crime rate or maybe we will be able to identify what a
240 Copyright © IEEE–2019 ISBN: 978-1-7281-3245-7
person is trying to say who could previously talk but lost
his voice due to any accident. For that we can just simply
convert the audio less video data into text or speech. In this
research we are going to discuss about converting this type
of video data into text data.
II. Literature Review
There is a numerous number of work done for audio
to video sync quandary but for audio less data to audio/
text (visual verbalization apperception) is still a rear case
now. These works are fundamentally done predicated on
some concrete methods by individual researchers. Zhou
et al. [1] surveyed a method in this regards on their research
with less detailing but efficacious. Many types of research
have been done on this field and everyone followed
relatively similar methods to extract the information
(features around lips) and then relegate these to a template.
Pfister et al. [2] distinguished lip movement by the state
of mouth openness by utilizing a single SIFT descriptor of
the facial region. Pei et al. [3] described in a research the
state of art on many databases, extract features and then
aligned these for motion patterns. Koller et al. [4] utilized
a deep convolutional neural network to extract dactylology
from effervescent mouth shapes. Similarly, Zoric et al. [5]
encoded the sample image, frame them, train them after that
relegate, to engender a word level classification. Chen [6]
represented a verbalization availed frame rate conversation
method for verbalization availed video encoding. Lavagetto
[7] developed a multimedia telephone for hard auricularly
discerning persons from the conversation by graphic
animation fit to lip-reading.
Word classification with a bag of word /lexicon has
not been endeavored in visual verbalization apperception,
however [8] has tackled the same drawback within the
context of text apperceiving. Their work shows that it’s

Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on July 21,2020 at 23:08:01 UTC from IEEE Xplore. Restrictions apply.
 Text Extraction through Video Lip ReadingUsing Deep Learning

possible to train a general and scalable word apperception

model for an immensely colossal pre-defined wordbook, as
a multi-class relegation quandary. Work on silent speech
recognition based on lip movement was done by Ke Sun
et al. (2018) and their purpose was to improve mobile
device interaction [9]. Joon Son Chun et al. (2018) worked
on lip reading technique and here only video data was Fig.1: Data Flow Diagram of Proposed System
provided without audio to monitor learning [10]. Researcher
Sumita Nainan et al. (2018) worked on lip tracking using IV. Word Identification
deformable models and geometric approaches [11]. Gelder The identification of word will be predicated on the
et al. [12] discussed about autistic children and compared training data set provided to the system. For each and every
the strength of their brain with normal people. The research word a sequence of image frame will be stored. A database
proves that, the control of memory for lip reading of autistic of word and their co-responding image frame sequence
need to be stored to the system as training data. When the
children are poorer then in general. Summerfield et al. [14]
system will take an input data set it will first separate video
reviewed the physiological understanding of lip reading
sequences predicated on the time we spent to verbalize
and audio visual speech prediction based on speech signal. a word after another word. Then it will split each video
Lee et al. [14] extracted features of lip motion from the sequence into image frames, so that it can be matched with
differences of consecutive frames. They have used Principle the training data set.
Component Analysis (PCA) and Independent Component Training data set will be built predicated on the amassed
Analysis (ICA). Luettin et al. [15] discussed about a data set. Data amassment must be for individual words.
latest approach with a view to identifying the speaker on For each word multiple video data will be accumulated.
lip reading from a small video database. They used Because the precision of the analysis will depend on the
spatio-temporal speaker based on specific features extracted amount of the data utilized for training. Additionally the
from image by identifying the grey level area of mouth and amount of frames for each word will be responsible for
using HMMs along with the mixture of Gaussians. Zhang the precision of the word detection. For data accumulation
some video will be accumulated and from it blank frames
et al. [16] presented a model that integrates the color
that are not responsible for the word spooked by the person
and spatial edge information with a view to addressing
will be abstracted. Because the more blank frames are
the problem of lip feature extraction for the concern of there, the more it will be arduous to detect a word. For this
speechreading. They have used prominent red hue as type of data system will first endeavor to match the blank
indicator with a view to locate the lip position. Depending frames with test data. So it will not be able to find the word
on the identified lip area, they further refined the exterior accurately. Consequently it is consequential to abstract
and interior lip boundary using the information of color those blank frames from the training video data. On the
and spatial edge, where those two were combined within other hand, if any audio data subsist with video data then
Markov random field (MRF) framework. Nguyen et al. [17] those audio additionally must be eliminated from the data.
proposed a latest technique for face detection and lip feature Because in this system there will be no role for the audio
extraction using a real time Field Programmable Gate Array data. A database must be engendered to store the word
(FPGA). So, we are intended to take the same approach. and its respective video data frames. This database will be
utilized in the system for word identification.
III. Proposed Method
In this research we are proposing a method to convert
video data without audio into text data. For that at first a
test data set will be needed to compare or identify the input
data. This system can be divided into four main sections.
They are – test data, data preprocessing, identification of
word and output. In the test data section raw video clips
will be considered as input for the system. This data will be
preprocessed in the data preprocessing section and data will
be split into frames. Those frames are the processed data
and will be matched with training data in the identification
of word section. Determinately the system will provide
result predicated on the analysis in the output section. The
data flow diagram is given below in the fig. 1. Fig. 2: Image Sequence Matching Process

Copyright © IEEE–2019 ISBN: 978-1-7281-3245-7 241

Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on July 21,2020 at 23:08:01 UTC from IEEE Xplore. Restrictions apply.
8th International Conference on System Modeling & Advancement in Research Trends, 22nd–23rd November, 2019
College of Computing Sciences & Information Technology, Teerthanker Mahaveer University, Moradabad, India
This system is entirely dependent on the training data
set provided to it. The more data is provided to it, the more
it becomes precise. Predicated on the angle or position
of the speakers face system will provide different output.
Because predicated on the angle the movement may seem
different form different side. So for a particular word
multiple training data will be required form different angle.
Moreover verbalizing a word may vary from person to
person. So it is withal paramount to take the training data
from multiple utilizer. As there will be an abundance of
training data for a single word, so amount of total training
data will be high additionally. But higher amount of data
will give us higher precision.
After building the training data set with enough data
items of word and its video frames system will be able to
deal with the test data. To convert the voiceless video data
into text data it is paramount to build such a system that can
handle video data as well as text data. For that the system
will require to map the lip of the utilizer and how it changes
during it is verbalizing about something. The system will
be able to break the video into image sequences first. Every
person takes a minimum time to verbalize a word after
another word. So to identify the word it is paramount to
identify the time a person takes. When someone verbalize
they take minuscule break, which is may be micro seconds.
May be someone takes longer than that, but it will not
transmute anything in the system. Because the minimum
time will not be transmuted. The longer break someone
takes the more accurately the system will be able to identify
a word.
After breaking the test video data into image sequences
like TD-X in fig. 2 the system will split them according
to the time space for verbalizing one word after another.
During this process the system will additionally eliminate
the frames in this time gape. Because in the training data
all blank or extraneous frames were withal expunged to
eschew any kind of error or mismatch of the word. When
splitting is done as fig. 2 the system is now yare to analyze
the test data. As mentioned afore system is already trained,
ergo now system will match every image sequence from
the test data set with the training database. For each match
system will engender a word and store it to a text file. Then
it will move for the next sequence matching.
In this system grammar is not the main priority, so
ascertaining what someone is verbalizing will be the
main focus. Because predicated on the rudimentary text
architecture it is quite possible to predict about what
someone is verbalizing about. In the text file all the words
must be placed in an order according to the sequence in the
video. Because if order is broken it can transmute entire
meaning of the sentence. When all the sequence are being
analyzed and their corresponding words are situated to the
text file the process will culminate and a text file will be
engendered as output.
242 Copyright © IEEE–2019 ISBN: 978-1-7281-3245-7
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on July 21,2020 at 23:08:01 UTC from IEEE Xplore. Restrictions apply.
As this system will identify words according to the
movement of lips, so it will be possible to detect any kind of
word from any language. Even emotions can be predicted
from the visage with the word. So it will be more facile to
identify what authentically the verbalizer meaning through
the sentence. This system has a profoundly and immensely
colossal use in our authentic life. Moreover it withal have
some back draw and complexities. The prodigious amount
of data cannot be stored in our traditional databases and if
not handled conscientiously data will not be able to provide
good result after analysis. There is no such limit for this
type of training data for better precision.
This work can be compared with the work of Ke
Sun’s teams Lip-Interact technique that allows interaction
between smart device and human [9]. The main similarity
between our researches that it includes language processing
from the lip movement of its user. According to the project
Lip-Interact collects silent command from its user where
the model is trained with some fixed inputs as do, undo,
screenshot, open camera and so on. The front camera of
the smart device gets the input while user looks at the
screen and says something. To improve the quality of
the command recognition they used Spatial Transformer
Networks. Their training model included real time user
inputs as data set instead of random data set and news data.
This can be a good solution for a small amount of words
because complexity will reduce because the possibility
of in accurate data is very low and noisy data is already
preprocessed. Therefore, the number of training data will
be in the good rang that can easily be handled. But for a
general conversation it will be difficult though our research
is still under process to improve the analysis.
An example of the data training can be explained
through the sample given below in figure 3-
Fig. 3: Data training for word recognition
According to the given sample which was collected
from the research regarding Lip-Interact shows that
the camera which works as a sensor in this research is
working as a lip gesture identifier and taking snap of every
movement from the video and there is a condition of time
gap about after how much time the image will be taken.
Figure 4 is showing the movement of lip movement while
a user is saying something from the research of Ahmad B.
A. Hassanat regarding the research of automated lip reading
[18]. Each word speaking have its own type of movement

and the movement sequence is different also. The change of
the position of the lip during speaking is expressed by the
figure given below-
Fig. 4: Lip Movement While Speaking
This technique can fail due to different reasons. One
of the most common problem researchers faced is the lip
gesture for similar words. Sometimes it is difficult to identify
what the speaker is trying to express and what actually the
machine is understanding. But through enough training and
sample data it is possible to reduce the error related to this
problem using machine learning. In that process machine
will be able to understand the sentence construction from
its training dataset. This analysis can also be done using
convolutional neural network.
Considering the previous study and several technique as
well as research projects the accuracy differs in a structured
way. In general based on the training data and model built
for analysis accuracy is used to be in between 85% to 95%
and this was studied from the references provided in this
research. Based on the study, this analysis will have about
90% of accuracy on the data set we are working on.
V. Conclusion
Day by day the amount of video data are incrementing
world wise. But the amount of systems that can analyze
those data are not enough yet. Those days are not far when
there will not be any language barrier among the people all
over the world. They will understand each other in their own
language and will be communicate in their native language.
Predicated on the lip reading and voice data analysis and
text data analysis it is possible to build such system that will
be able to impeccably translate one language to another. So
someone from one country will verbalize something in his
native language, then through the data analysis another one
from another country will understand in his own language.
Research on data science making our quotidian life more
facile every day. Lip reading has its own paramount role
to play in this development. Gregarious development,
crime reduction and so on are possible through this type
of analysis. So it is high time to commence working with
audio less video data for our own development.
Acknowledgement
This research was conducted with the help of DIL NLP
and Machine Learning Lab. Still some great and energetic
researchers are working in this lab on this topic. We hope
that this lab will be able to present good researches and
products in future.
Copyright © IEEE–2019 ISBN: 978-1-7281-3245-7 243
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on July 21,2020 at 23:08:01 UTC from IEEE Xplore. Restrictions apply.
Text Extraction through Video Lip ReadingUsing Deep Learning
References
[1] Z. Zhou, G. Zhao, X. Hong and M. Pietik¨ainen, “A review of recent
advances in visual speech decoding,” Image and vision computing,
vol. 32, Issue. 9, pp. 590–605, September 2014.
[2] T. Pfister, J. Charles and A. Zisserman, “Large-scale learning of
sign language by watching TV (using co-occurrences),” Proceedings
of the British machine vision conference, DOI:10.5244/c.27.20,
January 2013.
[3] Y. Pei, T.K. Kim and H. Zha, “Unsupervised random forest manifold
alignment for lipreading,” Proceedings of the IEEE International
Conference on Computer Vision, DOI: 10.1109/ICCV.2013.23, pp.
129–136, December 2013.
[4] O. Koller, H. Ney and R. Bowden, “Deep learning of mouth shapes
for sign language,” Proceedings of the IEEE International Conference
on Computer Vision Workshops. DOI: 10.1109/ICCVW.2015.69,
pp. 85–91, December 2015.
[5] G. Zoric and I. S. Pandzic, “A real-time lip sync system using a
genetic algorithm for automatic neural network configuration,” IEEE
International Conference on Multimedia and Expo. DOI: 10.1109/
ICME.2005.1521684, pp. 1366–1369, July 2005.
[6] T. Chen, “Audiovisual speech processing, lip reading and lip
synchronization,” IEEE Signal Processing Mag., vol. 18, pp. 9-21,
January 2001.
[7] F. Lavagetto, “Converting speech into lip movements: A multimedia
telephone for hard of hearing people,” IEEE Trans. on Rehab. Eng.,
vol. 3, March 1995.
[8] M. Jaderberg, K. Simonyan, A. Vedaldi and A. Zisserman, “Synthetic
data and artificial neural networks for natural scene text recognition,”
Workshop on Deep Learning, NIPS, June 2014.
[9] K. Sun, C. Yu, W. Shi, L. Liu and Y. Shi, “Lip-Interact: Improving
Mobile Device Interaction with Silent Speech Commands,”
Proceedings of the 31st Annual ACM Symposium on User Interface
Software and Technology, doi: 10.1145/3242587.3242599, pp: 581-
593, October 2018.
[10] J. S. Chung, A. Zisserman, “Learning to lip read words by watching
videos,” Computer Vision and Image Understanding, Vol. 173, pp.
76-85, August 2018.
[11] S. Nainan and V. Kulkarni, “Lip Tracking Using Deformable Models
and Geometric Approaches” Satapathy S., Joshi A. (eds) Information
and Communication Technology for Intelligent Systems. Smart
Innovation, Systems and Technologies, vol 106, pp. 655-663,
December 2018.
[12] Gelder, B. de, Vroomen, J., & van der Heide, L. (1991). Face
recognition and lip-reading in autism. European Journal of Cognitive
Psychology, 3(1), 69–86. doi:10.1080/09541449108406220
[13] Summerfield, Q. (1992). Lipreading and Audio-Visual Speech
Perception. Philosophical Transactions of the Royal Society B:
Biological Sciences, 335(1273), 71–78. doi:10.1098/rstb.1992.0009
[14] Lee, K. D., Lee, M. J., & Soo-Young Lee. (n.d.). Extraction of
frame-difference features based on PCA and ICA for lip-reading.
Proceedings. 2005 IEEE International Joint Conference on Neural
Networks, 2005. doi:10.1109/ijcnn.2005.1555835
[15] Luettin, J., Thacker, N. A., & Beet, S. W. (n.d.). Speaker identification
by lipreading. Proceeding of Fourth International Conference
on Spoken Language Processing. ICSLP ’96. doi:10.1109/
icslp.1996.607030
[16] Zhang, X., & Mersereau, R. M. (n.d.). Lip feature extraction towards
an automatic speechreading system. Proceedings 2000 International
Conference on Image Processing (Cat. No.00CH37101). doi:10.1109/
icip.2000.899336
[17] Nguyen, D., Halupka, D., Aarabi, P., & Sheikholeslami, A. (2006).
Real-time face detection and lip feature extraction using field-
programmable gate arrays. IEEE Transactions on Systems, Man and
Cybernetics, Part B (Cybernetics), 36(4), 902–912. doi:10.1109/
tsmcb.2005.862728
[18] Ahmad B. A. H., “Visual Speech Recognition.” Speech and
Language Technologies (2011): n. pag. Crossref. Web.