import re

import pickle
import time
from nltk.tokenize import word_tokenize
import tensorflow as tf
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
from tensorflow.keras.optimizers.legacy import Adam
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import load_model
from tensorflow.keras import backend as K
from tensorflow.keras.layers import Layer
from flask import Flask, request, render_template

#Attention Model for classification

class attention(Layer):
def init(self):
super(attention,self).__init__()
def build(self,input_shape):
self.W=self.add_weight(name='att_weight',shape=(input_shape[-
1],1),initializer="normal")
self.b=self.add_weight(name='att_bias',shape=(input_shape[-
2],1),initializer="zeros")
super(attention, self).build(input_shape)
def call(self,x):
e = K.tanh(K.dot(x,self.W)+self.b)
a = K.softmax(e, axis=1)
 output = x*a
return K.sum(output, axis=1)

def extra_space(text):
new_text= re.sub("\s+"," ",text)
return new_text

def sp_charac(text):
new_text=re.sub("[^0-9A-Za-z ]", "" , text)
return new_text

def tokenize_text(text):
new_text=word_tokenize(text)
return new_text

app = Flask(__name__)

with open('len_tokens_ATT6.pickle', 'rb') as handle:

length_tokens_6 = pickle.load(handle)

with open('len_tokens_ATT4.pickle', 'rb') as handle:

length_tokens_4 = pickle.load(handle)

with open('len_tokens_ATT2.pickle', 'rb') as handle:

length_tokens_2 = pickle.load(handle)
file="lstm_att_len6.hdf5"
model_len6 = load_model(file ,custom_objects={'attention': attention})
model_len6.compile(loss='categorical_crossentropy',
optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])

file="lstm_att_len4.hdf5"
model_len4 = load_model(file, custom_objects={'attention': attention})
model_len4.compile(loss='categorical_crossentropy',
optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])

file="lstm_att_len2.hdf5"
model_len2 = load_model(file , custom_objects={'attention': attention})
model_len2.compile(loss='categorical_crossentropy',
optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])

@app.route('/')
def my_form():
return render_template('my-form.html')

@app.route('/', methods=['POST'])
def predict_next():
text = request.form['text']
if not text:
return render_template('my-form.html', error="Please enter a new word /
sentence!")
start= time.time()
cleaned_text=extra_space(text)
cleaned_text=sp_charac(cleaned_text)
tokenized=tokenize_text(cleaned_text)
line = ' '.join(tokenized)
pred_words = []
if len(tokenized)==1:
encoded_text = length_tokens_2.texts_to_sequences([line])
pad_encoded = pad_sequences(encoded_text, maxlen=1, truncating='pre')
for i in (model_len2.predict(pad_encoded)[0]).argsort()[-3:][::-1]:
pred_word = length_tokens_2.index_word[i]
pred_words.append(text + " " + pred_word)
elif len(tokenized) < 4:
encoded_text = length_tokens_4.texts_to_sequences([line])
pad_encoded = pad_sequences(encoded_text, maxlen=3, truncating='pre')
for i in (model_len4.predict(pad_encoded)[0]).argsort()[-3:][::-1]:
pred_word = length_tokens_4.index_word[i]
pred_words.append(text + " " + pred_word)
else:
encoded_text = length_tokens_6.texts_to_sequences([line])
pad_encoded = pad_sequences(encoded_text, maxlen=5, truncating='pre')
for i in (model_len6.predict(pad_encoded)[0]).argsort()[-3:][::-1]:
pred_word = length_tokens_6.index_word[i]
pred_words.append(text + " " + pred_word)
print('Time taken: ',time.time()-start)
return render_template('my-form.html', pred_words=pred_words)
if __name__ == '__main__':
app.run()
 SYNOPSIS

SYNOPSIS
The "Next Word Prediction" project aims to leverage machine learning and
natural language processing techniques to develop a predictive model capable
of suggesting the next word in a sequence of text input. The project involves
collecting textual data from various sources, such as books, articles, and online
content, and preprocessing the data to extract meaningful features and
patterns. Feature selection and engineering techniques are employed to
identify key linguistic attributes and contextual information that contribute to
predicting the next word accurately.

The model training phase utilizes supervised learning algorithms, including

recurrent neural networks (RNNs), long short-term memory networks
(LSTMs), and transformers, to learn the underlying patterns and relationships
within the textual data. Evaluation of model performance is conducted using
metrics such as perplexity, accuracy, and word prediction accuracy on
validation datasets.

The predictive model's deployment aims to integrate it into text processing

applications, word processors, virtual keyboards, and other platforms where
predictive text input functionality is valuable. The objective is to provide users
with an intuitive and efficient tool for enhancing text input experiences,
improving typing speed, and reducing errors.

By harnessing the power of machine learning and natural language processing,

the next word prediction model seeks to facilitate seamless and intuitive
communication in various digital environments. The ultimate goal is to enhance
user productivity, streamline text input processes, and optimize user
experiences across different applications and devices.

INTRODUCTION

INTRODUCTION
Text prediction has become an integral part of modern communication, aiding
in typing efficiency and enhancing user experience across various digital
platforms. In this project, we delve into the development of a predictive model
focused on next word prediction, leveraging the advancements in machine
learning and natural language processing (NLP) techniques.

With the exponential growth of digital communication channels, the ability to

accurately predict the next word in a sequence has become paramount.
Whether in messaging applications, search engines, or word processors,
predictive text input significantly streamlines the typing process and improves
user productivity.

Our endeavor involves harnessing the power of machine learning algorithms

and NLP techniques to analyze vast amounts of textual data. By exploring
patterns, semantics, and context within the text, we aim to build a robust
predictive model capable of suggesting the most probable next word based on
the preceding words in a sentence or phrase.

Through the utilization of advanced algorithms such as recurrent neural

networks (RNNs), long short-term memory networks (LSTMs), and
transformer models, we strive to uncover intricate linguistic structures and
relationships embedded within the text corpus. By training the model on
diverse datasets encompassing a wide range of textual genres and styles, we
aim to enhance its predictive accuracy and adaptability to different writing
styles and contexts.
The objective of our predictive model is to empower users with a seamless and
intuitive text input experience, enabling faster typing speeds, reduced errors,
and enhanced communication efficiency. By integrating our model into various
text-based applications, virtual keyboards, and digital interfaces, we envision
facilitating fluid and natural interactions in the digital realm.

ORGANIZATION PROFILE

ORGANIZATION PROFILE
Our organization is dedicated to pioneering predictive analytics solutions for
various domains, including next word prediction, leveraging cutting-edge
artificial intelligence (AI) algorithms. With a firm commitment to enhancing
user experiences and optimizing digital communication, we aim to empower
individuals with intuitive and efficient text input mechanisms.

Company Background

Established in [Year], our organization emerged from a collective vision to

revolutionize the way people interact with digital platforms and applications
through predictive text technologies. Drawing upon a diverse pool of talent in
data science, machine learning, and natural language processing (NLP), our
team embarked on a mission to develop innovative solutions that streamline
text input processes and elevate user productivity.

Core Competencies

Our core competencies lie in the development and deployment of AI-driven

predictive models tailored for next word prediction tasks. By harnessing the
power of advanced machine learning algorithms and NLP techniques, we
analyze vast corpora of text data to identify patterns, semantics, and contextual
cues essential for accurate word prediction. Our expertise extends to the
integration of predictive text functionalities into a wide range of digital
platforms, including messaging applications, search engines, virtual keyboards,
and text editors.
OUR VISION

Our vision is to become a leading authority in next word prediction, reshaping

digital communication landscapes through innovative AI-driven solutions. We
envision a future where individuals seamlessly interact with digital interfaces,
leveraging predictive text technologies to enhance typing efficiency, reduce
errors, and foster smoother communication experiences across various digital
platforms.

OUR MISSION

Our mission is to develop cutting-edge AI solutions that empower users with

intuitive and efficient text input mechanisms. By harnessing the latest
advancements in machine learning and NLP, we strive to deliver predictive text
solutions that anticipate user intent, adapt to individual writing styles, and
enhance overall user satisfaction. We are committed to democratizing access to
predictive text technologies, making them widely available and accessible
across diverse digital environments.

QUALITY OBJECTIVES

1. Accuracy and Reliability: Develop predictive models with high accuracy rates
(>90%) for next word prediction, ensuring precise and reliable text
suggestions.
2. User-Centric Design: Design intuitive and user-friendly interfaces for
seamless integration of predictive text functionalities into digital platforms,
enhancing user experiences and productivity.

3. Continuous Improvement: Continuously monitor and refine predictive

models based on user feedback and evolving language patterns, driving
iterative enhancements and optimizations.

SYSTEM SPECIFICATION
SYSTEM SPECIFICATION

Hardware Configuration:
- RAM: 4GB or higher
- Operating System: Windows 10 or later

Software Specification:
- Python Environment: Anaconda or Miniconda
- Deep Learning Framework: TensorFlow or PyTorch
- Image Processing Libraries: OpenCV, Pillow
- Visualization Libraries: matplotlib, seaborn

Description:
The system specifications provided above are tailored for the development and
deployment of a next word prediction model using machine learning and
natural language processing techniques. These specifications ensure
compatibility and optimal performance throughout the development lifecycle
of the predictive model.

RAM: A minimum of 4GB RAM is recommended to handle the computational

requirements of running machine learning algorithms, processing large
datasets, and managing model training tasks efficiently. However, higher RAM
configurations may be beneficial for handling more extensive datasets and
complex model architectures.

Operating System: The system is designed to operate on Windows 10 or later

versions to ensure compatibility with the specified software components and
frameworks.
Python Environment: The system relies on the Python programming language
for its flexibility, extensive libraries, and robust ecosystem for machine learning
and natural language processing tasks. Anaconda or Miniconda distribution is
recommended for managing Python environments, dependencies, and package
installations efficiently.

Deep Learning Framework: TensorFlow or PyTorch serves as the primary deep

learning framework for building and training neural network models required
for next word prediction tasks. These frameworks offer comprehensive
support for building and training deep learning models, including recurrent
neural networks (RNNs) and transformers, which are commonly used for
natural language processing tasks.

Image Processing Libraries: OpenCV and Pillow are essential image processing
libraries utilized for preprocessing text data, handling image inputs if
applicable, and performing any required image-related tasks during the
development and training of the predictive model.

Visualization Libraries: Matplotlib and seaborn are employed for data

visualization tasks, enabling the visualization of model performance metrics,
analysis of training data distributions, and presentation of results in a clear and
informative manner.

Overall, these system specifications provide a robust and compatible

environment for developing, training, and evaluating next word prediction
models, ensuring efficient workflow management and optimal utilization of
computational resources throughout the development process.

SYSTEM STUDY

SYSTEM STUDY FOR NEXT WORD PREDICTION

EXISTING SYSTEM:
In traditional approaches to language prediction, basic statistical methods or
rule-based techniques are commonly used. These methods often rely on simple
language models and n-gram probabilities to predict the next word in a
sentence. However, they have several limitations:

1. Lack of Context Awareness: Basic statistical methods do not consider the

context of the entire sentence, leading to limited accuracy in predicting the next
word, especially in complex language contexts.

2. Limited Vocabulary Coverage: Rule-based approaches may struggle with out-

of-vocabulary words or rare language patterns, affecting the accuracy and
coverage of predictions.

3. Fixed Prediction Strategies: Traditional systems often use fixed prediction

strategies, which do not adapt to user-specific language patterns or evolving
linguistic trends over time.

PROPOSED SYSTEM:
The proposed next word prediction system leverages advanced machine
learning and natural language processing techniques to overcome the
limitations of traditional approaches. Key features of the proposed system
include:

1. Contextual Language Models: The system utilizes state-of-the-art language

models such as GPT (Generative Pre-trained Transformer) or BERT
(Bidirectional Encoder Representations from Transformers) to capture
contextual information from the entire sentence and make more accurate
predictions.

2. Large Vocabulary Coverage: By training on vast corpora of text data, the

system can effectively handle a wide range of vocabulary and linguistic
variations, improving the coverage and accuracy of word predictions.

3. Dynamic Prediction Strategies: The system employs dynamic prediction

strategies based on user input and real-time context analysis. It adapts to
individual user preferences, language style, and context, providing
personalized and contextually relevant word suggestions.

4. Continuous Learning: The system incorporates mechanisms for continuous

learning and adaptation to evolving language patterns and user behaviors over
time. It updates its language models and prediction algorithms based on user
interactions and feedback, improving prediction accuracy and relevance.

SYSTEM MODULES:

1. Data Collection: Gather large-scale text corpora from diverse sources,

including books, articles, social media, and web content, to train language
models and enrich vocabulary coverage.

2. Preprocessing and Tokenization: Clean and preprocess the text data,

including removing noise, tokenizing sentences and words, and handling
special characters and punctuation marks.
3. Model Training: Train advanced language models such as GPT or BERT using
deep learning frameworks like TensorFlow or PyTorch. Fine-tune the models
on specific language prediction tasks and optimize performance.

4. Prediction Engine: Develop the prediction engine to process user input,

analyze contextual information, and generate next word suggestions based on
the trained language models and prediction strategies.

5. User Interface: Design an intuitive and user-friendly interface for users to

input text and interact with the prediction system. Provide real-time word
suggestions and feedback to enhance user experience.

SYSTEM IMPLEMENTATION:

The implementation of the next word prediction system involves the following
steps:

1. Infrastructure Setup: Configure hardware resources and software

environments compatible with deep learning frameworks and natural language
processing libraries.

2. Data Acquisition and Preprocessing: Collect and preprocess large-scale text

corpora to prepare training data for language model training.
3. Model Training and Optimization: Train advanced language models using
high-performance computing resources and optimize model parameters for
prediction accuracy and efficiency.

4. Prediction Engine Development: Implement the prediction engine using

efficient algorithms and data structures to enable real-time word prediction
and user interaction.

5. User Interface Design: Develop a responsive and user-friendly interface for

users to input text, view word suggestions, and provide feedback to the
prediction system.

6. Testing and Evaluation: Conduct extensive testing and evaluation of the

prediction system using diverse language datasets and user scenarios to assess
accuracy, coverage, and usability.

7. Deployment and Maintenance: Deploy the prediction system in production

environments and monitor its performance and user feedback continuously.
Update and maintain the system to incorporate new language patterns and user
preferences over time.

SYSTEM DESIGN AND DEVELOPMENT

SYSTEM DESIGN AND DEVELOPMENT FOR NEXT WORD PREDICTION

INPUT DESIGN:
In the input design phase for next word prediction, the system collects and
preprocesses text data necessary for language modeling. This includes
gathering large-scale text corpora from diverse sources such as books, articles,
social media, and web content. The collected text data undergoes preprocessing
to clean noise, tokenize sentences and words, and handle special characters and
punctuation marks.

OUTPUT DESIGN:
The output design phase focuses on presenting the results of next word
predictions in a user-friendly format for seamless integration into text-based
applications and platforms. Predictions regarding the next word in a sentence
are communicated through intuitive interfaces or text suggestion boxes.
Visualizations such as probability distributions or word clouds may be used to
display the likelihood of various words as the next word in the sequence.

DATABASE DESIGN:
While next word prediction systems typically do not require a database for
prediction generation, they may utilize databases for storing and managing
large-scale text corpora used for model training. The database schema may
include tables for storing text documents, metadata, and language model
parameters. Proper indexing and optimization techniques are applied to ensure
efficient data retrieval and storage.

SYSTEM DEVELOPMENT:
The system development phase involves implementing the input, output, and
language model designs to create a functional next word prediction system.
This includes developing software modules for text preprocessing, language
model training, prediction generation, and user interface integration. Advanced
natural language processing techniques and deep learning frameworks such as
TensorFlow or PyTorch are employed to build language models based on text
data.

MODULES:
1. Data Collection and Preprocessing:
- Gather large-scale text corpora from various sources.
- Preprocess text data to remove noise, tokenize sentences and words, and
handle special characters and punctuation marks.

2. Language Model Training:

- Train advanced language models such as GPT or BERT using deep learning
frameworks.
- Fine-tune language models on specific next word prediction tasks and
optimize model parameters for prediction accuracy.

3. Prediction Engine:
- Develop the prediction engine to process user input, analyze contextual
information, and generate next word suggestions based on the trained language
models.
- Implement efficient algorithms and data structures for real-time word
prediction and user interaction.

4. User Interface:
 - Design an intuitive and user-friendly interface for users to input text and
view next word suggestions.
- Provide real-time word predictions and feedback to enhance user
experience across various text-based applications and platforms.

SYSTEM IMPLEMENTATION:
The implementation of the next word prediction system involves configuring
hardware resources and software environments compatible with deep learning
frameworks and natural language processing libraries. Data acquisition and
preprocessing are conducted to prepare training data for language model
training. Advanced language models are trained using high-performance
computing resources, and prediction engines are developed for real-time word
prediction. User interfaces are designed and integrated with the prediction
system to provide seamless user interaction and feedback.

Testing and evaluation are conducted to assess the accuracy, coverage, and
usability of the prediction system across diverse language datasets and user
scenarios. Continuous monitoring and optimization are performed to enhance
prediction accuracy and relevance over time. The deployed system is
maintained and updated to incorporate new language patterns and user
preferences, ensuring its effectiveness and reliability in real-world
applications.
 TESTING AND IMPLEMENTATION
TESTING AND IMPLEMENTATION FOR NEXT WORD PREDICTION

SYSTEM TESTING:
Testing and implementation are crucial phases in the development and
deployment of a next word prediction system. Testing involves ensuring the
accuracy, reliability, and robustness of the predictive model across various
datasets and scenarios. This includes different levels of testing such as unit
testing, integration testing, and system testing.

1. Unit Testing:
- Individual components of the next word prediction system are tested in
isolation to ensure they perform as expected.
- This involves testing preprocessing modules, language model training
algorithms, and prediction engines.

2. Integration Testing:
- Components of the system are combined and tested together to validate
their interactions and integration.
- Integration testing ensures that the preprocessing, training, and prediction
modules work seamlessly together.

3. System Testing:
- The complete next word prediction system is tested against specified
performance criteria and validation metrics.
- Metrics such as prediction accuracy, coverage, and response time are
evaluated to assess the system's performance.

IMPLEMENTATION:
The implementation phase involves deploying the next word prediction system
into a production environment and ensuring its seamless integration with
existing platforms and applications.

1. Deployment:
- Install and configure the necessary hardware and software components to
support the prediction system's operation.
- Set up servers, databases, and computational resources required for training
and inference.

2. Integration:
- Integrate the prediction system with existing text-based applications and
platforms.
- Ensure compatibility with different operating systems and software
environments.

3. Data Preparation:
- Preprocess and tokenize text data to prepare it for language model training.
- Handle special characters, punctuation marks, and noise in the input text.

4. User Training and Support:

- Provide user training and support to familiarize users with the next word
prediction system.
- Offer assistance and troubleshooting for any issues encountered during
system usage.
5. Continuous Monitoring and Optimization:
- Monitor the system's performance and user feedback to identify areas for
improvement.
- Optimize model parameters, algorithms, and user interfaces based on
feedback and usage patterns.

6. Documentation:
- Document system configurations, deployment procedures, and user
guidelines for reference and future maintenance.

LIMITATIONS:
Next word prediction systems may face limitations related to data quality,
model accuracy, and user expectations.
- Data Quality: The accuracy and relevance of predictions depend on the
quality and diversity of the training data.
- Model Accuracy: Language models may struggle with out-of-vocabulary
words or ambiguous contexts, leading to inaccurate predictions.
- User Expectations: Users may have varying preferences and writing styles,
making it challenging to cater to diverse prediction needs.

CONCLUSION
CONCLUSION FOR NEXT WORD PREDICTION
This project aimed to develop an advanced next word prediction system using
machine learning and natural language processing techniques. Through the
analysis of extensive text data and the application of state-of-the-art
algorithms, we have successfully built a robust predictive model. By
implementing data preprocessing, language modeling, and prediction
algorithms, we have achieved significant accuracy and efficiency in suggesting
the next word in a given sequence of text.

The developed next word prediction system holds substantial promise in

enhancing user experience across various text-based applications and
platforms. With its ability to anticipate the next word in a sentence or phrase,
the system can streamline text input processes, improve typing speed, and
enhance overall productivity for users.

Through continuous research and refinement, along with the integration of

additional linguistic datasets and advanced algorithms, we can further enhance
the predictive capabilities of the system. By exploring techniques such as
recurrent neural networks, attention mechanisms, and transformer
architectures, we can improve prediction accuracy and adaptability to diverse
linguistic contexts.

The deployment of the next word prediction system has the potential to
revolutionize text input methods and user interactions across a wide range of
devices and applications. By providing users with intuitive and efficient text
prediction capabilities, we can enhance communication, productivity, and user
satisfaction in various domains, including messaging apps, word processors,
and virtual assistants.

Continued research and development efforts are essential to further optimize

the performance and usability of the next word prediction system. By
addressing challenges such as vocabulary expansion, context sensitivity, and
user personalization, we can ensure that the system remains adaptive,
accurate, and responsive to the evolving needs and preferences of users.

In conclusion, the next word prediction system represents a significant

advancement in natural language processing technology, offering valuable
benefits in terms of user convenience, efficiency, and engagement. Through
ongoing innovation and collaboration, we can unlock the full potential of
predictive text technology and shape the future of human-computer interaction
in the digital age.

BIBLIOGRAPHY
 APPENDICES
APPENDICE
A.DATA FLOW DIAGRAM

B.SAMPLE CODE
C.SAMPLE INPUT
D.SAMPLE OUTPUT