Progress Report: Text Simplification Using Machine

Learning Techniques
Date: 15/05/2023

Project: Text Simplification Using Machine Learning Techniques

Project Lead: Indrani Paul Roy, Divyanshu Kaushik, Pratik Kumar

Project Status: On Track

Summary: This progress report provides an update on the project "Text

Simplification Using Machine Learning Techniques." The project aims to develop a
system that can simplify complex texts using various machine learning approaches.
The objective is to make text more accessible and comprehensible for a wider
range of readers. In this report, we highlight the accomplishments related to
tokenization, stopwords removal, POS tagging, and complex sentence
identification.

Accomplishments:

1. Tokenization:
 Successfully implemented tokenization techniques to break down the
input text into individual tokens or words.
 Utilized popular tokenization libraries or algorithms, such as NLTK,
SpaCy, or regular expressions, to handle various tokenization
challenges.
 Ensured that the tokenization process preserved the integrity and
meaning of the original text.
2. Stopwords Removal:
 Developed a stopwords removal mechanism to eliminate common and
insignificant words from the tokenized text.
 Identified and utilized appropriate stopwords lists, either from
existing libraries or by creating custom lists tailored to the project's
requirements.
 Improved the quality of the text simplification process by eliminating
noise and focusing on more meaningful words.
3. POS Tagging:
 Implemented part-of-speech (POS) tagging to assign grammatical tags
to each token in the text.
 Leveraged existing POS tagging libraries, such as NLTK or SpaCy, to
accurately assign POS tags based on context.
 Utilized the POS tags to gain insights into the syntactic structure of
the text, which can be helpful for subsequent simplification steps.
4. Complex Sentence Identification:
 Developed an algorithm or methodology to identify complex
sentences within the input text.
  Utilized linguistic or structural features, such as sentence length,
subordination, or syntactic complexity, to identify sentences that
require simplification.
 Successfully identified complex sentences to prioritize them for
further simplification efforts.

Challenges Faced:

1. Language Variations: Dealing with language variations, including idiomatic

expressions, compound words, or specific domain terminologies, posed
challenges in the tokenization process.
2. Ambiguity in POS Tagging: Resolving ambiguities in POS tagging, especially
for words with multiple possible tags based on context, required careful
consideration and fine-tuning of the tagging algorithms.
3. Complex Sentence Identification: Developing an algorithm or methodology
that accurately identifies complex sentences within a text proved
challenging, as complexity can be subjective and context-dependent.

Next Steps:

Simplification Strategies: Explore and implement machine learning techniques,

such as rule-based or sequence-to-sequence models, to simplify the identified
complex sentences.

Evaluation and Refinement: Evaluate the effectiveness of the simplification

process by comparing the simplified sentences with reference or gold standard
sentences. Refine the simplification strategies based on evaluation results and user
feedback.

Integration and User Interface: Integrate the tokenization, stopwords removal,

POS tagging, and complex sentence identification components into a cohesive
system. Develop a user-friendly interface for users to input complex texts and
receive simplified versions.

Performance Optimization: Optimize the efficiency and speed of the text

simplification system by identifying potential bottlenecks and implementing
performance improvements. Consider techniques like parallel processing or
algorithmic optimizations.

Simplification Strategies: Explore and implement various simplification strategies

to transform the identified complex sentences into simpler, more accessible
versions. Consider approaches such as sentence splitting, paraphrasing,
substitution, or simplification rules based on linguistic patterns. Experiment with
different techniques and evaluate their effectiveness in achieving the desired
simplification goals.

Evaluation and Refinement: Develop an evaluation framework to assess the

quality of the simplified texts. Compare the output of the simplification strategies
against reference or gold standard sentences to measure factors like simplicity,
coherence, and readability. Gather feedback from users and domain experts to
gain insights into the strengths and weaknesses of the simplification system. Use
this feedback to refine and improve the simplification strategies iteratively.

Documentation and Reporting: Maintain detailed documentation of the project's

progress, including methodologies, algorithms, datasets used, and experimental
results. Capture any modifications or improvements made to the original
techniques during the implementation process. Prepare a final project report
summarizing the entire development process, including the challenges faced,
solutions implemented, and future recommendations.

Conclusion: The project has made significant progress in implementing

tokenization, stopwords removal, POS tagging, and complex sentence
identification. By continuing with the next steps, such as developing simplification
strategies, evaluating and refining the system, integrating components, optimizing
performance, and documenting the project's findings, we are on track to create a
comprehensive text simplification system using machine learning techniques.