See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/335058609

Classifying non-functional requirements using RNN variants for quality

software development

Conference Paper · August 2019

DOI: 10.1145/3340482.3342745

CITATIONS READS
33 1,277

4 authors:

Md. Abdur Rahman Ariful Haque

University of Dhaka Masaryk University
19 PUBLICATIONS 185 CITATIONS 29 PUBLICATIONS 88 CITATIONS

SEE PROFILE SEE PROFILE

Md. Nurul Ahad Tawhid Md Saeed Siddik

Victoria University Melbourne University of Dhaka
22 PUBLICATIONS 177 CITATIONS 31 PUBLICATIONS 234 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Md. Nurul Ahad Tawhid on 09 September 2019.

The user has requested enhancement of the downloaded file.

 Classifying Non-functional Requirements using RNN Variants
for Quality Software Development
Md. Abdur Rahman
Centre for Advanced Research in Sciences
University of Dhaka, Bangladesh
mukul.arahman@gmail.com
Md. Nurul Ahad Tawhid
Institute of Information Technology
University of Dhaka, Bangladesh
tawhid@iit.du.ac.bd
ABSTRACT
Non-Functional Requirements (NFR), a set of quality attributes,
required for software architectural design. Which are usually scat-
tered in SRS and must be extracted for quality software development
to meet user expectations. Researchers show that functional and
non-functional requirements are mixed together within the same
SRS, which requires a mammoth effort for distinguishing them. Au-
tomatic NFR classification would be a feasible way to characterize
those requirements, where several techniques have been recom-
mended e.g. IR, linguistic knowledge, etc. However, conventional
supervised machine learning methods suffered for word represen-
tation problem and usually required hand-crafted features, which
will be overcome by proposed research using RNN variants to cate-
gories NFR. The NFR are interrelated and one task happens after
another, which is the ideal situation for RNN. In this approach, re-
quirements are processed to eliminate unnecessary contents, which
are used to extract features using word2vec to fed as input of RNN
variants LSTM and GRU. Performance has been evaluated using
PROMISE dataset considering several statistical analysis. Among
those models, precision, recall, and f1-score of LSTM validation
are 0.973, 0.967 and 0.966 respectively, which is higher over CNN
and GRU models. LSTM also correctly classified minimum 60% and
maximum 80% unseen requirements. In addition, classification ac-
curacy of LSTM is 6.1% better than GRU, which concluded that
RNN variants can lead to better classification results, and LSTM is
more suitable for NFR classification from textual requirements.
CCS CONCEPTS
• Software and its engineering → Requirements analysis.
KEYWORDS
Non-Functional Requirements, NLP, Deep Learning, RNN
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
MaLTeSQuE ’19, August 27, 2019, Tallinn, Estonia
© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-6855-1/19/08. . . $15.00
https://doi.org/10.1145/3340482.3342745
25
Md. Ariful Haque
Institute of Information Technology
University of Dhaka, Bangladesh
arifulmit17@gmail.com
Md. Saeed Siddik
Institute of Information Technology
University of Dhaka, Bangladesh
saeed.siddik@iit.du.ac.bd
ACM Reference Format:
Md. Abdur Rahman, Md. Ariful Haque, Md. Nurul Ahad Tawhid, and Md.
Saeed Siddik. 2019. Classifying Non-functional Requirements using RNN
Variants for Quality Software Development. In Proceedings of the 3rd ACM
SIGSOFT International Workshop on Machine Learning Techniques for Soft-
ware Quality Evaluation (MaLTeSQuE ’19), August 27, 2019, Tallinn, Estonia.
ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3340482.3342745
1 INTRODUCTION
Software requirements specification is the most important artifacts
that describe the features and behaviors of software application.
It includes a variety of components to define the intended func-
tionality required by users, which are composed of Functional Re-
quirements (FR) and Non-Functional Requirements (NFR). Where,
functional requirements describe software behavior directly listed
by stakeholders and the non-functionals are the expected non listed
requirements to perform functional tasks [1]. The functional and
non-functional requirements are mingled within the same docu-
ments, which are hard to detect manually. Since NFR are not given
by stakeholders, researchers showed that unaddressed and inap-
propriate NFR classification leads to project failure or increase
production cost, which may affect software quality evolution by
frequent change request [2, 3]. Therefore, early NFR detection en-
sures quality software development and development cost reduc-
tion. However, manual NFR classification escalates development
time and maintenance cost, which may ruin software quality.
For automatic textual data classification, various approaches
have been proposed such as information retrieval [2], linguistic
knowledge [4], machine learning [5], etc. Among those, very little
have been found to direct requirement classifying techniques, where
IR based NFR classification strategy was presented by [2]. Textual
requirements was classified as functional and non-functional us-
ing linguistic knowledge [4]. In 2017, Matsumato et. al proposed
method to identify ambiguity, inconsistency, incompleteness, and
redundancy of NFR in software requirement specification [6]. Lu et.
al proposed a new technique to classify app user reviews into three
categories named as functional, non-functional and others [7].
However, all of those models were developed using machine
learning algorithms, which usually stacked on prerequisite hand-
crafted features [8]. More significantly, word representation in ma-
chine learning approach has several problems such as context lost,
sparse representation, arbitrary encodings, etc. For example, Dhaka
MaLTeSQuE ’19, August 27, 2019, Tallinn, Estonia
and Estonia may be represented as Id224 and Id453 respectively,
meaning the 453r d entry of the long sparse vector is 1. Such repre-
sentation does not provide any useful information to the system
regarding similarities between individual symbols. This machine
learning shortcoming would be overcome using deep learning based
distributed vector representation, which attempts to learn multiple
levels of representation for handling increased complexity.
In the literature, several machine learning approaches have been
reported for text and NFR classification [2, 4, 5]. Even though few
research found in deep learning for NFR classification, including
text data representation in low dimensional vector space using Con-
volutional Neural Network (CNN) [9]. However, hardly found any
research direction for NFR classification with Recurrent Neural Net-
work (RNN). Where, RNN is one of the most popular architecture
used in NLP, because its recurrent structure is suitable to process
variable length text [10]. RNN utilized distributed representations
of words by comprising each text into vectors to form a matrix.
This paper presents an efficient NFR classification technique
using RNN variants to categories requirements into pre-defined
labels. At first, the textual requirement has been processed to elim-
inate unnecessary text, symbols, etc. from the dataset. Then the
processed documents were vectorized using word2vec algorithm
to fed in the neural network model. The RNN variants named as
Long Short Term Memory (LSTM) [8] and Gated Recurrent Unit
(GRU) [11] are used for NFR model construction. The implemented
classifiers have been applied to categorize NFR into pre-defined
labels. Finally, proposed classifier performance has been evaluated
using PROMISE [12] dataset considering several statistical analyses.
The investigated validation and testing results of proposed mod-
els denoted that LSTM performs highest score over CNN and GRU.
The reported average precision, recall, f1-score for LSTM validation
are 0.973, 0.967 and 0.966 respectively which is high and indicate the
model was trained well. In the model, the boundary scores are 0.95
and 1.00, which indicate model’s minimum classification validity
is 95% and maximum is 100%. The reported average accuracy for
LSTM is always higher over GRU, where, LSTM’s lowest and high-
est accuracy are 0.60 and 0.80 respectively, that means minimum
60% and maximum 80% unseen requirements are correctly classified
by the model. The reported average precision, recall, f1-score and
accuracy are 71.7%, 71.5%, 70% and 71.5% respectively. However,
LSTM’s standard deviation is always lower than GRU. The low de-
viation indicates model’s well stability, which are 0.081, 0.062, 0.057
and 0.62 for precision, recall, f1-score and accuracy respectively.
The main contribution of this paper is to investigate how well deep
learning methods perform for multi-class NFR classification.
The rest of this paper is organized as follows; Section II discusses
the existing work related to this research. Section III and IV illus-
trates the proposed method and result analysis respectively. Finally,
section V concludes this paper with future research direction.
2 RELATED WORK
Classifying non-functional requirements using analysis of textual
natural language is an emerging field in software engineering re-
search to evolve software quality. NFR may classify using informa-
tion retrieval based [2], linguistic knowledge based [4], CNN based
[9], etc. approaches which are briefly discussed in this section.
26
Md. Abdur Rahman, Md. Ariful Haque, Md. Nurul Ahad Tawhid, and Md. Saeed Siddik
Information Retrieval (IR) technique plays a vital part for de-
tecting and classifying NFR, which was presented in [2]. In IR
approach, a set of indicator terms was identified for each NFR cate-
gory to classify user requirements. A probabilistic weight for each
potential indicator term was calculated from the requirements. This
strategy requires less effort for NFR classification compared to semi-
automated classification methods. However, this technique suffers
for evaluating correctness of candidate NFR manually.
Requirements were classified into functional and non-functional
using linguistic knowledge in [4]. The method works for sentence
level as the characteristics of functional and non-functional require-
ments remain within the scope of sentences. The results were cross
checked ten times, and showed that their method significantly im-
proved classification performance. However, more training and
testing data could be introduced to justify performance.
Using requirements frame model, a method was proposed to de-
tect the redundancy, ambiguity, inconsistency and incompleteness
of NFR in SRS [6]. The research focused on the requirements re-
lated to response time and usability. Here, the specific requirements
were retrieved using respective keywords and multiline statements
were converted to single sentences manually. However, this manual
statements conversion is time consuming and error-prone.
Combining different feature extraction and machine learning
techniques, a NFR classification method was presented to find out
the best pair by [5]. BoW and variation of TF-IDF were used as fea-
ture extraction strategies, where eight machine learning algorithms
were applied as NFR classifier. The empirical results showed that
TF-IDF (character level) combined with SDG SVM performed best to
categorise NFR into pre-defined levels. However, NFR classification
efficiency could be improved incorporating RNN and word2vec.
Functional and non-functional requirements were classified us-
ing supervised machine learning approach in [13], where both bi-
nary and multi-class classifiers have been introduced. The research
focused on accurately identification of four NFR named as usabil-
ity, security, operational and performance. Problem of imbalance
dataset was handled using under and over sampling techniques. For
feature extraction, BoW strategies were used, where part of speech
tags were identified as most informative features. The experimen-
tation was performed using support vector machine classifier.
A machine learning technique was presented to classify app user
reviews in [7], where BoW, TF-IDF, CHI2 and AUR-BoW techniques
were applied with NB, J48, and Bagging algorithms. The research
focused on four types of NFR names as reliability, usability, porta-
bility, and performance. The paper concluded that imbalance and
smaller dataset effects badly on classification results in machine
learning environment. However, no direction has been given to
handle mentioned problem regarding dataset.
Winkler and vogelsang [1] presented convolutional neural net-
works based approach to automatically classify content elements
of a natural language requirements specification. Their approach
can be used to classify content elements in documents. To train the
neural network, this model used 10K content elements extracted
from 89 requirements specifications from industry and reported
precision of 0.73 and a recall of 0.89. However, requirements are
interconnected and one task happens after another, where RNN
may perform better than CNN.
Classifying Non-functional Requirements using RNN Variants for Quality Software Development MaLTeSQuE ’19, August 27, 2019, Tallinn, Estonia

Using CNN model a method was proposed to classify NFR into

different pre-defined categories in [9], where text was converted
to low dimensional vector space using word2vec algorithm. The
requirement text was represented using embedding vectors to fed
into CNN model. Effectiveness of the proposed model was evaluated
using PROMISE dataset. The research applied k-fold cross validation
strategy to improve accuracy and reduce biasness in prediction. The
technique used different statistical analysis to compare proposed
model results with others. The results showed CNN can classify
NFR into different categories efficiently, however, RNN could be
incorporated for better accuracy.
The analysis of existing approaches concluded that different
strategies have been proposed for NFR classification such as ma-
chine learning based, CNN based, etc. However, very few research
found for automatic NFR classification using deep learning method.
In addition, no guideline has been found yet to classify NFR using
RNN, which is significant in NLP domain.

3 METHODOLOGY
Multi-class NFR classification using RNN has been proposed in
this framework to facilitate quality software development. This
automatic NFR classification method considers word2vec algorithm
to extract features from requirement dataset. On the other hand,
twice RNN variants named as LSTM and GRU have been applied as
NFR classifier. Textual software requirements have been processed
to eliminate unnecessary text, symbols, etc. and vectorized using
word2vec algorithm for RNN modeling. The whole working process
of this framework is divided into following four steps.
• Dataset Pre-processing
• Word Vectorization
• RNN Model Construction
• Model Training and Testing
The details of above steps have been elaborately described in the
following sub-sections.
3.1 Dataset Pre-processing
The first step of this framework is cleaning and pre-processing
software requirement dataset. This involves removing special char-
acters, stop words, case-folding, lemmatization and tokenization.
Special characters and symbols are usually non-alphanumeric char-
acters, which add extra noise to the experimental dataset. To remove
noisy data, special characters have been removed using regular ex-
pression. Words appear in upper-case or lowercase having similar
meaning in Enlgish; therefore, case-folding is applied for unique
case consideration. On the other hand, stop words usually refer
to the most common words such as "a", "an", "the", which do not
influence the semantics of a software requirement. The most uses
natural language processing stop words corpus of nltk has been
used in this implementation. Lemmatization approach has been
utilized to get base form of word. It also grouped the inflected
words together. Numbers are also converted to word in order to
enrich dataset. Finally, tokenization process splits longer text into
smaller piece or tokens. The requirements have been tokenized into
sentences which then split into distinct words.
Figure 1: Overview of the Proposed Method
3.2 Word Vectorization
The deep learning model could not understand natural language
sentence to process, therefore, a word vectorization technique is
required. This allows the model to recognize patterns, even if the
words occur in the pattern vary slightly. The requirement text has
been converted to word vectors using word2vec model with skip-
gram feature extraction method. Word2vec maps a single word to
a vector v ∈ R, where R is the set of real numbers. In word2vec
technique, vector distance of two given words is small if these two
words are used in similar context, otherwise it is large. The sentence
transformation in word2vec can be written as m ∈ Rn,l , where m,
R, n, and l represents the matrix, set of real number, embedding
vector size and length of sentence respectively.
This strategy uses google pretrained word embedding model,
where the model is trained on over 100 billion words. At the time
of tokenization, each unique word is turned into a unique number,
which is used as the respective word index. The vector of a particular
word is extracted from the pretrained word embedding, and kept
in an embedding matrix, which is initialized by zero as elements.
Then, these elements are replaced by the embedding vectors of a
particular word. Thus the embedding matrix contains all the word
embedding vectors and some zero vectors for the words that are
not found in the pretrained word embeddings.
The unique word vectors were fetched into an embedding matrix
which size is measured using vocabulary size and embedding di-
mension. The features of the documents are extracted as embedded
word vectors with dimension of 300 for each word in each sentence.
The embedding vectors were imported as pre-trained word vectors
and transfer learning is used to learn the word embedding. The

27
MaLTeSQuE ’19, August 27, 2019, Tallinn, Estonia
embedding matrix is fed as weights in the embedding layer in the
neural network.
3.3 RNN Model Construction
The two RNN variants model has been implemented in this NFR
classification framework using word2vec vectorization algorithm.
The RNN variants LSTM and GRU algorithms are prominent in nat-
ural language domain especially for text classification [11], which
are implemented in NFR classification framework. The sequential
RNN architecture mainly consists of three layers named as input
layer, hidden layer and output layer.
3.3.1 Input Layer: The first layer prepares word embeddings ac-
cording to model’s input requirement using sequence length and
embedding dimension. The parameters in the embedding layer are
not trainable to ensure proper transfer learning. For training deep
learning model, transfer learning is used in the first layer in order
to improve context learning of the word, where vector weights
remain unchanged.
3.3.2 Hidden Layer: LSTM and GRU algorithms have been used
in hidden layer, which takes each word embedding in a time step
and processes to produce an output. The dropout and recurrent
dropout has been applied here to reduce over-fitting of the model.
The algorithms learn the context of the sentence and pass the last
time step output to the dense layer activation function. The leaky
relu (1) activation function has been used in this case, where α is a
small constant.
z>0
 
z The OpenScience tera-PROMISE software requirement dataset [12]
LeakyReLU (z) = (1)
αz αz <= 0
3.3.3 Output Layer: This is the last layer of the model, where
final output for each requirement has been produced. This dense
layer contains softmax (2) activation function to help predicting
the target class. The softmax function calculates the probabilities
distribution of the requirement over pre-defined different categories.
This function produces unique probability for each class, where the
maximum probability will be used to predict the decision class. The
produced probability value ranges between 0 to 1, and the sum of all
the probabilities will be equal to 1. The Equation (2) computes the
exponential (e-power) of the given input and the sum of exponential
values of entire input set. Then the ratio of the exponential of the
input value and the sum of exponential values lead the output of
the softmax function.
exp(wyt x)
p(y|x) = Í t (2)
y exp(wy x)
The model was built using the sequential API of keras. This API
stacks the layers one over another and uses one-layer output as
input of next layer. The dimensions for embedding, LSTM/GRU and
Leaky Relu were 300. On the other hand, the softmax dense layer
was 10 which is equal to total number of targeted NFR category.
3.4 Model Training and Testing
The implemented LSTM and GRU models have been trained to
evaluate how effectively the proposed framework can identify NFR
28
Md. Abdur Rahman, Md. Ariful Haque, Md. Nurul Ahad Tawhid, and Md. Saeed Siddik
into different given categories. This process makes an effective pre-
diction based on constructing current model, calculating prediction
incorrectness, and updating network parameters to minimize this
error and make the model better. This process is repeated until the
model has converged and can no longer learn. Following parameters
have been used to tune this process.
Metric: This is used to measure performance of proposed model,
where accuracy has been used as metric value for this framework.
Loss function: This function is used to calculate a loss value
of the model in the training process. It attempts to minimize the
value by tuning the network weights. This framework uses the loss
function which is suitable for multiclass categorical classification.
Optimizer: This is a significant function in the model that de-
cides how the network weights will be updated based on the output
of the loss function.
The actual training happens using the fit method. In each training
iteration, batch size, sample number and weight are updated single
time. The training process completes an epoch, when the model has
seen the entire training dataset. At the end of each epoch, validation
dataset is used to evaluate model’s learning accuracy. This process
is repeated for a predetermined number of epochs.
4 EXPERIMENT AND RESULT ANALYSIS
The dataset and experiment for RNN variants algorithms are dis-
cussed in this section with the result analysis and efficiency com-
parison of different techniques for automatic NFR classification.
4.1 Dataset
has been used for experimental analysis containing both func-
tional and non-functional (11 categories) requirements. The dataset
is small in size which consists of 625 requirement sentences. In
these sentences 255 are functional and 370 are non-functional re-
quirements. The NFR are labeled with eleven categories named
as availability, legal, look and feel, maintainability, operational,
performance, scalability, security, usability, fault tolerance, and
portability. However, the dataset contains majority number of func-
tional requirements which indicates class distribution is imbalanced.
Therefore, sampling strategies for dealing with imbalances in data
has been applied, as balanced distribution can improve the classi-
fication accuracy. Also, one category containing single value has
been removed to reduce class distribution biasness in the dataset.
4.2 Experiment
In this framework, the word embedding matrix is used as weights
in the embedding layer. The labels were converted to integers using
a dictionary to hold unique classes which are used to convert the
labels into specific classes representing each number. The training
and validation sentences were tokenized to find all the unique
words. After tokenizing the sentences, zero padding was applied
according to the length of the longest sentence in the dataset.
The training, validation and testing requirement sentences were
converted to same length to ensure that same length sentences
are fed as input in the model. Where, the highest length of a sen-
tence over the dataset has been considered. Then the class labels
were encoded to integer in the range of the total classes minus
Classifying Non-functional Requirements using RNN Variants for Quality Software Development MaLTeSQuE ’19, August 27, 2019, Tallinn, Estonia

one. The testing dataset was selected manually to ensure that all and 0.73 respectively which covered by CNN experiment. On other
class instances are present. The training set is split using k-fold hand, LSTM’s lowest and highest score in Table 1 are 0.95 and 1.00,
cross validation process. Before training started, class weights were indicates the minimum classification validity is 95% and maximum
calculated to give the model more attention on the classes with low is 100%. The reported average precision, recall, f1-score for LSTM
number of instances. The model was fitted using the training and are 0.973, 0.967, and 0.966 respectively. Analysis of demonstrated
validation data, where the validation data size was equal to 1/k results showed that LSTM over performed than other models. In
of k-fold cross validation. Class weights were used to produce a addition, it presents a very well stability that are reflected in the low
model giving more attention to the underrepresented classes. At the standard deviations 0.015, 0.017 and 0.015 in precision, recall and
training time, the model takes labels as one-hot encoded vector for f1-score respectively. The validation performance was recalculated
producing separate probabilities for each class in output softmax after each epoch and weights were recursively updated.
function as Equation (2). The model was trained for several epochs
with a fixed batch size and output was kept in a history object.
The dropout probability was set to 0.5 in order to avoid overfit-
ting during training. The learning algorithm selected to this problem
was Adam [14] with learning rate λ = 0.001, β1 = 0.9, β2 = 0.999,
ϵ = 1e − 08 and the loss function was categorical cross-entropy.
The number of epochs to train the LSTM and GRU were fixed to
200. The trained model showed total 1,144,810 parameters, where
814,510 as trainable and 330,300 as non-trainable.

4.3 Evaluation Metric

Precision, recall, f1-score, and accuracy matrix have been used to
measure the model learning for classifying NFR. The precision,
recall and f1-score values are the average of used ten classes named
as availability, legal, look and feel, maintainability, operational,
performance, scalability, security, usability and fault tolerance.

4.4 Result Analysis

The labeled and unlabeled requirement dataset was used for training
and testing respectively. The model was trained with k-fold cross
validation, where k was set to 10. The obtained results of validation Figure 2: Validation Result of CNN, GRU and RNN
and testing for each fold has been plotted in Table 1 and Table 2
The graphical representation based on validation results for CNN,
respectively. The comparative experimental results for CNN [9],
GRU and LSTM has been depicted in Figure 2. Where, Y-axis and
GRU and LSTM are showed in Table 1 and plotted in Figure 2.
X-axis represents validation values and techniques respectively.
Where, GRU and LSTM test accuracy are reported in Table 2.
The box whisker plots for precision, recall and f1-score have been
grouped by the classifier methods. According to the plot, lower,
Table 1: CNN[9], GRU and RNN Results
middle and upper quartile starting range is always better for RNN
compared to CNN.
Precision Recall F1-score
Fold CNN GRU LSTM CNN GRU LSTM CNN GRU LSTM
1 0.80 0.95 0.98 0.79 0.95 0.97 0.77 0.95 0.97 Table 2: GRU and LSTM Test Scores
2 0.75 0.97 0.95 0.76 0.97 0.97 0.73 0.97 0.96
3 0.79 0.99 0.98 0.76 0.99 0.94 0.75 0.99 0.95 Precision Recall F1-score Accuracy
4 0.81 0.97 0.98 0.81 0.97 0.97 0.80 0.97 0.97 Fold GRU LSTM GRU LSTM GRU LSTM GRU LSTM
5 0.81 0.97 0.98 0.78 0.97 0.97 0.76 0.97 0.97 1 0.77 0.79 0.60 0.75 0.64 0.74 0.60 0.75
6 0.83 0.97 0.95 0.78 0.97 0.97 0.76 0.97 0.96 2 0.83 0.67 0.75 0.70 0.74 0.66 0.75 0.70
7 0.84 0.94 0.96 0.85 0.94 0.94 0.82 0.94 0.94 3 0.78 0.84 0.70 0.80 0.68 0.79 0.70 0.80
8 0.85 0.88 0.98 0.77 0.88 0.97 0.76 0.88 0.97 4 0.67 0.60 0.65 0.60 0.61 0.60 0.65 0.60
9 0.85 1.00 1.00 0.81 1.00 1.00 0.81 1.00 1.00 5 0.83 0.73 0.70 0.75 0.71 0.71 0.70 0.75
10 0.80 0.97 0.97 0.75 0.97 0.97 0.74 0.97 0.97 6 0.75 0.76 0.70 0.75 0.68 0.72 0.70 0.75
Avg 0.813 0.961 0.973 0.785 0.961 0.967 0.77 0.961 0.966 7 0.77 0.62 0.70 0.65 0.68 0.65 0.70 0.65
Std 0.032 0.033 0.015 0.027 0.033 0.017 0.03 0.033 0.015 8 0.30 0.78 0.35 0.75 0.27 0.73 0.35 0.75
9 0.72 0.63 0.65 0.65 0.66 0.65 0.65 0.65
10 0.75 0.75 0.70 0.75 0.68 0.75 0.70 0.75
According to Table 1, the RNN model’s validation score are
higher than CNN and GRU, which indicate that the model is trained Avg 0.717 0.717 0.65 0.715 0.635 0.70 0.65 0.715
well. In Table 1, the lowest precision, recall and f1-score are 0.75, 0.75 Std 0.153 0.081 0.113 0.062 0.133 0.057 0.113 0.062

29
 MaLTeSQuE ’19, August 27, 2019, Tallinn, Estonia
The testing results of two RNN variants GRU and LSTM have
been plotted in Table 2 and depicted in Figure 3. Testing results
comparison has been carried out on GRU and LSTM only in Ta-
ble 2, because comparing with Ju et al. [9] the accuracy of CNN
was unreported. The precision, recall, f1-score and accuracy were
calculated from true and predicted labels. As stated in Table 2, aver-
age scores are always higher for LSTM over GRU. Where, LSTM’s
lowest and highest accuracy are 0.60 and 0.80 respectively that
means the model can classify unseen requirements minimum 60%
and maximum 80% correctly. The reported average precision, recall,
f1-score and accuracy are 71.7%, 71.5%, 70% and 71.5% respectively.
On the other hand, the LSTM’s standard deviation is always lower
than GRU, which are 0.081, 0.062, 0.057 and 0.062 for precision, re-
call, f1-score and accuracy respectively. The low deviations indicate
LSTM model’s well stability.
Figure 3: RNN Test Result of GRU and RNN
The variants of RNN model test results are graphically repre-
sented in Figure 3. Where, Y-axis and X-axis represents testing
values and techniques respectively. The box whisker plots for sta-
tistical analysis named precision, recall, f1-score and accuracy have
been grouped by the RNN variants LSTM and GRU. According
the plot, an outlier has been found in GRU, but the LSTM plotted
as packed. The lower, middle and upper quartile starting range is
always better for LSTM compared to GRU.
This experiment covers three deep learning methods named CNN
[9], LSTM, and GRU for finding the optimal NFR classifier. After
analyzing entire experiment, LSTM performs best for categorizing
NFR to ensure quality software development.
4.5 Threats to Validity
The classification results have been obtained for 10 NFR categories.
However, during this work it was evident the lack of publicly avail-
able large datasets to applied deep earning in software requirement
classification. Therefore, that limits the experimentation of more
complex and interesting models.
30
View publication stats
Md. Abdur Rahman, Md. Ariful Haque, Md. Nurul Ahad Tawhid, and Md. Saeed Siddik
5 CONCLUSION
An effective requirement identification framework has been pro-
posed using RNN variants in order to classify NFR into pre-defined
categories. This automatic NFR classification method considers
word2vec algorithm and RNN deep learning model for feature ex-
traction and text classification respectively. The requirement text
has been processed to eliminate unnecessary text, symbols, etc.
from the dataset. The processed documents were vectorized using
word2vec algorithm to fed on RNN. The standard PROMISE dataset
has been used to evaluate the proposed model. The reported re-
sults showed that the model’s minimum classification validity is
95% and maximum is 100%. Where, the average testing precision,
recall, f1-score and accuracy are 71.7%, 71.5%, 70% and 71.5% re-
spectively. The LSTM model reports a very well stability that are
reflected in the low standard test score deviation (<0.082) in preci-
sion, recall and f-measure. According to the experimental results, it
can be concluded that deep learning algorithms especially LSTM
is useful on software non-functional requirements classification.
Incorporating large scale dataset and significant feature extraction
techniques such as glove, fastText, and ensemble classifier are the
future research direction of this framework.
REFERENCES
[1] Jonas Winkler and Andreas Vogelsang. Automatic classification of requirements
based on convolutional neural networks. In 2016 IEEE 24th International Require-
ments Engineering Conference Workshops (REW), pages 39–45. IEEE, 2016.
[2] J Cleland-Huang, R Settimi, X Zou, and P Solc. The detection and classification
of non-functional requirements with application to early aspects. In 14th Int.
Requirements Engineering Conference (RE’06), pages 39–48. IEEE, 2006.
[3] Mirza Rehenuma Tabassum, Md Saeed Siddik, Mohammad Shoyaib, and
Shah Mostafa Khaled. Determining interdependency among non-functional
requirements to reduce conflict. In 2014 International Conference on Informatics,
Electronics & Vision (ICIEV), pages 1–6. IEEE, 2014.
[4] I Hussain, L Kosseim, and O Ormandjieva. Using linguistic knowledge to classify
non-functional requirements in srs documents. In Int. Conf. on Application of
Natural Language to Information Systems, pages 287–298. Springer, 2008.
[5] Md. Ariful Haque, Md. Abdur Rahman, and Md Saeed Siddik. Non functional
requirements classification with machine learning: An empirical study. In Inter-
national Conference on Advances in Science, Engineering and Robotics Technology
(ICASERT), pages 629–633. IEEE, 2019.
[6] Y Matsumoto, S Shirai, and A Ohnishi. A method for verifying non-functional
requirements. Procedia Computer Science, 112:157–166, 2017.
[7] Mengmeng Lu and Peng Liang. Automatic classification of non-functional re-
quirements from augmented app user reviews. In 21st International Conference on
Evaluation and Assessment in Software Engineering, pages 344–353. ACM, 2017.
[8] Jenq-Haur Wang, Ting-Wei Liu, Xiong Luo, and Long Wang. An lstm approach
to short text sentiment classification with word embeddings. In Proceedings of the
30th Conference on Computational Linguistics and Speech Processing (ROCLING
2018), pages 214–223, 2018.
[9] N Almanza R, Reyes Ju, and Guillermo Licea. Towards supporting software
engineering using deep learning: A case of software requirements classification.
In 5th International Conference in Software Engineering Research and Innovation
(CONISOFT), pages 116–120. IEEE, 2017.
[10] P Zhou, Zhenyu Qi, S Zheng, Jiaming Xu, H Bao, and Bo Xu. Text classification
improved by integrating bidirectional lstm with two-dimensional max pooling.
In 26th Int. Conference on Computational Linguistics, pages 3485–3495, 2016.
[11] Jakub Nowak, Ahmet Taspinar, and Rafał Scherer. Lstm recurrent neural networks
for short text and sentiment classification. In International Conference on Artificial
Intelligence and Soft Computing, pages 553–562. Springer, 2017.
[12] Openscience tera-promise software requirement (last accessed: July 01, 2019).
URL https://terapromise.csc.ncsu.edu/!/#repo/view/head/requirements/nfr.
[13] Zijad Kurtanović and Walid Maalej. Automatically classifying functional and non-
functional requirements using supervised machine learning. In 25th International
Requirements Engineering Conference (RE), pages 490–495. IEEE, 2017.
[14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.
International Conference on Learning Representations, 12 2014.