CCS366 SOFTWARE TESTING AND

AUTOMATION

Nirmala Santiago
ASP/CSE
LICET
COURSE OBJECTIVES AND OUTCOMES
·To understand the basics of software
testing
· To learn how to do the testing and planning
effectively
·To build test cases and execute them
·To focus on wide aspects of testing and
understanding multiple facets of testing
·To get an insight about test automation and
the tools used for test automation
CO1: Understand the basic concepts of
software testing and the need for software
testing
CO2: Design Test planning and different
activities involved in test planning
CO3: Design effective test cases that can
uncover critical defects in the application
CO4: Carry out advanced types of testing
CO5: Automate the software testing using
Selenium and Testing
UNIT I FOUNDATIONS OF SOFTWARE TESTING

● Why do we test Software?,

● Black-Box Testing and White-Box Testing,
● Software Testing Life Cycle,
● V-model of Software Testing,
● Program Correctness and Verification,
● Reliability versus Safety,
● Failures, Errors and Faults (Defects),
● Software Testing Principles,
● Program Inspections,
● Stages of Testing: Unit Testing, Integration Testing, System Testing
 Why do we test Software? - Case studies

Ariane 5 Flight 501, 4 June 1996

Ref: wikipediahttps://en.wikipedia.org/wiki/Ariane_flight_V88#Launch_failure
Ref: https://www.youtube.com/watch?v=2CP5zifJYDs
https://www.youtube.com/watch?v=2ngVl6iO94c
Y2K Problem
The USA Star-Wars Program
Failure of London Ambulance System
USS Yorktown Incidenthttps://www.youtube.com/watch?v=IFTUpnItt4A
Testing helps prevent downtime and ensures that users
have a positive experience with the software.
Software build process
TERMINOLOGIES
Software Fault: A static defect in the software.

Software Error: An incorrect internal state that is the manifestation of some fault.

Software Failure: External, incorrect behavior with respect to the requirements or another
description of the expected behavior.
GOALS OF TESTING SOFTWARE
Verification: The process of determining whether the products of a phase of the software
development process fulfill the requirements established during the previous phase.

Validation: The process of evaluating software at the end of software development to

ensure compliance with intended usage.
SOFTWARE TESTING - DEFINITIONS
● Testing is the process to ensure whether the software meets the customer / user
requirements
● Testing is the process of demonstrating that errors are not present
● The purpose of testing is to show that a program performs its intended functions correctly
● Testing is the process of of establishing confidence that a program does what it is
supposed to do.
● Testing is the process of of executing the program with the intent of finding faults
● Testing is a group of techniques to determine the correctness of the application

Testing includes an examination of code and also the execution of code in various
environments, conditions as well as all the examining aspects of the code
Attributes of software

Software testing is a process of identifying the

correctness of software by considering its all attributes
(Reliability, Scalability, Portability, Re-usability,
Usability) and evaluating the execution of software
components to find the software bugs or errors or
defects.
Types of testing contd..
● Manual testing
○ Software testing process in which test cases are executed manually without using any automated tool
○ It ensures whether the application is working, as mentioned in the requirement document or not.
○ Test cases are planned and implemented to complete almost 100 percent of the software application
○ Test case reports are also generated manually.
○ It can find both visible and hidden defects of the software.
○ Manual testing is mandatory for every newly developed software before automated testing.
 Types of Manual Testing

Black box testing The white box testing Gray Box testing
The black box testing is is done by Developer, Gray box testing is a
done by the Test They check every line combination of white
Engineer, where they of a code before giving box and Black box
can check the it to the Test Engineer. testing. It can be
functionality of an Since the code is performed by a person
application or the visible for the who knew both coding
software according to Developer during the and testing. And if the
the customer /client's testing, that's why it is single person performs
needs. In this, the code also known as White white box, as well as
is not visible while box testing. black-box testing for
performing the testing; the application, is
that's why it is known known as Gray box
as black-box testing. testing.
How to perform Manual Testing

● First, tester observes all documents related to software, to select testing areas.
● Tester analyses requirement documents to cover all requirements stated by the
customer.
● Tester develops the test cases according to the requirement document.
● All test cases are executed manually by using Black box testing and white box
testing.
● If bugs occurred then the testing team informs the development team.
● The Development team fixes bugs and handed software to the testing team for a
retest.
Advantages and disadvantages of manual testing
Advantages of Manual Testing
It does not require programming knowledge
while using the Black box method.
It is used to test dynamically changing GUI
designs.
Tester interacts with software as a real user
so that they are able to discover usability and
user interface issues.
It ensures that the software is a hundred
percent bug-free.
It is cost-effective.
Easy to learn for new testers.
Disadvantages of Manual Testing
It requires a large number of human
resources.
It is very time-consuming.
Tester develops test cases based on their
skills and experience. There is no evidence
that they have covered all functions or not.
Test cases cannot be used again. Need to
develop separate test cases for each new
software.
It does not provide testing on all aspects of
testing.
Since two teams work together, sometimes it
is difficult to understand each other's motives,
it can mislead the process.
Manual testing tools
 LoadRunner Citrus
It is most commonly used performance testing tools. Citrus is an integration testing tool, which is the most
LoadRunner is mainly used to support performance testing commonly used test framework. It is written in Java
for the wide range of procedures, number of approaches, and programming language. It is mostly used to request
application environments. and respond to server-side and client-side and validate
The main purpose of executing the LoadRunner tool is to the XML JSON files.
classify the most common sources of performance issues To accomplish the end-to-end use case testing, citrus
quickly. supports several HTTP, JMS, and SOAP protocols.
Features of LoadRunner
• LoadRunner tool contains n-numbers of applications,
which reduces the time to understand and describe the
reports.
• We can get thorough performance test reports by using
the LoadRunner tool.
• It will reduce the cost of distributed load testing and also
offer the operational tool for deployment tracking.
Citrus contd..
● Characteristics of Citrus
● Following are some of the important features of Citrus tool:
● It is used to send and receive messages.
● Citrus is available as both an open-source and a licensed in the market.
● It delivers a low-cost solution.
● We can authenticate the database by using the citrus tool.
● It will describe the sequence of messages, offer the test plan, and document the test
coverage.
● It creates the message and verifies the response
ZAP is an open-source web application security scanner. It is stands for
Zed Attack Proxy.
Just like some other tools, it is also written in the JAVA programming language.
It is the most effective Open Web Application Security Projects [OWASP].
Features of ZAP
● It supports many operating systems such as Windows, Linux, OS X.
● It has a plugin-based architecture.
● It contains an online marketplace that permits us to add new or updated features.
● ZAP's GUI control panel is easy to use.
 ○ NUnit is one of the most frequently used unit testing tools. It is an open-source tool and primarily derived
from the JUnit.
○ It was completely written in the C# programming language and suitable for all .Net languages.
○ In other words, we can say that the NUnit tool is entirely redesigned to become the advantage of many .Net
language qualities. For example:
○ Reflection-related capabilities.
○ Other custom attributes.
● Characteristics
○ It allows the assertions as a static method of the advantage class.
○ It sustains the data-driven tests.
○ It supports several platforms, like .NET core Xamarin mobile, Silverlight, and efficient framework.
○ The ability of NUnit help us to execute the tests simultaneously.
○ It uses a console runner to load and execute the tests.
● Features of JIRA
● It is a time-saving tool.
● Jira is used to track the
defects and issues.
● It is used to establish the
documentation tasks.
● Jira is a very useful tool in
tracking the improvement of
our documentation.
Another testing tool of manual testing is SonarQube, which
improves our workflow with continuous code quality and code
security. It is flexible with the use of plug-ins.
It is completely written in the JAVA programming
language. It offers fully automated evaluation and
integration with Ant, Maven, Gradle, MSBuild, and
constant integration tools. SonarQube has the ability to
record a metrics history and gives the evolution graph.

● JMeter is an open-source tool that is used to
test the performance of both static and
dynamic resources and dynamic web
applications.
● It is completely designed on the JAVA
application to load the functional test
behavior and measure the application's
performance.
● It facilitates users or developers to use the
source code for the development of other
applications.
Features of JMeter
Below are some of the essential characteristics of JMeter:
• It is platform-independent, which accepts a JVM
like Windows, Mac, and Linux, etc.
• It supports a user-friendly GUI, which is interactive and
straightforward.
• It is incredibly extensible to load the performance test in
multiple types of servers.
Bugzilla is an open-source tool that helps the customer and the client to keep track of the
defects. Bugzilla is also considered a test management tool because in this, we can easily
link other test case management tools such as ALM, Quality Centre, etc.
Features of Bugzilla
● Bugzilla has some additional features which help us to report the bug easily:
● It supports various operating systems such as Windows, Linux, and Mac.
● With the help of Bugzilla, we can list a bug in several formats.
● User preferences can measure email notification.
● Bugzilla has advanced searching capabilities.
Mantis is a web-based bug tracking system. ManitsBT stands for Mantis Bug Tracker.
It is used to follow the software defects and performed in the PHP programming
language. It is also an open-source tool.
Features of Mantis
● Some of the standard features of the particular tool are as follows:
● With the help of this tool, we have full-text search accessibility.
● Audit trails of changes made to issues.
● It provides the revision control system integration.
● Revision control of text fields and notes

Tessy is used to perform the integration and unit testing for the
embedded software.
It also helps us to discover the code coverage of the software or
an application.
It can easily manage the entire test organization, including
business needs, test management, coverage quantity, and
traceability.
Features of TESSY
The standard features of the TESSY are as follows:
• It produces the test report for the test execution
results.
• It supports various programming languages such
as C and C++.
• Tessy is used to evaluate the interface of the
function and describes the variable used by that
function.

Tessy contains three primary functions, which are as follows:

Test Interface Editor (TIE)

Test Data Editor (TDE)
Workspace.
Automation Testing

Automation testing refers to the automatic testing

of the software in which developer or tester write
the test script once with the help of testing tools
and framework and run it on the software. The
test script automatically test the software without
human intervention and shows the result (either
error, bugs are present or software is free from
them
Manual testing Automation testing
Testing in which a human tester executes test cases In automation testing, automation tools are used to
execute the test cases
In this testing, human resources are involved, that's why It is much faster than the manual testing
it is time-consuming
It is repetitive and error-prone Here automated tools are used that make it interesting
and accurate
BVT (build verification testing) is time-consuming and It's easy to build verification testing
tough in manual testing
Instead of frameworks, this testing use checklist, Frameworks like keyword, hybrid, and data drive to
guidelines, and stringent process for drafting test cases. accelerate the automation process.
The process turnaround time is higher than the It completes a single round of testing within record time;
automation testing process (one testing cycle takes lots therefore, a process turnaround time is much lower than
of time) a manual testing process.
The main goal of manual testing is user-friendliness or Automation testing can only guarantee a positive
improved customer experience. customer experience and user-friendliness.
It is best for usability, exploratory and adhoc testing It is widely used for performing testing, load testing and
regression testing.
Low return on investment The high return on investment
Automation testing types
Functional testing

● The first test performed by tester on newly revised software is called functional
testing, which verifies all the software functions' features per user requirement. This
testing works on the real-world business application and obtaining the expected
output from a given input. All application functions are tested and involve smoke,
unit, and integration testing.
Unit testing

The unit testing technique is

divided into three broad
categories: White box testing,
Black box testing and Grey
box testing.
● Integration testing
● Integration testing is more complicated to set up compared with other tests. All the
modules of the application communicate with each other to perform tasks. Therefore,
testers group them for testing and exposing the flaws in maintaining the interaction
between these modules. Another name for this testing is I&T or string testing,
considered end-to-end.
● Smoke testing
● This testing checks and defines the product's stability (whether stable or not). If the
product result is unstable, it is called an 'unstable build' and sent back to developers,
where they run more test cases to find out the root cause of the problem.
Non-functional testing

Non-functional testing focuses on how well application functions are doing, not on what the product does. It is the
opposite of functional testing, where application elements like reliability, usability, performance, etc., are tested. Some
types of non-functional testing are reliability testing, load testing, compatibility testing, performance testing, security
testing etc.
Performance testing
This non-functional testing tests the software's stability, responsiveness and speed under the workload. It finds out the
potential issues faced by critical software and medical programs used by the user, like slow operation of software under
stressful circumstances. It finds hurdles in the performance of software and removes them to increase the ability of
software to deliver the best results to the end user.
Regression testing
When some changes are made to the code of software or application, it needs to be tested to determine
whether the software is working as before the change; for this purpose, testers use automation regression
testing to automate scripts, applications of workflows, plans and other activities. It tests the system or
software workflow after its updation and functional error.
Non-functional testing

Keyword driven testing

Keyword-driven testing tests the application using the data files consisting of the keywords related to the
application, representing a set of actions needed to carry out the step. Here these specific keywords are identified
and connected with the specific action. Therefore during testing, when these keywords are used, their related
actions will automatically be done. This keyword testing is a popular choice for many businesses as it's flexible,
concise, easy to maintain and reusable. Keyword-driven testing is compatible with all kinds of automation tools
in the market. Instead of programming experts, functional testers can plan the testing before the application is
fully developed.
Data-driven testing
In data-driven testing, automation is inbuilt and very effective due to the few facilities provided, like the
reusability of code, change in the script doesn't affect the test cases, and this testing can be carried out in the
phase of the software development cycle. It provides consistency in results and reduces the investment of time
and resources. Test cases use the data separately stored in the table or spreadsheet format, and testers have
multiple data sets for testing.
Software testing principles
Objectives of testing and classifications
Black box testing is mainly focused on testing the functionality of the software, ensuring
that it meets the requirements and specifications

White box testing is mainly focused on ensuring that the internal code of the software is
correct and efficient.
Differences between Black Box Testing vs White Box Testing
Black Box Testing is a software testing
method in which the internal
structure/design/implementation of the
item being tested is not known to the
tester.
Only the external design and structure
are tested.
The tester only focuses on the input
and output of the software.
White Box Testing is a software
testing method in which the internal
structure/design/implementation of
the item being tested is known to the
tester.
Implementation and impact of the
code are tested
i.e individual code snippets,
algorithms and methods.
 Black Box Testing White Box Testing

It is a way of software testing in which the internal structure or It is a way of testing the software in which the tester has knowledge
the program or the code is hidden and nothing is known about it. about the internal structure or the code or the program of the software.

Implementation of code is not needed for black box testing. Code implementation is necessary for white box testing.

It is mostly done by software testers. It is mostly done by software developers.

No knowledge of implementation is needed. Knowledge of implementation is required.

It can be referred to as outer or external software testing. It is the inner or the internal software testing.

It is a functional test of the software. It is a structural test of the software.

This testing can be initiated based on the requirement This type of testing of software is started after a detail design
specifications document. document.
 No knowledge of programming is required. It is mandatory to have knowledge of programming.

It is the behavior testing of the software. It is the logic testing of the software.

It is applicable to the higher levels of testing of software. It is generally applicable to the lower levels of software testing.

It is also called closed testing. It is also called as clear box testing.

It is least time consuming. It is most time consuming.

It is not suitable or preferred for algorithm testing. It is suitable for algorithm testing.

Can be done by trial and error ways and methods. Data domains along with inner or internal boundaries can be better tested.

Example: Search something on google by using keywords Example: By input to check and verify loops
 Black-box test design techniques-
White-box test design techniques-
● Decision table testing
● Control flow testing
● All-pairs testing
● Data flow testing
● Equivalence partitioning
● Branch testing
● Error guessing

Types of Black Box Testing: Types of White Box Testing:

● Functional Testing ● Path Testing

● Non-functional testing ● Loop Testing
● Regression Testing ● Condition testing

It is less exhaustive as compared to white box testing. It is comparatively more exhaustive than black box testing.
Testing methods: Black box testing uses methods like equivalence partitioning,
boundary value analysis, and error guessing to create test cases. Whereas, white
box testing uses methods like control flow testing, data flow testing and statement
coverage.

Scope: Black box testing is generally used for testing the software at the functional
level. White box testing is used for testing the software at the unit level, integration
level and system level.
Advantages and disadvantages
Black box testing is easy to use, requires no programming knowledge and is effective in
detecting functional issues. However, it may miss some important internal defects that are
not related to functionality.

White box testing is effective in detecting internal defects, and ensures that the code is
efficient and maintainable. However, it requires programming knowledge and can be time-
consuming.
Types of Black box testing
● Functional testing
● Non-functional testing
Goal of functional testing
● The purpose of the functional testing is to check the primary entry function,
necessarily usable function, the flow of screen GUI.
● Functional testing displays the error message so that the user can easily navigate
throughout the application
What to test in functional testing?

● The main objective of functional testing is checking the functionality of the software
system. It concentrates on:
● Basic Usability: Functional Testing involves the usability testing of the system. It
checks whether a user can navigate freely without any difficulty through screens.
● Accessibility: Functional testing test the accessibility of the function.
● Mainline function: It focuses on testing the main feature.
● Error Condition: Functional testing is used to check the error condition. It checks
whether the error message displayed.
 Processing steps in functional testing
• Tester does verification of the requirement specification in the
software application.

• After analysis, the requirement specification tester will make a plan.

• After planning the tests, the tester will design the test case.

• After designing the test, case tester will make a document of the
traceability matrix.

• The tester will execute the test case design.

• Analysis of the coverage to examine the covered testing area of the

application.

• Defect management should do to manage defect resolving.

Steps to perform functional testing
● There is a need to understand the
software requirement.
● Identify test input data
● Compute the expected outcome
with the selected input values.
● Execute test cases
● Comparison between the actual and
the computed result
Types of functional testing
● Unit Testing:
A type of software testing, where the
individual unit or component of the
software tested.
By unit testing functional testing also
done, because unit testing ensures each
module is working correctly.
The developer does unit testing during
development phase of the application.
● Smoke Testing: Functional testing by smoke testing. Smoke testing includes only
the basic (feature) functionality of the system. Smoke testing is known as "Build
Verification Testing." Smoke testing aims to ensure that the most important function
work.
○ For example, Smoke testing verifies that the application launches successfully will check that GUI is
responsive.
● Sanity Testing: Sanity testing involves the entire high-level business scenario is
working correctly. Sanity testing is done to check the functionality/bugs fixed. Sanity
testing is little advance than smoke testing.
○ For example, login is working fine; all the buttons are working correctly; after clicking on the button
navigation of the page is done or not.
● Regression Testing: This type of testing concentrate to make sure that the code
changes should not side effect the existing functionality of the system. Regression
testing specifies when bug arises in the system after fixing the bug, regression testing
concentrate on that all parts are working or not. Regression testing focuses on is there
any impact on the system.
● Integration Testing: Integration testing combined individual units and tested as a
group. The purpose of this testing is to expose the faults in the interaction between
the integrated units.
● Developers and testers perform integration testing.
Bug Defect Fault Error Failure
When we have When the application The fault may occur The Problem in code Many defects lead
some coding error, it is not working as per in software leads to errors, to the software's
leads a program to the requirement is because it has not which means that a failure, which
its breakdown, knows as defects. added the code for mistake can occur means that a loss
which is known as a fault tolerance, due to the specifies a fatal
bug making an developer's coding issue in software/
the bug announced application act up. error as the application or in its
Test engineers call by Fault occurs due to: developer module, which
it a bug the programmer and 1. Lack of misunderstood the makes the system
inside the code is resources requirement or the unresponsive or
called a Defect. 2. An invalid step requirement was not broken.
3. Inappropriate defined correctly. if an end-user
data definition The developers use detects an issue in
the term error. the product, then
that particular issue
is called a failure.
Software Testing Life Cycle (STLC)

Systematic approach to testing a software application to ensure that it meets the

requirements and is free of defects.
A series of steps or phases, and each phase has specific objectives and deliverables
To ensure that the software is of high quality, reliable, and meets the needs of the end-
users.
Goal of STLC
The main goal of the STLC is to identify and document any defects or issues in the
software application as early as possible in the development process. This allows for
issues to be addressed and resolved before the software is released to the public.
Characteristics of STLC
STLC is a fundamental part of the Software Development Life Cycle (SDLC) but STLC
consists of only the testing phases.
STLC starts as soon as requirements are defined or software requirement document is
shared by stakeholders.
STLC yields a step-by-step process to ensure quality software.
Watch this video https://www.youtube.com/watch?v=OFAplk_DQto
Phases of STLC
Requirement analysis
● Reviewing the software requirements document (SRD) and other related documents
● Interviewing stakeholders to gather additional information
● Identifying any ambiguities or inconsistencies in the requirements
● Identifying any missing or incomplete requirements
● Identifying any potential risks or issues that may impact the testing process
Creating a requirement traceability matrix (RTM) to map requirements to test cases
Test Planning

● Identifying the testing objectives and scope

● Developing a test strategy: selecting the testing methods and techniques that will be
used
● Identifying the testing environment and resources needed
● Identifying the test cases that will be executed and the test data that will be used
● Estimating the time and cost required for testing
● Identifying the test deliverables and milestones
● Assigning roles and responsibilities to the testing team
● Reviewing and approving the test plan
Test Case Development

● Identifying the test cases that will be developed

● Writing test cases that are clear, concise, and easy to understand
● Creating test data and test scenarios that will be used in the test cases
● Identifying the expected results for each test case
● Reviewing and validating the test cases
● Updating the requirement traceability matrix (RTM) to map requirements to test
cases
Test Environment Setup
Test environment setup is a vital part of the STLC.
Test environment decides the conditions on which software is tested.
This is independent activity and can be started along with test case development.
In this process, the testing team is not involved. either the developer or the customer
creates the testing environment
Test Execution
Test execution: The test cases and scripts created in the test design stage are run against
the software application to identify any defects or issues.
Defect logging: Any defects or issues that are found during test execution are logged in a
defect tracking system, along with details such as the severity, priority, and description of
the issue.
Test data preparation: Test data is prepared and loaded into the system for test execution
Test environment setup: The necessary hardware, software, and network configurations
are set up for test execution
Contd..
Test execution: The test cases and scripts are run, and the results are collected and
analyzed.
Test result analysis: The results of the test execution are analyzed to determine the
software’s performance and identify any defects or issues.
Defect retesting: Any defects that are identified during test execution are retested to ensure
that they have been fixed correctly.
Test Reporting: Test results are documented and reported to the relevant stakeholders.
 Test Closure
Final stage of the Software Testing Life Cycle (STLC) where all testing-related activities are
completed and documented so that software quality and reliability is ensured.
The main objective of the test closure stage is to ensure that all testing-related activities have been
completed and that the software is ready for release.
Test summary report: A report is created that summarizes the overall testing process, including the
number of test cases executed, the number of defects found, and the overall pass/fail rate.
Defect tracking: All defects that were identified during testing are tracked and managed until they
are resolved.
Test environment clean-up: The test environment is cleaned up, and all test data and test artifacts
are archived.
Contd..
Test closure report: A report is created that documents all the testing-related
activities that took place, including the testing objectives, scope, schedule, and
resources used.
Knowledge transfer: Knowledge about the software and testing process is shared
with the rest of the team and any stakeholders who may need to maintain or
support the software in the future.
Feedback and improvements: Feedback from the testing process is collected and
used to improve future testing processes
Self study

Test cases for the program ‘Minimum’

Why should we test?
Fundamental principles of
software engineering:
Code and fix