Professional Documents
Culture Documents
Etl Testing Material
Etl Testing Material
Testing the backend databases like comparing the actual results with expected results.
Data base testing basically include the following.
1) Data validity testing.
2) Data Integrity testing
3) Performances related to database.
4) Testing of Procedure, triggers and functions.
For doing data validity testing you should be good in SQL queries
For data integrity testing you should know about referential integrity and different constraint.
For performance related things you should have idea about the table structure and design.
For testing Procedure triggers and functions you should be able to understand the same.
***************************************************************************
****
Below is the list of objects that are treated as essential for validation in ETL testing:
Verify that data transformation from source to destination works as expected
Verify that expected data is added in target system
Verify that all DB fields and field data is loaded without any truncation
Verify data checksum for record count match
Verify that for rejected data proper error logs are generated with all details
Verify NULL value fields
Verify that duplicate data is not loaded
Verify data integrity
***************************************************************************
So ETL Testing implies - Testing this entire process using a tool or at table level with the help of test
cases and Rules Mapping document.
In ETL Testing, the following are validated 1) Data File loads from Source system on to Source Tables.
2) The ETL Job that is designed to extract data from Source tables and then move them to staging
tables. (Transform process)
3) Data validation within the Staging tables to check all Mapping Rules / Transformation Rules are
followed.
4) Data Validation within Target tables to ensure data is present in required format and there is no
data
loss
from
Source
to
Target
tables.
Extract
In this step we extract data from different internal and external sources, structured and/or
unstructured. Plain queries are sent to the source systems, using native connections, message
queuing, ODBC or OLE-DB middleware. The data will be put in a so-called Staging Area (SA), usually
with the same structure as the source. In some cases we want only the data that is new or has been
changed, the queries will only return the changes. Some tools can do this automatically, providing a
changed data capture (CDC) mechanism.
Transform
Once the data is available in the Staging Area, it is all on one platform and one database. So we can
easily join and union tables, filter and sort the data using specific attributes, pivot to another
structure and make business calculations. In this step of the ETL process, we can check on data
quality and cleans the data if necessary. After having all the data prepared, we can choose to
implement slowly changing dimensions. In that case we want to keep track in our analysis and
reports when attributes changes over time, for example a customer moves from one region to
another.
Load
Finally, data is loaded into a central warehouse, usually into fact and dimension tables. From there
the data can be combined, aggregated and loaded into datamarts or cubes as is deemed necessary.
***************************************************************************
****
***************************************************************************
****
OLTP System
Online Transaction Processing
(Operational System)
OLAP System
Online Analytical Processing
(Data Warehouse)
Source of data
Purpose of data
Inserts and
Updates
Queries
Space
Requirements
Database Design
Backup and
Recovery
***************************************************************************
****
BI used for?
Organizations use Business Intelligence to gain data-driven insights on anything related to business
performance. It is used to understand and improve performance and to cut costs and identify new
business opportunities, this can include, among many other things:
Storing Data
Storing Data is concerned with making sure the data is filed and stored in appropriate ways to ensure
it can be found and used for analysis and reporting. When storing data the same basic principles
apply that you would use to store physical goods say books in a library you are trying to find the
most logical structure that will allow you to easily find and use the data. The advantages of modern
data-bases (often called data warehouses because of the large volumes of data) is that they allow
multi-dimensional formats so you can store the same data under different categories also called
data marts or data-warehouse access layers. Like in the physical world, good data storage starts with
the needs and requirements of the end users and a clear understanding of what they want to use the
data for.
Analyzing Data
The next component of BI is analysing the data. Here we take the data that has been gathered and
inspect, transform or model it in order to gain new insights that will support our business decision
making. Data analysis comes in many different formats and approaches, both quantitative and
qualitative. Analysis techniques includes the use of statistical tools, data mining approaches as well as
visual analytics or even analysis of unstructured data such as text or pictures.
Providing Access
In order to support decision making the decision makers need to have access to the data. Access is
needed to perform analysis or to view the results of the analysis. The former is provided by the latest
software tools that allow end-users to perform data analysis while the latter is provided through
reporting,
dashboard
and
scorecard
applications.
***************************************************************************
****
What is Metadata?
Metadata is defined as data that describes other data. Metadata can be divided into two main types:
structural and descriptive.
Structural metadata describes the design structure and their specifications. This type of metadata
describes the containers of data within a database.
Descriptive metadata describes instances of application data. This is the type of metadata that is
traditionally spoken of and described as data about the data.
A third type is sometime identified called Administrative metadata. Administrative metadata
provides information that helps to manage other information, such as when and how a resource was
created, file types and other technical information.
Metadata makes it easier to retrieve, use, or manage information resources by providing users with
information that adds context to the data theyre working with. Metadata can describe information at
any level of aggregation, including collections, single resources, or component part of a single
resource. Metadata can be embedded into a digital object or can be stored separately. Web pages
contain metadata called metatags.
Metadata at the most basic level is simply defined as data about data. An item of metadata
describes the specific characteristics about an individual data item. In the database realm, metadata
is defined as, data about data, through which the end-user data are integrated and
managed. Metadata in a database typically store the relationships that link up numerous pieces of
data. Metadata names these fields, describes the size of the fields, and may put restrictions on what
can
go
in
the
field
(for
example,
numbers
only).
Therefore, metadata is information about how data is extracted, and how it may be transformed. It
is also about indexing and creating pointers into data. Database design is all about defining metadata
schemas. Meta data can be stored either internally, in the same file as the data, or externally, in a
separate area. If the data is stored internally, the metadata is together with the data, making it more
easily accessible to view or change. However, this method creates high redundancy. If metadata is
stored externally, the searches can become more efficient. There is no redundancy but getting to this
metadata
may
be
a
little
more
technical.
All the metadata is stored in a data dictionary or a system catalog. The data dictionary is most
typically an external document that is created in a spreadsheet type of document that stores the
conceptual design ideas for the database schema. The data dictionary also contains the general
format that the data, and in effect the metadata, should be. Metadata is an essential aspect to
database design, it allows for increased processing power, due to the fact that it can help create
pointers and indexes.
***************************************************************************
****
If data exists or not in a table Primary Index once created in Teradata cannot be changed. We have
to drop the table and recreate the table with the column you want to be created as Primary Index
***************************************************************************
****
Explain
To
the
Parallel
Shared
Process
nothing
everything
is
architecture?
to Share
nothing
Shared nothing architecture (SNA) is a distributed computing architecture which consists of multiple
nodes such that each node has its own private memory, disks and input/output devices independent
of any other node in the network. Each node is self sufficient and shares nothing across the network.
Therefore, there are no points of contention across the system and no scope for data sharing or
system resources. This type of architecture is highly scalable and has become quite popular especially
in
the
context
of
web
development.
For instance, Google has implemented an SNA which evidentially enables it to scale web applications
effectively by simply adding nodes in its network of servers without slowing down the system.
***************************************************************************
****
Difference
between
SMP
and
MPP?
Symmetric Multiprocessing (SMP) is the processing of programs by multiple processors that share a
common operating system and memory. This SMP is also called as "Tightly Coupled Multiprocessing".
A Single copy of the Operating System is in charge for all the Processors Running in an SMP. This
SMP Methodology doesnt exceed more than 16 Processors. SMP is better than MMP systems when
Online Transaction Processing is Done, in which many users can access the same database to do a
search with a relatively simple set of common transactions. One main advantage of SMP is its ability
to dynamically balance the workload among computers ( As a result Serve more users at a faster rate
)
Massively Parallel Processing (MPP)is the processing of programs by multiple processors that work on
different parts of the program and share different operating systems and memories. These Different
Processors which run communicate with each other through message interfaces. There are cases in
which there are upto 200 processors which run for a single application. An Interconnect arrangement
of data paths allows messages to be sent between different processors which run for a single
application or product. The Setup for MPP is more complicated than SMP. An Experienced Thought
Process should to be applied when u setup these MPP and one shold have a good in-depth knowledge
to partition the database among these processors and how to assign the work to these processors.
An MPP system can also be called as a loosely coupled system. An MPP is considered better than an
SMP for applications that allow a number of databases to be searched in parallel.
***************************************************************************
****
***************************************************************************
**************************************************************************
***************************************************************************
****
***************************************************************************
****
CASE BASED[case_n]
4. HASH INDEX
5. JOIN INDEX
***************************************************************************
****
INDEXES
SET / MULTISET
COLUMN LEVEL
ROW LEVEL
***************************************************************************
****
Should
PI + SET
UPI + MULTI SET
Both
the
be
created
with
the
below
combinations
above
conditions
check
the
duplication
of
data.
***************************************************************************
****
TEMPORARY TABLES
THREE TYPES OF TEMPORARY TABLES
VOLTAILE TABLES
GLOBAL TABLES
DERIVED TABLES
***************************************************************************
****
***************************************************************************
****
USER SPACES
PERMANENT SPACE -- stores PERMANENT TABLES
SYSTEM TABLES
This is disk space used for storing user data rows in any tables located on the
database.
Both Users & databases can be given perm space.
This Space is not pre-allocated , it is used up when the data rows are stored
on disk.
TEMPORARY SPACE -- stores GLOBAL TABLES
It is allocated to any databases/users where Global temporary tables are
created and data is stored in them.
Unused perm space is available for TEMP space
SPOOL SPACE
-It is a temporary workspace which is used for processing Rows for given
SQL statements.
-Spool space is assigned only to users . -Once the SQL processing is complete the spool is freed and given to some
other query.
-Unused Perm space is automatically available for Spool .
***************************************************************************
****
Types of ETL Bugs
***************************************************************************
****
1)
Constraint Testing:
In the phase of constraint testing, the test engineers identifies whether the data is mapped from
source to target or not.
The Test Engineer follows the below scenarios in ETL Testing process.
a)
NOT NULL
b)
UNIQUE
c)
Primary Key
d)
Foreign key
e)
Check
f)
Default
g)
NULL
2)
Source to Target Count Testing:
In the Source to Target data is matched or not. A Tester can check in this view whether it is
ascending order or descending order it doesnt matter .Only count is required for Tester.
Due to lack of time a tester can follow this type of Testing.
3)
4)
Threshold/Data Integrated Testing:
In this Testing, the Ranges of the data, A test Engineer can usually identifies the population
calculation and share marketing and business finance analysis (quarterly, halferly, Yearly)
MIN
4
MAX
10
RANGE
6
5)
Field to Field Testing:
In the field to field testing, a test engineer can identify that how much space is occupied in the
database. The data is integrated in the table cum datatypes.
NOTE: To check the order of the columns and source column to target column.
6)
Duplicate
Check
Testing:
In this phase of ETL Testing, a Tester can face duplicate value very frequently so, at that time the
tester follows database queries why because huge amount of data is present in source and Target
tables.
Select ENO, ENAME, SAL, COUNT (*) FROM EMP GROUP BY ENO, ENAME, SAL HAVING COUNT (*)
>1;
1)
2)
Note:
There are no mistakes in Primary Key or no Primary Key is allotted then the duplicates may
arise.
Sometimes, a developer can do mistakes while transferring the data from source to target
at that time duplicates may arise.
3)
Due to Environment Mistakes also duplicates arise (Due to improper plugins in the
tool).
7)
8)
Incremental and Historical Process Testing:
In the Incremental data, the historical data is not corrupted. When the historical data is corrupted
then this is the condition where bugs raise.
9)
Control Columns and Defect Values Testing:
This is introduced by IBM
10) Navigation Testing:
Navigation Testing is the End user point of view testing. An end user cannot follow the friendly of the
application that navigation is called as bad or poor Navigation.
At the time of Testing, A tester can identify this type of navigation scenarios to avoid
unnecessary navigation.
14) Retesting:
Re executing the failed test cases after fixing the bug.
15)
Integration testing: After the completion of programming process . Developer can integrate the
modules there are 3 models
a)
Top Down
b)
Bottom Up
c)
Hybrid
**************************************************************************
*****
Comparing to primary indexes, secondary indexes allow access to information in a table by alternate,
less frequently used paths. Teradata automatically creates a secondary index subtable. The subtable
will contain:
Secondary index value
Secondary index row id
Primary index row id
When a user writes an sql query that has a si in the where clause, the parsing engine will
hash the secondary index value. The output is the row hash of the si. The pe creates a request
containing the row hash and gives the request to the message passing layer (which includes the
bynet software and network). The message passing layer uses a portion of the row hash to point to a
bucket in the hash map. That bucket contains an amp number to which the pe's request will be sent.
The amp gets the request and accesses the secondary index subtable pertaining to the requested si
information. The amp will check to see if the row hash exists in the subtable and double check the
subtable row with the actual secondary index value. Then, the amp will create a request containing
the primary index row id and send it back to the message passing layer. This request is directed to
the amp with the base table row, and the amp easily retrieves the data row.
Secondary indexes can be useful for :
Satisfying complex condition
Processing aggregates
Value comparison
Matching character combination
Joining tables
*********************************************************************