Professional Documents
Culture Documents
Pro&Ent
User manual
www.dbsoftlab.com
Advanced ETL Processor Professional & Enterprise User Manual
Contents
1. Introduction ........................................................................................................................... 10
2. What can Advanced ETL Processor do? ............................................................................ 12
2.1 Case Study: Automatic Processing of Financial Data ..................................................... 14
3. Requirements......................................................................................................................... 16
4. Key features ........................................................................................................................... 18
4.1 Extraction Process ........................................................................................................... 18
4.1.1 Multiple Data Formats.............................................................................................. 18
4.1.2 Multiple Databases and Table Processing................................................................ 18
4.1.3 Other Database Features........................................................................................... 18
4.2 Summary of the Extraction Process: ............................................................................... 18
4.3 Validation Process ........................................................................................................... 19
4.4 Summary of Validation Processes:.................................................................................. 19
4.5 Transformation Process ................................................................................................... 19
4.6 Powerful Data Transformation ........................................................................................ 19
4.7 Summary of Transformation Processes:.......................................................................... 20
4.8 Loading Process............................................................................................................... 21
4.9 Summary of the Loading Process:................................................................................... 21
4.10 Automation Process....................................................................................................... 22
4.11 Summary of the Automation Processes:........................................................................ 22
5. Advanced ETL Processor Architecture .............................................................................. 23
6. Demo Data............................................................................................................................. 25
7. Options .................................................................................................................................. 27
8. User Interface ........................................................................................................................ 32
8.1 System Menu ................................................................................................................... 32
8.2 Advanced ETL Processor Graphical User Interface: Main window ............................... 32
8.2.1 Objects tree description ............................................................................................ 33
8.2.2 Objects tree list ......................................................................................................... 33
8.2.3 Objects Groups ......................................................................................................... 34
8.2.4 Status Bar.................................................................................................................. 36
9. Directories and Connections.................................................................................................. 37
9.1 Directories ....................................................................................................................... 37
9.2 Database Connections ..................................................................................................... 38
9.2.1 Oracle Connection .................................................................................................... 39
9.2.2 Microsoft SQL Server Connection ........................................................................... 40
9.2.3 ODBC Connections .................................................................................................. 41
9.3 Ftp Connection ................................................................................................................ 44
9.4 SMTP Connection ........................................................................................................... 45
9.5 POP3 Connection ............................................................................................................ 46
10. Processing Data ................................................................................................................... 47
10.1 Screen Overview ........................................................................................................... 47
Main tool bar ......................................................................................................................... 48
10.2 Transformation Properties ............................................................................................. 49
10.3 Template tab .................................................................................................................. 50
10.4 Execution Log Tab ........................................................................................................ 51
10.5 Rejected Records Tab.................................................................................................... 52
10.6 Creating new transformation and working with Objects............................................... 53
10.7 Working with Reader .................................................................................................... 55
Copyright © 2009 DB Software Laboratory Page 2 of 276
Advanced ETL Processor Professional & Enterprise User Manual
Copyright
License Information
You have access to Advanced ETL Processor Professional and Enterprise software and
documentation pursuant to the terms of a Software License Agreement granted by DB
Software Laboratory Limited. As a user of this software and documentation, you are bound by
the terms of the Software License Agreement. All rights, title, and interest to this software
remain with DB Software Laboratory Limited.
Requests for copies of this publication and for technical information about DB Software
Laboratory products should be made directly to DB Software Laboratory Limited.
Disclaimer
All information in this manual is subject to periodic change and revision without notice.
While every effort has been made to ensure that this manual is accurate, DB Software
Laboratory Limited excludes its liability for errors or inaccuracies (if any) contained herein.
Registered Marks
Any products or services mentioned or depicted in this document are identified by the
trademarks or service marks of their respective companies or organisations.
Edition Information
This document refers to Advanced ETL Processor Professional and Enterprise version
3.3.2.
1. Introduction
The Advanced ETL Processor is a Codeless ETL Tool. The Advanced ETL Processor is
designed to automate all of your business processes. It saves you time and money so you can
concentrate on what is important for your business.
The Advanced ETL Processor is an end to end database extraction and importing tool. The
beauty of the system is that it saves the drudgery and manual tasks normally required for tasks
of this type, such as the writing of code, and all the transformations, validations and general
checks normally performed. Using traditional methods, operations of this type can only be
performed in stages, and not as one smooth operation!
For example, the traditional method of importing data from one system to another is to write
specific code to extract data from the source database, e.g. an Oracle database, by creating a
CSV comma de-limited file, and then writing code in the new language or system, for instance
Microsoft Access, and then performing the import. However, the operation does not end there.
Any data imported has to be sorted, duplicated and loaded into the database using appropriate
primary and foreign key constraints. This is only possible by creating code designed to achieve
this process. You then need to manually send an e-mail to the administrator when the process
is complete. In other words, each stage cannot be left to run in an automated fashion and has to
be completed before proceeding to the next.
As you can see, the process is not straightforward. The Advanced ETL Processor automates
all these processes in a simple and transparent fashion, and all without writing any code
whatsoever.
As stated, the tool to handle any kind of database, including Oracle, Microsoft Access, SQL
Server, DB2, MySQL, PostgreSQL, Interbase, Excel spreadsheets, and a wide range of others.
It is an excellent tool for those organisations who work with data warehouses, and where this
involves working with a number of disparate databases.
Existing users find that the tool provides several benefits over existing tools such as Oracle
SQL Loader, BCP, DTS or SSIS, such as the ability to update records automatically via
utilisation of the primary key.
For example typical usage would be to download an Excel file from the FTP server, load into
a database run sum calculations, email results a to manager and log to the administrator.
The Advanced ETL Processor would automatically carry out all the following steps for this
task using the following process:
The idea is to present the tool with a "roadmap" of what to do and a set of tools and
procedures to execute the end to end process in one swoop. Once you are ready to go, you
simply start the automation process and it completes all the necessary tasks required without
any further intervention by the user or developer. Unlike standard migrations, it may be
possible to leave the tool to do its job overnight, saving staff resources and daytime business
hours. If something went wrong Advanced ETL Processor writes detailed messages into the
log.
Since Advanced ETL Processor is a Codeless ETL Tool most of the tasks performed
without writing a single line of code
Enterprise environments benefit from this type of system, such as Fortune 100 companies and
large corporations. However, it is equally useful for the small business environment.
Advanced ETL Processor can handle a number of different IT functions and services. For
example, it can process databases, files, email protocols such as smtp, pop3 and imap, ftp,
Microsoft Office documents, and numerous others.
The system works by working to a schedule and set of tasks created and engineered by the
end user. Some of these tasks may involve the running of a shell script; dos batch file or a sql
script/stored procedure. The system is set up like a "conveyor belt" of tasks, each one to be
completed before the next one can take place. However, it is also possible to set up tasks to run
concurrently, and indeed this would increase the efficiency of some of the tasks.
Although the system itself is "codeless" i.e. the end user does not need to enter any code, it
does not prevent the execution of code or having code written to perform certain complex
actions. However, any code written is not part of the Advanced ETL Processor, and can be
run independently if required. This provides even more flexibility.
When planning out tasks, it is probably best to specify the requirements beforehand. This can
be documented in Standard English pseudo code, or simply provided by a list of actions.
Here are some examples of what our customers use this automations tool for:
Check Data
• Validate Post codes, Phone numbers, TAX Codes using regular expressions
• Validate Date Formats
• Validate Date and Time
• Validate Numbers
• Validate Strings
Transform Data
• Sort Data
• Group Data
• Summarize Data
• Pivot and UnPivot Data
• Translate coded values.
• Create surrogate keys
• Split, merge Fields
• Split data and load it into different tables
• Add, replace, and delete strings
Load Data
• Add Records
• Update Records
• Delete Records
• Add New and Update existing Records
• Run SQL scripts
Now let’s provide a case study and a list of actions to see how the system can work in
practice:
We are small Accounting Company based in Melbourne. Over last year our customer base
had grow twice and it is still growing. We spend a lot of time thinking how we can optimise
our business processes. First thing we did created standard Excel templates for everything. For
example standard expenses form will consist of customer id, employee id, expense date,
category and amount. Using standard forms saved us a lot of time but once all the forms are
filled in they had to be processed manually. Our people ware constantly under the pressure and
we had to employ an agency staff on temporary basis.
During routine Google search we discovered Advanced ETL Processor Ent which quickly
becomes a corner stone for automating our business processes.
By using Advanced ETL Processor Ent we were able to eliminate most of our manual tasks
It is like spider which sits in the middle connected to all our systems.
Carl Barret,
Accountant
The Advanced ETL Processor can be used to automate the process of transferring and
transforming data from one database to another, or from several databases to several other
databases. The Advanced ETL Processor is a specialised tool designed for this task.
For the Advanced ETL Processor the above tasks are very simple. It can perform tasks
much more complex than this, but it does provide a flavour of what the processor can achieve.
Additional validation checks can be built into the process, and any problems can be reported to
an error log. The process can be made to halt on error, or it can continue, so that any problem
areas can be re-executed at a later time.
Once you are ready to go, it is a simple matter to start the Advanced ETL Processor and
allow it to perform its tasks. Stages in the process can be changed or amended for different
environments and systems. The processing stages can also be changed depending on what
happens during the process, so it can make intelligent decisions based on what the creator of
the process would like to happen during execution. Therefore “what if” situations can be
handled and alternative processing routes included.
3. Requirements
Below is the list of Software that must be installed before installation of Advanced ETL
Processor:
Separate Downloads:
Oracle client 8.1.7 and Oracle Ole DB Provider for Oracle to use Oracle repository on Oracle
8-9 and load data into/from Oracle
Or
Oracle client 9 and Oracle Ole DB Provider for Oracle to use Oracle repository on Oracle 8-9
and load data into/from Oracle
Or
Oracle client 10 and Oracle Ole DB Provider for Oracle to use Oracle repository on Oracle 9-
10g and load data into/from Oracle.
Or
Oracle client 11 and Oracle Ole DB Provider for Oracle to use Oracle repository on Oracle
10-11g and load data into/from Oracle.
(ote:
Depending on the Requirements you may or may not need to have all components installed
There is no need to install clients for MySql and PostgreSQL they are integrated into the
software itself.
4. Key features
The ability of the Advanced ETL Processor to work with a number of disparate systems
means that it is provided with a rich set of tools and functionality, which can be used in
isolation or combined in a powerful way with other toolsets, either within the processor or with
other third party tools. It is in effect, an "engineering" environment for the movement of data
to and from different sources.
We will now explain and define the variety of features which are provided as part of the
toolset. Let us first have a look at the data extraction process.
The extraction process can handle a variety of data formats, including multiple delimited or
fixed width text files. The power of this system however, is in its ability to find files to load
using mask.
It can easily interpret and manipulate Microsoft Access data, from a number of different
databases. Again the end user can use a mask to find the tables to load the data from. It does
also apply to Excel and DBF/FoxPro files.
The Advanced ETL Processor also has other useful database features, such as the ability to
connect to any Object Database Connectivity (ODBC) database. ODBC was intended to enable
developers' access to any data through any application, regardless of the DBMS used for
managing that data. ODBC boasts platform independence since it has been purposefully
designed in a way that makes it distinct from database systems, programming languages and
operating systems. The Advanced ETL Processor utilises this technology to great effect.
So what if you don’t have an Oracle or MySQL database? What about SQL Server? No
problem, the Advanced ETL Processor can handle SQL Server data as efficiently as data
from any other type of database.
The Advanced ETL Process has a robust validation process built in. The types and nature of
the validations taking place can be tweaked and configured by the user. A full range of
validation functions are included. Validations can be performed on the basis of data type, lists
of values, and regular expressions, which can be individually changed according to
requirements.
Validation:
The process of “transformation” involves the manipulation of data so that it can be converted
from one format or data type to another. The Advanced ETL Processor provides a useful set
of “built in” transformation functions. Transformations can be performed on strings, numbers,
dates, coded values and whole chunks of data. The transformation functionality allows for the
sorting and grouping of data according to requirements. Data can also be duplicated as
required.
In addition to the standard data transformation abilities, the processor can also perform
complex translation functions. An example would be if an integer variable = “1”, then set a
text variable to “yes”. Once data is translated, it is possible to join data into a completely new
format or present it in a new way. The Advanced ETL Processor provides many flexible
alternatives for data manipulation, and these are not difficult to take advantage of.
The Advanced ETL Processor also provides the ability to derive calculated values, join data
together from multiple fields, summarise multiple rows at once, or can split or merge columns
at will.
The flexibility and power of the processor means that you can customize data transformation
and conversion functions according to your requirements with a click of the mouse. This saves
hours and hours of coding.
Transformation:
• 36 Transformation Functions
• String Transformation
• Number Transformation
• Date Transformation
• Sorting
• Grouping
• Deduplication
• Translating coded values (e.g., if the source system stores 1 for male and 2 for female,
but the warehouse stores M for male and F for female)
• Deriving a new calculated value (e.g., sale amount = qty * unit price)
• Joining together data from multiple fields
• Summarizing multiple rows of data (e.g., total sales for each store, and for each region)
• Generating surrogate key values
• Transposing or pivoting (turning multiple columns into multiple rows or vice versa)
• Splitting a column into multiple columns (e.g., putting a comma-separated list
specified as a string in one column as individual values in different columns)
• Customised Transformation
The loading capability of the Advanced ETL Processor is superior to other basic tools such
as SQL Loader, because it provides the administrator with several options and ways of
providing database load capability without creating any code.
Other tools allow you to load data into a single database table at a time, under a single
instance. The ETL processor on the other hand allows you to specify multiple upload targets
which mean you can save time loading each individual table. Another useful feature is that you
can execute SQL scripts either prior to the load or after the load has completed. This is useful
for tidying up the data or providing a report on the result of the load process once it completes.
It does not matter if the file to be loaded is a fixed or variable length text file, the Advanced
ETL Processor can handle it. It works with Access, DBF files, Oracle, SQL Server and any
ODBC compliant database.
Oracle
SQL server
This software uses the same API as Microsoft DTS and SSIS service.
The automation process of the Advanced ETL Processor is very powerful and generic.
If there is a process requiring automation there is a very good chance that the ETL Processor
can handle it. The ETL processor can automate a myriad of different technologies. In fact,
"automation ability" sits at the core of the Advanced ETL Processor.
This is a valid claim as the system caters for a number of database technologies and
protocols. The coding to perform this is built in; therefore there is no need for further
development by any programmers. The only actions required are that the definitions of the data
and the functionality are specified, and the system does the rest. As previously stated, the
automation functions allow you to set up a "conveyor belt" of actions. The administrator
simply sets the process in motion.
For instance, you may wish to run an SQL script, produce a report, copy a file, send a file
over to a server via FTP, backup the resulting data, and then inform the project manager of the
end result via email. The Advanced ETL Processor is an excellent automation system,
without the need for coding.
Automation:
• SQL Scripts
• Email Templates
• Report Designer
• Integrated Scheduler
• Logging
• Email Notifications
• PO3 Email Receiver
• File Operations
• Check Files
• Compare Files
• Full range of FTP operations
• ZIP
• Applications
• SQL Data Check
• Automatic Backups
The following graphical depiction of a typical ETL installation shows how the software
interacts with other components and interfaces belonging to various databases. The processor
sits in the middle of the various databases and carries out its tasks, such as converting,
transforming and validating data from various sources.
As you can see that the Advanced ETL Processor uses native low level API's for specific
databases, such as the Oracle Call Interface (OCI), or, in order to handle the Microsoft SQL
Server database the BCP API. Other API's can also be "plugged in" such as the ODBC API
which allows MS Access, DBF or plain flat files to be processed.
(ote:
One of major benefit of using native low level API' is a great performance boost it gives to
Advanced ETL Processor.
Oracle
Oracle OCI API
ETL Processor
SQL
SQL
BCP API
MySQL
MySQL Integrated
Clients
PostgreSQL
PostgreSQL Automation
Interbase
Interbase
Connections
Scheduler Flat Files
Flat Files Packages
SQL Scripts
Emails
Reports
FTP
Files
Check File
Extraction
6. Demo Data
In order to provide a flavour of the types of data the Advanced ETL Processor can handle, a
number of different file types have been provided for demonstration purposes.
A range of different types of data files have been provided, in addition to SQL scripts to
create database tables in Oracle, and MS SQL Server.
Text files
Access Databases
DBF Files
Excel Files
(ote:
If you want to use other locations please amend Directories properties.
Demo tables
Use SQL Scripts provided to create demo tables for Oracle and MS SQL Server.
Please adjust connection details before executing these scripts.
Most of the SQL Server Imports use DEMO database. You have to create demo tables within
this database first before executing import scripts.
ODBC connections
During the setup process, the following ODBC DSN is created during the installation:
• ODBC_FOXPRO
• ODBC_ACCESS_SOURCE
• ODBC_ACCESS_TARGET
• ODBC_MS
• ODBC_ORACLE
• ODBC_EXCEL
These ODBC connections provide the basis for the ETL Processor to perform its tasks and
process different types of data, while ensuring that the automation of these processes is
completely transparent.
7. Options
The Advanced ETL Processor allows you to tweak key settings. Using this provision you
can change the repository type, control the execution of packages and where the results will be
logged, set interface actions such as the prevention of accidental deletion and set email options
to report the outcome of the automation to individuals or groups of people.
To change Advanced ETL Processor settings click System menu-> File-> Options.
The Repository tab defines the repository connection. Once the logon parameters have been
verified via the “Test Connection” dialogue this information is saved so it is not necessary to
provide this information again.
• MS Access
• MS SQL Server
• Oracle
• Interbase (Firebird)
• MySql
• PostgreSQL
(ote:
Provided Default Repository [MS Access] contains all the SQL scripts required to create
Oracle, MS SQL server, Interbase, MySql and PostgreSQL repositories.
Execution tab defines settings related to the logging and Packages execution.
The Interface tab defines key settings. These operate globally during the activities of the
Advanced ETL Processor. For instance, in order to save time when creating packages to
automate tasks, the editor can be set so that it automatically opens. In order to ensure that any
changes made to scripts and packages upon exit, you can safeguard this by checking the “Save
all scripts and packages on exit” box. Accidental deletion is prevented by requesting that the
user is asked before deleting any objects. Another aspect of the interface actions is the
updating of the log. This can be set so that once a package has completed its actions the log is
updated with all the actions which took place during execution. Finally, if you wish to
automatically refresh table list when creating/opening transformation you simply check the
“Update Tables List” to ensure that you this is accurate, and up to date.
The Email tab defines settings required to send Emails. As the Advanced ETL Processor is
designed to work automatically and in batch mode, it will be necessary to specify where emails
will be sent from. Emails messages are used to convey when processes and activities have
been completed. It is also possible to specify the default recipient. The default recipient can be
set to a group email address, or it can be left blank.
8. User Interface
Upon starting the Advanced ETL Processor software, you will be presented with the main
window. This is where the main tasks are controlled such as processing cycles, and other
processing tasks taking place in the processor.
The system menu allows you access to the essential setting and tools you will need to use the
processor.
System Menu
Objects tree
Status bar
The "Objects" tree lists the main categories for the provision of objects. For instance, the
"Directories" category contains icons representing the location of files and related information
belonging to an individual item. For example, by clicking on the "Excel Files" icon you will
access the directory area where all the Excel files are stored for processing.
You may also find icons representing individual database connection services. For instance,
you can see connections available for both MS SQL Server and Oracle.
Other object types include the ODBC connections. You can see from the illustration that the
DEMO system has been configured, and as a result the ODBC connections created are listed in
the objects tree. The Advanced ETL Processor provides the Objects tree so that users have
easy access to the main objects. The full list of objects is described below:
The Advanced ETL Processor also has the ability to group objects together into “groups”.
The purpose of doing this is so that objects of a specific type can be viewed as a branch of the
object tree. Whenever a new object is added it will be placed under the appropriate branch for
its type.
Groups of Objects
To create a new object group select appropriate object type for example transformation and
click “new”.
Depending on the object type, the appropriate dialogue will be presented. In this example the
Transformation Group has been selected. The dialogue in this case requests a description and
comment about the object to be created. Some of the fields in a dialogue of this type will be
mandatory and will therefore have to be entered. However, comments will be optional, and just
provide the user with the ability to specify more detail about an object and what it does.
The Status Bar indicates various activities during processing. It provides an indication of
what the system is doing at any moment, including indicating when the processor is connected
to a repository and if it is processing or waiting for various actions to take place.
9.1 Directories
If a new directory is required, selecting “New” on the new directory icon brings up the
following dialogue:
• In the Name Text Box type in a new name for the directory you are about to create
• Fill in Directory path you wish to load data from
• Click OK to close the directory Properties Window
(ote:
The user may change connection or directory properties at any time by double clicking on it.
It is also possible to use sub directory in existing directory <Directory,ame>\Subdirectory.
When setting up database connections, you should ensure that the account you wish to use
has access to all the available functionality you may wish to use. For instance, it is not much
use, setting up a connection to an Oracle account to use certain stored procedures, if the
underlying Oracle permissions are not set for this facility. The account used in the Advanced
ETL Processor for the database should allow the user to perform underlying database actions,
which may not be under control of the processor directly.
To create a new connection select appropriate connection type and click “New”.
Once this step has been completed, you will be provided with an appropriate dialogue which
will request information about the new connection you wish to create, and the type of
connection you are creating.
For an Oracle connection you will need to specify the TNS name required, and provide the
user name and password for the connection.
(ote:
It is also possible to use Oracle instant client.
• In the Name Text Box type in a new name for the connection you are about to create
• Select a TNS Name from the Server Drop Down List
• Fill in Username/Password for the database you wish to connect to
• Click Test to ensure the details you have provided are correct
• Click OK to close the connection properties window
The procedure for many database connections is very similar. In the case of MS SQL Server it
is necessary to specify the server, database name and user name/password combination.
• In the Name Text Box type in a new name for the connection you are about to create
• Select a Server Name from Server Drop Down List
• Select a Database Name form the Drop Down List
• Fill in Username/Password for the database you wish to connect to
• Click Test to ensure the details you have provided are correct
• Click OK to close the connection properties window
(ote:
Leave user name and password blank to use Windows Authentication.
ODBC history
ODBC, standing for Open Data Base Connectivity, represents a standard database access
method working as SQL API software. It was developed back in 1992 with the sole purpose of
facilitating the work with database management systems (DBMS). It was created by the
Microsoft corporation and is since then included in every copy of Microsoft Windows. In 1995
the ODBC was included in the SQL Standard, which played a great role in making ODBC
more popular. Today, there are versions for almost any operating system currently used.
ODBC was intended to enable developers' access to any data through any application,
regardless of the DBMS used for managing that data. ODBC boasts platform independence
since it has been purposefully designed in a way that makes it distinct from database systems,
programming languages and operating systems.
Facilitating the data access from an application to a database management system through
ODBC is done through a specific mechanism. A common ODBC implementation contains one
or more applications, a core ODBC 'Driver Manager' library, and one or more database drivers.
The Driver Manager's role is to interpret the data queries coming from an application by using
the DBMS-specific details contained in database drivers. The latter represent a middle layer
inserted between an application and the DBMS in use. This way, the application's data queries
are translated into commands that can easily be read by the DBMS.
A basic requirement for an ODBC implementation to be run is that both the application and
the DBMS be ODBC-compliant. In other words, the application must be able to issue ODBC
commands and the DBMS must be capable of responding to them.
Thanks to its modular nature ODBC gives developers a great freedom in creating the separate
components of an ODBC implementation. Thus, a programmer can write applications that use
standard features without needing to worry about the type of DBMS used for administering the
database that the application tries to access. Likewise, the only thing the database driver
programmers need to keep in mind during the development process is how to attach their
database driver to the 'Driver Manager' library. The result is that currently there are hundreds
of ODBC drivers created for a large variety of data sources.
Thanks to the long period of existence and the fruitful efforts of its team of developers,
ODBC now offers access to a much wider range of data sources than any other database access
method available today. It has turned into a universal data access standard, working with a
great variety of operating systems and providing access to even non-relational data, including
text and XML files.
The ODBC Connection properties dialogue allows the administrator to create appropriate
connections to various databases. It is be necessary to create connections to databases,
whenever they are to be processed.
• In the Name Text Box type in a new name for the connection you are about to create
• Select a ODBC DSN Name from Server Drop Down List
• Fill in Username/Password for the database you wish to connect to
• Click Test to ensure the details you have provided are correct
• Click OK to close the connection properties window
It also possible to use ODBC connection strings for both Reader and Writer connections.
For example for MS SQL Server connection string is:
One of the major benefits of using connection strings that it makes it no longer necessary to
create ODBC Dsn’s manually on every single computer where Advanced ETL Processor is
installed. It also gives a greater control over the connection parameters.
(ote:
Leave user name and password blank and provide it within connection string
The simplest way to create ODBC connection string is to use ODBC Connection builder
dialog. Double click on ODBC driver name to create a connection string
FTP stands for File Transfer Protocol, the protocol for exchanging files over the Internet. FTP
works in the same way as HTTP for transferring Web pages from a server to a user's browser
and SMTP for transferring electronic mail across the Internet in that, like these technologies,
FTP uses the Internet's TCP/IP protocols to enable data transfer.
FTP is most commonly used to download a file from a server using the Internet or to upload a
file to a server (e.g., uploading a Web page file to a server).
• In the Name Text Box type in a new name for the FTP Connection you are about to create
• Fill in the host name
• Select TCP/IP port (default is 21)
• Type the user name and password
• Test connection
• Click OK to close the FTP connection properties window
SMTP stands for Simple Mail Transfer Protocol, a protocol for sending e-mail messages
between servers. Most e-mail systems that send mail over the Internet use SMTP to send
messages from one server to another; the messages can then be retrieved with an e-mail client
using either POP or IMAP. In addition, SMTP is generally used to send messages from a mail
client to a mail server.
• In the Name Text Box type in a new name for the SMTP connection you are about to
create
• Fill in host name
• Select TCP/IP port (default is 25)
• Type in user name and password
• Click OK to close the SMTP connection properties window
POP3 (Post Office Protocol 3) is the most recent version of a standard protocol for receiving
e-mail. POP3 is a client/server protocol in which e-mail is received and held for you by your
Internet server. Periodically, you (or your client e-mail receiver) check your mail-box on the
server and download any mail, probably using POP3. This standard protocol is built into most
popular e-mail products, such as Eudora and Outlook Express. It's also built into the Netscape
and Microsoft Internet Explorer browsers.
POP3 is designed to delete mail on the server as soon as the user has downloaded it.
However, some implementations allow users or an administrator to specify that mail be saved
for some period of time. POP can be thought of as a "store-and-forward" service.
To setup email access using POP as the protocol follow these steps:
• In the Name Text Box type in a new name for the POP3 connection you are about to create
• Fill in host name
• Select TCP/IP port (default is 110)
• Type in user name and password
• Click OK to close the POP3 connection properties window
In order to load data from the data source into the data target you must define data mapping
between data source and data target. Typical transformation consists of reader, transformer and
writer.
The data processing screen provides a number of settings, for the three stages of reading,
transformation and writing. This is facilitated by the main tool bar, and a list of available
objects. The main reader toolbar provides the user with all the necessary functionality to setup
the reading process, according to the settings provided.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
The main tool bar provides a number of icons which allow the user to create the steps
required to carry out the automatic functions required.
1. Transformation Properties
2. New Transformation
3. Load Transformation From the file
4. Saves Transformation to the file
5. Saves Transformation under the new name
6. Saves Transformation to the Repository
7. Cut
8. Copy
9. Paste
10. Delete
11. Arrange Horizontally
12. Arrange Vertically
13. Prints Transformation
14. Print Preview Transformation
15. Manage Versions
16. Add Version
17. Revert to the previous version
18. Process Data
Click to change Transformation properties, please note that Default Date format applies to
all templates. Default Date format only used for new Date related validation and
transformation functions for example Is Date, Is Date between ETC
The execution log provides information about the Advanced ETL Processor and the actions
it took during its operations. This is useful when you wish to analyse the activities of
individual processes during their execution.
Occasionally, records will be rejected by the processor. This may be due to things like corrupt
records which have been read in a format not expected by the processor. The rejected records
tab allows the user to see a list of all the rejected records, and other information about the
nature of the rejection and where this occurred in the process.
Objects Panel
To join two objects together, first click on the source Output button then drag it on to the
Input button
Output Buttons
Description
Every Transformation created must have one reader object. The Reader connects to the Data
source and extracts data from it. Depending on the Data source type some options may not be
available. To change the Reader properties click or double click on the Reader object.
One of the benefits of using Advanced ETL Processor is a concept of Universal Data
Reader
Provided that you are using Advanced ETL Processor all you need to do is to change
connection type no mapping will be lost.
Other tools use different connector for different databases and some of then even sell separate
licences for them. That mean that user have to recreate mapping for new files/ databases.
The Reader is capable of extracting data from delimited or fixed width files. All parameters
are user definable. It can also skip a number of Header and Footer lines
The Rejected records file can have a pre-defined format. The following dialogue allows the
user to set this up.
Data View
One of the useful features of the Advanced ETL Processor is the ability to view the
resultant data prior, or subsequently processed. The data view looks like a spreadsheet view as
follows:
Source file
Number of records to show
The data view toolbar allows you to change various aspects of the data view, such as
refreshing data as it changes and setting properties for how the data will look. You can also
switch between viewing of the data and checking to see how the data is defined i.e. the data
dictionary, via the “Switch to Data Definition View”.
1 2 3 4 5 6 7
1. Reader Properties
2. Refresh Data
3. Edit file in external editor
4. Add a column
5. Delete last column
6. Switch to Data View
7. Switch to Data Definition View
(ote:
You may rename fields and change field’s width here. (Works only for text files)
Within the Data Definition view you can perform a number of actions. These allow you to
change how you want data to be represented in the data view screens. The navigation also
allows switching between views.
1 2 3 4 5 6 7 8 9 10
1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Edit file in external editor
7. Add a column
8. Delete last column
9. Switch to Data View
10. Switch to Data Definition View
Query Builder
(ote:
It is also possible to use Query builder to design queries
Data View
1 2 3 4 5
1. Reader Properties
2. Refresh Data
3. Edit file in external editor
4. Switch to Data View
5. Switch to Data Definition View
1 2 3 4 5 6 7 8
1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Edit file in external editor
7. Switch to Data View
8. Switch to Data Definition View
Source Directory
Data View
Source Directory
1 2 3 4
1. Reader Properties
2. Refresh Data
3. Switch to Data View
4. Switch to Data Definition View
1 2 3 4 5 6 7
1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Switch to Data View
7. Switch to Data Definition View
Copyright © 2009 DB Software Laboratory Page 66 of 276
Advanced ETL Processor Professional & Enterprise User Manual
Data View
Source Table
1 2 3 4
1. Reader Properties
2. Refresh Data
3. Switch to Data View
4. Switch to Data Definition View
1 2 3 4 5 6 7
1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Switch to Data View
7. Switch to Data Definition View
(ote:
All properties are very similar to ODBC connection
(ote:
All properties are very similar to ODBC connection
Advanced ETL Processor Data Validator guarantees to your application database that every
data value is correct and accurate.
One of the simplest forms of data validation is verifying the data type. Data type validation
answers such simple questions as "Is the string alphabetic?" and "Is the number valid?"
As an extension of simple type validation, range checking ensures that the provided value is
within allowable minimums and maximums. For example, a character data type service code
may only allow the alphabetic letters A through Z. All other characters would not be valid.
Code checking is a bit more complicated, typically requiring a lookup table. For example,
maybe your application calculates sales tax for only certain state codes. You would need to
create a lookup object to hold the authorized, taxable state codes.
Pattern checking when you checking structure of the data field for example social security
number format or car registration number. Regular expressions used quite often for pattern
checks.
It does not matter which business you are in sooner or later you will discover that there is
something wrong with the data ant it has to be validated. Here when Advanced ETL
Processor Validation can help.
Records Rejected by validation Rules
Processed Records
Discarded Records
(ote:
• Records can also be rejected by the Server.
• If you have several validation rules and one of them rejects record and another
discards it, record will be discarded
Data Sample
Validator Toolbar
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
1. Cut
2. Copy
3. Paste
4. Delete
5. Arrange Horizontally
6. Arrange Vertically
7. Print
8. Print Preview
9. Delete All objects
10. Delete All Links
11. Process Data
12. First Record
13. Previous Record
14. Next Record
15. Last Record
16. Show Data
(ote:
Use <value> to include actual value into default value
To add new Validation rule drag and drop it from the Validation rules panel
There more than 190 Validation Functions at the moment. They are grouped in five different
categories
1. String
2. Number
3. Date
4. Time
5. Regular Expressions
Category Toolbar
It is also possible to apply several validation rules to Input field by joining them
Validated Data
Failed Data
(ote:
If you have several validation rules and one of them rejects record and another discards it,
record will be discarded
Copyright © 2009 DB Software Laboratory Page 76 of 276
Advanced ETL Processor Professional & Enterprise User Manual
Records Transformed
(ote:
To change Transformer properties double click on it
Transformer Toolbar
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
1. Cut
2. Copy
3. Paste
4. Delete
5. Arrange Horizontally
6. Arrange Vertically
7. Print
8. Print Preview
9. Auto Map
10. Delete All objects
11. Delete All Links
12. Process Data
13. First Record
14. Previous Record
15. Next Record
16. Last Record
17. Copies Inputs to Outputs (Only visible if transformer is connected to any object other
than writer)
18. Show Data
(ote:
If transformer is connected to any object other than writer it is possible to modify list of
Outputs. When transformer is connected to writer list of Outputs is taken from Writer.
Examples:
Example below splits date field into Day, Month and Year using ‘/’ as a delimiter
If the Inputs and Outputs have got the same names you may use Auto map feature.
Records Out
Records In
To change grouper properties double click on the object. Choose appropriate grouping order
and grouping type. Fields with grouping type Ignore are not passed to the next object. Grouper
does not perform sorting.
Records Out
Records In
About Sorting
In computer science and mathematics, a sorting algorithm is an algorithm that puts elements
of a list in a certain order. All sorting algorithms are very well documented and it is hard to
invent something new or much faster that existing algorithm. To achieve the best performance
is important to load all the data into the memory which is not possible for large amount of
data. Advanced ETL Processor uses combination of Quicksort and Merge sort. Portion of
Data is loaded in the memory and sorted using Quicksort algorithm than data is saved into
temporary file. Once all portions are sorted temporary files merged together using Merge sort
algorithm.
Quicksort
Quicksort is a divide and conquer algorithm which relies on a partition operation: to partition
an array, we choose an element, called a pivot, move all smaller elements before the pivot, and
move all greater elements after it. This can be done efficiently in linear time and in-place. We
then recursively sort the lesser and greater sublists. Efficient implementations of quicksort
(with in-place partitioning) are typically unstable sorts and somewhat complex, but are among
the fastest sorting algorithms in practice. Together with its modest O(log n) space usage, this
makes quicksort one of the most popular sorting algorithms, available in many standard
libraries. The most complex issue in quicksort is choosing a good pivot element; consistently
poor choices of pivots can result in drastically slower (O(n²)) performance, but if at each step
we choose the median as the pivot then it works in O(n log n).
Merge sort
Merge sort takes advantage of the ease of merging already sorted lists into a new sorted list.
It starts by comparing every two elements (i.e., 1 with 2, then 3 with 4...) and swapping them if
the first should come after the second. It then merges each of the resulting lists of two into lists
of four, then merges those lists of four, and so on; until at last two lists are merged into the
final sorted list. Of the algorithms described here, this is the first that scales well to very large
lists, because its worst-case running time is O(n log n).
To change Sorter properties double click on the object. Tick sort, select field order and sort
order. Field order must be unique for sort field. Data is loaded in the memory first than sorted
and passed to the next object.
Records Out
Records In
To change Deduplicator properties double click on the object. Tick Deduplicate for the fields
you wish to depuplicate, only ticked fields are passed to the next object.
(ote:
There is no need to sort data before deduplication
Records In Records In
To change UnPivot properties double click on the object. Fill in Description, Group field
name, create all necessary Groups and Outputs. Once it is done Map input fields to outputs and
Groups
UnPivoted Data
Group field
UnPivot Properties
Pivot Key
Set Key
Pivoted Data
Since there are more than one field to pivot let’s join Profit and Sales fields together first
using tab a delimiter than pivot the data and split it again
Joining Data
Every Transformation created must have at least one writer object. Writer connects to Target
database and loads data into it. Depending of the target type some options may not be
available. To change the Writer properties click or double click on the Writer object.
Writer is capable of saving data into delimited or fixed width files. All parameters are user
definable.
Data View
Target File
Number of records to show
1 2 3 4 5 6 7
1. Reader Properties
2. Refresh Data
3. Edit file in external editor
4. Add new column
5. Delete last column
6. Switch to Data View
7. Switch to Data Definition View
(ote:
You may rename fields and change field’s width here. (Works only for text files)
1 2 3 4 5 6 7 8 9 10
1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Edit file in external editor
7. Add new column
8. Delete last column
9. Switch to Data View
10. Switch to Data Definition View
Data View
1 2 3 4 5 6 7 8
1. Reader Properties
2. Refresh Data
3. Print
4. Print Preview
5. Find
6. Edit file in external editor
7. Switch to Data View
8. Switch to Data Definition View
1 2 3 4 5 6 7 8
1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Edit file in external editor
7. Switch to Data View
8. Switch to Data Definition View
Data View
1 2 3 4 5 6 7
1. Reader Properties
2. Refresh Data
3. Print
4. Print Preview
5. Find
6. Switch to Data View
7. Switch to Data Definition View
Check constraints
Ensure that any constraints on the destination table are checked during the bulk copy
operation. By default, constraints are ignored.
Keep identity
Specify that there are values in the data file for an identity column.
Keep (ULLS
Specify that any columns containing a null value should be retained as null values, even if a
default value was specified for that column in the destination table.
Batch size
Specify the number of rows in a batch. The default is the entire data file.
The following values for the Batch size property have these effects:
If you set Batch size to zero, the data is loaded in a single batch. The first row that fails will
cause the entire load to be cancelled, and the step fails.
If you set Batch size to one, the data is loaded a row at a time. Each row that fails is counted
as one row failure. Previously loaded rows are committed.
If you set Batch size to a value greater than one, the data is loaded one batch at a time. Any
row that fails in a batch fails that entire batch; loading stops and the step fails. Rows in
previously loaded batches are either committed or, if the step has joined the package
transaction, provisionally retained in the transaction, subject to later commitment or rollback.
(ote:
Option ‘Commit every Array’ works only for Oracle conventional path loading
Update Key
For the example provided below, Advanced ETL Processor will execute the following SQL
(Update key is CustomerId, OrderNo).
Select count(*)
from [DEMO].[dbo].[orders]
where CustomerId=? And OrderNo=?
Update [DEMO].[dbo].[orders]
set orderdate=?,
amount=?
where customerid=? And OrderNo=?
Update Records
Update [DEMO].[dbo].[orders]
set OrderDate=?,
Amount=?
where CustomerId=? And OrderNo=?
Delete Records
(ote:
“Add ew And Update Old Records” is not supported for SQL Server Connection use
ODBC connection instead.
Advanced ETL Processor is capable of running SQL Scripts before and after the
transformation.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
This chapter represents list of validation rules together with short description grouped by
category. Every Validation Rule/function could have some specific parameters and common
parameters to control data flow of Advanced ETL. Most if it is self explanatory see the
example below:
(ote:
To change validation rule properties double click on it
11.1 Strings
11.1.1 Is Null
11.1.3 Is Alpha
11.1.5 Is Hex
11.1.6 Is Equal To
11.1.9 Contains
11.1.10 In List
11.2 Numbers
11.2.1 Is Number
11.2.2 Is Integer
11.2.3 Is Positive
11.2.4 Is Negative
11.3 Date
11.3.1 Is Date
11.3.7 Is 1st Quarter, Is 2nd Quarter, Is 3rd Quarter, Is 4th Quarter, Is Current
Quarter,
Is Last Quarter, Is Next Quarter
11.3.10 Is Within Past Minutes, Is Within Past Hours, Is Within Past Days, Is
Within Past Weeks, Is Within Past Months
11.4 Time
11.4.1 Is Time
11.4.5 Is Second
11.4.6 Is Minute
11.4.7 Is Hour 24
11.4.8 Is Hour 12
11.4.9 Is PM
11.4.10 Is AM
Regular expression is a string that is used to describe or match a set of strings, according to
certain syntax rules.
Regular expressions are used by many text editors, utilities, and programming languages to
search and manipulate text based on patterns. For example, Perl and Tcl have a powerful
regular expression engine built directly into their syntax. Several utilities provided by Unix
distributions—including the editor ed and the filter grep—were the first to popularize the
concept of regular expressions. "Regular expression" is often shortened to regex or regexp
(singular), or regexes, regexps, or regexen (plural). Some authors distinguish between regular
expression and abbreviated forms such as regex, restricting the former to true regular
expressions, which describe regular languages, while using the latter for any regular
expression-like pattern, including those that describe languages that are not regular. As only
some authors observe this distinction, it is not safe to rely upon it.
As an example of the syntax, the regular expression \bex can be used to search for all
instances of the string "ex" that occur at word boundaries (signified by the \b). Thus in the
string, "Texts for experts," \bex matches the "ex" in "experts," but not in "Texts" (because the
"ex" occurs inside the word there and not immediately after a word boundary).
Source: Wikipedia.
Most of regular expressions were taken from www.regexlib.com. We are not able to include
full list of contributors because it is too big. We did our best to test and modify if necessary
pattern strings however as time goes by standards changes and some of them could became no
longer valid. Therefore Regular Expression Validation Rules should be used with caution. If
you have any useful Regular expressions please let us know we would be more than happy to
include it into the next release of Advanced ETL Processor.
11.5.2 Is IP Address V4
11.5.3 Is IP Address V6
11.5.4 Is Email
11.5.5 Is ISBN 10
11.5.6 Is ISBN 13
11.5.8 Is URL
11.5.9 Is UNC
11.5.20 Is US State
(ote:
To change Transformation function properties double click on it
12.1 Strings
12.1.4 First Up
12.1.6 Trim
12.1.10 Replace
12.1.19 Delete
12.1.20 Left
12.1.21 Right
12.2 Numbers
12.2.1 Round
12.2.2 Abs
12.2.3 Sign
12.3 Date
12.4 Miscellaneous
12.4.1 Length
12.4.2 Literal
12.4.2 User
12.4.3 Splitter
12.4.4 Joiner
12.4.5 Calculation
Below is the basic structure that every Calculation transformation must follow:
Var
VariableName : VariableType;
VariableName : VariableType;
...
// Some single line Comment if necessary
{
Multi line
Comment
}
Procedure ProcedureName;
variables here if necessary
Begin
Some Code;
End;
Begin
the main program block.
Result:= some calculation
End.
(ote:
The functions and procedures can appear in any order. The only requirement is that if one
procedure or a function uses another one, that latter one must have been defined already.
Declaring Variables
Variables are simply a name for a block of memory cells in main memory. If a value is
assigned to a variable, that value must be of the same type as the variable, and will be stored in
the memory address designated by the variable name. The assignment statement is the
semicolon-equal :=.
Example:
Var
i : Integer: { variable name is i, type is integer)
Begin
i := 10; { valid integer number assigned to variable i }
End.
Variable Types
Number variables can be assigned from other numeric variables, and expressions:
var
Age : Byte; // Smallest positive integer type
Books : SmallInt; // Bigger signed integer
Salary : Currency; // Decimal used to hold financial amounts
Expenses : Currency;
TakeHome : Currency;
begin
Expenses := 12345.67; // Assign from a literal constant
TakeHome := Salary; // Assign from another variable
TakeHome := TakeHome - Expenses; // Assign from an expression
end;
(umerical operators
When using these multiple operators in one expression, you should use round brackets to
wrap around sub-expressions to ensure that the result is obtained. This is illustrated in the
examples below:
var
myInt : Integer; // Define integer and decimal variables
myDec : Single;
begin
myInt := 20; // myInt is now 20
myInt := myInt + 10; // myInt is now 30
myInt := myInt - 5; // myInt is now 25
myInt := myInt * 4; // myInt is now 100
myInt := 14 div 3; // myInt is now 4 (14 / 3 = 4 remainder 2)
myInt := 14 mod 3; // myInt is now 2 (14 / 3 = 4 remainder 2)
myInt := 12 * 3 - 4; // myInt is now 32 (* comes before -)
myInt := 12 * (3 - 4); // myInt is now -12 (brackets come before *)
myDec := 2.222 / 2.0; // myDec is now 1.111
end;
Character Types
var
Str1 : Char; // Holds a single character, small alphabet
Str2 : WideChar; // Holds a single character, International alphabet
Str3 : AnsiChar; // Holds a single character, small alphabet
Str4 : ShortString; // Holds a string of up to 255 Char's
Str5 : String; // Holds strings of Char's of any size desired
Str6 : AnsiString; // Holds strings of AnsiChar's any size desired
Str7 : WideString; // Holds strings of WideChar's of any size desired
Variants
The Variant data type provides a flexible general purpose data type.
Variants are useful in very specific circumstances, where data types and their content are
determined at run time rather than at compile time.
Example:
var
myVar : Variant;
Date Variables
TDateTime
Description
It is stored as a Double variable, with the date as the integral part, and time as fractional part.
The date is stored as the number of days since 30 Dec 1899. Quite why it is not 31 Dec is not
clear. 01 Jan 1900 has a days value of 2.
(ote:
,o local time information is held with TDateTime - just the day and time values.
Example:
Finding the difference between two dates
var
day1, day2 : TDateTime;
diff : Double;
begin
day1 := StrToDate('12/06/2002');
day2 := StrToDate('12/07/2002');
diff := day2 - day1;
Result:='day2 - day1 = '+FloatToStr(diff)+' days';
end;
These are used in conjunction with programming logic. They are very simple:
Var
Log1 : Boolean; // Can be 'True' or 'False'
Boolean variables are a form of enumerated type. This means that they can hold one of a
fixed number of values, designated by name. Here, the values can be True or False.
Logical Operation
var
number : Integer;
text : String;
begin
number := Sqr(17); // Calculate the square of 17
if number > 400
then text := '17 squared > 400' // Action when if condition is true
else text := '17 squared <= 400'; // Action when if condition is false
result:=text;
end;
There are a number of things to note about the if statement. First that it spans a few lines -
remember that statements can span lines - this is why it insists on a terminating ;
Second, that the then statement does not have a terminating ; -this is because it is part of the if
statement, which is finished at the end of the else clause.
Third, that we have set the value of a text string when the If condition is successful - the
Then clause - and when unsuccessful - the Else clause. We could have just done a then
assignment:
Note that here, the then condition is not executed (because 17 squared is not > 400), but there
is no else clause. This means that the if statement simply finishes without doing anything.
Note also that the then clause now has a terminating ; to signify the end of the if statement.
We can have multiple conditions for the if condition. And we can have more than one
statement for the then and else clauses. Here are some examples:
We used And to join the if conditions together - both must be satisfied for the then clause to
execute. Otherwise, the else clause will execute. We could have used a number of different
logical primitives, of which And is one, covered under logical primitives below.
(ested if statements:
There is nothing to stop you using if statements as the statement of an if statement. Nesting
can be useful, and is often used like this:
if condition1
then statement1
else if condition2
then statement2
else statement3;
However, too many nested if statements can make the code confusing. The Case statement,
discussed below, can be used to overcome a lot of these problems.
Logical primitives
begin
if false And false
then Result:='false and false = true';
if false Or false
then Result:= 'false or false = true';
if true Or false
then Result:= 'true or false = true';
if false Or true
then Result:= 'false or true = true';
if true Or true
then Result:= 'true or true = true';
if Not false
then Result:= 'not false = true';
if Not true
then Result:= 'not true = true';
end;
Note that the Xor primitive returns true when one, but not both of the conditions are true.
Case statements
The If statement is useful when you have a simple two way decision. Ether you go one way or
another way. Case statements are used when you have a set of 3 or more alternatives.
var
i : Integer;
begin
i := [F1];
Case i of
15 : Resut := ('Number was fifteen');
16 : Resut := ('Number was sixteen');
17 : Resut := ('Number was seventeen');
18 : Resut := ('Number was eighteen');
19 : Resut := ('Number was nineteen');
20 : Resut := ('Number was twenty');
end;
end;
The case statement above routes the processing to just one of the statements. OK, the code is
a bit silly, but it is used to illustrate the point.
Supposing we were not entirely sure what value our case statement was processing? Or we
wanted to cover a known set of values in one fell swoop? The Else clause allows us to do that:
var
i : Integer;
begin
i := [F1];
Case i of
15 : Resut := ‘Number was fifteen';
16 : Resut := 'Number was sixteen';
17 : Resut := 'Number was seventeen';
18 : Resut := 'Number was eighteen';
19 : Resut := 'Number was nineteen';
20 : Resut := 'Nuumber was twenty';
else
Resut := 'Unexpected number‘;
end;
end;
Unexpected number : 10
One of the main reasons for using computers is to save the tedium of many repetitive tasks.
One of the main uses of loops in programs is to carry out such repetitive tasks. A loop will
execute one or more lines of code (statements) as many times as you want.
Your choice of loop type depends on how you want to control and terminate the looping.
This is the most common loop type. For loops are executed a fixed number of times,
determined by a count. They terminate when the count is exhausted. The count (loop) is held
in a variable that can be used in the loop. The count can proceed upwards or downwards, but
always does so by a value of 1 unit. This count variable can be a number or even an
enumeration.
Counting up
var
count : Integer;
begin
For count := 1 to 5 do
Result:= 'Count is now '+IntToStr(count);
end;
Counting down
var
count : Integer;
begin
For count := 5 downto 1 do
Result:= 'Count is now '+IntToStr(count);
end;
The For statements in the examples above have all executed one statement. If you want to
execute more than one, you must enclose these in a Begin and End pair.
The Repeat loop type is used for loops where we do not know in advance how many times
we will execute. For example, when we keep asking a user for a value until one is provided, or
the user aborts. Here, we are more concerned with the loop termination condition.
Repeat loops always execute at least once. At the end, the Until condition is checked, and the
loop aborts of condition works out as true.
A simple example
var
stop : Boolean; // Our exit condition flag
i : Integer;
begin
i := 1;
exit := False; // do not exit until we are ready
repeat
i := i+1; // Increment a count
if Sqr(i) > 99
then stop:= true; // Exit if the square of our number exceeds 99
until stop; // Shorthand for 'until exit := true'
result:=I;
end;
Here we exit the repeat loop when a Boolean variable is true. Notice that we use shorthand -
just specifying the variable as the condition is sufficient since the variable value is either true
or false.
var
i : Integer;
begin
i := 1;
repeat
i := i+1; // Increment a count
until (Sqr(i) > 99) or (Sqrt(i) > 2.5);
result:=i;
end;
Notice that compound statements require separating brackets. Notice also that Repeat
statements can accommodate multiple statements without the need for a begin/end pair. The
repeat and until clauses form a natural pairing.
While loops
While loops are very similar to Repeat loops except that they have the exit condition at the
start. This means that we use them when we wish to avoid loop execution altogether if the
condition for exit is satisfied at the start.
Var
i : Integer;
begin
i := 1;
while (Sqr(i) <= 99) and (Sqrt(i) <= 2.5) do
i := i+1; // Increment a count
result:=i;
end;
Notice that our original Repeat Until condition used Or as the compound condition joiner -
we continued until either condition was met. With our While condition, we use And as the
joiner - we continue whilst neither condition is met. Have a closer look to see why we do this.
The difference is that we repeat an action until something or something else happens. Whereas
we keep doing an action while neither something nor something else have happened.
Functions
Functions provide a flexible method to apply one formula many times to possibly different
values. They are comparable to procedures but
Begin
Result:= [F001] + [F002];
End;
2. Strings Concatenation
Begin
Result:= ‘[F001]’ + ‘[F002]’;
End;
2. If Statement
Begin
If '[F002]'='' then
Result:= 0
else If [F002]=0 then
Result:= 0
Else
Result:= [F001] mod [F002];
End;
3. Variables
var
MyVariable : integer;
Begin
MyVariable:=10;
Result :=[F001] mod MyVariable;
end;
Properties:
LowerCase(S)
LowerCase returns a string with the same text as the string passed in S, but with all letters
converted to lowercase. The conversion affects only 7-bit ASCII characters between 'A' and 'Z'.
To convert 8-bit international characters, use AnsiLowerCase.
AnsiUpperCase(S)
AnsiLowerCase(S)
AnsiLowerCase returns a string that is a copy of the given string converted to lower case.
AnsiCompareStr(S1,S2)
Conditio
n Return Value
S1 > S2 >0
S1 < S2 <0
S1 = S2 =0
AnsiCompareText(S1,S2)
AnsiStrLIComp (S1,S2,MaxLen)
AnsiLastChar(S)
Trim(S)
Trim removes leading and trailing spaces and control characters from the given string S.
TrimLeft(S)
TrimLeft returns a copy of the string S with leading spaces and control characters removed.
TrimRight(S)
TrimRight returns a copy of the string S with trailing spaces and control characters removed.
QuotedStr(S)
Use QuotedStr to convert the string S to a quoted string. A single quote character (') is
inserted at the beginning and end of S, and each single quote character in the string is repeated.
AnsiQuotedStr(S,Quote)
Use AnsiQuotedStr to convert a string (S) to a quoted string, using the provided Quote
character. A Quote character is inserted at the beginning and end of S, and each Quote
character in the string is doubled.
AnsiExtractQuotedStr(S,Quote)
AnsiExtractQuotedStr removes the quote characters from the beginning and end of a quoted
string, and reduces pairs of quote characters within the string to a single quote character. The
Quote parameter defines what character to use as a quote character. If the first character in S is
not the value of the Quote parameter, AnsiExtractQuotedStr returns an empty string.
The function copies characters from S to the result string until the second solitary quote
character or the first null character in S. The S parameter is updated to point to the first
character following the quoted string. If S does not contain a matching end quote character, the
S parameter is updated to point to the terminating null character.
IntToStr(S)
IntToStr converts an integer into a string containing the decimal representation of that
number.
IntToHex(I,Digits)
IntToHex converts a number into a string containing the number's hexadecimal (base 16)
representation. Value is the number to convert. Digits indicate the minimum number of
hexadecimal digits to return.
StrToInt(S)
StrToInt converts the string S, which represents an integer-type number in either decimal or
hexadecimal notation, into a number. If S does not represent a valid number, StrToInt raises an
exception.
StrToIntDef(S,Default)
StrToIntDef converts the string S, which represents an integer-type number in either decimal
or hexadecimal notation, into a number. If S does not represent a valid number, StrToIntDef
returns Default.
FileAge(File(ame)
Call FileAge to obtain the OS timestamp of the file specified by FileName. The return value
can be converted to a TDateTime object using the FileDateToDateTime function. The return
value is -1 if the file does not exist.
FileExists(File(ame)
FileExists returns true if the file specified by FileName exists. If the file does not exist,
FileExists returns false.
DeleteFile(File(ame)
DeleteFile deletes the file named by FileName from the disk. If the file cannot be deleted or
does not exist, the function returns false.
RenameFile(OldFile,(ewFile)
RenameFile attempts to change the name of the file specified by OldFile to NewFile. If the
operation succeeds, RenameFile returns true. If RenameFile cannot rename the file (for
example, if the application does not have permission to modify the file), it returns false.
ChangeFileExt(File(ame,EXT)
ChangeFileExt takes the file name passed in FileName and changes the extension of the file
name to the extension passed in Extension. Extension specifies the new extension, including
the initial dot character.
ChangeFileExt does not rename the actual file, it just creates a new file name string.
ExtractFilePath(File(ame)
The resulting string is the leftmost characters of FileName, up to and including the colon or
backslash that separates the path information from the name and extension. The resulting
string is empty if FileName contains no drive and directory parts.
ExtractFileDir(File(ame)
ExtractFileDrive(File(ame)
ExtractFileDrive returns a string containing the drive portion of a fully qualified path name
for the file passed in the FileName. For file names with drive letters, the result is in the form
"drive". For file names with a UNC path the result is in the form "\servername\sharename". If
the given path contains neither style of path prefix, the result is an empty string.
ExtractFile(ame(File(ame)
The resulting string is the rightmost characters of FileName, starting with the first character
after the colon or backslash that separates the path information from the name and extension.
The resulting string is equal to FileName if FileName contains no drive and directory parts.
ExtractFileExt(File(ame)
ExpandFile(ame(File(ame)
ExpandFileName converts the relative file name into a fully qualified path name.
ExpandFileName does not verify that the resulting fully qualified path name refers to an
existing file, or even that the resulting path exists.
ExpandU(CFile(ame(File(ame)
ExpandUNCFileName returns the fully-qualified file name for a specified file name.
ExtractRelativePath(File(ame)
Call ExtractRelativePath to convert a fully qualified path name into a relative path name. The
DestName parameter specifies file name (including path) to be converted. BaseName is the
fully qualified name of the base directory to which the returned path name should be relative.
BaseName may or may not include a file name, but it must include the final path delimiter.
DiskFree(Drive)
DiskFree returns the number of free bytes on the specified drive, where 0 = Current, 1 = A, 2
= B, and so on.
DiskSize(Drive)
DiskSize returns the size in bytes of the specified drive, where 0 = Current, 1 = A, 2 = B, etc.
DiskSize returns -1 if the drive number is invalid.
GetCurrentDir(Directory)
SetCurrentDir(Directory)
The SetCurrentDir function sets the current directory. The return value is true if the current
directory was successfully changed, or false if an error occurred.
CreateDir(Directory)
CreateDir creates a new directory. The return value is true if a new directory was successfully
created, or false if an error occurred.
RemoveDir(Directory)
Call RemoveDir to remove the directory specified by the Dir parameter. The return value is
true if a new directory was successfully deleted, false if an error occurred. The directory must
be empty before it can be successfully deleted.
FloatToStr(F)
FloatToStr converts the floating-point value given by Value to its string representation. The
conversion uses general number format with 15 significant digits.
StrToFloat(S)
EncodeDate(Year,Month,Day)
EncodeDate returns a TDateTime value from the values specified as the Year, Month, and
Day parameters. The year must be between 1 and 9999. Valid Month values are 1 through 12.
Valid Day values are 1 through 28, 29, 30, or 31, depending on the Month value. For example,
the possible Day values for month 2 (February) are 1 through 28 or 1 through 29, depending on
whether or not the Year value specifies a leap year.
EncodeTime(Hour,Min,Sec,MSec)
EncodeTime encodes the given hour, minute, second, and millisecond into a TDateTime
value. Valid Hour values are 0 through 23. Valid Min and Sec values are 0 through 59. Valid
MSec values are 0 through 999. If the specified values are not within range, EncodeTime
raises an EConvertError exception. The resulting value is a number between 0 and 1
(inclusive) that indicates the fractional part of a day given by the specified time or (if 1.0)
midnight on the following day. The value 0 corresponds to midnight, 0.5 corresponds to noon,
0.75 corresponds to 6:00 pm, and so on.
DayOfWeek(D)
DayOfWeek returns the day of the week of the specified date as an integer between 1 and 7,
where Sunday is the first day of the week and Saturday is the seventh.
Date
Use Date to obtain the current local date as a TDateTime value. The time portion of the value
is 0 (midnight).
Time
Use Time to return the current time as a TDateTime value. The two functions are completely
equivalent.
(ow
IncMonth(D)
IncMonth returns the value of the Date parameter, incremented by NumberOfMonths months.
NumberOfMonths can be negative, to return a date N months previous. If the input day of
month is greater than the last day of the resulting month, the day is set to the last day of the
resulting month. The time of day specified by the Date parameter is copied to the result.
IsLeapYear(D)
Call IsLeapYear to determine whether the year specified by the Year parameter is a leap year.
Year specifies the calendar year. Use YearOf to obtain the value of Year for IsLeapYear from a
TDateTime value.
DateToStr(D)
Use DateToStr to obtain a string representation of a date value that can be used for display
purposes.
TimeToStr(D)
DateTimeToStr(D)
StrToDate(S)
Call StrToDate to parse a string that specifies a date. If S does not contain a valid date,
StrToDate raises an exception.
StrToTime(S)
Call StrToTime to parse a string that specifies a time value. If S does not contain a valid time,
StrToTime raises an exception.
StrToDateTime(S)
Call StrToDateTime to parse a string that specifies a date and time value. If S does not
contain a valid date, StrToDateTime raises an exception.
FormatDateTime(Format,DateTime)
FormatDateTime formats the TDateTime value given by DateTime using the format given by
Format. See the table below for information about the supported format strings.
Abort
Beep
AnsiPos(Substr,S)
Call AnsiPos to obtain the byte offset of the Substr parameter, as it appears in the string S.
For example, if Substr is the string "AB", and S is the string "ABCDE", AnsiPos returns 1. If
Substr does not appear in S, AnsiPos returns 0.
Chr(X)
Abs(X)
Length(X)
Copy(S,Index,Count)
Count specifies more characters or array elements than are available, only the characters or
elements from S[Index] to the end of S are returned.
Round(X)
Trunc(X)
Pos(Substr,Str)
Pos searches for Substr within S and returns an integer value that is the index of the first
character of Substr within S. Pos is case-sensitive. If Substr is not found, Pos returns zero
Delete(S,Index,Count)
Delete removes a substring of Count characters from string S starting with S[Index]. S is a
string-type variable. Index and Count are integer-type expressions. If index is larger than the
length of the string or less than 1, no characters are deleted. If count specifies more characters
than remain starting at the index, Delete removes the rest of the string. If count is less than or
equal to 0, no characters are deleted
Insert(Substr,Dest,Index)
Insert merges Source into S at the position S[index]. Source is a string-type expression. S is a
string-type variable of any length. Index is an integer-type expression. It is a character index
and not a byte index. If Index is less than 1, it is mapped to a 1. If it is past the end of the
string, it is set to the length of the string, turning the operation into an append. If the Source
parameter is an empty string, Insert does nothing
Sqr(X)
The Sqr function returns the square of the argument. X is a floating-point expression. The
result, of the same type as X, is the square of X, or X*X.
Sqrt(X)
Exp(X)
Exp returns the value of e raised to the power of X, where e is the base of the natural
logarithms
Ln(X)
Sin(X)
Cos(X)
Cos returns the cosine of the angle X. X expression that represents an angle in radians
Tan(X)
ArcTan(X)
ArcTan returns the arctangent of X. X is a real-type expression that gives an angle in radians
SetLength(S,Length)
High(X)
Low(X)
PI
Represents the mathematical value pi, the ratio of a circle's circumference to its diameter. Pi
is approximated as 3.1415926535897932385.
ArcCos(X)
ArcCos returns the inverse cosine of X. X must be between -1 and 1. The return value is in
the range [0..Pi], in radians.
ArcCosh(X)
ArcCosh returns the inverse hyperbolic cosine of X. The value of X must be greater than or
equal to 1.
ArcCot(X)
ArcCotH(X)
ArcCsc(X)
ArcCscH(X)
ArcSec(X)
ArcSecH(X)
ArcSin(X)
ArcSin returns the inverse sine of X. X must be between -1 and 1. The return value will be in
the range [-Pi/2..Pi/2], in radians.
ArcSinh(X)
ArcTan(X)
ArcTan returns the arctangent of X. X is a real-type expression that gives an angle in radians.
ArcTanh(X)
ArcTanh returns the inverse hyperbolic tangent of X. The value of X must be between -1 and
1 (inclusive).
Ceil(X)
Call Ceil to obtain the lowest integer greater than or equal to X. The absolute value of X must
be less than MaxInt. For example: Ceil(-2.8) = -2 Ceil(2.8) = 3 Ceil(-1.0) = -1
Cosecant(X)
Use the Cosecant to calculate the cosecant of X, where X is an angle in radians. The cosecant
is calculated as 1/ Sin(X).
Cosh(X)
Cot(X)
Call Cot to obtain the cotangent of X. The cotangent is calculated using the formula 1 / Tan
(X).
Cotan(X)
Call Cotan to obtain the cotangent of X. The cotangent is calculated using the formula 1 / Tan
(X)
Do not call Cotan with X = 0
CotH(X)
Csc(X)
CscH(X)
Use the CscH to calculate the hyperbolic cosecant of X, where X is an angle in radians.
CycleToDeg(X)
CycleToDeg converts angles measured in cycles into degrees, where degrees = cycles * 360.
CycleToGrad(X)
CycleToRad(X)
CycleToRad converts angles measured in cycles into radians, where radians = 2pi * cycles.
DegToCycle(X)
Use DegToCycle to convert angles expressed in degrees to the corresponding value in cycles.
DegToGrad(X)
Use DegToGrad to convert angles expressed in degrees to the corresponding value in grads.
DegToRad(X)
Use DegToRad to convert angles expressed in degrees to the corresponding value in radians,
where radians = degrees(pi/180).
Floor(X)
Call Floor to obtain the highest integer less than or equal to X. For example: Floor(-2.8) = -3
Floor(2.8) = 2 Floor(-1.0) = -1
GradToCycle(X)
GradToDeg(X)
GradToRad(X)
GradToRad converts angles measured in grads into radians, where radians = grads(pi/200).
Hypot(X,Y)
Hypot returns the length of the hypotenuse of a right triangle. Specify the lengths of the sides
adjacent to the right angle in X and Y. Hypot uses the formula Sqrt(X**2 + Y**2)
IntPower(Base,Exponent)
Ldexp(X)
LnXP1(X)
LnXP1 returns the natural logarithm of (X+1). Use LnXP1 when X is a value near 0.
Log10(X)
Log2(X)
Log((Base,X)
Max(A,B)
Call Max to compare two numeric values. Max returns the greater value of the two.
Min(A,B)
Call Min to compare two numeric values in Delphi. Min returns the smaller value of the two.
Power(Base,Exponent)
Power raises Base to any power. For fractional exponents or exponents greater than MaxInt,
Base must be greater than 0.
RadToCycle(X)
Use RadToCycle to convert angles measured in radians into cycles, where cycles =
radians/(2pi).
RadToDeg(X)
RadToGrad(X)
RandG(Mean,StdDev)
RandG produces random numbers with Gaussian distribution about the Mean. This is useful
for simulating data with sampling errors and expected deviations from the Mean.
RandomRange(AFrom,ATo)
RandomRange returns a random integer from the range that extends between AFrom and
ATo (non-inclusive). RandomRange can handle negative ranges (where AFrom is greater than
ATo). To initialize the random number generator, add a single call Randomize or assign a
value to the RandSeed variable before making any calls to RandomRange.
Sec(X)
Call Sec to obtain the secant of X, where X is an angle in radians. The secant is calculated
using the formula 1 / Cos(X).
SecH(X)
Sinh(X)
Tan(X)
Tanh(X)
12.4.6 Lookup
12.4.9 Sequence
Properties:
Date/Time format strings control the conversion of strings into date time type.
Date/Time format strings are composed from specifiers which describe values to be converted
into the date time value.
In the following table, specifiers are given in lower cases. Case is ignored in formats, except
for the "am/pm" and "a/p" specifiers.
Specifier Description
tt Uses the 12-hour clock for the preceding h or hh specifier, 'am' for any hour
before noon, and 'pm' for any hour after noon.
Important thing is to understand that this format has nothing to do with your target database.
This is the format of the source data. It is there to help to covert string into date time type
inside of the software, so it can be loaded later into date or timestamp field
Once data is loaded into the database user may want to perform various tasks like
transformations, validations or calculations. User may perform any SQL command allowed by
the target database. SQL commands must be separated by a separator. SQL Scripts Screen is
designed to provide such functionality for the end user.
• To create a new SQL Script select SQL scripts group from objects tree and click New
• Dialog box will appear
• Fill in Description edit box with the name of SQL Script you are about to create
• Fill in comment if required
• Click OK to finish creation of SQL Script
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
17 18 19 20 21 22 23
15. Reports
This chapter illustrates the process involved in creating Advanced ETL Processor reports.
A report provides the end user with the ability to view data as a neatly formatted presentation
of data. A typical report will be specific to a set of parameters or business concepts, such as a
"Latest Orders" list, or in the case of accounts, an “Overdue Payments" report.
The type of report created will be dependant on the nature of the tasks and reporting
procedures required.
Advanced ETL Processor reports can be designed specifically, for individual business cases
or can be generated from the system. The following "Report Designer" section describes the
process in more detail:
Upon opening Report Designer you will be presented with a screen similar to the following:
The units of measurement in the report can be determined from the horizontal and vertical
rulers shown in the display. The units of measurement are usually based on centimetres, but
this can be changed if required. This enables the user to design the exact layout of the data and
titles on the report on paper before the report is designed. Therefore, it is possible to sue “X”
and “Y” measurements.
Reports have three main aspects to them, as you can see on this illustration. These are
described as follows:
As the name suggests, this is the name the report will be identified by, such as "Overdue
Orders Report". This title is global to the whole report and will not change.
This header will appear on every page of the report. The headings for each page should be
relevant for the data you intend to print on the report. For instance, the heading on this report
for each page will contain the "Product Name", "Quantity Per Unit" etc.
Each report can be grouped by a specific category e.g. "group all prices by product type". The
groupings usually take place in the SQL driving the report. Grouping can also take place at a
number of levels.
Hence, in this example there are two levels of grouping. The "CategoryName" and the
"CompanyName" reflect the order of the groupings, i.e. group the data by "Category Name",
then by "Company Name". The order of grouping can change, depending on the SQL or the
groupings physically defined in the report.
You will also note the other two tabs in the report. These are described as follows:
15.1.4 Code
This reflects the actual SQL code used in the report. Therefore, if you wish to change the data
in the report, or the way it is grouped, for example, you can amend the SQL to reflect the
changes you wish to implement in the report.
15.1.5 Data
This feature allows you to see a list of all the data used in the compilation of the report. The
information presented here will probably change as a result of any changes made to the "code"
section.
1 2 3 4 5 6 7
1. Report Properties
2. Saves Report to the Repository
3. Report Connection
4. Show only connections for current project
5. Manage Versions
6. Add Version
7. Revert to previous version
The Report Properties hold the basic information about the report, such as the report
description and any comments describing the report in more detail.
All reports are stored internally in the Advanced ETL Processor. This facility allows
provides the report writer with an easy way to save the report for later retrieval or amendment.
It is possible to have a number of different projects stored on the ETL configuration. If you
wish only to show only connections for the current project you would click this icon.
Every report can have a number of versions associated with it. For instance, one version may
show all the prices for a list of products, but a different version of the same report may only
show a subset of these prices. Hence, there may be a requirement for different versions of the
same report. The "Manage Versions" utility allows you to organize different versions of the
report according to your preferences. These may include subsets of your report which can be
Copyright © 2009 DB Software Laboratory Page 219 of 276
Advanced ETL Processor Professional & Enterprise User Manual
"branched" e.g. you can have a version 1.0 or a version 1.1. Version 1.1 may have features
which are slightly different from version 1.0.
In order to demonstrate how the report process works, let’s create very simple report which
returns a list of customers.
First, the user should select an appropriate report group from the objects tree and click
“new”.
This will bring up the Report Properties tab as shown. Fill in description and comment edit
box if necessary:
Provide a report description, and any comments in the Comment box. Once you are happy
with this, click “OK”. The report Wizard Form will then appear. Once the report wizard
commences, the user can enter the details necessary for production of the report. The first
dialogue box requests the data “source”. The data source can be based on an SQL statement or
it can be based on data contained within a table.
The example below will create a report based on a query. Therefore, click “next” and “new
query”. At this point we have a choice. We can type in SQL directly or we can use the query
builder.
If you are not very confident of your SQL skills, we recommend that the query builder it used
in the process. However, in this example we will generate the report from an SQL query:
Query Builder
The example illustrated shows that we have chosen the "Customers" table and chosen
from the list of fields within that table for the report.
Once you have the data you wish to include in the report the next step is to choose the
Copyright © 2009 DB Software Laboratory Page 223 of 276
Advanced ETL Processor Professional & Enterprise User Manual
report layout. The process of choosing the fields from the table does not actually mean
they will be displayed in the report.
The objective here is to include appropriate data, perhaps to include in the query.
However, not all fields will be needed in the report. This takes us on to the next step,
i.e. deciding from the sub-set of fields which of these will be printed on the final report:
The "Fields" dialogue allows you to select the fields you would like to be printed in the
report, from the query you have just created. In the field’s dialogue shown, you can see that all
fields in the query have been chosen for the report in this case.
We do not wish to group by anything in this report, therefore leave the "Groups" section
blank as shown:
Here you can define the preferred layout of the report, including the orientation, and if you
prefer a tabular (the standard report format), or if you prefer you can have the information
printed in a columnar format. In this case we have chosen a portrait layout, a tabular format
and so that the data in the report fits neatly on to the page, we have clicked the "Fit fields to
page width" checkbox.
The other information about the report is shown, such as the company information and the
address.
Next, we come the to "Style" dialogue. This allows us to specify how the report will "look
and feel", by the use of colours and fonts. In our example there are four report styles to choose
from but it is possible to create your own styles. For our report we chose "Corporate".
Finally, we click "Finish" to create the new report. Once we have done this we see the
original Report Designer screen. From here we can choose the "Preview" button as shown:
Preview Button
We are then presented with the report as it will look when it is printed like so:
Using the preview button, the report can be exported to PDF, Html or into an Excel file as a
CSV. Reports can also be emailed, either by saving them into one of the formats stated or
directly from the preview facility.
Email templates is a library of the most often used email texts. These can be used when
automating processes such as when a set of transformations have taken place. Individual
emails can then be set for various departments or management. For example most of the
packages will have notification email to administrator.
For instance, if a package fails it will be necessary to inform about the fact. An email
template allows specific variables and information to be provided relating to the package
failure. In addition email templates can be used to attach the actual execution log relating to
the specific problem occurrence. For instance, a typical message about this sent in an email
might be:
One of the advantages of the approach of using email templates is that they save a lot of time
designing and sending emails individually. Another factor is that the email system reports
incidents as they occur, and are not sent on after the event has finished.
In the example email template below you can see that certain variables have been defined.
These variables are replaced by “actual” values at run time, or when the incident occurs.
17. Packages
This chapter describes Packages - a great way of combining different Actions together.
Most of the actions are self-explanatory therefore only screenshots provided.
• Transformation
• SQL Script
• SQL Data check
• Package
• FTP Operation
• ZIP Operation
• Application
• File Check
• File operation
• Compare Files
• Backup Repository
• Pause
• Check Date
• Report
• Note
We are constantly working on improving our software if some of the functionality is missing
please let us know and we will be more than happy to add it.
• To create a new Package select a Package group from objects tree and click New
• Dialog box will appear
• Fill in Description edit box with the name of the Package you are about to create
• Fill in comment if required
• Click OK to finish creation of the Package
The package screen overview provides the user with the ability to create objects in the ETL
Advanced Processor. The objects are numerous and can be combined with any number of
other objects. For instance, you can combine an SQL script to output information to an email
and vice versa, to store information from an email into a database. There is a great deal of
flexibility in the system.
Every package must have a starting point (Action in blue box on the picture).
Objects can be joined to other objects as shown here. This is achieved by simple “drag” and
“drop” of the items from one to another. It is a very simple process.
(ote:
To remove join repeat same procedure again.
The package tool bar provides all the tools you need in order to create and manipulate
objects. It is also possible to change how objects are viewed in the interface.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1. Packages Properties
2. Save to the Repository
3. Print
4. Print Preview
5. Cut
6. Copy
7. Paste
8. Delete selected Action(s)
9. Clear all
10. Execute
11. Align Vertical
12. Align Horizontal
13. Manage Versions
14. Add Version
15. Revert to previous version
16. Execute Package
To create a new Transformation Action, Drag and Drop it from the Object tree. You will then
be presented with the following properties dialogue for transformations:
The SQL Script Action object allows the user to define and identify any SQL scripts to be run
(in this case Drop Demo Objects), the connection to be used:
The SQL Check Action object can be used to check data queries in the database, to ensure
that data integrity and ensure that results are returned as expected.
(ote:
SQL Data Check uses SQL scripts to check data in the database.
For example user may write following SQL:
Execution will be successful if the value of first field of lookup query is more than 0
Execution will fail if the value of first field of lookup query is equals to 0 or less than 0 or
field type is not numeric.
A “Package Action” object allows the user to specify “what if” conditions based on the
results of package executions. For instance, in the example below an email notification will be
sent when the package completes it’s processing.
(ote:
Recursion is not allowed
The Ftp Action performs various operations on the ftp server. For example, using the
Advanced ETL Processor you will be able to download data directly from the FTP server, as
in the following example, where the user is downloading Customer Sales figures:
The “Zip” object allows an automatic compression of data in the processor. This can be used
to backup log files, as in the following example, although compression can be used when
sending attachments via email, when saving space on servers, and so on.
With the compression facility you can compress items and uncompress them as required. The
location of the zip file and the files to compress are specified, as illustrated below:
The “Receive” email object sets up the configuration you need in order to receive emails. The
setting on here will be based on your particular server. A variety of connections are available
including POP, IMAP, HTTP, Exchange and others.
The dialogue below allows the user to specify a directory where attachments will be
downloaded to. This is useful in the automation process as attachments can be gathered in
batch mode and saved to a central area.
The “Send” email action object allows you to configure the settings required for sending
email, and is the converse objects to “Receive” email. The workflow settings allow the user to
specify email templates to use for sending emails. The configuration can be set to the users
preferred settings. In this case, the Simple Network Management Protocol (SMTP) has been
defined by the user as the preferred method of transmitting messages.
(ote:
Type <last email> in ‘TO’ box to reply to last sender
Another useful feature is the ability in the Advanced ETL Processor to call and use external
scripts and programs. This enhances the existing functionality already built into the system,
and provides an even greater degree of flexibility.
The Check File Action object is provided to check on the status of files, for things like their
existence, and attributes. The radio box dialogue below allows the user to specify which option
is required to check for the file specified in the “Files To Check” text box.
In addition to being provided to with the ability to check the properties of files, it is also
possible to decide the actions to take in relation to files and folders. For instance, the following
dialogue is set up to create a directory, in “c:\temp” as specified in the filename text box:
This “Compare” files object allows the user to perform a validation check against two sets of
files by comparing their properties, via a “check sum”. The comparison can be done via the
creation date as the comparison factor, or it can check to ensure that the MD5 checksums
match. The files are successfully compared when the comparisons of both files are the same.
From time to time, it will be necessary to perform backup procedures on the repository. This
object allows the user to set up a backup scheme. This returns the results of the backup to the
user, in terms of success or fail status.
• Every day we export data from our database and would like to save it into different file
using current date as a part of the file.
• We would like to load yesterday’s data from the folder which has date part in the name
(ote:
Only one {} pair is allowed.
LeftString(DecDateS(GETSYSTEMVARIABLE('SYSTEM_DATE'),'YYYYMMDDHH
NNSS', 'DAY',1),8)
(ote:
For complete reference of available functions consult chapter 23.
18. Scheduler
Once a Package is created the Scheduler allows user to execute it on a regular basis. A
Package may be executed once, daily, weekly, or monthly. The user may also specify a day of
the week or month when to execute the Package.
.
The user may specify day of the week when to execute the batch. User must specify at least
one day of the week.
The user may specify month when to execute the batch. User must specify at least one month.
Once a Package is running or completed the Execution Monitor screen allows checking status
or troubleshooting if any error happens.
The Bottom panel shows status of individual items within the Package.
Double click on Bottom panel to check item log.
1 2 3 4 5 6 7 8 9 10 11 12 13 14
20. SQL
The SQL Screen is a tool to run a free hand SQL or SQL scripts against the target databases.
1 2 3 4
21 Version Control
Of the major benefits of using Advanced ETL Processor is integrated version control
system. User can have multiple versions of transformations, SQL scripts, Reports, Email
templates and Packages.
1 2 3
1. Manage versions
2. Add version
3. Revert to the old version
To run a package from the command line type aetlcl.exe package id.
Package ID
Parallel Execution
Another benefit of using aetlcl.exe that the combination of Application Action and aetlcl
makes it possible to execute several packages in parallel. See the pictures below.
This chapter provides list of functions which can be used in file names or directories
Examples:
C:\Data{GetSystemDate('YYYYMMDD')}.vis
<LogDirectory>{GetSystemDate('YYYYMMDD')}.vis
(otes:
Only one pair of {} is allowed
Trim
Trim(String):string
TrimLeft
TrimLeft(String):string
TrimRight
TrimRight(String):string
UpperCase
UpperCase(String):string
UpperCase returns a string with the same text as the string passed in, but with all letters
converted to Uppercase
LowerCase
LowerCase(String):string
LowerCase returns a string with the same text as the string passed in, but with all letters
converted to LowerCase
Replace
Replace(String,OldPattern,NewPattern):string
Replace replaces all occurrences of the OldPattern by NewPattern within the String
SubString
SubString(String,Index,Count):string
SubString returns a substring containing Count characters or elements starting from Index.
RightString
RightString(String,Count):string
LeftString
LeftString(String,Count):string
AddCharLeft
AddCharLeft(Char,String,Count):string
AddCharRight
AddCharRight(Char,String,Count):string
MakeString
MakeString(Char,Count):string
DelSpaces
DelSpaces(String):string
Delete
Delete(String,Index,Count):String
DelSpaces returns string with count characters deleted starting from index.
Insert
Insert(Source,S,Index):String
ProperCase
ProperCase(String):string
ProperCase returns string, with the first letter of each word in uppercase and all other letters in
lowercase "proper case"->"Proper Case"
Abs
Abs(Integer):Integer
Round
Round(Float,Integer):Float
Expression Value
Round(1234567, 3) 1234000
Round(1.234, -2) 1.23
Round(1.235, -2) 1.24
Round(1.245, -2) 1.24
Sign
Sign(Integer):Integer
Day
Day(Date,Format):Integer.
Day('01012003','DDMMYYYY')
Hour
Hour(Date,Format):Integer.
Hour('01012003','DDMMYYYY')
Minute
Minute(Date,Format):Integer.
Minute('01012003','DDMMYYYY')
Month
Month(Date,Format):Integer.
Month('01012003','DDMMYYYY')
Second
Second(Date,Format):Integer.
Second('01012003','DDMMYYYY')
Year
Year(Date,Format):Integer.
Year('01012003','DDMMYYYY')
DayS
DayS(Date,Format):String.
DayS('01012003','DDMMYYYY')
HourS
HourS(Date,Format):String.
HourS('01012003','DDMMYYYY')
MinuteS
MinuteS(Date,Format):String.
MinuteS('01012003','DDMMYYYY')
MonthS
MonthS(Date,Format):String.
MonthS('01012003','DDMMYYYY')
SecondS
SecondS(Date,Format):String.
SecondS('01012003','DDMMYYYY')
YearS
YearS(Date,Format):String.
YearS('01012003','DDMMYYYY')
IncDateS
IncDateS(Date,Format,ChangeType,Increment):String.
ChangeType: YEAR,MONTH,WEEK,DAY,HOUR,MINUTE,SECOND
DecDateS
DecDateS(Date,Format,ChangeType,Decrement):String.
ChangeType: YEAR,MONTH,WEEK,DAY,HOUR,MINUTE,SECOND
IntegerToString
IntegerToString(Integer):String
NumberToString
NumberToString(Float):String
StringToInteger
StringToInteger(String):Integer
StringToNumber
StringToNumber(String):Float
Iif
Iif(expr1==expr2;expr3;expr4)
GetSystemVariable
GetSystemVariable('VARIABLENAME'):string
GetSystemDate
GetSystemDate(Format):String
Pos
Pos(Substr,String): Integer
Pos searches for Substr within String and returns an integer value that is the index of the first
character of Substr within String. Pos is case-sensitive. If Substr is not found, Pos returns zero.
GetFileHeaderLine
GetFileHeaderLine(LineNumber): String
GetFileHeaderLine function returns header line LineNumber from the source text file
The Execution Agent is Microsoft Windows service which allows user to schedule and
execute Advanced ETL Processor packages.
(ote:
Execution Agent is only available with Advanced ETL Processor Enterprise Version.
To install the Advanced ETL Processor Enterprise Execution Agent as a service run
following command
aetlagent.exe /INSTALL
aetlagent.exe /UNINSTALL
While submitting a bug or problem please include the following to make it easier to solve the
problem as soon as possible:
This End-User License Agreement ("EULA") is a legal agreement between you (either an
individual or a single entity) and DB Software Laboratory for the SOFTWARE PRODUCT
identified above, which includes computer software and may include associated media, printed
materials, and "online" or electronic documentation. By installing, copying, or otherwise using
the SOFTWARE PRODUCT, you agree to be bound by the terms of this EULA. If you do not
agree to the terms of this EULA, you may be subject to civil liability if you install and use this
SOFTWARE PRODUCT.
Once SOFTWARE PRODUCT is installed you may use it for 30 days. After evaluation period
ends, you have to purchase a license or stop using the SOFTWARE PRODUCT.
If this is an EVALUATION VERSION of the SOFTWARE PRODUCT, you may copy and
distribute an unlimited number of copies of the SOFTWARE PRODUCT; provided that each
copy shall be a true and complete copy, including all copyright and trademark notices, and
shall be accompanied by a copy of this EULA.
If this is a REGISTERED VERSION of the SOFTWARE PRODUCT, you may install and use
it for your personal use only. You may not reproduce or distribute the SOFTWARE
PRODUCT for use by others.
LICE(SI(G
1. A single computer usage license. The user purchases one license to use the SOFTWARE
PRODUCT on one computer.
2. A SITE usage license. The user purchases a single usage license, authorising the use of
SOFTWARE PRODUCT, by the purchaser, the purchaser's employees or accredited
agents, on an unlimited number of computers at the same physical site location. This site
location would normally be defined as a single building, but could be considered to be a
number of buildings within the same, general, geographical location, such as an industrial
estate or small town.
You may not reverse engineer, decompile, or disassemble the SOFTWARE PRODUCT,
except and only to the extent that such activity is expressly permitted by applicable law
notwithstanding this limitation. Any such unauthorized use shall result in immediate and
automatic termination of this license and may result in criminal and/or civil prosecution. All
rights not expressly granted here are reserved by DB Software Laboratory.
The SOFTWARE PRODUCT is licensed as a single product. Its component parts may not be
separated for use on more than one computer.
You may permanently transfer all of your rights under this EULA, provided the recipient
agrees to the terms of this EULA.
SEVERABILITY
In the event of invalidity of any provision of this license, the parties agree that such invalidity
shall not affect the validity of the remaining portions of this license.
COPYRIGHT
MISCELLA(EOUS
Should you have any questions concerning this EULA, or if you desire to contact the author of
this Software for any reason, please contact DB Software Laboratory (see contact information
at the top of this EULA).
LIMITED WARRA(TY
DB Software Laboratory expressly disclaims any warranty for the SOFTWARE PRODUCT.
The SOFTWARE PRODUCT and any related documentation is provided "as is" without
warranty of any kind, either express or implied, including, without limitation, the implied
warranties or merchantability, fitness for a particular purpose, or no infringement. The entire
risk arising out of use or performance of the SOFTWARE PRODUCT remains with you.
In no event shall DB Software Laboratory be liable for any damages whatsoever or refund any
money (including, without limitation, damages for loss of business profits, business
interruption, loss of business information, or any other pecuniary loss) arising out of the use of
or inability to use this product, even if DB Software Laboratory has been advised of the
possibility of such damages. Because some states/jurisdictions do not allow the exclusion or
limitation of liability for consequential or incidental damages, the above limitation may not
apply to you.
Users with a fully paid annual maintenance fee get the following benefits:
Priority Support
Free software enhancements, updates and upgrades during the maintenance period
Advanced and exclusive notification of software promotions
"Maintenance Owner ONLY" product promotions
E(TIRE AGREEME(T
This is the entire agreement between you and DB Software Laboratory which supersedes any
prior agreement or understanding, whether written or oral, relating to the subject matter of this
license.