Configuration Stager Documentation

The Configuration Stager enables the push and import of configurations from one server to
another, for example this can be used to copy setups between development, test, and production
systems. Configurations can be pushed directly between source and destination servers, or by
using file export / import to copy configurations when direct access between servers is not
possible.

SECTION PAGE

• System Requirements 2
• Installation, Upgrades, and Uninstall 3
• Configuration Stager Overview 5
• Setup Connections 7
• Direct Staging 15
• Export a Configuration 22
• Import a Configuration 27
• Checking Staged Configuration 33
• Log Files 34
• Appendix 35
Notifications and Error Messages 35
Limitations 38

1
System Requirements
Source and destination requirements

The source and destination systems must be running:

 Dotmatics Platform 22.1 or higher

 Oracle® version18c XE or 19 SE/EE (CDB or non-CDB mode)

Other versions of Dotmatics Platform and/or Oracle database (such as 21c XE) might work but
have not been fully tested by us.

Configuration Stager requirements

Any modern desktop/laptop computer should be capable of running the Configuration Stager.

Supported operating systems:

 Microsoft Windows® 10 or higher

 Apple OS® and Linux operating systems are not currently supported

Network recommendations

Access to the appropriate port on both the source and destination application servers. A low
latency, high-bandwidth LAN connection is optimal, but access over a good-quality WAN, VPN
or Internet connection is also supported. Revision to firewall settings to open a port may be
required to establish connections.

2
Installation, Upgrades, and Uninstall
Purpose

This section describes a standard installation of the Configuration Stager.

Scope

This installation procedure is intended to be used with Dotmatics Configuration Stager v22_1
and higher.

For any assistance, please contact support@dotmatics.com – see

https://www.dotmatics.com/contact-us for details.

Download installation files and create Configuration Stager directory

The Configuration Stager can be installed directly on the source or destination (target) servers or
on a local desktop/laptop computer.

1. Download the ConfigurationStager.exe file.

2. Create a folder named Configuration Stager on the install location.
3. Move the ConfigurationStager.exe file into the Configuration Stager directory.

Configuration Stager installation

1. Double click on the ConfigurationStager.exe to launch the application.

2. The “log”, “persist”, and “saved_configs” folders will be created in the parent directory.

3
3. Confirm two windows open, a console and a main Configuration Stager interface. The
main interface is used to select and stage configuration. The console displays a running
log of activity.

Console

Main Interface

4
If the installation fails, then please contact support@dotmatics.com, and include a zip archive of
your log files from …\Configuration Stager\log (as detailed in Log Files ).

Configuration Stager upgrade & uninstall

1. Uninstall the outdated version by deleting the directory and all the content where the
ConfigurationStager.exe is located. Save a copy of the connections.pickle file if you
wish to apply previously saved connections to the upgrade before deletion of the
installation.
2. Repeat the installation steps outlined above.

Configuration Stager Overview

Configuration staging is the copying and processing of system settings and configuration onto a
target Dotmatics environment. The Configuration Stager is a python based executable that
automates the staging of Browser projects and Studies protocols between different related
systems (referred to as servers or environments herein). The tool facilitates:

 Direct-connection staging: staging from one server to another (for example from Test to
Production)
 File-based configuration staging: Staging to file (also referred to as export staging) is
staging an existing configuration to a file for archiving, to create a back up, version
control, or to use for staging at a later date. Staging from a file (import staging) is staging
from a saved configuration file.

The tool stages the configuration of the source database, e.g. format and structure of tables
containing the staged data, but not the data itself. When staging to an existing Dotmatics
installation, checks are made to see if the configuration already exists. If not, it is created with
the exact metadata of the source. If it exists, further checks are completed and merges are made
before checking for consistency (see Limitations for full details on what is and what is not
staged).

Definitions

 Configuration staging: Sometimes referred to as “staging” or “porting”, configuration

staging is the copying and processing of system settings and configuration onto a target
Dotmatics environment.
 Destination environment / server: (Target environment) is the system were setup and
configuration are being recreated to mimic the source.
 Direct-connection configuration staging: (push configuration staging) is a method of
staging where the user authenticates on both the source and specified destination
environments and the configurations is passed between them automatically.

5
  File-based configuration staging: (import/export configuration staging) is a method of
staging where a file is exported from the source and imported onto the target
environment. The user moves the configuration file amongst environments manually.
 Migration / data migration: Import of records, files, and associated data from legacy or
3rd party systems into the Dotmatics platform. Migration is typically performed to enable
access to legacy data from within the Dotmatics platform. The Configuration Stager does
NOT support data migration processes.
 Source environment / server: is the original, reference setup and configuration being
applied to another environment.

User interfaces

When the Configuration Stager is run, two windows are opened. The main interface where you
configure the staging and a console, which reports (logs) events that occur when using the tool.
Keep both open while running the Configuration Stager. If you accidently close one of the
interfaces relaunch the Configuration Stager.

Displays the source and destination servers selected in Set up

Connection Databases

Copy configuration directly from source to destination via a

direct connection (direct-connection configuration staging)

Export configuration file from the source server (for file-based

configuration staging)

Upload configuration file on destination (target) server (for

file-based configuration staging)

6
 Click the “Br” button to retrieve a list of Browser Projects to
stage from the source environment or configuration file

Click the “St” button to retrieve a list of Studies Protocols to

stage from the source environment or configuration file

Launch the “Connections” popup to configure source and/or

destination database connections

Check the configuration differences between the source and

destination ( when performing direct-connection staging)

Click to begin configuration staging process of select

protocols and/or projects

Setup Connections
 Configure a new connection
 Copy connections from a previous version
 Set up connection databases

To configure connections and set up connections to databases (to tell the Configuration Stager
which connection is the source and which is the destination ), start by clicking Connections at
the bottom of the tool.

Configure a new connection

To configure a connection:

7
1. Under Configuring new connections set the dropdown to “New”. Ignore previously
saved connections in the list.

2. Enter a Connection name

3. Click on any of DS3_USERDATA, DS3_APPDATA or GATEWAY and enter the

schema details for your environment to set up the connection. Set the connection for ALL
THREE schema.

8
The DS3_USERDATA ,DS3_APPDATA and GATEWAY schema connection details can be
found in the browser.properties configuration file on the server. Open the file with a text editor
application to review:

 DS3_USERDATA = db2.account
 DS3_APPDATA = db.account
 GATEWAY = db3.account

The selected schema username for the connection. The Dotmatics

Username
defaults are ds3_userdata ,ds3_appdata or gateway.
The selected schema password for the connection. Click the “Auto”
Password
will populate the field with Dotmatics default passwords.

9
 The hostname of the connection, for example 10.0.1.60.
Hostname
This can be found in the db.description parameter of the
browser.properties configuration file.
Port of the server, for example 1521.
Port
This can be found in the db.description parameter of the
browser.properties configuration file.
The connection SID or Service name, for example “orclpdb”.
SID/Service Name
This can be found in the db.description parameter of the
browser.properties configuration file.
Connection The connection username. The Dotmatics defaults are
Username ds3_userdata ,ds3_appdata or gateway.
Checks you can connect to the server using the credentials given.

If you are unable to connect, view the console window for details.

Clears information in all the fields.

Adds connection details and closes the popup.

Auto populates the Username, Password and Connection Username for

the three schemas based on the details currently entered, or if no details
have been entered, with the Dotmatics default values.

10
4. Click Test and confirm a Connection Successful! is displayed in the bottom left corner. If
Connection Failed is returned edit your configuration and test again.

Pitfalls:

 Are you using the correct IP address? When connecting to target Dotmatics-managed
cloud environments inquiry with Dotmatics to obtain the pubic IP address to use to
establish a connection.
 Are firewall setting blocking connection? Firewall settings may result in restricted
access and may require modification in order to grant inbound connections.

5. Continue to enter setup parameters, test and add them for all three schemas (ds3_userdata,
ds3_appdata, and gateway).

6. Alternatively, if the configuration tool is running on the same machine as the target or the
source. Click Base Path and enter the full path to the WEB-INF folder containing the
browser.properties file (i.e. D:\Program Files\Apache Software Foundation\Tomcat
9.0\webapps\browser ). DO NOT include the WEB-INF folder and browser.properties file in
the specified path.

Click Test to confirm the path is correct and the file exists. Edit if invalid and retest

11
Once the Test is successful click Add to add the base path. Select Clear to remove the path.

7. Next, click the Add Connection Icon (database icon with the green +)to save the connection.
It will now be listed under the dropdown list. Repeat steps as necessary to add additional
connections (source and target environments). If updates to connections are required it is
recommend that you use select the connection name and use the Remove Connection Icon
(database icon with the red X) to drop the connection - Then repeat the steps to setup and
recreate the connection.

12
Copy connections from a previous version

NOTE: A connections.pickle file containing the configuration of connections is located in

...\Configuration Stager\persist. This file is not human-readable nor intended for direct editing.

Connections configured in a previous version of the Configuration Stager can easily be

transferred to a new version by copying the connections.pickle file located at ...\Configuration
Stager\persist from the old version to the new version.

Set up connection databases

Tell the Configuration Stager which database is the Source and which is the Destination:

1. Under Set up Connection Databases, use the dropdowns to select the source and the
destination databases. Note that the destination database dropdown is ignored when
exporting to file and the source database is ignored when migrating from file.

2. Click Ok.

13
3. The Source: and Destination: databases are now shown on the top left of the
Configuration Stager.

14
Direct Staging
1. Before proceeding with configuration staging fully backup your destination and target
environments. Do not run the Configuration Stager without performing this step.
2. Select Direct on the Configuration Stager and Set up connections to tell the
Configuration Stager which is the source and which is the destination server. If either
connection is not set, it will not be possible to run the staging.

3. Click Run Comparison

4. The Comparison dialog will popup, listing the differences between the source and
destination databases. Tabs list the Tables, Views, Procedures, Sequences and
Functions missing or extra on the destination server - where there are no differences
between the source and destination server, None is displayed on the tab. The system
comparison is executed against the entire ds3_userdata user on the target and source
environments and is not limited to the Browser projects and Studies protocols
configuration select for porting.

15
5. Click on the Browser icon (Br).

6. The Select Browser Project dialog will popup, where you can select the Browser Projects to
stage from the source database.

16
7. Projects moved to the right-hand panel will be staged. Multiple specific projects can be
selected using repeated Ctrl-click, or multiple contiguous projects using Shift-click/arrow key,
and then moved by using the arrow buttons between the two lists.

8. When projects have been selected, press Ok to close the popup. The selected projects will be
listed in the selected projects box.

9. Click on the Studies icon (St).

17
10. The Select Studies Protocol dialog will popup, where you can select the Studies Protocols to
Stage from the source database.

11. Protocols moved to the right-hand panel will be staged. Multiple specific protocols can be
selected using repeated Ctrl-click, or multiple contiguous projects using Shift-click/arrow key,
and then moved by using the arrow buttons between the two lists.

12. When protocols have been selected, press Ok to close the popup. The selected protocols will
be listed in the selected protocols box. The name of the protocol, the protocol type, and version
are displayed (recall protocols may have the same name and type - use the version to
differentiate amongst them). Note: The Configuration Stager supports comprehensive staging of
notebook type protocols. Other types (e.g. Screening, DMPK) may require manual addition of
protocol type-specific configuration.

13. After you have selected the desired configurations click Run Staging.

18
14. Confirm the details of the Staging and click Proceed.

15. If the Oracle version differs between the source and the destination, the migration will abort.
If the Browser/Platform versions differ or the source SQL references an object outside of the
schema , a warning will be displayed.

Dotmatics recommends staging across systems run the same Browser/Platform versions. Never
stage from a source environment running a more recent Dotmatics software version than
the target environment. Oracle objects outside of ds3_userdata should be identified and

19
created/updated on the target environment by your system administrator before running the
Configuration Stager.

16. Use the console window (that opened with the .exe) to monitor the staging progress.

The duration of the staging process is dependent on the amount of configuration data and the
complexity of the configuration to be staged. In some extreme instances the process may take a
few hours and log updates may appear infrequently. DO NOT CLOSE THE
CONFIGURATION STAGER APPLICATION WHILE IT IS RUNNING.

NOTE - If there is a mismatch between column data types on the source and destination and the
data type is VARCHAR2 on the source, it will be changed to CLOB on the destination. If the
data type is of any other format, such as DATE or BINARY_FLOAT, the staging will abort.

17. On successful completion of the staging, a popup is displayed and the console window
displays the message Configuration Staging Complete.

20
18. Restart Tomcat on the destination / target server and perform recommended quality assurance
tests.

21
Export a Configuration
When exporting a configuration, the Configuration Stager also exports Oracle and system
version information for that configuration. When importing the file, the tool uses that
information to determine whether it is possible to import that configuration onto the target
server. It is not possible to import a configuration when the Oracle versions do not match.

The export will be saved as a Python-based .pickle file located in the saved_configs folder of the
Dotmatics Configuration Stager.

1. Select Export on the Configuration Stager and Set up connections to tell the
Configuration Stager which is the source server. If the source is not set, it will not be
possible to export the staging.

2. Enter a filename for the export and click Ok.

3. Click on the Browser icon (Br).

22
4. The Select Browser Project dialog will popup, where you can select the Browser
Projects to stage from the source database.

5. Projects moved to the right-hand panel will be staged. Multiple specific projects can be
selected using repeated Ctrl-click, or multiple contiguous projects using Shift-click/arrow
key, and then moved by using the arrow buttons between the two lists.
6. When projects have been selected, press Ok to close the popup. The selected projects will
be listed in the selected projects box.

23
7. Click on the Studies icon (St).

8. The Select Studies Protocol dialog will popup, where you can select the Studies
Protocols to Stage from the source database.

9. Protocols moved to the right-hand panel will be staged. Multiple specific protocols can be
selected using repeated Ctrl-click, or multiple contiguous projects using Shift-click/arrow
key, and then moved by using the arrow buttons between the two lists.
10. When protocols have been selected, press Ok to close the popup. The selected protocols
will be listed in the selected protocols box. The name of the protocol, the protocol type,
and version are displayed (recall protocols may have the same name and type - use the
version to differentiate amongst them). Note: The Configuration Stager supports
comprehensive staging of notebook type protocols. Other types (e.g. Screening, DMPK)
may require manual addition of protocol type-specific configuration.

24
11. Click Run Staging.

12. Confirm the details of the Staging and Proceed.

13. Use the console window (that opened with the .exe) to monitor the staging progress.

25
 The duration of the staging export process is dependent on the amount of configuration
data and the complexity of the configuration to be staged. In some extreme instances the
process may take a few hours and log updates may appear infrequently. DO NOT
CLOSE THE CONFIGURATION STAGER APPLICATION WHILE IT IS
RUNNING.

14. On successful completion of the staging, a popup is displayed and the console window
displays the message Configuration Staging Complete

15. A Python .pickle file containing the staged configuration will be created, with the
filename entered earlier, in the saved_configs folder of the Dotmatics Configuration
Stager.

If making multiple exports, navigate to Direct and back to Export in order to adjust the filename
of the exported file.

26
Import a Configuration
After a configuration has been exported from a server, it can be imported onto a different server
as long as that server supports it.

 If the Oracle version differs between the file and the destination, the staging will abort.
 If the Browser/Platform versions differ, a warning will be displayed. Dotmatics
recommends staging across systems run the same Browser/Platform versions. Never
stage from a source environment running a more recent Dotmatics software version
than the target environment.
 Source SQL referencing an object outside of the schema are not ported. Oracle objects
outside of ds3_userdata should be identified and created/updated on the target
environment by your system administrator before running the Configuration Stager.

The import will be from a Python .pickle file.

1. Select From-File on the Configuration Stager and Set up connections to tell the
Configuration Stager which is the destination server. If the destination is not set, it will
not be possible to import the staging.

2. In the popup, navigate to the Python .pickle file to be used and click Open. The selected
file will be displayed on the tool. The pickle file to be used must be located
in the saved_configs folder of the Configuration Stager.

3. Click on the Browser icon (Br).

27
4. The Select Browser Project dialog will popup, where you can select the Browser
Projects to stage from the source file.

5. Projects moved to the right-hand panel will be staged. Multiple specific projects can be
selected using repeated Ctrl-click, or multiple contiguous projects using Shift-click/arrow
key, and then moved by using the arrow buttons between the two lists.
6. When projects have been selected, press Ok to close the popup. The selected projects will
be listed on the tool.

28
7. Click on the Studies icon (St).

8. The Select Studies Protocol dialog will popup, where you can select the Studies
Protocols to Stage from the source file.

9. Protocols moved to the right-hand panel will be staged. Multiple specific protocols can be
selected using repeated Ctrl-click, or multiple contiguous projects using Shift-click/arrow
key, and then moved by using the arrow buttons between the two lists.
10. When protocols have been selected, press Ok to close the popup. The selected projects
will be listed in the selected projects box.

29
11. Click Run Staging.

12. Confirm the details of the Staging and Proceed.

30
13. As noted above, if the Oracle version differs between the file and the destination, the
staging will abort. If the Browser/Platform versions differ or the source SQL references
an object outside of the schema , a warning will be displayed.

14. Use the console window (that opened with the .exe) to monitor the staging progress.

The duration of the staging process is dependent on the amount of configuration data and
the complexity of the configuration to be staged. In some extreme instances the process
may take a few hours and log updates may appear infrequently. DO NOT CLOSE THE
CONFIGURATION STAGER APPLICATION WHILE IT IS RUNNING.

NOTE - If there is a mismatch between column data types in the file and on the
destination and the data type is VARCHAR2 in the file, it will be changed to CLOB on
the destination. If the data type is of any other format, such as DATE or
BINARY_FLOAT, the staging will abort.

15. On successful completion of the staging, a popup is displayed and the console window
displays the message Configuration Staging Complete.

31
16. RRestart Tomcat on the destination / target server and perform recommended quality
assurance tests.

32
Checking Staged Configuration
It is important to confirm that ported configuration using the Configuration Stager is functional
on the destination / target environment. After a successful staging and restart of the system on
the destination server Dotmatics recommends a complete and comprehensive validation of all
ported configuration. Perform end-to-end testing on all ported protocols and projects.
Additionally, validate any pre-existing configuration that may share configuration and data
sources with ported projects and protocols.

Configuration errors may result in irreversible data capture issues and data corruption if
users attempt to use an impacted system. If issues are identified during testing immediately
stop use of the system, resolve the issues, and/or consider rolling the system back to a
backup taken prior to staging. Consider disabling users or shutting down the system (stopping
Tomcat) until you can correct the issues.

Recommended Configuration Staging Quality Assurance Checks for Administrators:

 Check the that target environment starts up and that the login page is accessible from
a web browser.
 Check that staged Browser protocols and Studies projects are accessible. An admin
may have to update user permissions and access to recently staged configuration.
 Check application settings. Intended behavior of configured protocols and projects may
be impacted by global Studies and Browser/Platform applications settings. Many may not
be ported during staging and require manual configuration.
 Check that staged Browser projects are functional. Confirm data sources, forms, and
project properties. Run queries using all displayed data sources, and preferably all fields.
Confirm buttons and display boxes on forms behave as expected. Test view-wizard views
and exports. Check data pivots.
 Check pre-existing Browser projects that share configuration with staged project.
Staged updates to configuration may cascade to associated projects utilizing common
configuration.
 Check that staged Studies protocols are functional. Create and experiment, add data
and files, complete the experiment. Check Check that forms and form objects
appropriately appear in the protocol. Check all buttons and triggered actions behave as
expected. Confirm settings and properties.
 Check Studies protocols that share configuration with staged protocols. Staged
updates to configuration may cascade to associated protocols utilizing common
configuration.
 Manually update non-notebook type Studies protocol configuration. The
Configuration Stager supports comprehensive staging of notebook type protocols. Other
protocol types (e.g. Screening, DMPK) may require manual addition of protocol type-
specific configuration.

33
  Check integrations with staged Studies protocols and Browser Projects. The
Configuration Stager will not impact linked applications and does not port data sources
outside of ds3_userdata. Confirm the functions of integrated applications and connected
data sources. This may include Dotmatics Register, Cascade, Bioregister, Inventory,
amongst others.

Log Files
The Configuration Stager keeps track of all the activity and transactions in a central log file.
This is typically useful during when there is a problem and Dotmatics support may ask for a
copy of the log file to help resolve a problem.

The log file can be useful for troubleshooting and is found in the extracted installation folder,
…\Configuration Stager\log directory.

The contents of the logfile.log can be viewed using Notepad++ or equivalent.

34
Appendix
• Notifications and Error Messages

• Limitations

Notifications and Error Messages

This reference lists the error messages and notifications displayed by the Configuration Stager in
case of compatibility problems or other conditions such as SQL violations or version issues.

 Errors
o Oracle Version Mismatch
o Table Column Data Type Mismatch
 Notifications
o Platform/Browser Version Mismatch
o External Project References

Errors

Errors will cause the staging to abort.

Oracle Version Mismatch

If the source and destination Oracle versions differ, a message highlighting the versions will be
displayed and the staging will abort.

The destination (or source) server will need updating before it will be possible to proceed with
the staging.

35
Table Column Data Type Mismatch

If there is a mismatch between column data types on the source and destination and the source
data type is not VARCHAR2, the tables and columns that differ will be displayed in a message
and the staging will abort.

The columns in the tables highlighted will need to be changed to the same type before it will be
possible to proceed with the staging.

Notifications

Notifications will not cause the staging to abort but will need user acknowledgement in order to
proceed.

Platform/Browser Version Mismatch

If the versions of Platform/Browser differ between the source and destination, a notification will
be displayed.

If the staging proceeds, any changes between the versions will need to be accounted for and
manual updates made on the destination.

External Project References

If SQL on the source references objects outside of the Schema, a notification will be displayed.

36
If the staging proceeds, these objects will need to be manually created or updated on the
destination.

37
Limitations
• What is staged?

• What is not staged?

What is staged?

 Browser:
o Browser projects
o Browser project types
o Browser project forms and form objects
o Browser project data sources (tables, views, dynamic data sources, data source
joins)
 Studies:
o Studies notebook protocols
o Studies protocol settings
 Basic settings
 Per experiment properties
 Processing script settings
 Studies protocol properties (SQL properties, Report Config,…)
o Associated column alters
o Associated column lookups

What is not staged?

 Browser:
o Objects outside the platform schema, including c2c/Bioregister
o Browser project property settings (see Project Properties menu)
 Editable project additional options
 Browser-project associated Vortex template files
 Other settings
o Browser project-associated pivots
o Platform system management settings (query templates, depiction SQL,
schedules, connections,…)
o Browser users, user groups, user permissions and access settings.
o Platform licenses
o Platform authentication settings
o Platform data source settings
o Platform depiction settings
o Platform email settings
o Platform event settings
o Platform IP filtering settings
o Platform application and application component settings
o Platform logging settings

38
 o Browser.properties file settings
 Studies:
o Objects outside the platform schema, including c2c/Bioregister
o Screening Ultra calculations and configuration
o Studies dictionary configuration (Conc Scheme, Properties, Reasons,…)
o Book naming masks and schemes, Notebook types, Notebook summary SQL
o Eln Disable Clone settings
o Experiment relationship types
o Folder mask manager settings
o Monitor alters and group settings
o Project access settings
o Studies users, user groups, user permissions and access settings.
 Bioregister - Bioregister hosts it’s own dedicated configuration staging toolkit.
 Other Dotmatics applications, including cascade, inventory, gateway, reaction
workflows, nucleus, vortex, and registry application settings

39