Regression Tool User Manual 2019-03-01

Regression Suite Automation Tool 03/01/2019
Regression Suite Automation

Tool
User Manual
1.200.25035.5
COPYRIGHT MICROSOFT CORPORATION 2019

1
Table of Contents
Overview .................................................................................................................................................................... 4
End-to-end Flow..................................................................................................................................................... 4
Intended Usage and Test Classification ................................................................................................................. 5
Business cycle (Business process) testing .......................................................................................................... 5
Data Integration testing ..................................................................................................................................... 6
Unit and component testing .............................................................................................................................. 6
Task Recordings and Validation of Expected Values.............................................................................................. 6
Prerequisites .............................................................................................................................................................. 7
Dynamics 365 for Finance and Operations test environment ............................................................................... 7
Excel ....................................................................................................................................................................... 7
Azure DevOps......................................................................................................................................................... 7
Authentication Certificate...................................................................................................................................... 7
Install .......................................................................................................................................................................... 9
Install the Tool ....................................................................................................................................................... 9
Install Selenium and Browser Drivers .................................................................................................................... 9
Run ........................................................................................................................................................................... 10
Configure.............................................................................................................................................................. 11
Load Test Cases and Create Automation Files ..................................................................................................... 12
Modify Test Parameters....................................................................................................................................... 13
Run a test as a specific user ............................................................................................................................. 14
Run a test in the context of a specific company .............................................................................................. 14
Infolog Validation ............................................................................................................................................. 14
Run ....................................................................................................................................................................... 14
Investigate Results ............................................................................................................................................... 15
Test Response Times ........................................................................................................................................ 17
Save your Work .................................................................................................................................................... 17
Chaining of Test Cases ............................................................................................................................................. 17
Best Practices ........................................................................................................................................................... 19
Authoring test cases using the Task Recorder ..................................................................................................... 19

2
Using RSAT ........................................................................................................................................................... 19

How to modify a Task Recording. ........................................................................................................................ 21
Troubleshooting and Known Issues ......................................................................................................................... 21

3
Overview
The Regression Suite Automation Tool (RSAT) significantly reduces the time and cost of user acceptance testing.
User acceptance testing is typically required before taking a Microsoft application update or applying custom
code and configurations to your Dynamics 365 for Finance and Operations production environment.
It enables functional power users to record business tasks using the Finance and Operations task recorder and
convert them into a suite of automated tests without the need to write source code. Test libraries are stored
and distributed in Lifecycle services using the Business Process Modeler (BPM) libraries and fully integrated with
Azure DevOps Services (Azure DevOps) for test execution, reporting and investigation. Test parameters are
decoupled from test steps and stored in Microsoft Excel files .
End-to-end Flow
This tool is part of the end to end flow described below. Finance and Operations, along with LCS and Azure
DevOps, provide a set of tools for test case authoring (using task recorder), configuration, execution,
investigation and reporting.
Get familiar with this process by reading this topic.
(https://docs.microsoft.com/en-us/dynamics365/unified-operations/dev-itpro/lifecycle-services/using-task-
guides-and-bpm-to-create-user-acceptance-tests)

4
Note
Using Lifecycle Services (and BPM) is recommended but not mandatory. You can manually create test cases in
Azure DevOps and attach developer recording files to your Azure DevOps test cases. Developer recording files can
be created directly from the task recorder pane in Finance and Operations.
You must name the developer recording file Recording.xml before attaching it to an Azure DevOps test case.
Intended Usage and Test Classification

Business cycle (Business process) testing
The Regression Suite Automation Tool is intended to be used for business cycle tests and scenario tests that
usually occurs at the end of the development lifecycle, sometimes referred to as user acceptance testing.
Business cycle testing consists of a smaller number of test cases than component or unit testing. This is
illustrated in the pyramid below.

5
Data Integration testing

For integration tests that rely on the data management framework (also known as DIXF), the data task
automation framework enables you to configure and automate the testing of your data integration scenarios.
Unit and component testing

For unit and component tests, we recommend using the SysTest framework and the build/test automation tools
of Finance and Operations (not covered in this document). For more details, go to the continuous delivery
homepage.
Task Recordings and Validation of Expected Values

An important component of a test case is validation of expected values. You can define validation parameters
during the authoring of your test cases using task recorder. While recording, right-click on a control and select
CurrentValue under the Task Recorder > Validate menu. This action becomes a validation step that you can use
with RSAT (The control value will become a validation variable in the automatically generated Excel parameters
file).

6
For more information on how to create task recordings, see this topic.
Prerequisites
Dynamics 365 for Finance and Operations test environment
Your Dynamics 365 for Finance and Operations test environment must be running Platform update 15 or newer.
The Regression Suite Automation Tool must have access to your Dynamics 365 for Finance and Operations test
environment via a web browser.
Excel
You need Microsoft Excel installed to generate and edit test parameters.
Azure DevOps
You must have an Azure DevOps project to store and manage your test cases, test plans and test case results.
You will need an Azure DevOps Test Manager license. For example, if you have a Visual Studio Enterprise
subscription, you already have a license to Test Manager. For more information, visit the pricing page for Azure
DevOps.
Authentication Certificate
To enable authentication, you need to generate and install a certificate. Below are step-by-step instructions. If
you are not familiar with this process, get help from your system administrator. Make sure you have Windows
Kits installed on your machine. If you do not have Windows Kits installed on your machine, you can download
the Windows 10 SDK from https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk . You
will need these two components only for the steps described in this document.
1) Windows SDK Signing Tools for Desktop Apps

2) Windows SDK for UWP Managed Apps.

7
Generate the certificate

To generate a certificate file, open a command line window as Administrator and run the following command
from a command window. When you are prompted to enter a private key password, enter None. Create the
C:\Temp folder if it does not already exist on your computer.
1. cd c:\Program Files (x86)\Windows Kits\10\bin\10.0.17763.0\x64

(Your exact folder maybe different, depending on where you have installed the windows SDK) You can
also use Windows Kits 8.1.
2. makecert.exe -n "CN=127.0.0.1" -ss My -sr LocalMachine -a sha256 -len 2048 -cy end -r -eku
1.3.6.1.5.5.7.3.1 c:\temp\authCert.cer
Note
The certificate needs to be generated on the same computer that the test tool is running on.
Install certificate to Trusted Root and extract the thumbprint

Double-click authCert.cer to install the certificate.
1. Click Install Certificate…

2. Select Local Machine > Place all certificates in the following Store > Browse > Trusted Root Certification
Authorities, clicking Next through each screen.
3. Leave the Password field blank.
4. In the Certificate dialog, browse to Details section and look for the Thumbprint.
5. Preserve the Thumbprint, you will need it later. See next sections for more details.
Configure the AOS machine to trust the connection

Configure the Dynamics 365 for Finance and Operations AOS to trust the test automation connection.
1. Modify l to make the AOS trust this certificate. To do that, open IIS and find Dynamics 365 for
Operations in the list of sites. Click Explore to open and find the file wif.config.

8
+
2. Update the wif.config file by adding a new authority entry for your certificate and authority name as
shown here.
<issuerNameRegistry type="Microsoft.Dynamics.AX.Security.SharedUtility.AxIssuerNameRegistry,
Microsoft.Dynamics.AX.Security.SharedUtility">
<authority name="CN=127.0.0.1">
<keys>
<add thumbprint="ccbc124d0a119xxxxxxxxxxxxxxxxxxxx841e797" />
</keys>
<validIssuers>
<add name="CN=127.0.0.1" />
</validIssuers>
</authority>
3. Restart IIS.
Install
Install the Tool
Download Regression Suite Automation Tool.msi to your machine and double-click it to run the installer.
Install Selenium and Browser Drivers

RSAT requires Selenium and web browser driver libraries. It will prompt you if needed libraries are missing and
will automatically install them for you. Select yes when you see the following (or similar) dialogs.

9
Alternatively, follow these steps for manual installation:
1. Download Selenium 3.13.1 from here.

 Or go to http://docs.seleniumhq.org/download/ > "Previous Releases" (the top left corner) > 3.13 and download
"selenium-dotnet-strongnamed-3.13.1.zip"
2. Install the Selenium libraries
 Unzip the downloaded file.
 Unpack the file dist\Selenium.WebDriver.StrongNamed.3.13.1.nupkg (To unpack this file, add
the .zip suffix to the file and unzip it).
 Copy the contents of the folder Selenium.WebDriver.StrongNamed.3.13.1.nupkg\lib to
"C:\Program Files (x86)\Regression Suite Automation Tool\Common\External\Selenium"
3. Download the Internet Explorer driver version 3.4.0 from here.
 Or go back in the browser, open the 3.4 folder and download "IEDriverServer_x64_3.4.0.zip"
4. Unzip the downloaded file and move the contents to "C:\Program Files (x86)\Regression Suite
Automation Tool\Common\External\Selenium".
If you want to use Google Chrome as your browser:
1. Go to https://sites.google.com/a/chromium.org/chromedriver/downloads
2. Download chromedriver_win32.zip from the latest/current release.
3. Unzip the downloaded file and move the contents to "C:\Program Files (x86)\Regression Suite
Automation Tool\Common\External\Selenium”
Run
Open the tool from your desktop.
The following sections explain how to configure the tool, load test cases from Azure DevOps, generate
automation files, modify test parameters, run tests, investigate results, and save your work back to Azure
DevOps.

10
Configure
Click “Settings” button on the top right and fill the dialog as follows:
1. Azure DevOps Url: This is the URL of your Azure DevOps project. Example:
https://yourAzureDevOpsUrlHere.visualStudio.com
2. Access Token: Access token that allows the tool to connect to Azure DevOps. You need to create
a Personal Access Token or use an existing one that you have saved. For more information, see
https://www.visualstudio.com/en-us/docs/setup-admin/team-services/use-personal-access-
tokens-to-authenticate
3. Project Name: The name of your Azure DevOps project.
4. Test Plan: The Azure DevOps test plan that contains your test cases. For more information, see
https://www.visualstudio.com/en-us/docs/test/manual-exploratory-testing/getting-
started/create-a-test-plan
5. Click “Test Connection” to test your connection to Azure DevOps.
6. Hostname: Hostname of the Dynamics 365 for Finance and Operations test environment.
Example: <myaos>.cloudax.dynamics.com. Do not include the https:// or http:/ prefix.
7. SOAP Host Name: SOAP Hostname of the Dynamics 365 for Finance and Operations test
environment. The SOAP hostname is typically the same as the Hostname with a soap suffix.
Example: <myaos>soap.cloudax.dynamics.com. Do not include the https:// or http:/ prefix.
8. Admin User Name: Email address of an admin user on the Finance and Operations test
environment.
9. Thumbprint: Thumbprint of the authentication certificate described earlier in this document.
10. Working directory: Folder location for storing test automation files, including Excel test data
files. For example: C:\Temp\RegressionTool

11
11. Test Run Timeout: Timeout (in minutes) of a test run. All active windows are closed, and
pending test cases are failed.
12. Company name: Specify a company name to use as you default Finance and Operations
company during creation of Excel Parameters files. It can be changed later my editing an Excel
file.
13. Click Apply to apply and save your settings.
14. Click Close to close the dialog.
The Save and Open buttons allow you to save your settings for reuse later. Click Save to save your current
settings into a configuration file on your computer. Click Open to restore your settings from a configuration file
on your computer.
Load Test Cases and Create Automation Files

Click Load to download test cases and test case automation files from Azure DevOps. All test cases belonging to
the test plan specified in the Settings dialog are downloaded.
Test cases are organized by test suites under a common test plan. These are test suites you created in your
Azure DevOps project. Using this tool, you can work with one test suite at a time.
If the tool fails to load any test case, verify that your test plan in Azure DevOps is properly created and contains
the desired test suites and test cases.
If this is the first time you load this test plan, the Parameters File column will be blank. You will need to create
test automation files for your test cases. Test automation files consist of:
12
 Test parameter files (Microsoft Excel files contain test case parameters)
 Other binary and XML files needed to execute the tests.
When you click the New button, test automation files are generated in your working directory. The Excel test
parameter files will appear on the grid under Parameters File.
You can also generate binary and XML files only without overwriting your parameter files: Use submenu New-
>Generate Execution Files only to re-generate execution files only and leave Excel files untouched. This is typical
when you install a new version of the tool, you can update your execution files while preserving the test
parameter files.
Modify Test Parameters

This section describes how to modify Excel files to specify input and validation parameters for your test run.
Select one or more test cases you want to modify and click Edit. This will open an Excel window for each
selected test case. Alternatively, you can open the Excel files directly from the working directory.

13
In addition to the General tab, the Excel file contains a data tab for every Finance and Operations form that the
test case visits.
Select the desired form (Excel tab) you want to edit and identify the parameter values that you want to change.
Values are identified by their control name. If you are not sure which control is the right one, open the form in
Finance and Operations, right-click the control whose value you want to change and select form information.
Save the Excel files.
Run a test as a specific user

By default, tests are executed in Finance and Operations using the admin role. If you want to run the test as a
specific security role, specify the e-mail address of a Finance and Operations user under the Test User parameter
in the General tab of the Excel parameter file. The Test User must be a valid user of the Finance and Operations
environments you are connecting to. The test will run under the security roles that the specific user belongs to.
You need version 1.200 or newer for this feature to be functional.
Run a test in the context of a specific company

The General tab of the Excel parameter file also allows you to specify the name of a Finance and Operations
legal entity (Company). The test will run in the context of this company. You can specify your default company in
the Settings dialog of the tool.
Infolog Validation
Excel parameter files that are generated using version 1.200 or newer contain an InfologValidation tab.
You can enter messages in this tab under Infolog Validation. After a test case completes execution, it validates
that the messages specified here appear in the Finance and Operations info log. The test case will fail if these
messages are not found.
Run
Click Run to execute the selected test cases. Only test cases with existing automation files can be run. The tool
will open Finance and Operations and execute these tests with the data you entered in Excel.
Note: You can modify the order in which test cases are executed using the up and down arrow buttons.

14
Investigate Results
When all test cases complete execution, Pass or Fail will be populated in the Result column. You can click on the
result to see error messages.
More investigation details are available in Azure DevOps. From your Azure DevOps project page, go to Test >
Runs.
Select the desired test run. It will include the results of all tests that were executed during that run.

15
You can open a failed test result and review the ErrorMessage section for information about the failure.

16
All error messages are also available locally under

C:\Users\$YourUserName\AppData\Roaming\regressionTool\errormsg-<TestCaseId>.txt
Test Response Times

In addition to execution logs, the duration of a test case is also available in the test result.
You can also review the response time of each step of the test case by opening the BaseTime.xml file attached
to the test result.
You need version 1.200 or newer for response times to be available.
Save your Work

To preserve your work, click Upload. This will upload test automation files (Including Excel test parameter files)
of all selected test cases to Azure DevOps for future use.
Once test automation files are uploaded to Azure DevOps, the next time your use RSAT, even from a different
computer, you will simply need to Load then Run without the need to generate test execution files or edit Excel
parameter files.
Chaining of Test Cases

One of the key features of the Regression Suite Automation tool is the chaining of test cases, i.e. the ability of a
test to pass values to other tests. Test cases are executed according to their defined order in the Azure DevOps
test plan (This can also be updated in the test tool itself), so it is very important to correctly order the tests if you
want to pass variables from one test case to the other.
To save the value of a variable while recording the test in Task Recorder, right click the field and click Task
recorder > Copy as shown below. This will save the variable in the recording file; this variable can be used in
subsequent tests.

17
In the Excel data file, these saved values appear in the Saved variables table of the General Tab.

18
To reuse these variables during test playback, copy the variable name and use it in place of a parameter value in
the data file of another test (or the same test) as shown below.
Note that variables can only be passed between tests during the same test run and the name of the variables
must match exactly.
Best Practices
Authoring test cases using the Task Recorder
1. Make sure all your recordings start on the main dashboard of Finance and Operations (or Retail).
2. Keep individual recordings short and focus on a business task performed by one user, like creating a
sales order. This simplifies maintainability and reusability of test cases.
3. Chart controls are not supported, any task recording actions related to charts will be ignored by RSAT
during test case playback.
4. When creating a recording make sure to click on a tab header even if the tab is already open. For
example, you can switch to another tab and then click back on the needed tab to activate it before using
a control on it. This will make your recording more reliable during test case playback.
5. If you edit your recording after it has been created, do not change the order of the steps. You can add or
delete a step instead.
6. RSAT cannot playback any test step that is not recognized by the task recorder. For example: You cannot
upload a file from the local disk during playback of a test case.
7. RSAT cannot playback a “page refresh” step, avoid refreshing a page while recording your test.
Using RSAT
1. Upon opening the tool for the first time, click on Settings and ensure you have all the desired settings.
2. Before installing a new version of the tool, it is recommended to uninstall the previous version.
3. When you install a new version of the tool, regenerate test execution files.

19
It is not necessary to regenerate Excel parameter files unless you want to take advantage of new
features available in a newer format of parameter files.
4. For test parameters needing a unique value (e.g. product receipt number in the Product Receipt form or
invoice number in the Vendor Invoice form), use the RandBetween(a,b) Excel function to generate a
unique number every time the test case is executed.
5. The default values in Excel come from the task recording. For certain Reference Group controls (such as
storage dimensions, tracking dimensions etc…), it stores the key of the lookup instead of the value (e.g.
2 instead of SiteWH). It is recommended to update these fields with the actual value in Excel so that the
test is more robust and resilient to changes.
The following are typical usage sequences:
Sequence A: Load->New->Edit->Run->Upload
1. Click Load.
2. Select desired test case(s).
3. If some test cases have no parameter file, click New to generate them.
4. Optional step: Edit Excel file to run tests with different parameters.
5. Click Run.
6. Optional step: After a successful run, click Upload to upload all generated automation files
(including Excel test parameter files) to Azure DevOps.
Sequence B: Load->Edit->Run->Upload
1. Click Load.
(If all parameter files already exist, there is no need to click New to regenerate)
2. Optional step: Edit Excel file to run tests with different parameters.
3. Click Run.
4. Optional step: After a successful run, click Upload to upload all generated automation files
(including Excel test parameter files) to Azure DevOps.
Sequence C: Load->New (for All Test Cases)->Run->Upload
This sequence is recommended after installing a new version of the tool.
1. Click Load.
2. Select all test cases in the test plan and click New.
3. Edit Excel parameter files
4. Click Run
5. After a successful run, click Upload to upload all generated test artifacts to Azure DevOps.

20
How to modify a Task Recording.

If you wish to modify an existing task recording, note these best practices.
In the Finance and Operations web client, open the Task recorder pane and start editing the recording using
the Edit Recording option.
You can add or remove any step; however, do not re-order steps because RSAT will not be able to playback
the edited recording correctly. If you must reorder steps, re-create the recording from scratch or re-order by
deleting and re-adding a step.
When you finish editing, playback the recording in the Finance and Operations client verify that all steps pass
and then save it. It is then safe to playback the edited recording using RSAT.
Troubleshooting and Known Issues

1. To troubleshoot issues with the tool during Playback operation, open the developer error log located at
C:\Users\$YourUserName\AppData\Roaming\regressionTool\playback\[TestName]Log.txt .
Analyze the error message to determine the possible cause of a failure.
2. The authentication certificate must be created and installed by an administrator. If it is not created by
an admin, you will observe an error message when you try to run a test case: "Cannot access Finance
and Operations environment. Verify your settings and make sure the environment is available."
3. Your desktop resolution should be set to 100% to run the tests successfully. Windows display settings >
Scale and layout.

21
4. Excel data tabs display the system identifiers for data variables. It does not display friendly names.
5. You can’t initiate a run while Excel test parameter files are open.
6. If you have used a previous version of the test tool, please uninstall it before installing the new version.
7. If you are testing against a Standard Acceptance Test Sandbox environment (Tier2) or any other multi-
box environment, and you get the following error when you run a test:
There was no endpoint listening at

https://<yourURL>soap.sandbox.operations.dynamics.com/Services/AxUserManagement/Service.svc
/ws2007FedHttp that could accept the message. This is often caused by an incorrect address or SOAP
action…
Assuming you have specified the correct SOAP hostname in the settings dialog, run the following
Powershell script on your client computer where the test tool is installed:
if ((Test-Path HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319))
Set-ItemProperty HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319 -Name

SchUseStrongCrypto -Value 1 -Type dword -Force -Confirm:$false
You can also set the registry key manually.
8. Sometimes Chrome browser don’t work with the tool due to AD security settings. In this case, change
your settings to use Internet Explorer browser.

22

Regression Tool User Manual 2019-03-01

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Regression Tool User Manual 2019-03-01

Uploaded by

Copyright:

Available Formats

Regression Suite Automation Tool 03/01/2019

Regression Suite Automation

COPYRIGHT MICROSOFT CORPORATION 2019

COPYRIGHT MICROSOFT CORPORATION 2019

Using RSAT ........................................................................................................................................................... 19

COPYRIGHT MICROSOFT CORPORATION 2019

Get familiar with this process by reading this topic.

COPYRIGHT MICROSOFT CORPORATION 2019

Intended Usage and Test Classification

COPYRIGHT MICROSOFT CORPORATION 2019

Data Integration testing

Unit and component testing

Task Recordings and Validation of Expected Values

COPYRIGHT MICROSOFT CORPORATION 2019

1) Windows SDK Signing Tools for Desktop Apps

COPYRIGHT MICROSOFT CORPORATION 2019

Generate the certificate

1. cd c:\Program Files (x86)\Windows Kits\10\bin\10.0.17763.0\x64

Install certificate to Trusted Root and extract the thumbprint

1. Click Install Certificate…

Configure the AOS machine to trust the connection

COPYRIGHT MICROSOFT CORPORATION 2019

Install Selenium and Browser Drivers

COPYRIGHT MICROSOFT CORPORATION 2019

Alternatively, follow these steps for manual installation:

1. Download Selenium 3.13.1 from here.

If you want to use Google Chrome as your browser:

COPYRIGHT MICROSOFT CORPORATION 2019

COPYRIGHT MICROSOFT CORPORATION 2019

Load Test Cases and Create Automation Files

Modify Test Parameters

COPYRIGHT MICROSOFT CORPORATION 2019

Save the Excel files.

Run a test as a specific user

You need version 1.200 or newer for this feature to be functional.

Run a test in the context of a specific company

COPYRIGHT MICROSOFT CORPORATION 2019

COPYRIGHT MICROSOFT CORPORATION 2019

COPYRIGHT MICROSOFT CORPORATION 2019

All error messages are also available locally under

Test Response Times

You need version 1.200 or newer for response times to be available.

Save your Work

Chaining of Test Cases

COPYRIGHT MICROSOFT CORPORATION 2019

COPYRIGHT MICROSOFT CORPORATION 2019

COPYRIGHT MICROSOFT CORPORATION 2019

The following are typical usage sequences:

Sequence C: Load->New (for All Test Cases)->Run->Upload

This sequence is recommended after installing a new version of the tool.

COPYRIGHT MICROSOFT CORPORATION 2019

How to modify a Task Recording.

Troubleshooting and Known Issues

COPYRIGHT MICROSOFT CORPORATION 2019

There was no endpoint listening at

Set-ItemProperty HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319 -Name

You can also set the registry key manually.

COPYRIGHT MICROSOFT CORPORATION 2019

You might also like