RPA Design &

Development
Image, Text and
Advanced Citrix
Automation
Agenda
Image and Text Based Automation

Keyboard Based Automation

Information Retrieval

Native Citrix Automation Challenges

Best Practices
Learning Objectives

Understand the particularities of

information retrieval

Learn how Image and Text

Based Automation operate in
UiPath Studio
Learn how to perform Virtual
Machine Application
Automations

Learn how Keyboard Based Automation

operate in UiPath Studio
Image and Text Based Automation

Keyboard Based Automation

Information Retrieval

Native Citrix Automation Challenges

Best Practices
Image and Text Based Automation
Image based automation Text based automation

Start App Stop App Start App Stop App

Relies on user interface Relies on identifying the

element recognition. text from certain user
interface elements.
Image and Text Based Automation (Contd.)
The Screen Scrapping is a UiPath feature which can extract text from running apps.

The Screen Scrapping

tool is found in the
UiPath Studio ribbon.
The Native Citrix
application is found by
expanding the
Recording drop down
button.
 Image Based Automation
Image recognition activities can simulate human behavior by using images as a means
of identifying UI elements.
Comparing captured
image to predefined
image:

80% match
Comparing captured
image to predefined
image:
Comparing captured
image to predefined
image:

100% match 65% match

Start App Stop App

Image Based Automation (Contd.)
Image Based Automation can be deployed in cases where scripting is not enabled.

Scripting
Disabled

Only Image Based Automation

can be used since scripting is
Start App Stop App
disabled.
Start App Stop App

UiPath will search images

resembling the action buttons of
this application.
Find Activity

Wait Image
Find Image Find Image Image Exists Vanish  
Match

Monitoring Events File Activities

Image and Text Based Automation

Keyboard Based Automation

Information Retrieval

Native Citrix Automation Challenges

Best Practices
Keyboard Based Automation
UiPath Studio features activities that simulate any type of keyboard input that a regular
human user would use.

Send Hotkey Type Into Type Secure Text

Keyboard Based Automation (Contd.)

Start App Stop App

With the use of hotkeys

and shortcuts, specific
actions can be set to
reduce the number of Start App Stop App
steps performed.
Start App Stop App
Image and Text Based Automation

Keyboard Based Automation

Information Retrieval

Native Citrix Automation Challenges

Best Practices
 Information Retrieval
In a Virtual Machine (VM), to retrieve information out of an application, we can either
copy the data from an editable text or use Optical Character Recognition (OCR) to
extract the relevant information.

Copy Text:
Selects editable texts
and copies it to the
UiPath Environment

Screen Scraping:
Scrapes a portion of an
image and uses OCR to
extract the information.
Image and Text Based Automation

Keyboard Based Automation

Information Retrieval

Native Citrix Automation Challenges

Best Practices
 Native Citrix Automation Challenges

• Selectors are natively

generated for Citrix
• UiExplorer cannot App elements
extract selectors for
elements inside the • The selectors can be
application in a Citrix used for full range of
Desktop application UiAutomation activities
• In these scenarios, we
rely on Image, Text and
Keyboard automation
When and Where to Perform Certain Actions

Wait for a specific Verify that the app Wait for the
application state is downloading or application’s
ready loading icon to
disappear

Visual Anchor
Example

Visual Anchor
Example
App Ready
Image and Text Based Automation

Keyboard Based Automation

Information Retrieval

Native Citrix Automation Challenges

Best Practices
Best Practices

Exclude icon
Use find background
  image/text to look
for elements
Use shortcut keys
Be in the correct to navigate
state of the
Use shortcut keys
  to launch
application
Be sure the
applications
application is in
the foreground
Takeaways
Image and Text based Automation,
Information Retrieval
There are two main processes
Image based automation
involved in Retrieving Information:
relies on user interface
 Copy Text
element recognition. Text
 Screen Scraping
based automation relies on
identifying text from user
interface elements.
.

Slide 20
/
Takeaways
Native Citrix Automation Challenges, Best
Practices
UIExplorer cannot extract
selectors for elements
inside the applications in a
Citrix Desktop application.
Proper variable and argument should
be named by following the correct
naming conventions.
The Best way to open an
application in the virtual
environment is to create a
shortcut for the respective
applications.

Slide 21
/
Questions & Answers
What is the full form of OCR?

a) Optical Character Recognition.

b) Optimum Character Recognition

c) Optical Code Recognition.

Why do we use image -based automation?

a) We use the image-based automation where we

want to ensure that we are able to capture the Ui
element
b) Where the selector fails to capture the Ui element

c) All of the above.

The best way to open an application in virtual
environment is……………………….

a) By creating shortcut

b) By assigning Hot key to the application

c)All of the above

Next Steps

Excel Data Tables & PDF