Power Virtual Agents Chatbot Review

A guide to chatbot development

Mallika Mainali
Table of contents

Overview ……………………………………………………………………………………………. 2
Objective ……………………………………………………………………………………………. 2
Functional Requirements …………………………………………………………………… 3
User Interface ……………………………………………………………………………………. 4
Technical Requirements …………………………………………………………………….. 8
Chatbot Development ……………………………………………………………………….. 9
Publish chatbot ……………………………………………………………………………….. 15
App manifest ……………………………………………………………………………………. 16
Performance Analysis ……………………………………………………………….……… 18
Conclusion ……………………………………………………………………………………….. 21

1
Overview
The purpose of the creation of this MS Teams chat bot is to explore possible ways
to utilize Power Virtual Agents in providing login related assistance to users and
make the information on OTIS SharePoint more accessible in Teams. The goal is to
make the chat bot function as a compilation of various knowledge bases.
The chat bot will be accessible on Microsoft Teams. It will be a third-party add
that can be downloaded from App Studio in Microsoft Teams.

Objective
This document will provide an analysis of the functional requirements and
technical requirements for this project. It will also provide a description for the
user interface of the application and the process of chatbot development.
Additionally, it will explain different ways Power Virtual Agents allows admins to
monitor the bot’s performance.

2
Functional Requirements
The following list provides a detailed description of the general functional
requirements that have been considered in developing this chatbot:
1. Chatting:
The system should allow users to chat with the bot.
The system should be able to inform users if an information is not available
to it and return a generic error message such as “Sorry, I am not able to
find a related topic. Can you rephrase and try again?”

2. Input format:
The system should be able to store user input in both structured data types
like boolean values, and unstructured data types such as a paragraph.

3. Logs:
The system should maintain a log of the questions asked during each
session.
It should also be able to keep a record of the analysis of each session
including if the user was satisfied, abandonment rate, resolve rate and
escalation rate.

4. Feedback:
The user should be able to leave a feedback in the form of a rating (1-5) and
provide suggestions if their questions were not answered.

5. Administrate:
The administrator should be able to update, delete and add the topics for
the chatbot, and view the analysis, feedbacks and logs provided by users
during their sessions.

3
User Interface
The chat bot is available on the Microsoft Teams application. Once logged in on
the demo tenant, user can go to Apps and search for ‘demo bot’ to install the chat
bot. It is also available under ‘Built for your org’ category on Apps.

4
Here is a demonstration of a conversation with the chatbot:

5
For the question ‘Was the information helpful?’, if the user selects ‘no’, the
system will ask the user to provide feedback so that it can improve. This feedback
will be stored in the backend for administrator to view so that they can make
necessary modifications.

The system will also ask for user’s rating on each session which will be available
for the administrator for better assessment of the bot’s performance.

6
7
Technical Requirements
The chatbot is created using Power Virtual Agents and Bot Framework Composer.
It is supported by Microsoft Teams. It also uses other Power applications such as
Power Automate to perform some tasks. The following represents the technical
requirements for the chatbot development:

1. Availability
The system is available in MS Teams 99.99% of the time for any 24-hour
period.

2. Authentication
The system is protected from unauthorized access. To prevent
unauthorized access, only users with UT System credentials can install the
chatbot.

3. Portability
The system is able to run on different operating systems since it only
requires MS Teams.

4. Third party interactions

The system can interact with other Power apps and Office 365 apps when
needed.

8
Chatbot Development

Overview
Chatbots can be a powerful tool to improve the information flow in a workplace.
Since they are available around the clock, chatbots can easily process requests at
any time. Chatbots can also be useful in gaining insights into user behavior. As the
developers can access the feedbacks and issues of users, these feedbacks can
help in analyzing the concerns and improving the system. One of the ways to
create powerful chatbots is by using Power Virtual Agents. These bots are easy to
create and maintain and offer a comprehensive analysis of user behavior.

Power Virtual Agents is available as both a standalone web app, and as a discrete
app within Microsoft Teams. Although most of the functionality between the two
is the same, the web app is usually used by an IT admin or to create complex
chatbots while Microsoft Teams app can be used by any member of an
organization to create efficient chatbots.

Creating chatbot using Power Virtual Agents

Resources utilized
Power Virtual Agents license (or trial)
Microsoft Teams
Bot Framework Composer (optional)

Total time
The preparation and bot-creation takes a short time (less than 15 minutes) since
Power Virtual Agents does the backend processing while creating the bot. Once
the bot is created, the time required to add respective topics, develop an app
manifest to deploy it to MS Teams, and test and maintain the system depends on
the type of bot and developer’s requirements.

9
Instructions

Environment Preparation

1. Go to Power Virtual Agents marketing page

(https://powervirtualagents.microsoft.com/en-us/) and select Start free.

2. Login using your work account. For a trial chatbot, admin test account can be
used. Once logged in, you will be issued a license for Power Virtual Agents.

10
3. Once you click on ‘Get Started’, you will be brought to a screen to create a new
bot. After the bot has been named, you can select ‘Create’ to start the bot-
building process.

As the environment for bot development has been set up, the bot is in the
process of creation. During the building process, you will see a notification at the
top of the Power Virtual Agents portal indicating that the bot is still being built.

Once the bot is created, the banner will change, and you will have full access to
the Power Virtual Agents portal.

11
Now that the bot is created, it can be customized to fit the user’s needs. Under
‘Topics’, you will be able to view topics that have been created by Power Virtual
Agents to show the use of different types of conditions and entities. You can add
a topic, delete it, or update an existing topic in this channel.

Creating a new topic

To create a new topic, you will need to select ‘New topic’ and add trigger phrases
for that topic. Once that is done, you will be able to customize its functions in the
authoring canvas.

In the authoring canvas, you will be able to customize the topic by asking question
to the user, displaying a message, going to another topic, or calling an action such
as creating a flow using Power Automate or connecting the chatbot to a third-
party application.

12
In the figure below, I have added a question and have offered four options for the
user to choose from. I have also redirected each option to its respective topic so
that the chatbot will prompt a particular topic when the user chooses that option.

Fig: Customizing the authoring canvas by asking a question, giving 4 options, and redirecting to
another topic based on each option

13
One thing that makes Power Virtual Agents powerful is that we can call an action
within a topic and connect it to external applications. One of the ways we can
utilize this feature is by making the bot send an email on behalf of the user. In the
figure below, an action is called using Power Automate that takes two variables
‘Reason’ and ‘employee’ and uses them to send an email to an email address
saved as a variable ‘email’. The first variable gets its value from user input to the
previous question. Similarly, the second variable gets its value from the user’s
display name which will be the same as their display name on MS Teams, and the
third variable has been assigned within the flow. Therefore, when this action is
called, it will automatically send an email on the employee’s behalf with a reason.

14
Publishing the chatbot
Once the chatbot has been created and tested, it will need to be published for
other users to access it. To publish it on MS Teams, it needs to be turned on in the
‘Channels’ section in homepage.

So, on the homepage, once you select Manage > Channels > Microsoft Teams,
click on ‘Turn on Teams’. This will enable MS Teams channel for Power Virtual
Agents and allow the chatbot to be published on Teams platform.

Once it is on, you will be able to share it within Teams and publish it organization-
wide by submitting it for admin approval.

15
App Manifest

The application manifest contains a definition of all the attributes of an

application object in the Microsoft identity platform. It also serves as a
mechanism for updating the application object. Bots in Teams are defined by an
app manifest JSON file that provides the basic information about your bot and its
capabilities.

Since our chatbot is created using Power Virtual Agents, all content authored in it
renders naturally in Teams. The IT administrators, business analysts, domain
specialists, and app developers can design, develop, and publish intelligent virtual
agents for Teams without having to setup a development environment. So, its app
manifest is automatically created by Power Virtual Agents once we publish the
chatbot.

With the use of app manifest, we can either upload our chatbot directly to Teams
App Studio or share it to our organization by requesting admin approval. To view
and edit the bot’s app manifest, you will have to click on ‘Submit for admin
approval’ option in Power Virtual Agent. To do this, go to Manage, then Channels
on the side navigation pane. Select Microsoft Teams > Submit for admin approval.

16
From there, you can either submit it for admin approval or download the app
manifest and use that manifest to publish your app to a wider platform. If you
choose to submit for admin approval, admins will be able to feature the chatbot
prominently as an app in Microsoft Teams, and pre-install for people in the
organization. On the other hand, by downloading the bot’s app manifest, you can
make detail changes to the manifest directly and share with admin or users in the
tenant. You can also directly upload the app manifest into Microsoft Teams if your
organization's policy allows you to upload custom applications.

Here, you will also be able to see the App ID for your application which is a unique
Microsoft generated identifier for your bot. This will be useful if you decide to
publish the bot in Azure portal, create an app package for Teams or use Bot
Framework.
When you download the manifest, you will be asked if you want to make any last
updates to your bot information. Once you have confirmed everything is correct,
select Download manifest. The manifest will be generated as a .zip file and will be
saved according to your browser's save settings.

17
Analytics
Power Virtual Agents offers many ways for developer analysis. The Summary page
gives you a broad overview of the bot's performance. It uses artificial intelligence
technology to show you which topics are having the greatest impact on escalation
rate, abandon rate, and resolution rate. It includes a variety of charts with
graphical views of the bot's key performance indicators.

The Engagement Rate Drivers, Abandon Rate Drivers, and Resolution Rate Drivers
charts use natural language understanding to group issues as topics. These charts
show you the topics that are having the most impact on the performance of the
chatbot. By default, the page shows you key performance indicators for the last
seven days.

18
Summary charts

The Summary charts summarize the key performance indicators for the specified
time and the percent change over that period.
The table below explains how different drivers show various aspects of the bot’s
performance.
Description Details

Total The total number of sessions within the specified time period.
sessions

Engagement The percentage of total sessions that are engaged sessions. An engaged
rate session is a session in which a user-created topic (as opposed to a system
topic) is triggered, or the session ends in escalation. Engaged sessions can
have one of three outcomes—they are either resolved, escalated, or
abandoned.

Resolution The percentage of engaged sessions that are resolved. A resolved session is
rate an engaged session in which the user receives an end-of-conversation
survey that asks the question "Did that answer your question?" and the user
either does not respond or responds Yes.

Escalation The percentage of engaged sessions that are escalated. An escalated session
rate is an engaged session that is escalated to a human agent.

Abandon The percentage of engaged sessions that are abandoned. An abandoned

rate session is an engaged session that is neither resolved nor escalated after one
hour from the beginning of the session.

CSAT The graphical view of the average of customer satisfaction (CSAT) scores for
sessions in which customers respond to an end-of-session request to take
the survey.

A blue up-and-down indicator next to the value indicates the percent change in a
positive direction. A red indicator indicates the percent change in a negative
direction.

19
We can also customize our chatbot so that users can provide necessary feedbacks
that might not be shown in the Summary Page. For example, when a user wants
the issue to be escalated, we can customize the chatbot to ask if the information
provided in the session was useful. If the information is not useful, the user can
describe ways to improve the bot’s performance. This feedback will be visible to
the admin so that they can make necessary changes to the bot and improve its
performance. This technique can help make a distinction between an
uninformative-escalated session and an informative-escalated session.
The screenshot below shows an application of this technique in the authoring
canvas.

Power Virtual Agent also offers the option to download the sessions and look at
the conversations the bot has engaged in. This file is downloaded as a .csv file that
gives session ID, time, initial user message, topic name the bot utilized, topic ID,
and chat transcript. It also shows the session outcome (Abandoned, Unengaged,
Resolved, or Escalated).

20
Below is an example of a session file. This session file gives an in-depth analysis of
the bot’s interaction and can help admins understand its language processing
capabilities. It can also give a better insight into any changes that need to be
made to enable the bot to be more effective.

Conclusion
Power Virtual Agents enables anyone in an organization to build chatbots with
built-in natural language processing and no-code graphical interface. With its
Teams integration, it allows developers to easily create, author, test, and publish
chatbot into Teams all within the Microsoft Teams interface. It can also be used to
create a chatbot that can be shared externally, such as one that interacts with
customers outside an organization using Power Virtual Agents web app. It is also
beneficial because it provides multiple options of evaluating the bot’s
performance for system admins. Thus, utilizing this Microsoft tool is an efficient
way of fulfilling any scenario that arise within an organization.

For more information on Power Virtual Agents and chatbot development, here is
a guide from Microsoft to build a sample HR bot.

21