O N

T I
B U
R I
S T
D I
O R
F
O T
Andri Apriyana SA, S.Si, M.M., CISA, CEH, CRMP, FSAI (Cand.)

N
Digital/AI Controls/Algorithms - Risk Analytics Associate Director @ Deloitte

Jakarta, 4 November 2020

 Covid-19 and The Future of Work

O N
The Future of Internal Audit T I
B U
Agenda

R I
T
Audit Planning in New Normal
S
D I
Analytics F O R
Embedded Internal Audit
O T
N
Discussion, Q&A
Confusion Matrix is not That Confusing…

O N
T I
B U
R I
S T
D I
O R
F
O T
N
“Type II Error” could be fatal to you and your business…

O N
T I
B U
I
R punah…
Auditor = berinovasiTatau
I S
D
O R
F
O T
N
Covid-19 and The Future of Work

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Source: Deloitte Research
Covid-19 and The Future of Work

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Source: Deloitte Research
Covid-19 and The Future of Work

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Source: google.com
Covid-19 and The Future of Work
The Future of Work, is not just about technology change and …new talent and organization arrangements are redefining
operational efficiency…. business models and creating opportunities

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Being prepared for the Future of Work is a key to enabling human centric change and future proofed environments.
Covid-19 and The Future of Work
Disruptors accelerating how we consider the future of work globally
Technology is everywhere… enables entirely different ways of working and the need for all
employees, not just IT leaders, to be “tech fluent”
O N
T I
unlock customer insights and remain competitive
B U
Tsunami of data… requires that all organizations access the power of big data and analytics to

R I
Cyber-Physical World… renews focus on efficiency and automation of manual tasks in
order to keep pace with competitors
S T
D I
Automation of jobs… creates new jobs with essentially human skills and a focus on
higher level value creation

O R
7 F Change in the nature of a career… lengthening life spans result in a
60-70 year career and continuous upskilling and career shifts
DISRUPTORS

O T Explosion in contingent work… provides access to a distributed

N talent pool that improves productivity and speed

Diversity and generational change… makes open

dialogue, employee empowerment, and inclusion an
organizational imperative
Covid-19 and The Future of Work
Emerging Technologies will continue to disrupt business-as-usual
Human augmentation

N
Brain-computer interfaces

O
> 10 years

Quantum computing

T I
Virtual Reality
Augmented Reality
B U Social robotics

Blockchain,
R I Conversational Computing
Virtual Assistance

Smart Contracts

S T Artificial Intelligence
2-10 years

Cognitive Computing
Sensors & The Internet of Things

D I Driverless vehicles
Commercial drones

O R
3D printing
Autonomous things

F
Additive manufacturing Algorithmic automation

O T
Social media & Digital platforms Big Data analytics
Past 20 years

N
Cloud computing

Internet
Enterprise systems
Covid-19 and The Future of Work
Emerging Technologies will continue to disrupt business-as-usual
With new realities known

81% 66% N
O90
T I %
of respondents
expect the use
of respondents
expect to reskill
B U of respondents
reported using
of AI to increase
or increase
significantly over
current
employees due
to automation
R I alternative
workers within
their organizations
the next 3 years

S T
over the next
three years
today

But action has been slow D I

26% O R 18% 25%
F
O T
of survey
respondents are
ready or very ready
of respondents said
their organizations
are significantly
respondents
reported having little
to no processes in

N to address the
impact of AI and
robotics on the
increasing their
investment in
reskilling
place for sourcing
and managing
alternative
nature of work programs workforce sources

Source: Deloitte Global Human Capital Trends Report

Covid-19 and The Future of Work
What is The Future of Work?

Forces of change are affecting three major

O N
dimensions of work: the work itself, who does
the work, and where work is done.
T I
To create value from these changes,
B U
organizations should take a broader perspective.

R I
The Future of Work is broader than just human S T
and machine collaboration, and includes all the
trends (technological, social, demographic, etc.) D I
workforce, and the workplace.
O R
that affect changes we are seeing in work, the

F
O T
N
Covid-19 and The Future of Work
Question to consider within your organizations…
WORK
74% of firms believe the skills gaps in
their workforce and local labor markets
are barriers to adoption of new
technology in their businesses
Every technology transformation is innately a human transformation
• Do you know the skills and
capabilities required to perform
the work you must do?
O T
each role?
N
• How are new technologies
changing the nature and need of
• How are you approaching the
redesign of jobs?
WORKFORCE
O N
WORKPLACE
T I
Executives expect talent shortages
across BU’s responsible for innovation
B U From 2012-2016, the percentage of
employees who did at least some work
and growth (e.g., R&D (75%) and
R I remotely rose (from 39% to 47%)
leadership (47%))
S T
D I
R
Questions to consider within your organizations
O
F • Do you have the talent that you
need?
• Do you have the business case to
transform how you operate?
• How are you upskilling/reskilling • How are you leveraging data and
the talent that you have?
• Do you know where you will find
the talent you do not have?
technology?
• How are you building a great
place to work?
• Do you have the right incentive
structures?
Source: The Future of Jobs Report 2018, World Economic Forum
Covid-19 and The Future of Work
Question to consider within your organizations…

O N
T I
B U
R I
S T
D I
O R
F
O T Who has an understanding of what
work can and can’t be done by smart

N machines and bots?

Responding to new realities will require you to strike a new balance in choices across cost,
value, and meaning
Covid-19 and The Future of Work
Impact of Covid-19 to The Future of Work

O N
The future of work trends are leading to an unleashed organization, connected on digital platforms, and able to
work anywhere at anytime. These changes are impacting all industries as advances in technology fundamentally
change the work itself, who does the work, and where it needs to be done.

T I
An organization with a Future of Work mindset is more resilient at coping with uncertain times and more able to
adapt and thrive, include in pandemic situation.

B U
WORK WORKFORCE R I WORKPLACE

In a digital world, machines and
humans will be enabled to do what
they do best independently and in
concert with each other. Companies
will challenge the idea that work
should be performed in a specific
S T
D I
Future organizations will identify,
attract and engage people with the
The future workplace will blend the
physical and virtual environments
O R
right skills and experience to
achieve its mission and purpose. An
and will deliver a consistent employer
identity to all types of employees,
F
organization will be made up of many
types of workers and will engage
contractors, and other stakeholders.
Personalized experiences will

O T
physical location by exploring ways partners and other key stakeholders empower people to be their best,

N
to digitize some or all work
through advances in robotics, IoT,
cloud, and other technologies.
anywhere in the world. Continuous
learning will support the upskilling
needed across the workforce to
address looming talent shortages.
balanced selves. Collaboration tools
and platforms will support dynamic
work locations and asynchronous
collaboration.
Covid-19 and The Future of Work
Practical Step to accelerate transition to the Future of Work

Start from where you are

O N
01 and extend it across the
enterprise
T I
B U
Understand the digital
R I
capabilities you need to scale
success
02 03
S T Explore your people
experience

D I
O R
Apply what you’ve learned
in practices, places, & with 04 05
Measure your success with
data to ensure continuous

F
technology improvement

O T
N
Covid-19 and The Future of Work

O N
The Future of Internal Audit T I
B U
R I
T
Audit Planning in New Normal
S
D I
Analytics F O R
Embedded Internal Audit
O T
N
Discussion, Q&A
Key Organizational Success

Risk & Assurance

O N
Too few controls,
T I
limited oversight.
excessive risk
B U
Performance
R I
Too many controls,
S T
excess oversight.
throttled business.
D I
O R
F
O T
Performance are at the core of business effectiveness and
financial earning. N
Risk & Assurance are at the core of business integrity and
efficiency.
Yet bullet-proof control systems are not realistic.
Key Organizational Success

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Key Organizational Success

O N
T I
B U
R I
S T
D I
O R
F
O T
N
The Future of Internal Audit

O N
T I
B U
R I
S T
D I
O R
F
O T
N
The Future of Internal Audit

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Don’t let “Type II Error” caught you in the act…

O N
T I
B U
I
R punah…
Auditor = berinovasiTatau
I S
D
O R
F
O T
N
The Future of Internal Audit

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Internal Audit 3.0
The Future of Internal Audit
TODAY TOMORROW
3rd Lines 3rd
LODLines
3

O N
T I 2nd Lines

B U LOD 2

R I
S T
D I
O R Automated Core Assurance

2nd Lines F
O T
1 Lines
N 1st
LODLines
1

Assurance by design
The Future of Internal Audit

O N
T I
B U
R I
S T
The IIA’s Three Lines Model: An update of the Three Lines of Defense - July 2020

D I
O R
F
O T
N
Covid-19 and The Future of Work

O N
The Future of Internal Audit T I
B U
R I
T
Audit Planning in New Normal
S
D I
Analytics F O R
Embedded Internal Audit
O T
N
Discussion, Q&A
Internal Audit - Mission

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Internal Audit – Core Principles

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Audit Plan Process
Other Assurance
1st Line : Operational Management
Regulator
2nd Line : Risk management & compliance/controllership function
Providers
3rd Line : Internal Audit
Risk Assessment
Understand
The Organization
• Analyse Objectives,
Strategies, Initiatives, KPI,
Priorities, and Structures.
• Analyse business process
changes, past assurance
Key Activities
results (incl. From other
assurance provider).
• Consult with key
stakeholders (BOD, BOC
O T
Management).
• Surveys, Interviews,
N
or Audit Committee and
Brainstorming & Research.
• Revisit Audit Universe.
Assurance
Combined
Management
O N
Committee
Audit
Planning Fieldwork Closing & Reporting
T I Monitoring
DEVELOPMENT&QUALITY ASSURANCE
B U
R I
Identify, Assess, Coordinate with
S T Estimate Draft, Communicate &
Prioritize Risks &
I
Other Assurance
D
Resources Finalize Plan
on risks and audit plan
ongoing update
Implement plan with
Opportunities Providers
• Understand and validate • Identify, coordinate • Assess and analyse • Drafting, present, solicit feedback
register/universe.
O R
organization updated risk and validate the result
of other assurance
internal audit team
knowledge, skill, and
and revise annual audit plan.
• Discuss and accommodate BOD &
F
• Linking objectives and strategies
to risk and opportunity.
providers. competencies.
• Revisit audit frequency • Analyse the need to
Audit Committee assurance and
consulting request.
• Validate identified risks: fraud, and timing. call upon internal • Propose, get approval, and
strategic, internal, external,
strategic, IT, environmental,
• Revisit Assurance Map. expert/specialist
• Calculate FTE,
Mandays/hours to
present final plan to BOD & Audit
Committee.
social, governance, third-party.
• Validate risk assessment
approach vs measurement vs
coverage.
• Validate risk assessment with
implement plan.
• Reprioritise plan.
• Analyse requirement
for manpower & skill
• Implement upon Approval.
• Maintain ongoing communication
with BOD & Audit Committee.
• Communicate changes on the
audit plan.
Management. fulfilment, technology,
• Revisit Risk Profile & Universe. schedule and funding.
Audit Plan in New Normal Play as a “Looking Ahead” roles
to support crisis response teams
- Providing objective and valuable
organizational knowledge
- Ensure lessons learnt exercises are Help organization look forward
prformed and recorded
Calibrate audit planning and risk coverage approach
Embrace short-term prioritization and regular review/updates
- Help ensure key risk areas identified
by crisis response teams.

O N
Provide an objective voice and real time assurance to teams who need to
make decisions quickly. Consider
- Attend project steering groups and provide an independent, objective
to the audit plan to mirror the changing pace of risk and
assurance needs.

T I
voice to help challenge management’s thinking on risk
- Feedback the design of new or amended controls prior to

Adjust coverage to ensure pragmatic and balanced consideration of risk

B U implementation as a result of changing work environment

- Improve the long and time-consuming reports to deliver insights sooner
- Anticipate by supplementing the first and second line feedback in
Collaborate with key stakeholders to understand any new or elevate
risks, and to assess how best to support with the provision of assurance.

R I identifying potential risk areas arising from theimmediare business

impact, and increased digital environment

S T Assure without disrupt critical operational areas

Timing to engage with broader stakeholders
Early discussion and early engagement with Audit Committee and other
interested stakeholders such as Regulators is imperative to help IA functions
understand their perspectives and key priorities to best add value/agree D I at a time of crisis
-
-
-
Consider remote auditing
Adopt or increase the use of technologies to deliver the work
Accelerate the deployment of analytics to deliver IA work
changes on the IA plan.

O R remotely

Work closely with other assurance providers to reduce

business disruption F For certain sectors, gap coverage will impact the ability for Audit
Committees meet annual reporting requirements on controls
When IA work may be reduces, adopt a limitation of scope approach

O T
Reduce overlap with other assurance provides including External
Audit, Compliance and Enterprise Risk Management. Identify
Cancel or defer activities where it’s
necessary to support critical
where needed.
where non-essential IA work is deferred or re-shaped

N
opportunities to collaborate and assist. organization’s response to Covid-19

Discuss with the Board and formally agree with the Audit
Committee of re-purposed IA resources
- Supporting the front line
-
-
-
-
Consider regulatory/statutory requirements impact
Consider CAEs ability to comment on control environments
Consider limited vs reasonable assurance
Must be clear on what has and has not been considered
- Contingency/pandemic plans review
- Contingency response review across the year to adopt scope limitation where needed.
- Scenario planning review
- Key incident scenarios exercises
- Key metric real time reporting for under stress process
- Secondments into first line
Covid-19 and The Future of Work

O N
The Future of Internal Audit T I
B U
R I
T
Audit Planning in New Normal
S
D I
Analytics F O R
Embedded Internal Audit
O T
N
Discussion, Q&A
Nature of Risks

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Move more effort from reactive post risk event
to proactive pre risk event
using analytics methods
Nature of Controls
Assurance without Analytics Assurance with Analytics
O N
T I
B U
R I
S T
D I
O R
F
O T
N
source : Handscombe, K. (2007). Continuous
Auditing From a Practical Perspective
Data Analytics Culture
“Statistical Modelling : The Two Cultures” (Breiman, 2001)
Statistics start with data, Data generated by “NATURE” blackbox :

O N
y NATURE x
T I
B U
Two different culture & apporach in analyzing data :
1. Parametric modelling approach (descriptive, inference, econometrics)
R I
linear regression -
S T
Culture population 98% of all statisticians
y logistic regression
etc
x -
-
I
y : by given stochastic model & validate by goodness-of-fit
D
Conclusion on data model not about nature mechanism
-
R
Statistical community use with exclusive commitment

O
F
2. Algorithmic modelling approach (algorithm, machine learning , deep learning)

y unknown T
O x -
-
Culture population 2% of all statisticians
Treats data mechanism is complex & unknown
N
Decision Trees
Neural Network
-
-
Validation using accuracy
Conclusion about nature mechanism

etc
Data Analytics Culture
“To Explain or to Predicts” (Shmueli, 2010)

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Algorithmic Approach
Google Statistics (DMR, 2016)
- Monthly searches : + 100 billion searches
- Monthly unique searchers : 1,2 billion users

N
- Searches per Second : +2,3 million query

O
T I
B U
R I
S T
D I
O R
F
O T
N
Data Analytics Path
Automation
Speed with precision
With the advantage of robotic & cognitive automation (R&CA)
O N
technologies, organizations can transform business processes,
augmenting human labor with a digital workforce to drive better
T I
outcomes.

B U
Insights
R I
Intellect with purpose
With the advantage of advanced pattern recognition and machine
S T
learning, organizations can find previously unrecognized patterns
in data, continuously learn from those patterns, and execute
actions and report on them. What’s more, the entire system is
able to get smarter and improve over time. D I
O R
Engagement
Connection with meaning F
O T
With the advantage of intelligent technologies such as natural
language processing with speech recognition, as well as advanced

N
engagement principles, such as predictive, proactive, and real-
time interactions, organizations can deliver mass personalization
at scale.

Source: Deloitte Research

Analytics Embedded Internal Audit
1st Line of Defense : Operational Management
External Audit

Assurance
Combined

Management
Regulator

2nd Line of Defense : Risk management & compliance/controllership function

Committee
Audit
3rd Line of Defense : Internal Audit

I O
Risk Assessment Planning
DEVELOPMENT&QUALITY
Fieldwork Closing & Reporting
ASSURANCE
U T Monitoring

•
•
Establish Audit Universe
Risk Profiling
• Communicate Audit Plan
• Scoping & assignment letter
•
•
Opening meeting
Desk Analysis
•
•

I
Review report
B
Generate draft report • Auditee feedback
• Issue status recording &
Key Activities

•
•
•
Assurance Requirement
Develop Assurance plan
Business update monitoring
• Understand Process, Key
Performance, Inherent Risk &
Key Control
•
•
•
Perform working paper
Data Analysis

T
Listing draft issues
•
•
R
Closing meeting
Audit Document
Management
Follow Up
• Release monitoring report
• Conduct quality assurance
• Regulatory update
monitoring
• Combined assurance map
• Develop Risk Control Matrix
• Generate Audit Program &
Test Strategy
•
•

I
Fieldwork review
S
Issue confirmation •
•
•
Issue final report
Update risk assessment
Combine assurance report
update
• Risk Assessment review
• Perform detail audit plan
• Planning review
D •
•
Management report
Audit Committee report

R
Analytics Use Case

F O
O T
KPI Dashboard
N RPA (Robotic Process Integrated Audit & Risk
Management System Tools
Dynamic Reporting/
Report intelligence
Diagnostic Natural Language
Processing
Automation)
Digital Assets

Assurance Chatbot
KRI Dashboard Automated QA Fraud Risk Indicator Predictive Automatic Sampling
(virtual assistant)
Automated Data Analysis Tools (TABK) Prescriptive
KCI Dashboard Digital boardroom
Follow Up
Case Study

O N
T I
AutomationIB U
T R
(Robotic ProcessSAutomation)
D I
O R
F
O T
N
Case Study -– Automation
Efficiency
One Click Desk Audit
O N
T I
B U
R I
C O L L E C T - D ATA CLEAN-UP

S T SAMPLING
Before : Proses sampling manual
C O N F I R M AT I O N
Before : Login, pilih menu, pilih parameter,
dan download data manual
After : Proses download data Otomatis
D I
Before : Proses merapikan data
terdownload secara manual di excel
After : Proses merapikan data Otomatis
berdasarkan kriteria yang disepakati
After : Proses sampling otomatis dengan
stratified random sampling
Before : Permbuatan file konfirmasi
sample data exception dipilih secara
manual

O R After : Permbuatan file konfirmasi

sample data exception otomatis

F (36 Data Points)

tergenerate.

O T
N Impact : Desk Audit Efficiency from 3 mandays to 3 manhours
Case
CaseStudy
Study– Efficiency
-–Automation
Efficiency
Report Mining using NLP
O N
T I
B U
R I
S T
D I
35

O R
F
20

O T
10 10
8 7
N
5
3 3 2 2 2 2 2 2 1 1 1 1 1 1

A B C D E F G H I J K L M N O P Q R S T U
Case Study

O N
T I
Insights B U
R I
S T
D I
O R
F
O T
N
Case Study -– Insights
Continuous Auditing

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Case Study -– Insights
Continuous Auditing

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Case Study -– Insights
Continuous Auditing

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Case Study -– Insights
Continuous Monitoring

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Predictive Analytics using Machine Learning

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Predictive Analytics using Machine Learning

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Predictive Analytics using Machine Learning

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Predictive Analytics using Machine Learning

O N
T I
Size : 25.599 Size : 83.015
B U
Period : 2016-2017
Keyword : vulnerability, exploits,
cve, attack , hack, zeroday, 0day.
Period : 1999-2017

R I
S T
D I
twitteR package

O R ROAuth package
RMySQL package
F
O T
N
Predictive Analytics using Deep Learning

O N
T I
B U
R I
S T
D I
O R
F
O T
N
Covid-19 and The Future of Work

O N
The Future of Internal Audit T I
B U
R I
T
Audit Planning in New Normal
S
D I
Analytics F O R
Embedded Internal Audit
O T
N
Discussion, Q&A
 Thank You
Andri Apriyana SA, S.Si, M.M., CISA, CEH, CRMP, FSAI (Cand.) O N
Email : andri.apr@gmail.com T I
WA : 0811 979 1511 B U
R I
S T
D I
O R
F
O T
N