Adam Toth Quality Architect: Improve Quality by Monitoring and Observing

Improve quality by monitoring and observing
Played different roles on different projects for various customers:
• Performance Engineering
• DevOps, Infrastructure testing
• Testing/Test Automation
• Solution/Quality Architecture
• Product Owner
Adam Toth
Quality • Resource Manager
Architect • Sales consultant
• Mentor, teacher and much more…

CONFIDENTIAL. Copyright © 1
Why do we need it?
Agenda Continuous testing
Shift-Right
Monitoring and Observing
Monitoring Challenges
When and How?
Conclusion
Why do we need it?
Why do we need to improve QA processes continuously?
Early feedback
Time To Market
CI/CD
Agile Transformation
Product
quality
Monitoring != Observing
Monitor: Observe:
to watch and check a to watch carefully the way something
situation carefully for a happens or the way someone does
period of time in order to something, especially in order to
discover something about it learn more about it:
What do we monitor? What do we observe?
• We monitor HTTP transactions • We observed that HTTP error rate is increased

• We monitor User Experience • We observed that Application rating is decreased
• We monitor fuel level in fuel tank at a car • We observe that the fuel tank is empty
Continuous testing
Traditional model compared to continuous testing
to Quality
Attention
Shift-Left Traditional Shift-Right

model model model
Plan & Develop & Test Deploy & Monitoring &

Design Build Release Alerting
What is continuous testing?
Automating builds, tests, processes,
reports, anything
Plan Release Deploy

Code
DEV OPS
Test
Monitor
Opearate
Build
Continous frequent changes

Shift-Right
Shift-right is the practice of performing testing, quality, and performance
evaluation in production under real-world conditions. Shift-right methods
ensure that applications running in production can withstand real user load
while ensuring the same high levels of quality.
Shift-Right != We test in Prod

The goal of shift-right testing is to ensure correct
behavior, performance and availability over the
production of an application.
Shift right is the practice of moving some testing later in the
DevOps process to test in production. Testing in production
uses real deployments to validate and measure an
application's behavior and performance in the production
environment.
How do I do this “Shift-Right”?
Deploy to „Shift-Right" Share

PROD testing feedback
Post Monitoring
Chaos
release &
testing
tests Observing
How do I do this “Shift-Right”?
Deploy to „Shift-Right" Share

PROD testing feedback
Monitoring
Post release Chaos
&
tests testing
Observing
Monitoring and Observing
It was invented before IT and DevOps…
Technology Out of Tech Everyday things
• Quality of products in any • Michelin Guide for • Weather

factory restaurants • Available movies at
• Traffic jam • Market trends Streaming providers
• Environmental footprint • Customer satisfaction • Evolving of AI
A few metrics what can be monitored, but there are much more
Infrastructure Application Users
• CPU, Memory, Disk • Errors/error rate • Frequent flows, dataset
• Networking • Response time • Diversity of users
• Scaling & uptime • Throughput • Customer tickets

created/rating
*See more details in the appendix! CONFIDENTIAL. Copyright © 15

What issues can be resolved by monitoring and observing?
Poor product quality Unhealthy system Lack of knowledge Unproper

about real users Nonfunctional
requirements
Monitoring Challenges
Challenges with Infrastructure monitoring
Pod
Pod
Pod
Pod
Increase horizontal/vertical Multiple systems and Infrastructure issues cannot

scaling instead of monitoring solutions be transferred to Business
application optimization level
Challenges with Application monitoring
NullPointerException
AI ArrayIndexOutOfBoundsException
ConstraintViolationException
IllegalStateException
Business and Dev team Lack of TestOps expert…and There are good patterns,
needs to be involved to less who can share feedback but new challenges can
create effective monitoring effectively appear anytime
Challenges with User monitoring
Feedback is not shared with Users monitoring cause Wrong implementation of

Application monitoring performance issues/alerts monitoring solution
team
When and How?
When do I do this?
SHOULD CONSIDER FOR T H I N K T W I C E I F YO U H AV E
• Complex system with many users, parameters • Lack of business flows, small amount of users
• Quality in focus • Production Feedback is not used
• Proper communication channels across • Quality is not important
business and technical teams • Business is not involved in testing, monitoring
• Long planned application lifetime activities
• Proper monitoring • Broken observability
Plenty of monitoring tools
What major factors need to be considered for tool selection?
AI or
Plans for
Traditional Confident
Business growth
choice
In house experts Expected
Technology stack benefits
Budget
Architecture
How to do it properly?
Metrics and measurement

Tools and observability
Prerequisites Monitoring experts' operational team
Train technical and business teams

Implementation (Dashboards, Alerts)
Installation Establish communication channels
Perform After changes

Regular basis
Operation Compare larger major releases
Conclusion
Benefits & Challenges
Short feedback loop Lack of experience
Cost effective Observability
Product confidence Broken feedback loop
Thank you!
IMPROVE QUALITY BY MONITORING AND
OBSERVING
Q&A
Appendix
References
Shift-Right Testing: The Emergence of TestOps

https://devops.com/shift-right-testing-the-emergence-of-testops/
Continuous Testing: Shift Left & Shift Right, Get it Right

https://www.inflectra.com/Ideas/Whitepaper/Continuous-Testing-Shift-Left-Shift-Right.aspx
Shift left vs shift-right: A DevOps mystery solved

https://www.dynatrace.com/news/blog/what-is-shift-left-and-what-is-shift-right/
Continuous Performance Management Basics

https://www.splunk.com/en_us/blog/learn/continuous-performance-management.html
Cambridge logos
https://dictionary.cambridge.org/
Key takeaways from Infrastructure monitoring
Goals
• React and detect of HW failures

• Optimize HW resources
• Decrease downtime
Approaches Benefits
• Areas to look: CPU, Memory, Disk, Network, Scaling • Reliability, security, performance
• Analysis together with DevOps/Ops teams • Early testing and outage prevention
• Cost optimization
Key takeaways from Application monitoring
Goals
• React and detect on SW failures

• Share feedback for further optimization
• Find hidden issues
Approaches Benefits
• Areas to look: Error rate, Response time, Throughput, • Better planning, highlight risky areas
Integration • Better product quality by finding hidden bugs
• Analysis together with Dev/Business teams. • Real time feedback about Business Health
Key takeaways from Users monitoring
Goals
• React and detect on issues reported

• Better user experience
• Improve planning, testing
Approaches Benefits
• Understand average visit duration and users' behavior • Better planning capabilities by knowing
• Crosscheck hidden/heated areas production data, scenarios
• Collect different datasets and parameters • Improved regression by risk-based approach
• Better and faster reaction to resolve or prevent
client issues
Do it as early as possible
How to start? Benefits

• Logging
• More product confident from the early stages
• Monitoring
• Alerting • It is cheaper to maintain the code
• Easier to develop new functionality
• Team can resolve issues faster
Why?
• Teams can start proactively think on:

• How to react if incident happens?
• How to optimize?
• How to improve performance and security?

Adam Toth Quality Architect: Improve Quality by Monitoring and Observing

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Adam Toth Quality Architect: Improve Quality by Monitoring and Observing

Uploaded by

Copyright:

Available Formats

Improve quality by monitoring and observing

Played different roles on different projects for various customers:

• DevOps, Infrastructure testing

Architect • Sales consultant

• Mentor, teacher and much more…

Monitoring and Observing

When and How?

What do we monitor? What do we observe?

• We monitor HTTP transactions • We observed that HTTP error rate is increased

Shift-Left Traditional Shift-Right

Plan & Develop & Test Deploy & Monitoring &

Plan Release Deploy

Continous frequent changes

Shift-Right != We test in Prod

Deploy to „Shift-Right" Share

Deploy to „Shift-Right" Share

Technology Out of Tech Everyday things

• Quality of products in any • Michelin Guide for • Weather

• Environmental footprint • Customer satisfaction • Evolving of AI

Infrastructure Application Users

• CPU, Memory, Disk • Errors/error rate • Frequent flows, dataset

• Networking • Response time • Diversity of users

• Scaling & uptime • Throughput • Customer tickets

*See more details in the appendix! CONFIDENTIAL. Copyright © 15

Poor product quality Unhealthy system Lack of knowledge Unproper

Increase horizontal/vertical Multiple systems and Infrastructure issues cannot

Feedback is not shared with Users monitoring cause Wrong implementation of

SHOULD CONSIDER FOR T H I N K T W I C E I F YO U H AV E

• Quality in focus • Production Feedback is not used

• Proper communication channels across • Quality is not important

business and technical teams • Business is not involved in testing, monitoring

• Long planned application lifetime activities

• Proper monitoring • Broken observability

What major factors need to be considered for tool selection?

Metrics and measurement

Train technical and business teams

Perform After changes

Short feedback loop Lack of experience

Cost effective Observability

Product confidence Broken feedback loop

Shift-Right Testing: The Emergence of TestOps

Continuous Testing: Shift Left & Shift Right, Get it Right

Shift left vs shift-right: A DevOps mystery solved

Continuous Performance Management Basics

• React and detect of HW failures

• React and detect on SW failures

• React and detect on issues reported

How to start? Benefits

• Teams can start proactively think on:

You might also like