Professional Documents
Culture Documents
life-cycle
100 %
List: query.do, user=Bob, action=showAll
Call
PaymentService.transferCash
Performance and Impact Analysis with PureModel
PureModel = PurePath + PureStack
PureStack
Technology ™
Execution times, CPU, Remoting,
Automatic Process
and Host Monitoring
Zero Config
26.7s Load Time 33! Service Calls 171! Total SQL Count
5kB Payload 99kB - 3kB for each call!
LEARN MORE: http://apmblog.dynatrace.com/2016/03/02/build-optimize-test-operate-highly-scalable-applications-with-dynatrace/
Confidential, Dynatrace, LLC
All-In Agile: Use application metrics as Feedback Loop & Quality Gates
Automated Tests: Identify Non-Functional Continuous Integration: Auto-Stop Bad Builds based on
Problems by looking at App Metrics AppMetrics from Unit-, Integration, - Perf Tests
Tip: Feed data back into your test tool! Tip: integrate with Jenkins, Bamboo ...
I am a Developer ...
App Metrics
DB, Exceptions, CPU, ...
for each Test Step
#3: Or Stream Data to your Tools!
#4: Integration Powered by REST API
I am a (Dev/App)Ops ...
6. How healthy is
3. What’s our load on 5. Do we have a
4. Do we have any the underlying
that Application? severe application performance infrastructure?
failures (HTTP 500, problem?
Exceptions, …)?
Analyze End User Performance
Diagnose Web Request Performance
Analyze Performance: Visits
Transaction: Infrastructure
• Process and Host Health involved in these transactions
Transactions and Business Transactions: Mapped
1. Performance
Overview per
Application
2. Alerts on unusual
throughput based on
baseline
3. Automatic
Failure Detection
4. Alerts on
performance
degradations
5. Is it Application or
Infrastructure related?
Correlate Process and
Host Health per
Application
6. Automatically 7. Baselines and Violations on Failure Rate, Response 8. Is the Business Transaction
Time and Throughput are calculated per Business impacted by Process or Host
identify critical Health Issues?
Transaction Splitting
Business Transactions
Memory Diagnostics & CPU
Diagnostic
Main Pains for Problem Hunters
The JVM/CLR contains a lot of objects.
How to analyze, especially in a load or
production environment?
Component/Layer Component/Layer
Component/Layer
How can single users create so How can certain business transactions create
many objects? so many objects?
Typical Anatomy of Java Memory Space
Maximum Process Space
Eden Space
(Young Objects)
Heap
Code Area
Generation 2
Native Memory (Old Objects/Potential Leak)
(Native Libraries, COM Objects, ...)
Generation 1
(Objects survived GC)
Large Object Heap
Generation 0
(Young Objects)
Code Area
Analyze Trend
Analyze Allocations
Host and Process Health
Identified a Host
Memory Issue
Host Memory
Over Time
Process Health
Impact of GC on
App Server
App Server
Memory Usage
over Time
Memory Dashboards / Dashlets
476ms
Garbage Collector
…and knows precisely
how long the PurePath
21ms
was interrupted
455ms
View suspension
time in detail
Runtime Suspension
Methods with high suspension
Methods suspended by counter are suspected to allocate
garbage collection large objects
Suspension
counter
Response Time Hotspots
Memory
Monitor Memory Behavior Consumption
Trending
Snapshot
Analyze Trend
Analyze Allocations
Trending Snapshot vs. Leak Analysis Snapshot
JVM/CLR JVM/CLR
Find Objects
increasing in number
Dynatrace Memory Diagnostics Process
Memory
Monitor Memory Behavior Consumption
Trending
Snapshot
Analyze Trend
Analyze Allocations
Creating Memory Consumption Trending Snapshots
• Contains Instance Count
per Class
• Schedule periodic
Snapshots
• Compare Snapshots to
find Objects that
increased in number over
time
Comparing Multiple Trending Snapshots
Select your
Memory
Snapshots…
Press Compare, to
analyze how instances
developed over time
Analyze Trend
Analyze Allocations
Direct vs. Keep Alive References
Direct Reference Keep Alive
This is the
1 REFERRER
1
For THIS
2 3 object 2 3 5
And this is
4 5 the REFEREE 4
Creating Deep Memory Leak Analysis Snapshots
• Contains full Heap Data
• Analyze Snapshot:
• Hotspots
• Keep-Alive Sets
• Collections
• Sessions
• Duplicate Strings
Automatic Memory Hotspots
• Biggest Memory Hotspots
• Biggest Collections and Maps
• HTTP Sessions
• Duplicate Strings
Follow link to
show path to
root by keep alive
Analyze Hotspots
Analyze with a single Mouse click
• What keeps the Hotspot Alive?
Select “Show
Keep Alive Set” in
context menu
How big?
How big?
Choose between
analyzing referrers
and referees
Shortest Paths to Root
Select “Show Shortest
Root Paths”/“Direct” in
context menu
Direct References
Select “Follow
References”/“Direct” in
context menu
Choose between
analyzing referrers
and referees
References to/from the Class
Choose between
analyzing referrers
and referees
Direct References
Choose between
analyzing referrers
and referees
Additional Automatic Analysis
• Biggest Collection/Maps
• HTTP Sessions
• Duplicated Strings
Native Memory Leak Warning
Switch to “group
by thread”
Switch to “All
Threads”
Identify Methods
that spend most
of their time in
CPU
Hands On: CPU Sampling
• Goals
• Use CPU Sampling to increase PurePath coverage of Transactions
• Scenario
• Core Training – CPU Sampling
• Steps
• Visit http://localhost:9010/Report/Start?minutes=1
• Use CPU Sampling on the dotNetBackend
• At the end of the Hands On visit http://localhost:9010/Report/Stop
• Questions
• Did you find any interesting Threads?
• Did you find any candidates for instrumentation?
• If so, create new Entry Point Sensors for them and verify PurePaths are being
captured.
Thread Diagnostics
Shared Resources
Thread 1 Thread 2
Thread 1 Thread 2
Ask for R1
Use Ask for R2
Ask for R2 Use
R1 R2
Wait Ask for R1
Wait
Threads
or
…or define
scheduled tasks
Create Dumps
manually…
Overview of a Single Thread Dump
Select your just
created Thread
Dump…
…and overview
results
Analyze Thread Dump Trends
Select your just
created Thread
Dumps…
Press compare
to further dig
into analysis
…and analyze how the
number of threads and their
states evolve over time
Compare Thread DumpsEasy grouping by
thread group, thread state,
monitor usage, class, method
Browse through
the stack trace
What is a monitor?
Deep Dive Analysis: Drill Down to PurePaths
Hands On: Thread Diagnostics
• Scenario
• Core Training – Thread Diagnostics
• Steps
• Use easyTravel to search for trips by clicking on the pictures
• Identify if any thread related problems occur
• If so, capture the appropriate Thread Dumps
• Questions
• Which Tier is experiencing a problem?
• What is the problem?
• Which Threads? Which methods?
Dynatrace Plugin
Dynatrace AppMon & UEM FastPacks and Plugin
Confidential, Dynatrace, LLC 87
Telegram Plugin
Assumption this step already done
• Create a bot Using TheBotFather
• Start a chat and finds it’s ID
• Test it – make sure you can send “Hello World” to group.
On Demand Streaming
dynaTrace 6.3
Setup JDK
Setup ElasticSearch
Setup Kibana
Accessible locally via port 5601
Setup Nginx as reverse proxy
Accessible from outside (port 80)
Demo VM
Setup LogStack
vi /etc/elasticsearch/elasticsearch.yml
service
• Verify theelasticsearch start
service (http://localhost:9200 ) elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=https://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
Install Kibana (from Repository)
vi /etc/kibana/kibana.yml
• Define
servicethe default
kibana index pattern (dt_*)
start
• Verify the service (http://localhost:5601)
kibana.repo
[kibana-4.4]
name=Kibana repository for 4.4.x packages
baseurl=http://packages.elastic.co/kibana/4.4/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
Kibana Quick Tutorial – First Time Setup
• Uncheck “Index contains time-based events” and enter “dt_*” as index pattern, and hit Create
Jumps to the source code line in your IDE if you have the Visual Studio Solution or Eclipse Project open.
Integration Step with IDE - Eclipse
Install dynaTrace Eclipse Integration Plugin
Step 1
Configuration
Start Application
unchecked
• Developers try to reproduce the problem and try to fix it based on log output
• Additional log refining is added to capture more information on next test run
• Another Load Test follows another „Try to Reproduce, Refine Log output and Try Fix“ Session until problem is eventually fixed
Developing
Problem Solving
Getting Insight Into the Box
• Virtual Users see your application as Black Blox
• Combining it with dynaTrace gives you Insight into the Black Box
Deployed Enterprise Application
Virtual Users
Problem Solving
time
Developing
Problem Solving
time
Why Linking Synthetic Transactions with
• Load Testing Tools use Transaction or Test Step Names
• Visit Home Page PurePath
• Login User
• Search Flight
• Purchase
• Logout
dynaTrace: RS=session20060620091846;PT=31;PA=1;PS=76562898
Key Description
RS Identifies name of the dynaTrace Recorded Session that is currently being recorded
SP System Profile Name
PT PurePath identifier - Trace#
PA PurePath identifier - entry point Agent#
PS PurePath Server identifier - identifies the dynaTrace Server which captured this PurePath
Other Integration Options – Record a Session
• Record a dynaTrace Session for every Load Test
• Start/Stop Session Recording before/after the Test using the Load Test Name
• Manually Start/Stop or use REST Interface and trigger recording from Tool
• https://community.dynatrace.com/community/display/DOCDT40/Server+REST+Interfaces
Start Recording
POST rest/management/profiles/GoSpace/startrecording HTTP/1.1
Host: localhost:8020
Content-Length: 80
presentableName=easyTravelTestRun1&description=MyTestRun1&isTimeStampAllowed=true
Stop Recording
GET rest/management/profiles/GoSpace/stoprecording HTTP/1.1
Other Integration Options – Query Data
• Query Data from dynaTrace to integrate in your existing reports
• Define a Dashboard that contains data you want to query
• Use Server-Side REST Interfaces to query this data after the test is done
• https://community.dynatrace.com/community/display/DOCDT40/Server-+and+Clientside+XML+Reporting
Example Request
GET rest/management/dashboard/My%20Test%20Dashboard HTTP/1.1
Example Response
<?xml version="1.0" encoding="utf-8"?>
<dashboardreport name=“My Test Dashboard" version="3.1.0.1781" reportdate="2009-07-22T10:57:10.199+02:00" description="">
<source name=“easyTravel" />
<data>
<apibreakdowndashlet name="API Breakdown" description="" displaysource="Base">
<apis>
<api name="Servlet" exec_avg="157.20263700354496" exec_sum="12104.603049272962"
cpu_avg="12.975964260238069" cpu_sum="999.1492480383313"
sync_avg="8.608057804695971E-4" sync_sum="0.06628204509615898"
wait_avg="0.0" wait_sum="0.0" />
<api name="JDBC" exec_avg="187.71199448524803" exec_sum="5819.0718290426885"
cpu_avg="148.24450352138393" cpu_sum="4595.579609162902"
sync_avg="0.0" sync_sum="0.0" wait_avg="0.0" wait_sum="0.0" />
</apis>
</apibreakdowndashlet>
</data>
</dashboardreport>
Business Transactions per Tagged Values
• Out of the Box: Synthetic Web Requests per Timer Name
• Groups Transactions by Timer Name (NA) value
• Chart Response Time – either on Avg or Count
https://www.dynatrace.com/services-support/#get-support
University
https://university.dynatrace.com/?_ga=2.132429130.519927234.1527478646-278173520.1491430282