Performance Analysis Methodology

A performance analysis methodology is a procedure that you can follow to analyze system or application performance. These generally provide a starting point and then guidance to root cause, or causes. Different methodologies are suited for solving different classes of issues, and you may try more than one before accomplishing your goal.

Analysis without a methodology can become a fishing expedition, where metrics are examined ad hoc, until the issue is found – if it is at all.

Methodologies documented in more detail on this site are:

• The USE Method: for finding resource bottlenecks
• The TSA Method: for analyzing application time
• Off-CPU Analysis: for analyzing any type of thread wait latency
• Active Benchmarking: for accurate and successful benchmarking

The following briefly summarizes methodologies I've either created or encountered. You can print these all out as a cheetsheet/reminder.

Summaries

I first summarized and named various performance methodologies (mostly developed by me) for my USENIX LISA 2012 talk: Performance Analysis Methodology (PDF, slideshare, youtube, USENIX), then later documented them in my Systems Performance book, and the ACMQ article Thinking Methodically about Performance, which was also published in Communications of the ACM, Feb 2013. More detailed references are at the end of this page.

The following is my most up to date summary list, with methodologies enumerated. These begin with anti-methods, which are included for comparison, and not to follow.

Anti-Methodologies

Blame-Someone-Else Anti-Method

1. Find a system or environment component you are not responsible for
2. Hypothesize that the issue is with that component
3. Redirect the issue to the responsible team
4. When proven wrong, go to 1

Streetlight Anti-Method

1. Pick observability tools that are:
• familiar
• found on the Internet
• found at random
2. Run tools
3. Look for obvious issues

Drunk Man Anti-Method

1. Change things at random until the problem goes away

Random Change Anti-Method

1. Measure a performance baseline
2. Pick a random attribute to change (eg, a tunable)
3. Change it in one direction
4. Measure performance
5. Change it in the other direction
6. Measure performance
7. Were the step 4 or 6 results better than the baseline? If so, keep the change; of not, revert
8. Goto step 1

Passive Benchmarking Anti-Method

1. Pick a benchmark tool
2. Run it with a variety of options
3. Make a slide deck of the results
4. Hand the slides to management

Traffic Light Anti-Method

1. Open dashboard
2. All green? Assume everything is good.
3. Something red? Assume that's a problem.

Methodologies

Ad Hoc Checklist Method

1. ..N. Run A, if B, do C

Problem Statement Method

1. What makes you think there is a performance problem?
2. Has this system ever performed well?
3. What has changed recently? (Software? Hardware? Load?)
4. Can the performance degradation be expressed in terms of latency or run time?
5. Does the problem affect other people or applications (or is it just you)?
6. What is the environment? What software and hardware is used? Versions? Configuration?

RTFM Method

(Read The Fine Manual) How to research performance tools and metrics:

1. Man pages
2. Books
3. Web search
4. Co-workers
5. Prior talk slides/video
6. Support services
7. Source code
8. Experimentation
9. Social Media

Scientific Method

1. Question
2. Hypothesis
3. Prediction
4. Test
5. Analysis

OODA Loop

1. Observe
2. Orient
3. Decide
4. Act

Workload Characterization Method

1. Who is causing the load? (PID, UID, IP addr, ...)
2. Why is the load called? (code path)
3. What is the load? (IOPS, tput, type)
4. How is the load changing over time? (time series line graph)

Drill-Down Analysis Method

1. Start at highest level
2. Examine next-level details
3. Pick most interesting breakdown
4. If problem unsolved, go to 2

Process of Elimination

1. Divide the target into components
2. Choose a test which:
• Can exonerate many untested components (ideally, half of those remaining)
• Is quick to perform
3. Perform test
4. Were the tested components exonerated?
• Yes: go to 2
• No: problem found?
• Yes: done
• No: how many components were tested?
• one: target = tested component; go to 1
• multiple: go to 2
• Not sure: consider components untested; go to 2 and choose a different test

Time Division Method

1. Measure operation time (or latency)
2. Divide time into logical synchronous components
3. Continue division until latency origin is identified
4. Quantify: estimate speedup if problem fixed

(I previously called this the "Latency Analysis Method")

5 Whys Performance Method

1. Given delivered performance, ask, "why?", then answer this question
2. ..5 Given previous answer, ask, "why?", then answer this question

By-Layer Method

Measure latency in detail (eg, as a histogram) from:

1. Dynamic languages
2. Executable
3. Libraries
4. Syscalls
5. Kernel: FS, network
6. Device drivers

Investigate the lowest layer that latency is introduced

Tools Method

1. List available performance tools (optionally add more)
2. For each tool, list its useful metrics
3. For each metric, list possible interpretation
4. Run selected tools and interpret selected metrics.

USE Method

For every resource, check:

1. Utilization
2. Saturation
3. Errors

RED Method

For every service or microservice, check:

1. Request rate
2. Errors
3. Duration

CPU Profile Method

1. Take a CPU profile (especially a flame graph)
2. Understand all software in profile > 1%

Off-CPU Analysis

1. Profile per-thread off-CPU time with stack traces
2. Coalesce times with like stacks
3. Study stacks from largest to shortest time

Stack Profile Method

1. Profile thread stack traces, on- and off-CPU
2. Coalesce
3. Study stacks bottom-up

TSA Method

1. For each thread of interest, measure time in operating system thread states. Eg:
• Executing
• Runnable
• Swapping
• Sleeping
• Lock
• Idle
2. Investigate states from most to least frequent, using appropriate tools

Active Benchmarking Method

1. Configure the benchmark to run for a long duration
2. While running, analyze performance using other tools, and determine limiting factors

Method R

1. Select user actions that matter for the business workload
2. Measure causes of response time for user actions
3. Calculate best net-payoff optimization activity
• If sufficient gain, tune
• If insufficient gain, suspend tuning until something changes
4. Goto 1

Performance Evaluation Steps

1. State the goals of the study and define system boundaries
2. List system services and possible outcomes
3. Select performance metrics
4. List system and workload parameters
5. Select factors and their values
6. Select the workload
7. Design the experiments
8. Analyze and interpret the data
9. Present the results
10. If necessary, start over

Capacity Planning Process

1. Instrument the system
2. Monitor system usage
3. Characterize workload
4. Predict performance under different alternatives
5. Select the lowest cost, highest performance alternative

Intel Hierarchical Top-Down Performance Characterization Methodology

• Are UOPs issued?
• If yes:
• Are UOPs retired?
• If yes: retiring (good)
• If no: investigate bad speculations
• If no:
• Allocation stall?
• If yes: investigate back-end stalls
• If no: investigate front-end stalls

Performance Mantras

1. Don't do it
2. Do it, but don't do it again
3. Do it less
4. Do it later
5. Do it when they're not looking
6. Do it concurrently
7. Do it cheaper

Benchmarking Checklist

1. Why not double?
2. Did it break limits?
3. Did it error?
4. Does it reproduce?
5. Does it matter?
6. Did it even happen?

References

• Blame-Someone-Else Anti-Method and the USE Method were developed by me and first in print in [Gregg 13a]: Gregg, B., "Thinking Methodically about Performance", Communications of the ACM, Volume 56 Issue 2, Feb 2013. This article also included the Streetlight Anti-Method and the Problem Statement Method for the first time in print, also developed by me, but based on earlier work (the streetlight effect, https://en.wikipedia.org/wiki/Streetlight_effect, and initial performance checklists used by Sun Microsystems support).
• Random Change Anti-Method, Passive Benchmarking Anti-Method, Ad Hoc Checklist Method, Tools Method, TSA Method, and Active Benchmarking, were developed by me and first in print in [Gregg 13b]: Gregg, B., Systems Performance: Enterprise and the Cloud, Prentice Hall, Oct 2013.
• My USENIX LISA 2012 talk: "Performance Analysis Methodology", first summarized and named many of these (before they were in print).
• Method R is from [Millsap 03]: Millsap, C., Holt, J., Optimizing Oracle Performance, O'Reilly, 2003.
• Performance Evaluation Steps and Capacity Planning Process are from pages 26 and 124 of [Jain 91]: Jain, R., The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, Wiley, 1991.
• Workload Characterization Method, and Drill-Down Analysis Method, were documented as specific methodologies in [Gregg 13a] and [Gregg 13b], but the general process has been known in IT for many years. You could glean it from [Jain 91], at least. I don't yet have an earlier reference.
• Intel Hierarchical Top-Down Performance Characterization Methodology was developed by Intel and documented in B.3.2 of the Intel 64 and IA-32 Architectures Optimization Reference Manual, 248966-030, Sep 2014.
• OODA Loop is from Joyd Boyd of the US Air Force, and popularized in technology by Roy Rapoport of Netflix.
• Performance Mantas is from Craig Hanson and Pat Crain.
• Benchmarking Checklist was developed by me.
• RED Method is from Tom Wilkie.