What to Test

Performance Testing Requirements

• Non functional requirements come from the business strategy or contract with clients
  • e.g. If the strategy is to have few rich visitors test should focus on reliability rather than expect billions of visitors
• Subset of functional testing (Use Cases, User Stories and their Scenarios)
• Most frequent scenarios
• Scenario which potentially impacts performance
• Prioritized by business people (managers) not testers
• Not everything should be tested
• Setting goals and scope of test is crucial (otherwise it is never done)
• Testing can be a continuous process

Performance Requirements

• Specific Hardware (8 Athlon Processors, 512MB RAM, SSD, etc..)
• Software Confiugration
• Max no of records in the database (e.g. 100 000 000)
  • Can be express in business terms (number of clients, contacts, invoices, enquiries, etc...)
• Max database size (e.g. 100 GB)
• Maximum number of registered user (e.g. 4500)
• Distribution of scenarios: scenarios 1 (20%), scenario 2 (30%), scenario 6 (70%)
  • Average Response Time (e.g. 1500ms)
  • Maximum page load time (e.g. 10 sec)
  • Minimum throughput (e.g. 2400 pages/minute)
  • Concurrency users (300 users working concurrently)

Performance Test results

Client side:

• Response Time: distribution shape, min, max, average, standard deviation, median, skewness
• Throughput (scenarios/min)
• Error Level, error responses in details

Servers side:

• Memory utilization
• CPU utilization
• Network throughput
• IO waiting time
• Storage free space

Performance Testing Process

• How to identify your testing needs
• Defining the steps of your test
  • Gathering requirements (use cases, user stories)
  • Scoping (choosing and prioritize scenarios)
  • Designing tests
  • Preparing data
  • Preparing environment
  • Recording tests
  • Perform the test
  • Analysis and reports
  • Conclusions and improvement