As your development efforts take shape, unit testing and integration testing provide the foundation for ensuring that the system meets the business requirements and is free from obvious errors. If you had to deploy to only one user, you might be able to deliver your application now. However, that is rarely the case. So you need system testing to determine whether the application is at the proper level of industrial strength and ready for the prime-time production environment.

Actually, there is no specific “system test.” Instead, a variety of tests fall under the general system test umbrella. Some of these tests can be difficult to set up and execute because they require you to simulate the production environment. Even if your company has the hardware and software required to simulate the live environment, it can be difficult to stage the proper set of people, transaction, and circumstances you see in production. For instance, although a Web application works fine and meets the business requirements, can it resist the attack of a hacker who wants to read your databases or deface your site? That event will be hard to test for.

In fact, if you look at major systems failures that have made the press, they typically are not brought on by code errors or logic problems. As often as not, the problems are caused by poor response times, weak security, lost data, screens people cannot understand, and so forth. These are all aspects that the system test addresses.

Catch up on previous software testing articles from this series

Types of system tests
A number of events fall within system testing. Of course, not all of them are necessary or appropriate for every system.

  • Performance: Test the application with transaction volume that is within and at the top of the range expected from the business requirements. Make sure that the resulting system behavior is within expectations or any formal service level agreements.
  • Stress: Expose the system to transaction volume substantially higher than what would normally be expected and over a concentrated time period.
  • Security: Ensure that people have the access level that is required and no more and that unauthorized people cannot access the system.
  • Requirements: Track each business requirement through the development process and make sure that it is included in the final system.
  • Usability: Determine that people can use the system easily and without frustration.
  • Documentation: Check that hard-copy and online documentation are understandable and accurate.
  • Training: Ensure that online or in-person training is effective and meets the training requirements.
  • Interface: Test your application interfaces with external databases or third-party companies.
  • Disaster recovery: See whether you can recover the system from a simulated disaster.
  • Multiple locations: Verify your system can function between multiple locations, if necessary.

It is important to note that all of the system tests start with the assumption that the system is complete and stable. It is difficult to do system testing if you are running into a large number of program bugs as well.

We’ll focus specifically on performance testing and stress testing here. Future articles will deal with the other testing events.

Performance testing
The development team must understand the level of transaction volume the system needs to handle. This includes low volume, normal volume, and high volume. Examples of transaction volume include the number of simultaneous users, the number of database reads and writes, the number of batch transactions, and the number of Web pages accessed. Of course, you need to know the volume over a specific time frame. A system that expects 100 online transactions per hour has vastly different considerations from one that has to handle a million transactions an hour.

Performance testing simulates the volume of a live system to make sure that it responds within expectations. If you have specific service level agreements in place, you must ensure that you can achieve them. It is very important to test at the top end of the volume requirements and not just test the typical volume level. The areas you are looking for include:

  • Online response time for screen navigation.
  • Online response time for processing data against databases.
  • The time needed to run batch processes.
  • Delivery time for hard-copy reports (e.g., are they ready by 8:00 A.M.?).
  • Printer capacity to handle requests.
  • Downtime for batch processing or periodic maintenance.
  • File sizes and storage capacity.

Stress testing
While performance testing is designed to ensure that the system performs as expected against low-, normal-, and high-volume requirements, stress testing determines how the system behaves when it receives volume higher than the requirements. For instance, if you expect no more than 10 users on the system at one time, what happens when 20 log on simultaneously? If you expect your Web site to have no more than 1,000 hits per hour, what will happen if it receives 1,000 hits within 10 minutes or 10,000 hits per hour?

The stress test shows how your system holds up against the unexpected. To be sure, you expect system performance to degrade if the resources are stressed. But do they crash and burn? If they do, at what point do they do so? The question then is whether this break point is acceptable. If you expect 1,000 transactions per hour, it may be acceptable for the system to break if it has to process 10,000 in an hour. However, it may not be acceptable if it breaks at 1,500 transactions per hour. In many cases, stress testing highlights flaws in your technical architecture and may point out the need for upgrades to hardware, software, or tools.

Just as with performance testing, you are looking to see how the system responds to an overload in terms of response time, storage capacity, file sizes, database response time, etc. Because of the volume requirements, you usually need a good testing tool to simulate many aspects of stress testing. The tool will be able to simulate user loads without having to line up hundreds of actual users to participate in the tests. You also want to make sure tests can be repeated often. Again, a testing tool is usually the best way to handle this.

Summary
Let’s recap what we’ve covered about system testing so far:

  • System testing consists of multiple individual tests designed to validate that your system is ready to go live in a production environment.
  • Performance testing validates that the system behaves as expected under low, normal, and high volumes.
  • Stress testing determines how the system reacts when its resources are overloaded and at what point a meltdown may occur.