Performance testing is conducted to check the behavior of application under test (AUT) under different load levels and its post-crash recovery process. AUT performance results extensively depend upon the test environment. First rule of thumb in Software Performance Testing is to have a test environment which is the exact replica of the production system as even the minor differences between the production and test environment can completely invalidate the entire test activity.
Complete knowledge of the AUT production environment, test environment isolation from other activities, network isolation, proxy removal from the network path, full access to the server and warming up the server before testing are some of the approaches that are necessary in order to successfully execute a performance test.
Various test environment options are available with each having its own advantages and disadvantages and selection of a particular option depends upon the requirements. Following is the brief list of test environment options:
1) A subset of the production system with fewer servers of lower specification
This is a very common test environment preparation approach where performance testing is conducted on an already established test environment regardless of its production environment specification. Although testing in such environment can help you identify some application bottlenecks, however complete application and its infrastructure will never be tested.
2) A subset of the production system with fewer servers of same specification
This is comparatively a quite better approach where testing is done on smaller application infrastructure but the results can be extrapolated to map them on full application scale. Although this environment produces better results but the post testing bottlenecks can be missed very easily in the extrapolation process.
3) An exact replica of the production system
This is the most desired test environment which is extremely difficult to achieve due to its complexities and financial constraints but if it is achieved, the test results are 100% guaranteed. Some of the challenges which the performance testing teams face while preparing the exact replica of the live system are:
- Replicating the production servers architecture
- Production system network infrastructure replication
- Replicating all application tiers of the production system
- Replicating the production system network bandwidth
- Replicating the production database with all its dataset
Some of the system components like the third party services, firewall impact, working of load balancers, application internet connection and DNS lookup time etc cannot be tested on the test environment.
4) Testing on Production System
Another approach which is also largely practiced is to conduct the testing on the production system due to the concerns and complexities of the test systems. Although the best way to validate the system performance is to run the test directly on it, however there are numerous concerns about this approach i.e. the system availability during test execution, its impact on actual users, protection and mixing of test and production data etc but if all these issues are handled successfully then the optimal performance test results can be achieved.
Furthermore, performance testing on production system should be conducted based on its following advantages:
- Test results validation
- Its cost effective
- Helps in testing the database back-ups and post-crash recovery
- Easier to setup the test infrastructure
The impact of testing on production system on actual users can be minimized by conducting performance testing during off-hours of off-days (i.e. around mid night during the weekends), during maintenance window, before officially realizing the application and by performing read only operations.
At Kualitatem, initially we conduct the performance testing on our test environment and involve the development and network teams to fix all the identified bottlenecks. Once all the identified bottlenecks are fixed and verified on the test environment, we test the production system to validate the test environment results and test the missing components as well.