Selection of performance testing tool is imperative for testing performance of applications. These tools are used to measure the performance, scalability, load and stress capacity of software. For software performance testing what do you think are some immediate concerns of test engineers? The testers have to ponder over the selection of performance testing tools, the reliability of performance testing tools and their efficiency while making selection. Testers always make comparison among top performance testing tools offered by HP, IBM and APACHE, for performance testing. This blog aims at providing a quick comparison of performance testing tools and how they support the goals of any performance test plan. A sharp contrast of some famous Performance Testing Tools (which collectively hold a large market share) is provided below.
The performance parameters for each tool have been discussed one by one to explore diverse functionalities of these Performance Testing Tools.
Supported Protocols
HP Load Runner (LR) supports a wide variety of protocols which includes Dot Net, Database (Oracle/SQL Server), Web and Multimedia, Web 2.0, Network, Oracle E-Business, SAP, Rich Internet Applications, DCOM, GUI Virtual Users, Java Record/Reply, Remote Access, Wireless and SOA etc.
On the other hand by default RPT supports HTTP, SAP, TCP Socket, Citrix, Web Services (SOA), Siebel protocols. The list can be extended by purchasing few additional component protocols. While JMeter has limited protocol support and only supports Web (HTTP) and Network (FTP).
Script Playback
LR provides rich playback features. LR extended logging feature makes the parameter values and server messages easy to view. We can also view the comparison snapshots of record and playback script. The RPT playback feature is quite useful and it provides the detail of all the identified issues. JMeter doesn’t have any such integrated functionality.
Scripting
LR default scripting language is C but it supports VB, VB Scripting and Java Scripting. LR script can be easily enhanced with custom commands. LR captures scripts in high level context and low level HTTP view. RPT is script-less application and works with record and play only. JMeter is a pure Java application and supports only Java language. We can add new Java functions to extend our script. JMeter captures scripts on low level HTTP protocol view.
Correlation
LR provides automated correlation by comparing recorded results with playback results. Moreover it also provides manual correlation of functions wherever automated correlation is not possible. RPT has a very powerful correlation support and can correlate dynamic server responses manually and automatically. JMeter supports only manual correlation.
Parameterization
LR has a very powerful parameterization feature and facilitates the users to provide dynamic values in multiple ways like in Table, File etc. and users can also view the dynamic values simulation before executing the script. RPT provides support of replacing dynamic script parameters with user defined values. JMeter supports standard functions for accessing data through files.
Monitoring
When it comes to software testing, LR has one of the most powerful monitoring capabilities. It provides a vast range of online monitors of all the relevant resources ranges from Transaction Response Time, Hits per Seconds, Throughput to Server, Database monitors and much more. RPT provides online monitoring information of all the application performance parameters like page performance, page throughput and Server health etc. JMeter doesn’t support online monitors. You can add plug-ins to view client side monitors only.
Distributed Load Test
All applications under discussion have more or less very sophisticated mechanism of Load Test.
IP Spoofing
LR provides the IP Spoofing feature and you can generate load from different IP addresses. RPT provides IP aliasing feature to generate load from different IP addresses. JMeter doesn’t have inbuilt IP Spoofing feature.
WAN/LAN Emulation
LR provides facility to emulate different network infrastructures like latency, packet loss, dynamic routing and user connection speed. RPT has the WAN/LAN emulation feature. JMeter doesn’t have inbuilt WAN/LAN emulation feature.
Caching
LR facilitates you to control the browser cache emulation during playback to control individual virtual user settings.
RPT has built in cache clearing feature. JMeter doesn’t have inbuilt caching feature but it can be added with script code.
Reporting and Analysis
LR has very powerful reporting capabilities and you can generate various types of charts and graphs through it. You can also overlay different graphs to make useful analysis. Moreover you can generate detailed and summary reports of your test in multiple formats. RPT provides online charts and graphs of application under test and provides the facility to import test reports in HTML, XML and CSV formats. JMeter provides running test charts and graphs for application analysis. You can also use different plug-ins to extend the list of available graphs.
By comparing the above features of these tools, there is no doubt HP Load Runner is simply the best amongst all of them with its large set of features and ability to support application from various domains and technologies. But the importance of all other performance testing tools cannot be ignored. It is suggested that testers should made a clever and careful selection of performance testing tools. If you are interested in knowing more about software testing and performance testing tools then you should check our blog for more tips and tricks to follow.
Note:
One factor has purposely been excluded from the comparison sheet i.e. the ‘cost’. Of course LR and RPT are commercial software (LR is very expensive as compared to RPT) while JMeter is an open source application. There isn’t any rule of thumb to decide which tool is the best. It all depends on your need and budget constraints. So while going through Performance Testing tool selection process, compare all tool features with their costs and perform a proper Cost Benefit analysis to make the final choice.
