The right selection for any automation tool is an art; that requires detailed analysis and search. It requires a lot of effort and planning to select a tool by performing detailed analysis across various tools. The effort and time invested for tool evaluation will eventually help in ensuring the success of test automation.
Many organizations get deceived by the sales executives of tool vendors, giving demo on how easy it is to create test scripts using the record/playback functionality. In many cases, organizations do not investigate how the tool is capturing the application and automating the functionalities and they buy the licenses after being inspired by the flowery words of the marketing personnel.
Eventually, they find out that the tool captures by identifying the application and its controls using coordinate positions, which is not a reliable method, and the tool gets shelved. Hence a systematic approach must be taken to evaluate tools. I recommend that the selection of automation tool be divided in to three phases:
- Requirements Gathering
- Tool selection
- Proof of Concept (POC)
Requirements Gathering
We have to gather all the requirements or the expectations from the tool under evaluation respective to your environment of application. Some of the features that should be checked during requirements phase are:
- The testing tool must be compatible with operating systems that your product supports.
- The tool must be compatible with third party software that your product is integrated with.
- The tool must be version control respective to build so as to update scripts with lesser man effort.
- The organization must have the skill set for operating the respective tool.
- Management must be ready to invest on licenses or training the resources for tool.
- The extent of test coverage that can be done using the tool.
- Degree to which regression tests are possible with this tool.
- Test data setup should be supported with the tool in different formats.
- The testing tool should increase the efficiency of the testing process.
There could be more points that will help in order to map your requirements during the selection process as per company processes and requirements. These all points will set the evaluation criteria for the tool.
Tool Selection
The second main step requires the selection of the tool as per the revelations from our evaluation criteria made in the requirements phase. One thing that must be must be known is that no tool will ever support your environment completely. So, you have to prioritize your requirements in a way that clearly identifies the most valuable features to you and the tool that best supports them.
Categorize the features for tools under consideration on following basis:
Compulsory Features: These are the features that are mandatory in order to accomplish the automation goals as per your requirements within all constraints.
Unique Features: These will be all those features that will show value/difference/uniqueness of one tool compared to the other one.
There will be some features within the tool that you do not require or that are of no use so just ignore all of them. Hence, rate all features for the tools and prioritize them as per your requirements. Shortlist the most feasible tools and perform a POC on it.
Proof of Concept
The POC is always performed on the tool(s) that you have short listed to map all of your requirements for automation testing. In order to perform the POC you have to identify some core features of the application under test along with technical constraints. Select any one scenario from a sample of common features that will be a part of POC too. Once the POC has been performed; document all the analysis, failures and success that will lead to choose the tool for automation testing. The same process will be applied on all the tools in the list.
Selecting the automation tool for testing requires intensive focused research following the process described above. This effort will cut down the time and cost risks that have the probability to appear without proper analysis of the tool.
