1:31 PM
Unknown
Test automation has matured over a period of time; from the crude form of automated tests that were set up just to run specific or stop gap runs, we today have much refined and well defined automation tests. According to a study, in the year 2004 the number of automated tests that were conducted globally as against manual tests were a mere 5 percent. However, by 2006 this increased to 20 percent. Going by this rate, we may be around 50 percent today.
According to a study conducted by the US NIST, software producers lose 21.2 billion dollars annually because of inadequate testing. Based on market estimates, software companies worldwide invested 931 million dollars in automated software testing tools in 1999, with an estimate of at least 2.6 billion dollars in 2004. This certainly tells us one thing: Customers are seeing the real benefits of test automation. The last 5-6 years has seen a considerable run in commercial testing tools and we now are seeing the trend of customers requesting for open-source tools, thanks to changing economic situations.
One of the key factors in choosing a test automation tool is the technology that is used to build the application. Not all tools support all technologies and that throws up the challenge of maximizing test automation returns with limited capabilities of tools. For example, an application developed in .NET or Java may require test automation tools to support such technology. Some of the tools come with specific plugins to address such challenges. So, choosing the right tool goes a long way in getting the expected RoI.
Technology, a Show-Stopper?
One would certainly run out of business if technology becomes a show-stopper. While that could be a very generic statement specific to test automation, technology has been one of the biggest challenges. With every new patch or release of an OS, browser, languages, or for that matter even test automation tools, the challenges to maintain automation scripts are undeniably compounded. By the time you start a test automation project and complete it, you may find changes in the environment it is set to work. The script that you have developed, while the customer agreed for a specific version, will not be of any use to him if the customer is going to upgrade their systems to the new version of the browser, for instance. Here is when a robust framework and good coding practice facilitate a smooth transition.
Develop framework that is independent of test automation tools. Since every tool has some limitation, it is best to use a combination of tools to take the maximum returns out of test automation investment. A combination of commercial and open-source tools is a good mix. Having a framework that best supports such a mix will give the best advantage over other frameworks.
Never Ending CRs
CRs or change requests have become the norm of the day. Today, CRs not necessarily include change in functionality and features due to changing business requirements but also the impact the application goes through because of the technological changes. Let us consider a situation like this: an application that is stable has about 1,250 automated test scripts that are run every other day and being used by 100s of customers worldwide. Due to business reasons the company decides to change the technology or architecture of the application. Now this is going to have an impact on the automation scripts as well. Either the scripts will have to be discarded completely, which is money into the drain, or evaluate the possibility to modify the scripts to suit the new architecture or technology. In case the latter being a possibility, an evaluation of time and effort required for such modification and associated costs need to be considered. Since users are realizing the benefits of automation, this aspect should not be missed when someone is considering changes to the application. Thus, it is important to integrate test automation in the purview of CRs, i.e., whenever a CR is raised, it is important to evaluate its impact on the automation scripts. A robust test automation framework will make such impacts minimal.
Build Framework that Works
Test automation framework is a set of rules, procedures, and tools that drive automated testing. One of the main advantages of using a framework is the low cost of maintenance. For any changes to the application, only the concerned test case(s) need to be updated while the rest of the script remains the same.
Some of the most common frameworks are data driven, keyword driven, modular, and hybrid. There are again custom built frameworks that fit to specific needs of the application. Spending enough time on choosing the right framework and designing it is the key to test automation success.
A good and robust framework should consider as many assumptions as possible. This will eventually make the framework quite flexible. While it will be difficult to prove due to lack of data but the general belief is that most of the test automation projects fail in the long run as the framework is not flexible enough to adapt to the changes that the application or the environment has undergone.
Let us take an example to explain the above situation. Consider a Web application that is developed in .NET technology compatible to WinXP, Vista, and Win7 and browsers such as IE and Firefox. A version 5.0 of a test automation tool is used to automate a set of sanity test cases.
A test automation framework is developed to ensure that as the application undergoes changes, there are re-usable functions that are developed so that with less effort you can address those changes. Most of the frameworks today address this basic need. However, consider a situation where a new version of OS, browser, and the test automation tool need to be addressed in the framework itself. It is not an easy situation or assumption to imagine or incorporate within a framework.
The next generation frameworks should address the best of the following:
* Coverage
* Business Process
* Extended Environment
* Multi OS
* Multi Test Automation Tools
* Best of Reporting and Analysis
* Defect Analysis
* Metric
* Compliance such as HIPPA or SOX
* Recovery Scenario
* Addressing Technology Challenges
Conclusion
Building a test automation framework needs skills such as designing, analytical, and a lot of foresight. While testers bring in a lot of value and insight into developing the framework, it is the designers and developers who eventually build it. Considering the development of framework as a typical development project goes a long way in getting the required RoI. Build the best software engineering practices while developing a framework. Some of the critical aspects, as the fig 1.0 depicts, make the framework dependable and robust in the long run.
via Test Automation Framework - Challenges in the Ever Changing Technology Scenario - QA Expert Columns.
According to a study conducted by the US NIST, software producers lose 21.2 billion dollars annually because of inadequate testing. Based on market estimates, software companies worldwide invested 931 million dollars in automated software testing tools in 1999, with an estimate of at least 2.6 billion dollars in 2004. This certainly tells us one thing: Customers are seeing the real benefits of test automation. The last 5-6 years has seen a considerable run in commercial testing tools and we now are seeing the trend of customers requesting for open-source tools, thanks to changing economic situations.
One of the key factors in choosing a test automation tool is the technology that is used to build the application. Not all tools support all technologies and that throws up the challenge of maximizing test automation returns with limited capabilities of tools. For example, an application developed in .NET or Java may require test automation tools to support such technology. Some of the tools come with specific plugins to address such challenges. So, choosing the right tool goes a long way in getting the expected RoI.
Technology, a Show-Stopper?
One would certainly run out of business if technology becomes a show-stopper. While that could be a very generic statement specific to test automation, technology has been one of the biggest challenges. With every new patch or release of an OS, browser, languages, or for that matter even test automation tools, the challenges to maintain automation scripts are undeniably compounded. By the time you start a test automation project and complete it, you may find changes in the environment it is set to work. The script that you have developed, while the customer agreed for a specific version, will not be of any use to him if the customer is going to upgrade their systems to the new version of the browser, for instance. Here is when a robust framework and good coding practice facilitate a smooth transition.
Develop framework that is independent of test automation tools. Since every tool has some limitation, it is best to use a combination of tools to take the maximum returns out of test automation investment. A combination of commercial and open-source tools is a good mix. Having a framework that best supports such a mix will give the best advantage over other frameworks.
Never Ending CRs
CRs or change requests have become the norm of the day. Today, CRs not necessarily include change in functionality and features due to changing business requirements but also the impact the application goes through because of the technological changes. Let us consider a situation like this: an application that is stable has about 1,250 automated test scripts that are run every other day and being used by 100s of customers worldwide. Due to business reasons the company decides to change the technology or architecture of the application. Now this is going to have an impact on the automation scripts as well. Either the scripts will have to be discarded completely, which is money into the drain, or evaluate the possibility to modify the scripts to suit the new architecture or technology. In case the latter being a possibility, an evaluation of time and effort required for such modification and associated costs need to be considered. Since users are realizing the benefits of automation, this aspect should not be missed when someone is considering changes to the application. Thus, it is important to integrate test automation in the purview of CRs, i.e., whenever a CR is raised, it is important to evaluate its impact on the automation scripts. A robust test automation framework will make such impacts minimal.
Build Framework that Works
Test automation framework is a set of rules, procedures, and tools that drive automated testing. One of the main advantages of using a framework is the low cost of maintenance. For any changes to the application, only the concerned test case(s) need to be updated while the rest of the script remains the same.
Some of the most common frameworks are data driven, keyword driven, modular, and hybrid. There are again custom built frameworks that fit to specific needs of the application. Spending enough time on choosing the right framework and designing it is the key to test automation success.
A good and robust framework should consider as many assumptions as possible. This will eventually make the framework quite flexible. While it will be difficult to prove due to lack of data but the general belief is that most of the test automation projects fail in the long run as the framework is not flexible enough to adapt to the changes that the application or the environment has undergone.
Let us take an example to explain the above situation. Consider a Web application that is developed in .NET technology compatible to WinXP, Vista, and Win7 and browsers such as IE and Firefox. A version 5.0 of a test automation tool is used to automate a set of sanity test cases.
A test automation framework is developed to ensure that as the application undergoes changes, there are re-usable functions that are developed so that with less effort you can address those changes. Most of the frameworks today address this basic need. However, consider a situation where a new version of OS, browser, and the test automation tool need to be addressed in the framework itself. It is not an easy situation or assumption to imagine or incorporate within a framework.
The next generation frameworks should address the best of the following:
* Coverage
* Business Process
* Extended Environment
* Multi OS
* Multi Test Automation Tools
* Best of Reporting and Analysis
* Defect Analysis
* Metric
* Compliance such as HIPPA or SOX
* Recovery Scenario
* Addressing Technology Challenges
Conclusion
Building a test automation framework needs skills such as designing, analytical, and a lot of foresight. While testers bring in a lot of value and insight into developing the framework, it is the designers and developers who eventually build it. Considering the development of framework as a typical development project goes a long way in getting the required RoI. Build the best software engineering practices while developing a framework. Some of the critical aspects, as the fig 1.0 depicts, make the framework dependable and robust in the long run.
via Test Automation Framework - Challenges in the Ever Changing Technology Scenario - QA Expert Columns.
