Test Automation Framework - Challenges

Test automation has matured over a period of time; from the crude form of automated tests that were set up just to run specific or stop gap runs, we today have much refined and well defined automation tests. According to a study, in the year 2004 the number of automated tests that were conducted globally as against manual tests were a mere 5 percent. However, by 2006 this increased to 20 percent. Going by this rate, we may be around 50 percent today. 

According to a study conducted by the US NIST, software producers lose 21.2 billion dollars annually because of inadequate testing. Based on market estimates, software companies worldwide invested 931 million dollars in automated software testing tools in 1999, with an estimate of at least 2.6 billion dollars in 2004. This certainly tells us one thing: Customers are seeing the real benefits of test automation. The last 5-6 years has seen a considerable run in commercial testing tools and we now are seeing the trend of customers requesting for open-source tools, thanks to changing economic situations.

One of the key factors in choosing a test automation tool is the technology that is used to build the application. Not all tools support all technologies and that throws up the challenge of maximizing test automation returns with limited capabilities of tools. For example, an application developed in .NET or Java may require test automation tools to support such technology. Some of the tools come with specific plugins to address such challenges. So, choosing the right tool goes a long way in getting the expected RoI.

Technology, a Show-Stopper?

One would certainly run out of business if technology becomes a show-stopper. While that could be a very generic statement specific to test automation, technology has been one of the biggest challenges. With every new patch or release of an OS, browser, languages, or for that matter even test automation tools, the challenges to maintain automation scripts are undeniably compounded. By the time you start a test automation project and complete it, you may find changes in the environment it is set to work. The script that you have developed, while the customer agreed for a specific version, will not be of any use to him if the customer is going to upgrade their systems to the new version of the browser, for instance. Here is when a robust framework and good coding practice facilitate a smooth transition.

Develop framework that is independent of test automation tools. Since every tool has some limitation, it is best to use a combination of tools to take the maximum returns out of test automation investment. A combination of commercial and open-source tools is a good mix. Having a framework that best supports such a mix will give the best advantage over other frameworks.

Never Ending CRs

CRs or change requests have become the norm of the day. Today, CRs not necessarily include change in functionality and features due to changing business requirements but also the impact the application goes through because of the technological changes. Let us consider a situation like this: an application that is stable has about 1,250 automated test scripts that are run every other day and being used by 100s of customers worldwide. Due to business reasons the company decides to change the technology or architecture of the application. Now this is going to have an impact on the automation scripts as well. Either the scripts will have to be discarded completely, which is money into the drain, or evaluate the possibility to modify the scripts to suit the new architecture or technology. In case the latter being a possibility, an evaluation of time and effort required for such modification and associated costs need to be considered. Since users are realizing the benefits of automation, this aspect should not be missed when someone is considering changes to the application. Thus, it is important to integrate test automation in the purview of CRs, i.e., whenever a CR is raised, it is important to evaluate its impact on the automation scripts. A robust test automation framework will make such impacts minimal.

Build Framework that Works

Test automation framework is a set of rules, procedures, and tools that drive automated testing. One of the main advantages of using a framework is the low cost of maintenance. For any changes to the application, only the concerned test case(s) need to be updated while the rest of the script remains the same.

Some of the most common frameworks are data driven, keyword driven, modular, and hybrid. There are again custom built frameworks that fit to specific needs of the application. Spending enough time on choosing the right framework and designing it is the key to test automation success. 

A good and robust framework should consider as many assumptions as possible. This will eventually make the framework quite flexible. While it will be difficult to prove due to lack of data but the general belief is that most of the test automation projects fail in the long run as the framework is not flexible enough to adapt to the changes that the application or the environment has undergone.

Let us take an example to explain the above situation. Consider a Web application that is developed in .NET technology compatible to WinXP, Vista, and Win7 and browsers such as IE and Firefox. A version 5.0 of a test automation tool is used to automate a set of sanity test cases. 

A test automation framework is developed to ensure that as the application undergoes changes, there are re-usable functions that are developed so that with less effort you can address those changes. Most of the frameworks today address this basic need. However, consider a situation where a new version of OS, browser, and the test automation tool need to be addressed in the framework itself. It is not an easy situation or assumption to imagine or incorporate within a framework.

The next generation frameworks should address the best of the following:

* Coverage
* Business Process
* Extended Environment
* Multi OS
* Multi Test Automation Tools
* Best of Reporting and Analysis
* Defect Analysis
* Metric
* Compliance such as HIPPA or SOX
* Recovery Scenario
* Addressing Technology Challenges

Conclusion

Building a test automation framework needs skills such as designing, analytical, and a lot of foresight. While testers bring in a lot of value and insight into developing the framework, it is the designers and developers who eventually build it. Considering the development of framework as a typical development project goes a long way in getting the required RoI. Build the best software engineering practices while developing a framework. Some of the critical aspects, as the fig 1.0 depicts, make the framework dependable and robust in the long run.

via Test Automation Framework - Challenges in the Ever Changing Technology Scenario - QA Expert Columns.

Read more »

Things you need to let go before joining a startup

A lot of people want to join a startup. From the outside, it looks like a sexy thing to do (i.e. next to starting up your own company) and looks like a natural choice to be in the game. But, things are different and well, strange as well.

As a early employee of a startup, you have the same responsibility as the founding team and you are expected to do a lot more than what you have probably imagined. But we all carry the baggage of past and have a certain expectation of the future. In the early days of the startup, when there’s ambiguity and amorphousness about the company’s structure, roles and responsibilities, it is understandable when the founders and early team members do everything and anything possible to get the job done right, on time. After a while, as the startup grows, a more formal structure comes into being.

Based on my experience (of being an early employee at a startup to running a startup), here are a few pointers that you probably will have to let go, before joining a startup:

Ego

Would you have ever imagined cleaning office, sending courier, running around government offices etc etc? Well, your engineering degree (or what’er/MBA etc) was way too fancy to shield you from such menial job. Moreover, the startup founder has a reason to do all the shit work (heck! its his company), but why should you?

   And there is only 1 reason why should you do that : “You pray for rain, you gotta deal with the mud too. That’s a part of it.” [Quote by Denzel Washington].

Excuses

“I can’t do this because <fill in the blank>”. Well, if you expect a perfect world, do not join a startup. It’s like saying ‘I want a girl-next-door’ – so maybe, you should just go next door?

Startups are amorphous and anything but perfect. There is an unorganized chaos and if you find yourself trying to poke holes/find excuses to not do things, it’s a reminder that you are better suited for a role somewhere else.

So go somewhere else.

Money, honey?

If all you are thinking about is how much do I make (vs. my friends working in Infosys/Yahoo/Google etc), you are comparing apples and oranges. In startups, even founders do not make a lot of money (read: Start-Up Salaries and Myths About Them). The fun lies in creating something, disrupting in your definition of the world and in the process, having fun, meeting other interesting people.

If all you look for is money, then go fish somewhere else. Agree that you need to make two ends meet, but expecting skyrocketing salary doesn’t really happen in a startup.

The “ONLY” Phenomena

If you are the one who is not a multi-tasking person, and want to stick to only 1 thing, you probably should join a company that is hiring for your skill set. Startups do not hire for skill set, they hire for attitude and ‘I can only do <X>’ doesn’t help. You need to move out of your comfort zone and do things which you have never even imagined [read: My first 90 days in a startup].

If all you are looking for is RoI (hell! what do I get by learning <X>, which isn’t even related to my core skill set), then you should never join an early stage startup. Join when the process is setup and you know what you are going to do.

The Sexy Title.

Titles are distributed randomly in a startup. From nowhere, you become ‘director of engineering’, ‘product manager’, ‘VP of sales’ etc etc., but ask yourself – do you even deserve the title?

If you are CTO of a small company, would you join Google as an entry-level engineer or somebody senior (after all, you were the CTO of a small startup!). Can you really handle same level of complexity (in Google/Apple) and justify the “CTO title”? Ask yourself.

And always remember that it’s never about the size of operation, but it’s the capability of the person that ‘title’ should stand for. (Recommended Read: Of Roles & Titles In Startups [First Deserve, Then Desire])

If you cannot live up to the title, then why are you even carrying it? Do a reality check and you will be fine.

If you are the VP, Sales in a startup and manage INR 20L/month of sales, can you also handle the complexity associated with managing INR 20 Cr when the startup grows up?

Quit. Or let go of the title. Better, fight to deserve the title.

Your Real CTC

You know your real cost to the company (CTC)? Apart from salary element, a person’s real cost to the company  is whether he/she is dependable or not. If you aren’t dependable, you are increasing the overall cost to the company as you are creating a process where the founder/senior members need to keep following up with you (on each and every small aspect).

This kind of attitude works in a big company (owing to the level of incompetency across the board) , but hiring in a startup is done with an assumption that

..You can be on your own.

You will eventually learn to run with the task you are supposed to do (+ tasks you aren’t supposed to do), without any followups.

-

Parting words

Startups are not fun, they are actually a paranoid-ic place to be in. And only paranoids survive this game. So be in for a bigger cause and learn to enjoy the bumpy ride.

And mind you, there is nothing wrong in working with a big company and the echo-chamber created by the likes of Pluggd.in (echo-chamber that startups are the next big thing) is frankly,  just another echo-chamber. If you are not in for a startup life, you are just normal.

In short, do not join a startup for the heck of joining one.

What’s your take?

via Things you need to let go before joining a startup.

Read more »