Test Automation Framework - Challenges

Test automation has matured over a period of time; from the crude form of automated tests that were set up just to run specific or stop gap runs, we today have much refined and well defined automation tests. According to a study, in the year 2004 the number of automated tests that were conducted globally as against manual tests were a mere 5 percent. However, by 2006 this increased to 20 percent. Going by this rate, we may be around 50 percent today. 

According to a study conducted by the US NIST, software producers lose 21.2 billion dollars annually because of inadequate testing. Based on market estimates, software companies worldwide invested 931 million dollars in automated software testing tools in 1999, with an estimate of at least 2.6 billion dollars in 2004. This certainly tells us one thing: Customers are seeing the real benefits of test automation. The last 5-6 years has seen a considerable run in commercial testing tools and we now are seeing the trend of customers requesting for open-source tools, thanks to changing economic situations.

One of the key factors in choosing a test automation tool is the technology that is used to build the application. Not all tools support all technologies and that throws up the challenge of maximizing test automation returns with limited capabilities of tools. For example, an application developed in .NET or Java may require test automation tools to support such technology. Some of the tools come with specific plugins to address such challenges. So, choosing the right tool goes a long way in getting the expected RoI.

Technology, a Show-Stopper?

One would certainly run out of business if technology becomes a show-stopper. While that could be a very generic statement specific to test automation, technology has been one of the biggest challenges. With every new patch or release of an OS, browser, languages, or for that matter even test automation tools, the challenges to maintain automation scripts are undeniably compounded. By the time you start a test automation project and complete it, you may find changes in the environment it is set to work. The script that you have developed, while the customer agreed for a specific version, will not be of any use to him if the customer is going to upgrade their systems to the new version of the browser, for instance. Here is when a robust framework and good coding practice facilitate a smooth transition.

Develop framework that is independent of test automation tools. Since every tool has some limitation, it is best to use a combination of tools to take the maximum returns out of test automation investment. A combination of commercial and open-source tools is a good mix. Having a framework that best supports such a mix will give the best advantage over other frameworks.

Never Ending CRs

CRs or change requests have become the norm of the day. Today, CRs not necessarily include change in functionality and features due to changing business requirements but also the impact the application goes through because of the technological changes. Let us consider a situation like this: an application that is stable has about 1,250 automated test scripts that are run every other day and being used by 100s of customers worldwide. Due to business reasons the company decides to change the technology or architecture of the application. Now this is going to have an impact on the automation scripts as well. Either the scripts will have to be discarded completely, which is money into the drain, or evaluate the possibility to modify the scripts to suit the new architecture or technology. In case the latter being a possibility, an evaluation of time and effort required for such modification and associated costs need to be considered. Since users are realizing the benefits of automation, this aspect should not be missed when someone is considering changes to the application. Thus, it is important to integrate test automation in the purview of CRs, i.e., whenever a CR is raised, it is important to evaluate its impact on the automation scripts. A robust test automation framework will make such impacts minimal.

Build Framework that Works

Test automation framework is a set of rules, procedures, and tools that drive automated testing. One of the main advantages of using a framework is the low cost of maintenance. For any changes to the application, only the concerned test case(s) need to be updated while the rest of the script remains the same.

Some of the most common frameworks are data driven, keyword driven, modular, and hybrid. There are again custom built frameworks that fit to specific needs of the application. Spending enough time on choosing the right framework and designing it is the key to test automation success. 

A good and robust framework should consider as many assumptions as possible. This will eventually make the framework quite flexible. While it will be difficult to prove due to lack of data but the general belief is that most of the test automation projects fail in the long run as the framework is not flexible enough to adapt to the changes that the application or the environment has undergone.

Let us take an example to explain the above situation. Consider a Web application that is developed in .NET technology compatible to WinXP, Vista, and Win7 and browsers such as IE and Firefox. A version 5.0 of a test automation tool is used to automate a set of sanity test cases. 

A test automation framework is developed to ensure that as the application undergoes changes, there are re-usable functions that are developed so that with less effort you can address those changes. Most of the frameworks today address this basic need. However, consider a situation where a new version of OS, browser, and the test automation tool need to be addressed in the framework itself. It is not an easy situation or assumption to imagine or incorporate within a framework.

The next generation frameworks should address the best of the following:

* Coverage
* Business Process
* Extended Environment
* Multi OS
* Multi Test Automation Tools
* Best of Reporting and Analysis
* Defect Analysis
* Metric
* Compliance such as HIPPA or SOX
* Recovery Scenario
* Addressing Technology Challenges

Conclusion

Building a test automation framework needs skills such as designing, analytical, and a lot of foresight. While testers bring in a lot of value and insight into developing the framework, it is the designers and developers who eventually build it. Considering the development of framework as a typical development project goes a long way in getting the required RoI. Build the best software engineering practices while developing a framework. Some of the critical aspects, as the fig 1.0 depicts, make the framework dependable and robust in the long run.

via Test Automation Framework - Challenges in the Ever Changing Technology Scenario - QA Expert Columns.

Read more »

Automating in Agile Environment

One of the major barriers to test automation is the volatility of the application to be tested. Even benign changes such as moving a button to a different part of the screen or changing a label from Next to Continue can cause test scripts to fail. This problem is especially acute when testing an applications functionally at the GUI level because this tends to be the area in which changes are most frequent.

For this reason, test teams have historically tended to avoid functional test automation until an application has become stable. However, this approach does not work when building SaaS products or developing products in an Agile environment where change is a constant. In this situation, a more sophisticated test approach is required to avoid incurring technical debt (i.e., quality issues or tasks that are deferred to be addressed later or not at all)

Develop a Reusable Subroutine Library

The first level of sophistication in test automation is to develop a library of reusable subroutines that will encapsulate and hide an applications implementation details from the test scripts themselves. For example, you might implement a test subroutine called login_to_my_app() that accepts two parameters: UserName and Password. 

At runtime, this test subroutine will first find the appropriate text fields in the application GUI and then execute the keystrokes and mouse movements required to fill in the given user name and password, thus completing the login operation. If the applications login GUI changes (e.g., if the on-screen label for User Name is changed to Login Name), then the login_to_my_app() test function might also need to be updated. However, the scripts that called this subroutine would not need to change.

The reason this type of abstraction is a win for some projects is that, in this scenario, a change to the application now only requires you to maintain a single subroutine rather than potentially hundreds or thousands of test scripts. Since complex applications frequently have thousands of test scripts, the savings in effort and time is clear. The trade-off is that the level of skill required to maintain a subroutine library is generally higher and of a different type than that needed to maintain test scripts. Obviously this requirement will change your staffing profile and cost structure.

However, the benefit of this approach is that you can accommodate change much more readily. More importantly, it will allow you to safely write test cases before the application has been fully implemented. Test script writers only need to know (a) the softwares functionality and requirements and (b) the stubs of the test subroutines that will be implemented. You can significantly compress your implementation schedule by enabling your development and test automation processes to function in parallel.

Conclusion

Without a doubt, developing a library of reusable subroutines to test your system requires an up-front investment. 

However, in situations where you:
1. Need to build or customize a SaaS product
2. Wish to truly gain the benefits of an Agile, test-driven development methodology
3. Need to compress your test and development schedule as much as possible while still ensuring high quality; or
4. Must cope with constant change during pre or post-deployment, taking a more sophisticated approach to test automation can be a lifesaver for your project.

via Making Automated Functional Testing Work in SaaS and Agile Environments

Read more »

Learn Selenium For Starters

If you are a starter and are looking into dwell into Selenium World. Then first of all, I would like to welcome you to this excellent tool.

Check out the PDF document from the Selenium Site:

http://seleniumhq.org/docs/book/Selenium_Documentation.pdf

It explains very clearly the basic concepts about Selenium and will surely let you get started immediately.

Read more »

Childhood Video - Created for "Sangeeth Sandhya"

Introduction:

This video was created by me to be presented before my marriage function on 25 February 2011 held at Chennai. It really relishes our childhood memories.

Read more »

Things you need to let go before joining a startup

A lot of people want to join a startup. From the outside, it looks like a sexy thing to do (i.e. next to starting up your own company) and looks like a natural choice to be in the game. But, things are different and well, strange as well.

As a early employee of a startup, you have the same responsibility as the founding team and you are expected to do a lot more than what you have probably imagined. But we all carry the baggage of past and have a certain expectation of the future. In the early days of the startup, when there’s ambiguity and amorphousness about the company’s structure, roles and responsibilities, it is understandable when the founders and early team members do everything and anything possible to get the job done right, on time. After a while, as the startup grows, a more formal structure comes into being.

Based on my experience (of being an early employee at a startup to running a startup), here are a few pointers that you probably will have to let go, before joining a startup:

Ego

Would you have ever imagined cleaning office, sending courier, running around government offices etc etc? Well, your engineering degree (or what’er/MBA etc) was way too fancy to shield you from such menial job. Moreover, the startup founder has a reason to do all the shit work (heck! its his company), but why should you?

   And there is only 1 reason why should you do that : “You pray for rain, you gotta deal with the mud too. That’s a part of it.” [Quote by Denzel Washington].

Excuses

“I can’t do this because <fill in the blank>”. Well, if you expect a perfect world, do not join a startup. It’s like saying ‘I want a girl-next-door’ – so maybe, you should just go next door?

Startups are amorphous and anything but perfect. There is an unorganized chaos and if you find yourself trying to poke holes/find excuses to not do things, it’s a reminder that you are better suited for a role somewhere else.

So go somewhere else.

Money, honey?

If all you are thinking about is how much do I make (vs. my friends working in Infosys/Yahoo/Google etc), you are comparing apples and oranges. In startups, even founders do not make a lot of money (read: Start-Up Salaries and Myths About Them). The fun lies in creating something, disrupting in your definition of the world and in the process, having fun, meeting other interesting people.

If all you look for is money, then go fish somewhere else. Agree that you need to make two ends meet, but expecting skyrocketing salary doesn’t really happen in a startup.

The “ONLY” Phenomena

If you are the one who is not a multi-tasking person, and want to stick to only 1 thing, you probably should join a company that is hiring for your skill set. Startups do not hire for skill set, they hire for attitude and ‘I can only do <X>’ doesn’t help. You need to move out of your comfort zone and do things which you have never even imagined [read: My first 90 days in a startup].

If all you are looking for is RoI (hell! what do I get by learning <X>, which isn’t even related to my core skill set), then you should never join an early stage startup. Join when the process is setup and you know what you are going to do.

The Sexy Title.

Titles are distributed randomly in a startup. From nowhere, you become ‘director of engineering’, ‘product manager’, ‘VP of sales’ etc etc., but ask yourself – do you even deserve the title?

If you are CTO of a small company, would you join Google as an entry-level engineer or somebody senior (after all, you were the CTO of a small startup!). Can you really handle same level of complexity (in Google/Apple) and justify the “CTO title”? Ask yourself.

And always remember that it’s never about the size of operation, but it’s the capability of the person that ‘title’ should stand for. (Recommended Read: Of Roles & Titles In Startups [First Deserve, Then Desire])

If you cannot live up to the title, then why are you even carrying it? Do a reality check and you will be fine.

If you are the VP, Sales in a startup and manage INR 20L/month of sales, can you also handle the complexity associated with managing INR 20 Cr when the startup grows up?

Quit. Or let go of the title. Better, fight to deserve the title.

Your Real CTC

You know your real cost to the company (CTC)? Apart from salary element, a person’s real cost to the company  is whether he/she is dependable or not. If you aren’t dependable, you are increasing the overall cost to the company as you are creating a process where the founder/senior members need to keep following up with you (on each and every small aspect).

This kind of attitude works in a big company (owing to the level of incompetency across the board) , but hiring in a startup is done with an assumption that

..You can be on your own.

You will eventually learn to run with the task you are supposed to do (+ tasks you aren’t supposed to do), without any followups.

-

Parting words

Startups are not fun, they are actually a paranoid-ic place to be in. And only paranoids survive this game. So be in for a bigger cause and learn to enjoy the bumpy ride.

And mind you, there is nothing wrong in working with a big company and the echo-chamber created by the likes of Pluggd.in (echo-chamber that startups are the next big thing) is frankly,  just another echo-chamber. If you are not in for a startup life, you are just normal.

In short, do not join a startup for the heck of joining one.

What’s your take?

via Things you need to let go before joining a startup.

Read more »

Small hack to run different versions of Internet Explorer

Introduction:

While doing testing across multiple browsers, we tend to have an interest of testing on different versions of browsers.  But due to lack of adequate hardware resources, this is pushed as a low priority and often not tested. But this hack will help you run different versions of Internet Explorer, i.e. IE7, IE8 and IE9.

Pre-requisite for this hack:

Ensure IE9 should be installed on your machine.

Steps:

1. Open IE.


2. Press F12 to Open IE Developer Toolbar.


3. Once IE Developer Toolbar is opened, change the browser mode option as shown in the image.


4. That’s it. Now start typing the webpage url and render. The pages will be rendered as if it was rendered in IE7 or IE8.


5. Note: Do not close the browser after changing the browser mode. Everytime the browser is opened, it is always rendered in IE9 mode by default.

Read more »

Reasons why I joined a startup

I have recently joined a startup company named MarketShare @ Bangalore office. It has become a common trend these days to leave a well secured job in an established company and join a startup. Well, I did not leave my previous job just because of this trend. There are several reasons why I had joined a startup. This article is to list down these reasons.

Why I wanted to join a startup:

1. I always wanted to start my own company

One of the main reasons why I always wanted to join a startup was because I always wanted to start my own company, some day or the other. I have a belief that if I work for a startup, I can gain more knowledge and experience than if I am working in a big established firm.

2. I cannot leave my fixed paying job due to my current commitments

I cannot leave my fixed paying job because I have a lot of commitments to carry forward and if I just quit and start, all my current commitments would get affected. Currently I am not in a position where I have good savings in my bank account to run the show without any impacts.

3. Faster Career Growth

I always believe that the personal career growth should always be dependent on the individual performance and contribution. Working in a startup gives you this advantage, as the overall performance and contribution directly impacts the company's performance. And as the company grows, so do the individual. This effect is somehow diluted or nullified in bigger companies. Moreover you get immediately recognized within a startup for your contributions and thereby it increases your value in the company.

Some other advantages of joining a startup:

• Flexible Work Timings


• Flexible  Process


• Work from Home


• Sense of achievement when company grows because of your contributions


• Explore, Learn and Adapt - Latest trends in technologies


• Option to drive the company as your own but at the risk of others.


• Option to make mistakes and learn, without getting directly impacted.

Read more »

Idea: Social Networking for Freelancers/Entrepreneurs

Benefits of Social Networking for Freelancers:
1. New Ideas – Staying connected with social networks is an excellent way to keep updated on the news as well as new trends.
2. Writing sources – When writing human interest stories writers look for quality sources for quotes regarding their articles.
3. Instant Help - Find other freelancers for supporting your project and thereby getting an instant help.
4. New clients – Through social networking, clients can find you. For example, they can search for specific skills or find you through people that you know. The beauty of this is that you don’t have to find them. Another way of finding you is through specialized groups or discussions by looking through someone’s contacts.

Some other benefits which will help the entire community is:

• Logging: Passing on your own knowledge or contributing useful comments.


• Forums: Showcasing your skills, helping others, getting support and more.


• Conferences: Word of mouth, chatting with experts and learning new skills.


• Social Networks: Gaining support, friends and followers with links or content.


• Side Projects: Money earners, extra-curricular activities or contributory work.


• Mentoring: Potentially gaining potential employees or job opportunities.


• Meet-ups: Getting like-minded people together to interact and gain leads.


• Skill-swaps: Offering services for others in return (bi-directional outsourcing).

Activities which the Freelancers can do to help each other:

1. Freebies section which will list down all the freebies across different sites. Users can post their contributions towards this list, thereby increasing the knowledge of each other.


2. Share an Update section where users can post information on their current projects, completed projects, issues faced, lessons learnt etc.. It is just like twitter to tell others on what they are currently working upon.


3. Blogs sections where users can create blogs about several topics and get paid.


4. Get a Help section where users can ask a question and get answered by others immediately.


5. Mail section where users can send personal mails to each others.


6. Swap section where users can post their request and get other service in return continuously (Bi-Directional Outsourcing)


7. Instead of photos, users can upload screenshots of the corresponding project and can store it in their gallery.


8. Google Calendar can be integrated to showcase the schedule of that user.


9. My Favourites website application can be added which will allow users to add their favourite websites and thereby allow other users to have a look into it. This application should option to categorize Favourites list.


10. A Feed application which will feed across major freelancer sites and publish the job details here. This will help users to get project details at a single place.

Features to be present in the site

• Dashboard Overview


• Inbox


• Share an Information


• Share a bookmarkable site along with comments


• Status Update


• Just Joined Block (Tutorials, Helplines etc.. )


• People You May Know


• Profile Statistics (Visits to your profile)


• Search People


• Profile Page


• Groups


• Get Helped


• Swap Skills


• Blogs


• Mentor and Get Mentored


• Bid for Projects (Optional)


• Photo Gallery (To upload screenshots)


• Google Calendar can be integrated to showcase the schedule of that user.


• Coupons/Giveaways


• Freebies

Members

1. An important way of finding friends is through the field user enters “Looking in for”. Several options can be added here and based on user’s response, other users can send friend request when they search.


2. Another way of sending friends request is through scanning the list of friend’s friend list. Instead of showing the common friends between the two members just like normal sites, option can be given to show uncommon friends between both of them. This will help both of them to increase their network by adding these members as their friends.

Advantages of having a bigger friends list:

1. Any activity performed by a user will be posted in the Activity Feed. Thereby more the friends, more is the chance of them looking into your activities and respond to you as soon as possible.


2. Any help request posted will also be posted with Title in the Activity Feed. Hence their friend’s list can

How to increase the friend’s network:

• Find friends in the groups where you are interested. Try to be active member in that group.


• Find Members through

Activity Feed

1. Option should be provided for users to forward any activity feed to all their friends list. This will give us a feed post like “My Friend <Actual Username along with link to his profile> had posted this: <activity post details>”. This will help users to forward any help/request/information to other friends. This will be posted as a normal post with the above specified text added before it, just to differentiate

Mentoring Requirements:

• Block - Mentors of the month and link to view all featured mentors in this category


• Seperate Rating option on several factors for Mentoring


• Latest Mentor Shout Outs (Option where mentors can create a short description about the cause


• In Search for a Mentor screen, a block should be present which displays “Can't find the right mentor? Just send a request!”.


• In Be a Mentor page, firstly option should be given for users to accept for being a mentor like by submitting a confirmation form which will contain a list of categories which the user is ready to mentor others. On submission, the user will be added and maintained in a list. When some one submits a request for mentoring, depending upon the categories selected, posts will be made to all the corresponding mentors and whoever is ready can start mentoring them.


• Separate section should be displayed indicating the list of members whom the user is mentoring (i.e their followers) and list of members whom the user is being mentored (i.e following)

Project Feeds

• Option for feeds from several sites should be provided. User should have the option of customizing the list of blocks which the user wants to follow from the feed list. This will help them to follow projects from a single place.

User Points for performing various activities on the site

• Creating a New Blog - 5 Points


• Each users liking the blog: 1 Point


• Mentoring Someone - 5 Points


• Users liking the mentoring - 1 Point


• Posting Project Updates - 1 Point


• Daily Visit to Site - 1 Point


• Contributing a Freebie - 5 point


• Each person liking the freebie contributed - 1 point


• Each New Friend Request accepted - 1 point for each member


• Completing 100% profile information - 5 points


• Contribution to Groups - 1 point


• Sharing an Event - 5 point


• Each person liking the event - 1 point


• Sharing the link of sites along with description - 5 point


• Each person liking the event - 1 point


• Each person liking the user’s comments on blogs etc... - 1 point

Read more »

Idea: Become a Brand Ambassador Online

Overview:

This is a platform for users to become a brand ambassador of certain companies online and thereby market for that brand in the social network using his/her contacts. Once a user becomes an ambassador, the user will get 5 giveaways from that company. The company has the choice to select the giveaways according to several categories. There should be several options available for users to choose from while taking away the giveaway option.

What should the Brand Ambassador Do (Draft Ideas):
The Brand Ambassador will do the following things online using his social network. Note that these are just raw ideas and are not final ones. Some of them might sound very stupid and can be ignored.

1. They should reveal out online that they are a brand ambassador for the brand when spreading the word publicly about the brand.
2. A tag should be associated to their names and signatures “Brand Ambassador of <Brand Name>”
3. Link should be posted on their profiles linking to the brand.
4. Separate Photo Gallery should be created for that brand and the user should upload photos regarding that brand.
5. In Facebook, the user should post a minimum number of posts regarding the brand name daily. Posts should be live commentary about the brand.
6. In Twitter, the user should tweet a minimum number of posts regarding the brand name daily.
7. Have Ad space reg. their brand in their page.

What can the users post:

• Live commentary about the brand


• Headlines and link about certain news about the brand

Revenue Generation Options:
1. Registration fees for companies who wants brand ambassadors.
2. Different Membership levels
3. Advertisements

Problems expected:
1. How to track if someone is doing the brand ambassador job properly or not.
2. How will the company give the compensation to the brand ambassadors.

Brainstorming:
1. Need to research more on what all activities the brand ambassador should do online.
2. Need to research more on how to select a brand ambassador i.e criterias for selection of a brand ambassador.
3. Need to research more on benefits of becoming a brand ambassador.
4. How to track if someone is doing the brand ambassador job properly or not.

Reference Articles:
http://gigaom.com/collaboration/how-to-know-a-good-fan-on-facebook/
http://gigaom.com/collaboration/how-to-convert-your-facebook-superfans-into-brand-ambassadors/
http://en.wikipedia.org/wiki/Net_Promoter
http://gigaom.com/collaboration/10-social-media-marketing-questions-answered/

Read more »

Idea: Get sponsored online

Overview:

Users posts their requirements and these are submitted to companies for reviewing. Based on company’s interest, they decide to sponsor the request or ignore them.

Two kinds of registrations will be required - One is users who needs sponsorship and the other is companies who are ready to sponsor.

Revenue Generation Options:

1. The sponsor will have to pay a certain amount of money to register to our website and then get details matching his interest where he can sponsor. This way we filter out the bad ones who are there to cause trouble in our system.
   (and/or)


2. The users seeking sponsorship will have to pay a certain amount to register and request for sponsorship.
   (and/or)


3. The process of requesting for sponsorship and viewing the details is all free. Once a sponsor accepts the sponsorship, the details of the sponsor will not be revealed out to the end user. For getting these details, the user has to pay X% of the sponsorship amount to us as a consultancy fee.
   (and/or)

Brainstorming:

1. Different categories or kinds of sponsorship has to be thought of. Some of them which I can think of are:

• Sponsor my studies


• Sponsor an event


• Sponsor a NGO


• Sponsor a program

Read more »

Idea: Option to earn money through SMS

Option to earn money through SMS just like mGinger does. mGinger sends ads from clients to interested people and thereby they get paid.  A reverse of this can be thought where people can send sms and then get paid. Now why will people get paid on sending sms?

Answer: Marketing for the client.
People can use their free sms service and send sms and get paid for it. But there should be a tracking mechanism for tracking the number of sms being sent with that content. This may be very difficult if user sends it through their phone. But this would definitely be possible if user sends it through an online site. There can be a platform in the site, just like way2sms.com, where the user’s messages are sent using their numbers to their contacts.

More research can be done to improve this idea.

Read more »