Cloud Testing: How to test on Cloud

Unless you've been living under a rock you must already be knowing that 'Cloud Computing' has been making a lot of buzz over past couple of years -- whether it your peer meeting, a client interview, a demo POC session with a prospect, the recent Red Hat Enterprise Virtualization Online Event on January 18, 2012 -- the talk about cloud is everywhere.

And why wouldn't it be? After all, if industry analysts and virtualization experts are to be believed then cloud based computing and business solutions are going to be the NEXT BIG thing of this decade.

So I guess it is only natural if you find yourself to be asking yourself questions like 'what is cloud testing?', 'how to test on cloud?', 'how can we use cloud to better our testing?', 'how does cloud impact how we used to test before?' etc.

However, since all these queries pose different questions, the answers to them would be unique. For starters, if you are looking for cloud testing, it simply means a testing environment that utilizes cloud infrastructure for performing software testing.

How to leverage Cloud to Transform Software Testing?

If you are someone who heavily use tools while testing then IBM (IBM Cloud) and Hewlett-Packard have already jumped into the market for software testing in the cloud. Thankfully, if done smartly, cloud based computing can prove to be a great value-addition for both software development and testing. The reason is simple -- the very nature of a cloud based infrastructure allows for great team collaboration.

As an added advantage, cloud based testing (as well as programming) environments are easy to setup (on-demand). In today's tight budgeted IT world, this can be a much bigger advantage than it appears at first. It is no secret that IT managers are operating under a very tight budgetary constraint and when it comes to testing phase, the budget is even smaller.

Traditional approaches to setting up a test environment involves high cost to setup multiple servers with various OS, hardware configuration, browser versions etc. And if you are going to simulate user activity from different geographic locations you will have to setup test servers with localized regional language OS, which in turn can add up to the cost. But using cloud based infrastructure, the team wouldn't have to setup expensive physical servers -- rather, setting up new testing environment will be fast and efficient and VMs (virtual machines) and test servers can be launched and decommissioned as needed.

On the other hand, as a tester you might also be required to one of those ever emerging cloud based SaaS applications that aim to cater to various large and small customer base, on-demand. If you are testing such a cloud based application then your challenges are double-fold. Because, testing all the layers - from your application to the cloud service provider - is something that as a tester you will have to become efficient in.

As a closing note, if you are a tester and if you are intrigued by all these buzz surrounding cloud testing, then here are 2 main reasons why you might consider trying it out -- Cloud based software testing infrastructure greatly helps in reducing capital expenditure and these testing setups are highly scalable , thus allowing your team to expand or decommission your test servers on-demand, as needed.

Tags: Software testing, IBM, Cloud Computing, Hewlett-Packard

Read more »

A beautiful poem on Testing.

Imagine there’s no testing
It’s easy if you try
No bugs below us
Above us only sky
Imagine all the developers living for today

Imagine there’s no test cases
It isn’t hard to do
Nothing to pass or fail
And no test reports too
Imagine all the developers living life in peace

You, you may say
I’m a dreamer, but I’m not the only one
I hope some day you’ll bug us
And the world will be as one

Imagine no certification
I wonder if you can
No need for politics or money
A brotherhood of testers
Imagine all the testers sharing all as one

You, you may say
I’m a tester, but I’m not the only one
I hope some day you’ll join us
And the testers will live as one

Read more »

How Google Instant Pages was tested

Google Instant Pages are a cool new way that Google speeds up your search experience. When Google thinks it knows which result you are likely to click, it preloads that page in the background, so when you click the page it renders instantly, saving the user about 5 seconds. 5 seconds is significant when you think of how many searches are performed each day--and especially when you consider that the rest of the search experience is optimized for sub-second performance.

The testing problem here is interesting. This feature requires client and server coordination, and since we are pre-loading and rendering the pages in an invisible background page, we wanted to make sure that nothing major was broken with the page rendering.

The original idea was for developers to test out a few pages as they went.But, this doesn’t scale to a large number of sites and is very expensive to repeat. Also, how do you know what the pages should look like? To write Selenium tests to functionally validate thousands of sites would take forever--the product would ship first. The solution was to perform automated test runs that load these pages from search results with Instant Pages turned on, and another run with Instant Pages turned off. The page renderings from each run were then compared.

How did we compare the two runs? How to compare pages when content and ads on web pages are constantly changing and we don't know what the expected behavior is? We could have used cached versions of these pages, but that wouldn’t be the realworld experience we were testing and would take time setting up, and the timing would have been different. We opted to leverage some other work that compares pages using the Document Object Model (DOM). We automatically scan each page, pixel by pixel, but look at what element is visible at the point on the page, not the color/RGB values. We then do a simple measure of how closely these pixel measurements match. These so-called "quality bots" generate a score of 0-100%, where 100% means all measurements were identical.

When we performed the runs, the vast majority (~95%) of all comparisons were almost identical, like we hoped. Where the pages where different we built a web page that showed the differences between the two pages by rendering both images and highlighting the difference. It was quick and easy for the developers to visually verify that the differences were only due to content or other non-structural differences in the rendering. Anytime test automation scales, is repeatable, quantified, and developers can validate the results without us is a good thing!

How did this testing get organized? As with many things in testing at Google, it came down to people chatting and realizing their work can be helpful for other engineers. This was bottom up, not top down. Tejas Shah was working on a general quality bot solution for compatibility (more on that in later posts) between Chrome and other browsers. He chatted with the Instant Pages developers when he was visiting their building and they agreed his bot might be able to help. He then spend the next couple of weeks pulling it all together and sharing the results with the team. 

And now more applications of the quality bot are surfacing. What if we kept the browser version fixed, and only varied the version of the application? Could this help validate web applications independent of a functional spec and without custom validation script development and maintenance?

Read more »

Conversation with Google Test Engineer

Alan Faulkner is a Google Test Engineer working on DoubleClick Bid Manager, which enables advertising agencies and advertisers to bid on multiple ad exchanges. Bid Manager is the next generation of the Invite Media product, acquired by Google in 2010. Alan Faulkner has been focused on the migration component of Bid Manager, which transitions advertiser information from Invite Media to Bid Manager. He joined Google in August 2011, and works in the Kirkland, WA office. 

Are you a Test Engineer, or a Software Engineer in Test, and what’s the difference?
Right now, I’m a Test Engineer, but the two roles can be very similar. As a Test Engineer, you’re more focused on the overall quality of the product and speed of releases, while a Software Engineer in Test might focus more on test frameworks, automation, and refactoring code for testability. I think of the difference as more of a shift in focus and not capabilities, since both roles at Google need to be able to write production quality code. Example test engineering tasks I worked on are introducing an automated release process, identifying areas for the team to improve code coverage, and reducing the manual steps needed to validate data correctness.

What is a typical day for you?
When I get in, I look at any code reviews I need to respond to, look for any production bugs from technical account managers that are high priority, and then start writing code. In my current role, I focus my development effort on improving the efficiency and coverage of our large scale integration tests and frameworks. I also work on adding additional features to our product that improve our testability. I typically spend anywhere from 50% to 75% of my time either writing code or participating in code reviews.

Do you write only test code?
No, I write a lot of code that is included in the product as well. One of the great things about being an SET or TE at Google is that you can write product code as easily as test code. I write both. My test code focuses on improving test frameworks and enabling developers to write integration tests. The production code that I write focuses on increasing the verification of external inputs. I also focus on adding features that improve testability. This pushes more quality features into the product itself rather than relying on test code for correctness.

What programming languages do you use?
Both the test and product code are mostly Java. Occasionally I use Python or C++ too.

How much time to do you spend doing manual testing?
Right now, with the role I am in, I spend less than 5% of my time doing manual testing. Although some exploratory testing helps develop product knowledge and find risky areas, it doesn’t scale as a repeated process. There are a small amount of manual steps and I focus on ways to help reduce this so our team does not spend our time doing repeated manual steps as part of our data migration.

Do you write unit tests for code that isn’t yours?
At Google, the responsibility of testing is shared across all product engineers, not just Test Engineers. Everyone is responsible for writing unit tests as well as integration tests for their components. That being said, I have written unit tests for components that are outside of what I developed but that has been to illustrate how to write a unit test for said component. This component usually involved a abnormally complex set of code or to illustrate using a new mocking framework, such as Mockito.

What do you like about working on the Google advertising products?
I like the challenges of the scalability problems we need to solve, from handling massive amounts of data to dealing with lots of real time ad requests that need to be responded to in milliseconds. I also like the impact, since the products affect a lot of people. It’s rewarding to work on stuff like that.

How is testing at Google different from your experience at other companies?
I feel the role is more flexible at Google. There are fewer SET’s and TE’s in my group at Google per developer, and you have the flexibility to pick what is most important. For example, I get to write a lot of production code to fix bugs, make the code more testable, and increasing the visibility into errors encountered during our data migrations. Plus, developers at Google spend a lot of time writing tests, so testing isn’t just my responsibility.

How does the Google Kirkland office differ from the headquarters in Mountain View?
What I really like about the offices at Google is that each of them has their own local feel and personality. Google encourages this! For instance, the office here in Kirkland has a climbing wall, boats and all the conference rooms in our building are named after local bands in the area. The office in Seattle has kayaks and the New York office has an actual food truck in its cafeteria.

What’s the future of testing at Google?
I think the future is really bright. We have a lot of flexibility to make a big impact on quality, testability and improving our release velocity. We need to release new features faster and with good quality. The problems that we face are complex and at an extreme scale. We need engineering teams focused on ensuring that we have efficient ways to simulate and test. There will always be a need for testers and developers that focus on these areas here at Google.

Read more »

Small hack to run different versions of Internet Explorer

Introduction:

While doing testing across multiple browsers, we tend to have an interest of testing on different versions of browsers.  But due to lack of adequate hardware resources, this is pushed as a low priority and often not tested. But this hack will help you run different versions of Internet Explorer, i.e. IE7, IE8 and IE9.

Pre-requisite for this hack:

Ensure IE9 should be installed on your machine.

Steps:

1. Open IE.


2. Press F12 to Open IE Developer Toolbar.


3. Once IE Developer Toolbar is opened, change the browser mode option as shown in the image.


4. That’s it. Now start typing the webpage url and render. The pages will be rendered as if it was rendered in IE7 or IE8.


5. Note: Do not close the browser after changing the browser mode. Everytime the browser is opened, it is always rendered in IE9 mode by default.

Read more »