Welcome!

iPhone Authors: Carmen Gonzalez, Elizabeth White, David H Deans, Liz McMillan, Scott Hirsch

Blog Feed Post

Is It Fair to Compare Manual and Automated Test Results?

​The article below was originally published by Rajini Padmanaban on the fantastic QAInfoTech blog.

Manual and automated testing have long co-existed and have contributed significantly in successfully signing off on a product’s quality. Over the years test teams have talked about the value of one over another, how testers need to groom their test automation skills and so on. A tester who really understands software quality and the intricacies it entails, will be one that appreciates the value of both these testing approaches and that one cannot be under or over-estimated over the other. With more organizations embracing the Agile style of development, test automation’s scope has definitely gone up. There is an increasing push for newer areas to be automated, newer test engineers to be trained to take on automation, newer tools to help with the process etc. Test managers and leads are being cautious in doing so, because an unplanned test automation effort can soon become a mammoth suite that is overwhelming to manage and maintain.

This is where an objective comparison between manual and test automation efforts is becoming important to determine their true value and make any adjustments that may be needed to further optimize them for the product’s benefit. You may ask – is it fair to compare the manual and automated test efforts and this is a valid question. Often times, it may not even be an apple to apple comparison. For example, the manual test focus may have been on the UI intensive portions of the application, which have been buggier. As a result, the test team may have reported a lot of valid defects through their manual efforts. On the other hand, test automation’s focus may have been API intensive areas, application’s performance, database level functionality etc. where defects are more difficult to find. It could also be the case, that APIs have been re-used from the past and are very stable that the tester has not reported as many bugs through test automation. Does this mean that the automation was not effective?  

While this argument holds weight, and that manual and automated test results comparison is not always apples to apples, it does not mean they cannot be compared. They need to compared on the right grounds to help the team arrive at the right balance between the two test approaches and also what, when, how and how much to automate. If so, how can they be objectively compared? Here are some tips:

  1. Pick areas that have been currently automated, see what bugs are reported through them and also have some of those areas tested manually, in parallel. Compare the results to see if the automation scenarios need any modification. This can also help add the manual tester’s instinctive creativity into the automated suite
  2. Understand the % of valid defects reported by manual and automated test efforts. This will help understand if automation has a lot of test/data/configuration issues that need to be attended to and fixed  
  3. After an exploratory round of testing or a bug bash, compare manual test results with the corresponding automation that may exist in those areas to see how the test automation suite can further be strengthened
  4. Look at overall numbers to a certain extent. If manual tests are yielding way too many bugs but hardly any are coming in through automation, it may be an indication for further analysis. This may end up being a false alarm, but is at least worth a quick look
  5. Compare the kind of defects reported by the two test approaches. If manual is reporting more UI and functional defects in a certain area, while automation is not catching them, this is a good checkpoint to see how the automation’s can be enhanced
  6. Use automation to also evaluate the manual testing effort’s efficiency. The comparison is not a one sided story. Sometimes the manual testers’ on the team may not be very efficient and may be careless in their test effort. Automation, once tested for its reliability and consistency in results, is a great approach to minimize human errors. So, say for instance a certain regression has been automated and the tester also happens to play around with that area manually, a manager can compare the results to understand the tester’s efficiency and areas of improvement

The above is certainly not an exhaustive list. However, when this mindset is established in the team, they will begin to appreciate the need for such an objective comparison between the manual and automated test approaches. This will be a new category that can be incorporated in the metrics that they use in line with the need of the current day, helping the two test approaches complement each other.

About the author:

Rajini Padmanaban is a Sr. Director of Testing Engagements at QA InfoTech and an active software testing evangelist. She has more than twelve years of professional experience, primarily in the software quality assurance space.

Read the original blog entry...

More Stories By Noel Wurst

I am the managing content editor at Skytap. Skytap provides SaaS-based dev/test environments to the enterprise. Our solution removes the inefficiencies and constraints that companies have within their software development lifecycle. As a result, our customers release better software faster. My aim is to publish engaging, thought provoking stories that revolve around agile enterprise applications and cloud-based development and testing.

@ThingsExpo Stories
The definition of IoT is not new, in fact it’s been around for over a decade. What has changed is the public's awareness that the technology we use on a daily basis has caught up on the vision of an always on, always connected world. If you look into the details of what comprises the IoT, you’ll see that it includes everything from cloud computing, Big Data analytics, “Things,” Web communication, applications, network, storage, etc. It is essentially including everything connected online from hardware to software, or as we like to say, it’s an Internet of many different things. The difference ...
Cloud Expo 2014 TV commercials will feature @ThingsExpo, which was launched in June, 2014 at New York City's Javits Center as the largest 'Internet of Things' event in the world.
An entirely new security model is needed for the Internet of Things, or is it? Can we save some old and tested controls for this new and different environment? In his session at @ThingsExpo, New York's at the Javits Center, Davi Ottenheimer, EMC Senior Director of Trust, reviewed hands-on lessons with IoT devices and reveal a new risk balance you might not expect. Davi Ottenheimer, EMC Senior Director of Trust, has more than nineteen years' experience managing global security operations and assessments, including a decade of leading incident response and digital forensics. He is co-author of t...

ARMONK, N.Y., Nov. 20, 2014 /PRNewswire/ --  IBM (NYSE: IBM) today announced that it is bringing a greater level of control, security and flexibility to cloud-based application development and delivery with a single-tenant version of Bluemix, IBM's platform-as-a-service. The new platform enables developers to build ap...

The major cloud platforms defy a simple, side-by-side analysis. Each of the major IaaS public-cloud platforms offers their own unique strengths and functionality. Options for on-site private cloud are diverse as well, and must be designed and deployed while taking existing legacy architecture and infrastructure into account. Then the reality is that most enterprises are embarking on a hybrid cloud strategy and programs. In this Power Panel at 15th Cloud Expo (http://www.CloudComputingExpo.com), moderated by Ashar Baig, Research Director, Cloud, at Gigaom Research, Nate Gordon, Director of T...
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at Internet of @ThingsExpo, James Kirkland, Chief Architect for the Internet of Things and Intelligent Systems at Red Hat, described how to revolutioniz...
Technology is enabling a new approach to collecting and using data. This approach, commonly referred to as the "Internet of Things" (IoT), enables businesses to use real-time data from all sorts of things including machines, devices and sensors to make better decisions, improve customer service, and lower the risk in the creation of new revenue opportunities. In his General Session at Internet of @ThingsExpo, Dave Wagstaff, Vice President and Chief Architect at BSQUARE Corporation, discuss the real benefits to focus on, how to understand the requirements of a successful solution, the flow of ...
The security devil is always in the details of the attack: the ones you've endured, the ones you prepare yourself to fend off, and the ones that, you fear, will catch you completely unaware and defenseless. The Internet of Things (IoT) is nothing if not an endless proliferation of details. It's the vision of a world in which continuous Internet connectivity and addressability is embedded into a growing range of human artifacts, into the natural world, and even into our smartphones, appliances, and physical persons. In the IoT vision, every new "thing" - sensor, actuator, data source, data con...
"BSQUARE is in the business of selling software solutions for smart connected devices. It's obvious that IoT has moved from being a technology to being a fundamental part of business, and in the last 18 months people have said let's figure out how to do it and let's put some focus on it, " explained Dave Wagstaff, VP & Chief Architect, at BSQUARE Corporation, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Focused on this fast-growing market’s needs, Vitesse Semiconductor Corporation (Nasdaq: VTSS), a leading provider of IC solutions to advance "Ethernet Everywhere" in Carrier, Enterprise and Internet of Things (IoT) networks, introduced its IStaX™ software (VSC6815SDK), a robust protocol stack to simplify deployment and management of Industrial-IoT network applications such as Industrial Ethernet switching, surveillance, video distribution, LCD signage, intelligent sensors, and metering equipment. Leveraging technologies proven in the Carrier and Enterprise markets, IStaX is designed to work ac...
"There is a natural synchronization between the business models, the IoT is there to support ,” explained Brendan O'Brien, Co-founder and Chief Architect of Aria Systems, in this SYS-CON.tv interview at the 15th International Cloud Expo®, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
C-Labs LLC, a leading provider of remote and mobile access for the Internet of Things (IoT), announced the appointment of John Traynor to the position of chief operating officer. Previously a strategic advisor to the firm, Mr. Traynor will now oversee sales, marketing, finance, and operations. Mr. Traynor is based out of the C-Labs office in Redmond, Washington. He reports to Chris Muench, Chief Executive Officer. Mr. Traynor brings valuable business leadership and technology industry expertise to C-Labs. With over 30 years' experience in the high-tech sector, John Traynor has held numerous...
Bit6 today issued a challenge to the technology community implementing Web Real Time Communication (WebRTC). To leap beyond WebRTC’s significant limitations and fully leverage its underlying value to accelerate innovation, application developers need to consider the entire communications ecosystem.
The 3rd International @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades.
The Internet of Things is not new. Historically, smart businesses have used its basic concept of leveraging data to drive better decision making and have capitalized on those insights to realize additional revenue opportunities. So, what has changed to make the Internet of Things one of the hottest topics in tech? In his session at @ThingsExpo, Chris Gray, Director, Embedded and Internet of Things, discussed the underlying factors that are driving the economics of intelligent systems. Discover how hardware commoditization, the ubiquitous nature of connectivity, and the emergence of Big Data a...
Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity.
SYS-CON Events announced today that Windstream, a leading provider of advanced network and cloud communications, has been named “Silver Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. Windstream (Nasdaq: WIN), a FORTUNE 500 and S&P 500 company, is a leading provider of advanced network communications, including cloud computing and managed services, to businesses nationwide. The company also offers broadband, phone and digital TV services to consumers primarily in rural areas.
SYS-CON Events announced today that IDenticard will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. IDenticard™ is the security division of Brady Corp (NYSE: BRC), a $1.5 billion manufacturer of identification products. We have small-company values with the strength and stability of a major corporation. IDenticard offers local sales, support and service to our customers across the United States and Canada. Our partner network encompasses some 300 of the world's leading systems integrators and security s...
IoT is still a vague buzzword for many people. In his session at @ThingsExpo, Mike Kavis, Vice President & Principal Cloud Architect at Cloud Technology Partners, discussed the business value of IoT that goes far beyond the general public's perception that IoT is all about wearables and home consumer services. He also discussed how IoT is perceived by investors and how venture capitalist access this space. Other topics discussed were barriers to success, what is new, what is old, and what the future may hold. Mike Kavis is Vice President & Principal Cloud Architect at Cloud Technology Pa...
Cloud Expo 2014 TV commercials will feature @ThingsExpo, which was launched in June, 2014 at New York City's Javits Center as the largest 'Internet of Things' event in the world. The next @ThingsExpo will take place November 4-6, 2014, at the Santa Clara Convention Center, in Santa Clara, California. Since its launch in 2008, Cloud Expo TV commercials have been aired and CNBC, Fox News Network, and Bloomberg TV. Please enjoy our 2014 commercial.