Eliminating Mobile Test Automation Flakiness and More

Mobile testing by definition is an unstable, flaky and unpredictable activity.

When you think you covered all corners and created a “stable” environment, still, your test cycle often get stuck due to 1 or few items.

In this post, I’ll try to identify some of the key root causes for test automation flakiness, and suggest some preventive actions to eliminate them.

What Can Block Your Test Automation Flow?

From an ongoing experience, the key items that often block test automation of mobile apps are the following:

  • Popups – security, available OS upgrades, login issues, etc.
  • Ready state of DUTs – test meets device in a wrong state
  • Environment – device battery level, network connectivity, etc.
  • Tools and Tets Framework fit – Are you using the right tool for the job?
  • Use of the “right” objects identifiers, POM, automation best practices
  • Automation at Scale – what to automate, on what platforms?

All of the above contribute in one way or the other to the end-to-end test automation execution.

We can divide the above 6 bullets into 2 sections:

  1. Environment
  2. Best Practices

Solving The Environment Factor in Mobile Test Automation

In order to address the test environment contribution to test flakiness, engineers need to have full control over the environment they operate in.

If the test environment and the devices under test (DUT) are not fully managed, controlled and secured the entire operation is at risk. In addition, when we state the term “Test Environment Readiness” it should reflect the following:

  1. Devices are always cleaned up prior to the test execution or are in a known “state”/baseline to the developers and testers
  2. If there are repetitive known popups such as security permissions, install/uninstall popups, OS upgrades, or other app-specific popups, they should be accounted for either in the pre-requisites of the test or should be prevented proactively prior to the execution.
  3. Network stability often is a key for unstable testing – engineers need to make sure that the devices are connected to WiFi or cellular network prior to testing execution start. This can be done either as a pre-requisite validation of the network or through a more generic environment monitoring.

 

Following Best Practices

In previous blogs, I addressed the importance of selecting the right testing frameworks, IDEs as well as leveraging the cloud as part of test automation at scale. After eliminating the risks of the test environment in the above section, it is important to make sure that both developers and test automation engineers follow proper guidelines and practices for their test automation workflow.

Since testing shifted left towards the development team, it is important that both dev and test align on few things:

  1. What to automate?
  2. On what platforms to test?
  3. How to automate (best practices)?
  4. Which tools should be used to automate?
  5. What goes into CI, and what is left outside?
  6. What is the role of Manual and Exploratory testing in the overall cycle?
  7. What is the role of Non-Functional testing in the cycle?

The above points (partial list)  covers some fundamental questions that each individual should be asking continuously to assure that his team is heading in the right direction.

Each of the above bullets can be attributed to at list one if not many best practices.

  1. To address the key question on what to automate, here’s a great tool (see screenshot below) provided by Angie Jones. In her suggested tool, each test scenario should be validated through some kind of metric that adds to a score. The highest scored test cases will be great candidates for automation, while the lowest ones obviously can be skipped.
  2. To address the 2nd question on platform selection, teams should monitor their web and mobile ongoing traffic, perform market research and learn from existing market reports/guides that addresses test coverage.
  3. Answering the question “How to automate” is a big one :). There is more than 1 thing to keep in mind, but in general – automation should be something repetitive, stable, time and cost efficient – if there is something preventing one or more of these objectives, it’s a sign that you’re not following best practices. Some best practices can be around using proper object identifiers, others can reflect building the automation framework with proper tags and catches so when something breaks it is easy to address, and more.
  4. The question around tools and test frameworks again is a big one. Answering it right depends on the project requirements and complexity, the application type (native, web, responsive, PWA), test types (functional, non-functional, unit). In the end, it is important to have a mix of tools that can “play” nicely together and provide unique value without stepping on each other.
  5. Tests that enter CI should be picked very carefully. Here, it is not about the quantity of the tests but about the quality, stability, and value these tests can bring focusing on fast feedback. If a test requires heavy env. setup is flaky by nature, takes too much time to run, it might not be wise to include it in the CI.
  6. Addressing the various testing types in Qs 6 and 7 – it depends on the project objectives, however, there is a clear value for each of the following tests in  mobile app quality assurance:
    1. Accessibility and performance provide key UX indicators about your app and should be automated as much as possible
    2. Security testing is a common oversight by many teams and should be covered through various code analysis, OWASP validations and more.
    3. Exploratory, manual and crowd testing provide another layer of test coverage and insights into your overall app quality, hence, should be in your test plan and divided throughout the DevOps cycle.

Happy Testing

The Rise of Automation: Why Coding is Becoming a Job for Everyone

Guest Blog by Wendy Dessler

Photo by Alex Knight on Unsplash

According to USA Today, as many as 73 million jobs may not exist by 2030; however, we could see benefits like an increase in productivity and economic growth “counteract” job losses. But, will it? Will it not? That is a question several economists mull over.

What we can say, given our increasing reliance and incorporation of technology, we predict that coding will be a skillset job seekers will list on their applications—along with the usual: typing, Excel-savvy, multitasking, you name it.

But does everyone have to be proficient in coding? Are all of our jobs reliant on how well we know to insert code? Read on to find out!

Automation Affects the Job Market   

Because of automation, we are seeing more self-checkouts in grocery stores and convenience stores. Even some fast food stores—like Wendy’s—are planning to install not one but more than 1,000 self-ordering kiosks in its stores.

The Why

The truth is, automation cuts market costs and saves companies a lot of money. Think about it: unlike an employee, self-ordering kiosks and self-checkouts, you name it aren’t going to get tired; they won’t come in late to work, and the chances of getting an order wrong is a lot lower. In other words, it’s safe to say automation minimizes human error.

More Automation Threatens Certain Industries

According to CNN Money, cashiers and toll booth operators, drivers, and fast food jobs are all at risk of becoming fully automated; however, other jobs like nurses, doctors, youth sports coaches, hairstylists and cosmetologists, songwriters, and social workers are far from being at risk.

Pretty much any job that incorporates a high degree of “humanness”—like empathizing with the client or using emotional nuances to do a better job—are in the clear.

 

But what how are tech industries faring? How is automation affecting the way developers and quality assurance do their jobs?  

A Shift in the Tech Sector

Thanks to automation, we even see a shift in the tech sector. DevOps teams, for instance, are responsible for deploying automation (not to mention making sure the site is up and the server doesn’t crash, etc.).

While those in the quality assurance sectors are relying more and more on coding to test products before they are distributed.

 

In some instances, they even do some degree of auto-testing, and some may use error monitoring software to rule out bugs, which you can learn more about at https://stackify.com/error-monitoring/.  

But Where Does Coding Factor In?

The age of automation is upon us, and with that, employees and job seekers can strengthen their skill sets by learning how to code. However—and this is a big, however—unless you work in the tech sector, you probably don’t have to be a coding expert.

 

Even Low-Risk Jobs Can Benefit from Knowing Some Coding Basics

As we mentioned, songwriters aren’t at risk at automation (as of yet), but that doesn’t mean they can’t benefit from learning some code and using automation in their song recording processes.

Long story short, learning how to code can only make you are a  more in-demand employee. But, like we said, unless your job is tech-related, it isn’t an end-all-be-all.

Photo by John Schnobrich on Unsplash

Final Thoughts

Automation doesn’t have to because for concern. Remember, we’ve gone through several economic transitions—such as the Industrial Revolution or even the shift from letter writing to email—that have changed how we do business (for the better).

What are your thoughts on automation? Will AI take little, some, or all of our jobs? Do you think it’s a positive or a negative? We want to know your thoughts; be sure to leave a comment in the comments section below.

Create a website or blog at WordPress.com

Up ↑