Using Smoke Tests in Automated Testing
First off this article is not meant to convince you of the benefits of automated testing. You know the benefits or else you wouldn’t be reading this. Instead, let’s focus on a type of automated test called a smoke test.
Many types of tests fall under the umbrella of automated testing. Your use case is unique and will not need every automated test type implemented. Our site healthline.com must be reliable, fast, and appealing. We do not use QA technicians so automated testing is important. Our delivery pipeline includes unit, visual, smoke, and integration tests.
What can be confusing is the difference between smoke, integration, and functional (end-to-end) tests. What is a smoke test? What do they entail? When should smoke tests run? What should smoke tests include?
What is a smoke test?
Smoke? Kind of a strange name, right? Imagine this. A house is on fire a few blocks away. What is the first thing you see? Smoke! You see the smoke and know something wrong is happening. You know the result is going to be disastrous if it isn’t stopped. The same goes for smoke tests. Failure of a smoke test indicates something is wrong with the build.
Wikipedia states smoke tests can be functional tests or unit tests. They test a slice of functionality of the whole system.
What to include
In researching smoke tests, I came upon a company called Functionize. They produce AI-based automated testing. Mastercard, Salesforce, and Cisco are a few of their customers, so in short, they know their stuff. They are a major proponent of smoke testing.
According to Functionize, “the best fit for smoke testing is at the Functional UI layer. Functional testing is all about testing the user interface (UI) in order to uncover new bugs (exploratory functional testing) or ensure that previously known bugs have not found their way back into the application (functional regression testing).”
They recommend you have 20-50 smoke tests. They also gave the following examples of what to include in smoke tests.
Navigating through foundational pages and clicking key areas.
Verifying correct layout and accuracy of all visual elements.
Exercising key functionality such as signup and login forms, additions/subtractions to shopping carts, checkout, return to shopping, and file exports.
When to run
It’s a given that longer running tests expensive to run. You lose productivity and incur more payroll expenses as your developers wait. Smoke tests run slower than your unit tests since they test functionality and not code. Run them after the unit tests for that reason. Make sure to run them before running your full test suite. With having fewer tests they run faster than your integration & remaining functional tests. Functionize states that your smoke tests should run for 5 minutes or less. We have our smoke tests running after our integration tests because of cost. I’ll explain later.
For no other purpose than to stop bugs, of course.
But why include smoke tests in your automated testing? According to Functionize, its customers found show-stopping bugs much earlier using smoke tests. Smoke tests covered 20% of their system and yet caught 80% of bugs. At Healthline, we’ve found smoke testing to be successful too. Our smoke tests have prevented several bugs from entering our production code. Consider the alternative.
We use a service called Sentry to log errors and notify us when an error is being logged repeatedly. It could take hours before an error’s frequency reached notification status. Then add in the time it takes to reproduce and fix and now that bug has cost the company a sizable amount.
Our Smoke Tests
We check the page’s meta tags for correctness too. Search engine performance draws the traffic that clicks and views ads. Negative changes to our search engine position and we lose revenue too!
Our last set of tests ensure our pages load in our most hostile environment. Old browsers are hot beds for bugs and Internet Explorer 11 is the most prevalent. I won’t hold back in telling you how much I look forward to Internet Explorer’s demise. It won’t die!
Internet Explorer users make up 1.77% of our traffic, coming in at #4 behind #1 Chrome, #2 Safari, and #3 Samsung Internet. We use BrowserStack for Internet Explorer 11 testing. BrowserStack can get a pricey if you run a lot of tests so that’s why we run our complete testing suite first.