Using Smoke Tests in Automated Testing

August 15, 2019  •  5 min read
  • Testing
  • Automated Testing
  • Jest

First off this article is not meant to convince you of the benefits of automated testing. You know the benefits or else you wouldn’t be reading this. Instead, let’s focus on a type of automated test called a smoke test.

image2

Many types of tests fall under the umbrella of automated testing. Your use case is unique and will not need every automated test type implemented. Our site healthline.com must be reliable, fast, and appealing. We do not use QA technicians so automated testing is important. Our delivery pipeline includes unit, visual, smoke, and integration tests.

What can be confusing is the difference between smoke, integration, and functional (end-to-end) tests. What is a smoke test? What do they entail? When should smoke tests run? What should smoke tests include?

What is a smoke test?

Smoke? Kind of a strange name, right? Imagine this. A house is on fire a few blocks away. What is the first thing you see? Smoke! You see the smoke and know something wrong is happening. You know the result is going to be disastrous if it isn’t stopped. The same goes for smoke tests. Failure of a smoke test indicates something is wrong with the build.

Wikipedia states smoke tests can be functional tests or unit tests. They test a slice of functionality of the whole system.

What to include

In researching smoke tests, I came upon a company called Functionize. They produce AI-based automated testing. Mastercard, Salesforce, and Cisco are a few of their customers, so in short, they know their stuff. They are a major proponent of smoke testing.

According to Functionize, “the best fit for smoke testing is at the Functional UI layer. Functional testing is all about testing the user interface (UI) in order to uncover new bugs (exploratory functional testing) or ensure that previously known bugs have not found their way back into the application (functional regression testing).”

They recommend you have 20-50 smoke tests. They also gave the following examples of what to include in smoke tests.

  • Navigating through foundational pages and clicking key areas.

  • Verifying correct layout and accuracy of all visual elements.

  • Exercising key functionality such as signup and login forms, additions/subtractions to shopping carts, checkout, return to shopping, and file exports.

When to run

It’s a given that longer running tests expensive to run. You lose productivity and incur more payroll expenses as your developers wait. Smoke tests run slower than your unit tests since they test functionality and not code. Run them after the unit tests for that reason. Make sure to run them before running your full test suite. With having fewer tests they run faster than your integration & remaining functional tests. Functionize states that your smoke tests should run for 5 minutes or less. We have our smoke tests running after our integration tests because of cost. I’ll explain later.

Why?

For no other purpose than to stop bugs, of course.

But why include smoke tests in your automated testing? According to Functionize, its customers found show-stopping bugs much earlier using smoke tests. Smoke tests covered 20% of their system and yet caught 80% of bugs. At Healthline, we’ve found smoke testing to be successful too. Our smoke tests have prevented several bugs from entering our production code. Consider the alternative.

We use a service called Sentry to log errors and notify us when an error is being logged repeatedly. It could take hours before an error’s frequency reached notification status. Then add in the time it takes to reproduce and fix and now that bug has cost the company a sizable amount.

Our Smoke Tests

Since healthline.com runs JavaScript we had several choices for a testing framework. We found the Jest testing framework was the right fit. We use the Jest framework for our unit, smoke, and integration testing. Our site consists of articles and newsletter sign-up pages. We also have various advertising and health tool pages. We check one of each type in our smoke tests.

Our smoke tests are tight-scoped. They answer the basic question, “Does it load?” We check whether a page returns an HTTP error such as a 404 or 500. Using Jest snapshots we ensure the page content matches previous loads. JavaScript exceptions can break an entire page. Other times an exception causes a partial load. Advertising is our bread and butter. Any decrease in ad views and clicks due to ads failing to load results in us losing revenue!

We check the page’s meta tags for correctness too. Search engine performance draws the traffic that clicks and views ads. Negative changes to our search engine position and we lose revenue too!

Our last set of tests ensure our pages load in our most hostile environment. Old browsers are hot beds for bugs and Internet Explorer 11 is the most prevalent. I won’t hold back in telling you how much I look forward to Internet Explorer’s demise. It won’t die!

Internet Explorer users make up 1.77% of our traffic, coming in at #4 behind #1 Chrome, #2 Safari, and #3 Samsung Internet. We use BrowserStack for Internet Explorer 11 testing. BrowserStack can get a pricey if you run a lot of tests so that’s why we run our complete testing suite first.

Credit

Cover Photo by Eberhard Grossgasteiger from Pexels


Copyright © Rebecca Vest and Camp Code Blog, 2022. Unauthorized use and/or duplication of this material without express and written permission from this site's author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Rebecca Vest and the Camp Code Blog with appropriate and specific direction to the original content. Built with Gatsby.