Automated Tests, Test-Driven Development and QA Automation
Topic: Tech
March 14, 2024
What Is Automated Testing?
Software testing has been an important part of development since the early years of end-user software. The market demanded the quality of paid applications to be high. High quality had a close connection with success of the business.
The software engineering world responded with an automated testing concept that facilitated work of engineers and even boosted possibilities of development through a new approach.
The new approach implied that tests were written at the first step of a feature engineering. An automated test would appear as a vision of the future - then the developers would need to make the future happen.
The process of defining the desired behavior through automated tests and then making it a reality got a name of test-driven development - also known as TDD.
I started practicing TDD in 2009 and have been practicing it in all projects I participated in. Over the years I could see how TDD would take the quality of software to the next level and even allow the businesses to grow more.
How Does Automated Testing Work in a Pipeline?
Modern software development has the form of a special process run by several groups of people. In Scrum terminology, there are:
- Stakeholders who own the business;
- Product owners who form the requirements;
- Developers who implement the requirements;
- Scrum masters who play the role of coaches and control the process.
These groups of people exchange information in a very particular way. The most valuable exchanges happen between product owners and developers.
The product owners provide the requirements to the developers. The developers provide new features and software updates in return.
Since it is important that the new features and updates match the company quality standards, the development camp should have a procedure for quality assurance.
In this way, quality assurance - both manual and automated - gets integrated in the common project development pipeline. A typical pipeline is:
- Product owners write user stories with acceptance criteria (using Gherkin or plain English).
- Developers (or QA engineers) write tests.
- Developers write code and build features.
- Developers push their updates to a continuous integration system (CI) that runs the test suite and verifies that all tests pass.
- The app is uploaded to a server where the added stories can be checked by manual QA and product owners.
As one can see, the development process is enhanced with automated and manual testing. What kind of tests can be written and what do they look like?
What Are the Types of Automated Testing?
It was already said that tests can be manual and automated. Automated tests, in their turn, can be:
- Unit tests - small code snippets that check the functionality on the level of classes and methods (or functions in procedural or functional programming).
- API requests tests - functions that send HTTP requests and parse the responses.
- Acceptance tests that test user-visible behavior of the system.
Automated testing simplifies but does not eliminate manual testing (manual quality assurance). Manual QA has the advantage of having a human eye that can give an opinion beyond automated procedures - like an overall impression of the usability, visual appearance, etc.
Writing useful unit tests requires experience. Writing useful acceptance tests requires tools. What tools are suitable for acceptance testing?
What Tools Are Used in Quality Assurance?
The mainstream frameworks and libraries for acceptance testing in a browser are:
- Selenium - https://www.selenium.dev/
- Cypress - https://www.cypress.io/
- Playwright - https://playwright.dev/
- Puppeteer - https://pptr.dev/
- HtmlUnit - https://htmlunit.sourceforge.io/
Selenium has been a long-time classic solution used in combination with various frameworks such as Ruby on Rails, Django, and Laravel.
Another classic solution for Java and Scala-based applications is HtmlUnit that is simpler yet much faster than Selenium.
An alternative solution is Cypress that is ideal for use in connection with Node.js and Next.js.
There are also other solutions in the modern market such as Playwright and Puppeteer that snap at Selenium’s heels.
For mobile testing, there are XCUItest on iOS, Espresso on Android and universal solutions like Appium.
Real-Life Examples of Acceptance Testing
Here are several examples of advanced acceptance testing on various platforms. All examples below are taken from real projects developed by me and other developers from our company.
Sports Backoffice and Apps
A good example of a project where we have all types of tests is a web application with a backend for sports facilities.
The backend has over 1000 unit/request tests and more than 600 acceptance tests including cases for online payments, direct messages, etc.
The tests are pushed to Github Actions.
The test coverage is 99.88%.
Acceptance tests written in Ruby on Rails, Capybara (https://github.com/teamcapybara/capybara), and Selenium look as follows:
There are even tests to verify the number of SQL queries performed by an API call:
An example of acceptance tests for the iOS client (Swift and XCUITest):
Accounting Software
Journal is an accounting system written in Scala.
The unit test suit has more than 1600 tests and takes 11 seconds to run.
The acceptance test suite has 614 tests and takes 1 minute 35 seconds to run. That extraordinary speed is possible because of the nature of the project - which is merely a management tool with quite a simple interface - as well as the HtmlUnit browser.
Compared to a standard full-featured browser (Chrome, Firefox) in headless mode, HtmlUnit runs much faster. So the full test suite (unit plus acceptance tests) runs in less than two minutes.
An acceptance test may look like this:
Time Tracking Software
Another example is our company’s product - a project management and time tracking tool called Tracker. Its unit test suite contains about 2000 tests and the acceptance tests suite - 6500 scenarios that take around 26 minutes to run.
Below are two examples of acceptance tests written for the Tracker. The first one is of interest because it uses Gherkin syntax to define the expected behavior.
In the background, it is converted into a running code. The idea of using this syntax is to allow product owners to write acceptance tests using predefined steps or define their own steps.
Another interesting example uses the traditional syntax. The interesting part of this example is the execute_with_dump function that comes from a Ruby on Rails gem that has been developed in our company and is available at https://github.com/alvir/dump_hook (written and maintained by our senior engineer Alexander Ryazantsev).
The purpose of that gem is to support the clean acceptance testing concept which assumes that all data in the database should come from the user input - even if that is the user input in the admin area.
However, since adding data in the admin area in every automated scenario is time-consuming, common scenarios are recorded, dumped into a local storage, and reused in multiple automated scenarios.
Content Conversion Platform
Another example is a content conversion platform that allows one to store images, run them through OCR and AI, get various translations of it, and deliver those translations to end users.
The project is developed in Next.js and Cypress for acceptance testing. The scenarios look as follows:
Other Examples of Acceptance Tests
Selenium can be used in combination with almost any mainstream language and framework - and its tests would look clean and natural.
This is how it looks in PHP/Laravel:
Another example is its usage in Python/Django:
This way, acceptance tests can be written with the help of various tools and can improve any web or mobile project.
What You Can Do to Improve Quality of Your Software
If you do not have automated tests in your project, start writing them right away. Automated tests can boost up the quality of your software significantly.
If you have unit tests, but do not have acceptance tests - start writing acceptance tests. If you have tests for the backend but do not have them for the mobile applications - start writing tests for the mobile applications.
Start writing tests! If you write tests, maybe it is possible to write them better.
Automated tests imply a specific development procedure. If you do not have an established development procedure, it is strongly suggested that you have one. You can follow Scrum or a simpler agile process, but the development process can impose order on the features being developed, which will instantly create a better impression of end users.
You may also want to run automated tests in a continuous integration environment and prevent updates without test coverage from being merged into the main code branch.
It is worth mentioning that automated testing does not fully eliminate manual testing. Remember that manual testing has the advantage of a real human reviewing the changes and sharing an opinion about the way everything looks and feels.
Additionally, automated testing is as complex as software engineering. It forms a code base that has all features as every code has: engineers need to maintain the code, scale it, and care about its readability and speed. The code should be decomposed into multiple modules, contexts, etc.
The common pipeline in large projects assumes that an automated test suite is accountable for regression testing while manual quality checks are more suitable for the new features.
If you are ready to deal with this challenge, the results will certainly be visible to your end users.