Selenium tests in parallel

Selenium is an automated testing framework to test applications via a web browser (see https://www.selenium.dev/). Other forms of automated testing like unit testing only test one part of an application whereas Selenium tests the whole application – frontend and backend combined. This is a great feature because it proves that the application works in the same way that user access it.

With automated tests running as part of a continuous integration environment, stakeholders can be more confident that the application has a high quality with few bugs. More and more automated tests mean fewer manual tests are needed before deploying each release to production, which therefore speeds up the application’s time to production.

The problem: Selenium tests often become too slow

At first there are only a few tests.

The picture below shows one block for each test. So here we see three tests running after each other. This works fast because there are only a few tests.

Unfortunately, Selenium tests can be slow and as more and more tests are created the test suite takes a long time to run.

This next picture shows many tests running after each other and so taking too much time:

Each time a code change is pushed, the developer must wait for the continuous integration environment to run, including the Selenium tests. That can be a long and inefficient wait. So how can we speed up the Selenium test suite?

The solution: run them in parallel

A good way to speed up the Selenium tests is to run then in parallel.

This final picture shows the tests running 4 times in parallel. Now the tests finish nice and fast:

Let’s see how to implement that:

• Selenium tests written in Java can be run using the maven-surefire-plugin. It has a handy option to run in parallel, see https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html.
  Create a Maven property for the number of times parallel and set the attribute of the maven-failsafe-plugin to this property. Now, in the command line, this property can be overridden with another value so that the value can be easily tuned. For other languages than Java use an equivalent test runner.
• Start a hub Selenium server (see https://www.selenium.dev/documentation/legacy/selenium_3/ with command: java -jar selenium-server-standalone-x.jar -role hub -port 4444
• Start a node Selenium server with command: java -jar selenium-server-standalone-x.jar -role hub -port 4444 and java -jar selenium-server-standalone-x.jar -role node -port 4445 -hubhttp://localhost:4444/grid/register -browser browserName=chrome,maxInstances=10 -maxSession 10 -Dwebdriver.chrome.driver=./chromedriver. Chrome is used here but, of course, other browsers can be used instead.

Hints and tips:

• Each test class runs on its own JVM separate for the other test classes. Therefore, there is no danger of (static) variables begin shared between parallel runs.
• One test class might take much longer to run than the others. Watch out for this and make the test classes smaller and more numerous to allow them to run better in parallel.
• Sometimes tests are dependent on each other. For example, one test asserts that a field has a certain value and another test updates that field. One way to handle this is to put tests that depend on each other in the same test class. Then you can be sure that they will run after each other and never in parallel.
• How many times parallel? That depends on your hardware. Experiment with different values for the command line property. When the value is too high, 100% of CPU and memory will be used and the tests will time out and fail. When the value is too low the Selenium tests will take too long. Find the ideal value in between.
• This heavier use of CPU and memory means that tests will sometimes run slower and so have timing issues. The tests must be made more resilient. Waiting for some fixed number of seconds is not advised. Instead wait until a certain element is present. For example, when navigating to a page, check for the presence of an element like a button and check for this in a loop until the element is present. If using the Page object model (https://www.selenium.dev/documentation/guidelines/page_object_models/) it can be useful to create an abstract method like waitForPageLoaded() in the superclass of the pages. This forces developers to implement this method for each page and so think about which element indicates that the page is loaded and so make the tests more resilient.

Summary

Selenium tests are a great way to automatically test your application. Unfortunately, they can become too slow. Speed them up by running them in parallel. Problem solved.