Case study: Improving the Performance of Automated Acceptance Testing with Selenium
Name
Stenver Jerkku
Abstract
In the current world, where everything is on the web, it is nearly a requirement for a company to manage its website. Building a complex website has its costs though – it is difficult and expensive to maintain and develop. Developers often use browser based acceptance tests to assure that the page behaves as dictated by the requirements. The problem with browser based acceptance tests is that they are slow. It is often difficult to use them in continuous integration since the running time of the test suite is so long that the developer moves on to the next task. This increases deployment cycle times, reduces bug-catching rate, makes it hard to trace a bug back to the source commit, and frustrates developers in general.
SaleMove Inc has high standards of quality and follows the TDD principles. In addition to doing TDD, SaleMove has an extensive Selenium-based acceptance test environment, running over 220 tests in 9 different browsers in parallel. Selenium is an automated tool for creating functional tests for web applications [1]. We conducted a case study to analyse and improve SaleMove’s acceptance test environment. The goal of the study was to reduce the total acceptance test time of about 50 minutes to around 10 minutes. This allowed us to shorten release cycles, make bug fixing faster and developers happier.
During the study, we analysed the bottlenecks of the SaleMove’s acceptance test environment. First, we analysed the test suite and test case startup and the teardown times. Then we identified the bottlenecks of the acceptance test environment and made improvements. We analysed the effect of running Selenium in full cluster mode, i.e., each test running in a clustered hub in parallel with other tests. In this thesis, we describe in detail how we implemented the clustered mode, and how we aggregated test results into a readable format. Finally, we document the gains and costs of this clustered setup and suggest future improvements.
In summary, we managed to shorten SaleMove’s acceptance test time to around 13 minutes in full cluster mode. This reduces deployment cycle times, increases developer satisfaction and has other benefits. This comes at the cost of higher complexity of the acceptance test environment and of additional machines that are needed for the clustered mode.
SaleMove Inc has high standards of quality and follows the TDD principles. In addition to doing TDD, SaleMove has an extensive Selenium-based acceptance test environment, running over 220 tests in 9 different browsers in parallel. Selenium is an automated tool for creating functional tests for web applications [1]. We conducted a case study to analyse and improve SaleMove’s acceptance test environment. The goal of the study was to reduce the total acceptance test time of about 50 minutes to around 10 minutes. This allowed us to shorten release cycles, make bug fixing faster and developers happier.
During the study, we analysed the bottlenecks of the SaleMove’s acceptance test environment. First, we analysed the test suite and test case startup and the teardown times. Then we identified the bottlenecks of the acceptance test environment and made improvements. We analysed the effect of running Selenium in full cluster mode, i.e., each test running in a clustered hub in parallel with other tests. In this thesis, we describe in detail how we implemented the clustered mode, and how we aggregated test results into a readable format. Finally, we document the gains and costs of this clustered setup and suggest future improvements.
In summary, we managed to shorten SaleMove’s acceptance test time to around 13 minutes in full cluster mode. This reduces deployment cycle times, increases developer satisfaction and has other benefits. This comes at the cost of higher complexity of the acceptance test environment and of additional machines that are needed for the clustered mode.
Graduation Thesis language
English
Graduation Thesis type
Master - Software Engineering
Supervisor(s)
Dietmar Pfahl; Carlos Paniagua
Defence year
2016