Re-run tests with Test Observability

Learn how you can trigger re-runs of your tests.

Supported scenarios for re-runs

Automation engineers, especially those who run functional end-to-end tests, know the importance of re-running test cases that fail on the first attempt. SDETs often re-run tests multiple times as part of the same CI job or re-trigger the same job with different parameters. Sometimes, the re-run is handled directly by the test framework.

Among the tests that you re-run, the common use case is to see only the latest status of the test. Test Observability solves this problem as you can not only re-run tests but also automatically map the re-runs with previous runs of the same test case and reflect only the latest status of the test.

The following scenarios are supported:

1. The framework automatically retries a failed test case.
2. You re-trigger a CI job with failed test cases.
3. The same CI job invokes the test runner again with the failed test cases.
4. You want to re-run test cases (a single test or multiple tests) during manual analysis.

Map re-run CI jobs with existing build runs

Test Observability maps the test case runs with a previous run instance if it happens to be a re-run. If the test framework itself is re-running the test, you need not do anything and you’d see the retries of a test automatically.

If you happen to be re-running tests either through approach (2) or (3) as mentioned above, you need to set the following environment variable in your runner environment just before triggering the second and subsequent runs:

Re-run specific tests

You can also trigger re-runs of specific tests while you’re manually analyzing the failures, right from within Test Observability.

You’d need to configure re-runs and the CI integration (shown in the steps below) to be able to achieve the above.

Configure re-run setting

You can go to Settings and enable/disable the Re-run configuration as shown below (it is enabled by default):

Configure CI integration for enabling job triggers

After enabling re-run from the dashboard setting, the next step is to configure a CI integration such that BrowserStack can trigger CI jobs on your behalf when you choose to re-run specific tests.

You need to visit the Integrations page and configure one of the supported CI integrations. We currently support Jenkins and Azure Pipeline.

Configure Jenkins integration for enabling job triggers

Follow these steps to configure your Jenkins integration. Every user in your organization who intends to use this functionality has to do this:

1. Go to configuring Jenkins integration
2. You’d need to add the BrowserStack plugin in your Jenkins setup, if not already done. Or, update to the latest version of the plugin if you’re using one.
3. To do that, from your Jenkins dashboard navigate to Manage Jenkins > Manage Plugins and select the Available tab. Locate the plugin by searching for “browserstack-integration” and install it. You’ll need v1.2.7 or above.
4. Add your Jenkins username and pre-generated auth token so that BrowserStack can trigger jobs on your behalf.
5. To generate an Auth Token, you’d need to go to Jenkins UI > Your name on the top right > Configure > API Token > Add new token.
6. Click Submit.

Configure Azure Pipelines integration for enabling job triggers

Follow these steps to configure your Azure Pipelines integration. Every user in your organization who intends to use this functionality has to do this:

1. Go to configuring Azure Pipelines integration
2. BrowserStack Test Observability would need your Azure DevOps Personal Access Token (PAT) that can be used to trigger builds remotely.
3. To generate a PAT, you’d need to login to Azure Org > User settings > Personal access tokens > New token > set expiry and give access to Build read & execute. You can read more about generating PAT.
4. Enter the PAT and click Submit.
   
5. Switch the toggle button Limit variables that can be set at queue time Off in Azure Pipelines settings as per the instructions here.

Trigger re-runs

After completing the CI configuration steps, follow these steps to trigger re-runs of any failed tests cases:

1. Navigate to the Test listing to see any failed test cases.
2. Hover your mouse pointer over the name of the test to see actions come up on the right-hand side (where duration shows up generally).
   
   
3. Hit the re-run icon to see the action modal show up as shown below:
   
   

Map re-runs using JUnit reports

Mapping re-runs of test cases with the previous run instances is possible via JUnit Reports as well (if you are using JUnit XML reports to ingest data into Test Observability). To achieve this, ensure you are using the same buildIdentifier across uploads that belong to the same build run.