Mute problematic test cases

Learn how you can mute problematic test cases and keep CI green.

What is the use case for muting a test case?

There are cases when you know that a test case is stale or it is flaky and it cannot be relied upon to tell whether the underlying scenario under test is passing or failing. Such test cases need to be fixed but they in the meantime result in failed build status.

In all such scenarios where you know that a test case cannot be relied upon, you can “Mute” the test case to exclude it from the overall build run execution results and all metric calculations that Test Observability provides.

If you’re a QA leader and you’d want to set thresholds for the number of days a test case can remain muted, then you can set alerts to that effect within the Test Observability tool.

How can you mute or unmute a test case?

1. Navigate to the Test listing page within any build run.
2. Hover your mouse pointer over the name of the test to see actions come up on the right-hand side (where duration shows up generally).
   
   
3. Click the “mute” icon and confirm the action on the next screen.
   
   
4. You can similarly “unmute” a test case but for that, you’ll first have to apply the “Muted” filter from the all filters dropdown and only then you’d see all muted tests from where you can un-mute any test case you want.

What does muting a test case do?

• Muting a test case removes the test from all views on that build run and all subsequent build runs.
• To find muted tests, you’d need to apply the “Muted” filter on any test listing as shown below:
  
  
• The aggregate number of tests that ran would be reduced and the pass/fail statuses will also not reflect the muted test.
• All stability-related calculations would omit the muted tests.
• Build status might change if you end up muting all failed tests in a build.

Why should you use it?

Muting a test case has several advantages:

1. If you mute all problematic test cases, you know that all test failures have to be investigated. It helps to focus better.
2. It is better than removing test cases, which some teams end up doing. You will still have the data and you can go and fix it and unmute it later.
3. In the world of functional testing with so many causes of flakiness, muting test cases help keep a green CI which is almost impossible to achieve otherwise.