Quarantine tests

Quarantine tests

Some CI tools (eg. Atlassian Bamboo) have the ability to quarantine tests that are failing for either expected reasons (or fail intermittently).  Jenkins also has an open feature request for a similar thing.


Is this something that has (or would be) considered for QUnit?

Currently if a test is failing, your options are either:

  1. Fix the code (or the test)
  2. Comment out (or permanently remove) the failing test

Sometimes a test may be failing for a known reason, and a fix is either not ready or depends on something else;  so it may not always be possible to fix immediately.  Having a test suite consistently failing because of one (potentially non-critical) failing test can increase 'noise' so that devs start to ignore the build results.

By the same token, commenting out a failing test is a risk; because by just looking at the test results it not clear that some tests may have been skipped.   It is easy to forget to uncomment the test at a later time.

I would like to propose the idea of being able to mark one or more tests as quarantined.  The tests would still run; but their failure would be treated more like a warning than an error.

On the results page, where the number of tests & assertions passed/failed is displayed along with the duration; the number of quarantined tests could be displayed (perhaps in red, to make it stand out).

The suite itself would still be considered 'green' if all non-quarantined tests pass; but it would be evident that some tests were skipped.

I'm not sure what would be the best way to mark a test as quarantined....perhaps wrapping it in a quarantine() call? eg.
  1. // non-quarantined test
  2. test("testSomething", function() { /* test code here */ });

  4. // quarantined test
  5. quarantine(test("testSomething, function() { /* test code here */ }));