Inexplicably failing tests when run in a full suite: what can I do?
I currently have 40 tests with approximately 200 assertions running. When all of them are run, the same two are almost always failing. Run individually, or with about half of all the available tests, they pass. The total running time is typically 8-10 seconds.
If I run *all* the tests with the developer console open (set up so that the browser cache is not used), the same two tests *always* fail - the total running time is closer to 30 seconds).
Almost all of these tests involve the following steps:
1. (re)set some default values
2. define the following:
var
done
=
assert
.
async
();
3. load some files using ajax calls, with async set to false; I'm running these tests locally, using Python http.server.
4. executing some code using eval() on the content of one of these files. (Most often, it is Brython's exec() which is used; Brython translates Python code in the browser into Javascript and executes it. As such, it is slower than simply using eval() on some pure javascript code.)
5. check some assertions
6. end the test with a call to done().
By inspecting closely the result, I found that, when a test has failed, the code run by exec/eval appeared to have been cut short - as though the state was corrupted by another test running (but this may be a red herring). [For example: a successful run would create an array containing 360 objects but a failed one, with the same test, would create an array containing less than 300 objects.]
Questions:
1. am I using async()/done() correctly?
2. Any suggestion as to what could be done to fix this?
For those interested, the main failing test can currently (July 6, 2017) be found at
https://github.com/aroberge/reeborg/blob/master/tests/integration_tests/js/all_qunit_tests.js#L366
In spite of the file name (all_qunit_tests.js), there are a few other tests that are included in the suite. Also, I do plan to split up the test files in the not too distant future so that the above link will likely be wrong.
It points to a test that begins with
QUnit
.
test
(
"Storm 3; also tests library"