1023198 - gaia-unit tests are run in a non-deterministic order due to use of os.walk()

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::TestAgent, defect)

Description

[Ed Morley [:emorley]]

On this push:
https://tbpl.mozilla.org/?tree=Fx-Team&rev=410f509e9913&jobname=gaia-unit

B2G desktop gaia-unit tests were run twice (the second being a retrigger).

On the first run, the tests runs prior to ftu/test/unit/tutorial_test.js (the failing test under investigation) were:

https://tbpl.mozilla.org/php/getParsedLog.php?id=41416128&tree=Fx-Team
  ...
  communications/contacts/test/unit/contacts_merger_test.js
  communications/contacts/test/unit/navigation_test.js
  ftu/test/unit/wifi_test.js
  ftu/test/unit/tutorial_test.js

For the second run:

https://tbpl.mozilla.org/php/getParsedLog.php?id=41427395&tree=Fx-Team
  ...
  ftu/test/unit/operatorVariant_test.js
  ftu/test/unit/wifi_test.js
  ftu/test/unit/ui_manager_test.js
  ftu/test/unit/tutorial_test.js

Given that both of these jobs are on the same gaia and gecko revisions, the list of enabled tests should be the same. 

We appear to walk the filesystem here:
https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-unit-tests/gaia_unit_test/main.py#L223

I would have thought this would be deterministic - though I guess we could be mangling it later?

Comment 1

[Ed Morley [:emorley]]

Ah, on:
https://blog.torproject.org/category/tags/deterministic-builds

> Reordering due to inode ordering differences (exposed via Python's os.walk())
> Several places in the Firefox build process use python scripts to repackage both compiled
> library archives and zip files. In particular, they tend to obtain directory listings
> using os.walk(), which is dependent upon the inode ordering of the filesystem. 
> We fix this by sorting those file lists in the applicable places.

Comment 2

[Ed Morley [:emorley]]

It appears that elsewhere we do sort after the os.walk():
http://mxr.mozilla.org/mozilla-central/source/js/src/tests/lib/jittests.py#196

Though will likely be worth filing a meta bug once this is fixed, to vet other uses throughout the tree.

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Sort Gu tests after os.walk

Comment 4

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8439507 [details] [review]
Sort Gu tests after os.walk

ty!

Comment 5

[Andrew Halberstadt [:ahal]]

https://github.com/mozilla-b2g/gaia/commit/6405d9ba1eedf1e28f690ae9cc511180d796e02f