gaia-unit tests are run in a non-deterministic order due to use of os.walk()

RESOLVED FIXED

Status

Firefox OS
Gaia::TestAgent
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: emorley, Assigned: ahal)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
On this push:
https://tbpl.mozilla.org/?tree=Fx-Team&rev=410f509e9913&jobname=gaia-unit

B2G desktop gaia-unit tests were run twice (the second being a retrigger).

On the first run, the tests runs prior to ftu/test/unit/tutorial_test.js (the failing test under investigation) were:

https://tbpl.mozilla.org/php/getParsedLog.php?id=41416128&tree=Fx-Team
  ...
  communications/contacts/test/unit/contacts_merger_test.js
  communications/contacts/test/unit/navigation_test.js
  ftu/test/unit/wifi_test.js
  ftu/test/unit/tutorial_test.js

For the second run:

https://tbpl.mozilla.org/php/getParsedLog.php?id=41427395&tree=Fx-Team
  ...
  ftu/test/unit/operatorVariant_test.js
  ftu/test/unit/wifi_test.js
  ftu/test/unit/ui_manager_test.js
  ftu/test/unit/tutorial_test.js

Given that both of these jobs are on the same gaia and gecko revisions, the list of enabled tests should be the same. 

We appear to walk the filesystem here:
https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-unit-tests/gaia_unit_test/main.py#L223

I would have thought this would be deterministic - though I guess we could be mangling it later?
Flags: needinfo?(ahalberstadt)
(Reporter)

Comment 1

4 years ago
Ah, on:
https://blog.torproject.org/category/tags/deterministic-builds

> Reordering due to inode ordering differences (exposed via Python's os.walk())
> Several places in the Firefox build process use python scripts to repackage both compiled
> library archives and zip files. In particular, they tend to obtain directory listings
> using os.walk(), which is dependent upon the inode ordering of the filesystem. 
> We fix this by sorting those file lists in the applicable places.
(Reporter)

Comment 2

4 years ago
It appears that elsewhere we do sort after the os.walk():
http://mxr.mozilla.org/mozilla-central/source/js/src/tests/lib/jittests.py#196

Though will likely be worth filing a meta bug once this is fixed, to vet other uses throughout the tree.
Summary: The order in which gaia-unit tests are run appears to be non-deterministic → gaia-unit tests are run in a non-deterministic order due to use of os.walk()
(Assignee)

Comment 3

4 years ago
Created attachment 8439507 [details] [review]
Sort Gu tests after os.walk
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
Attachment #8439507 - Flags: review?(jgriffin)
Flags: needinfo?(ahalberstadt)
Comment on attachment 8439507 [details] [review]
Sort Gu tests after os.walk

ty!
Attachment #8439507 - Flags: review?(jgriffin) → review+
(Assignee)

Comment 5

4 years ago
https://github.com/mozilla-b2g/gaia/commit/6405d9ba1eedf1e28f690ae9cc511180d796e02f
(Assignee)

Updated

4 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.