1227219 - Gij Test Suite may not include one test file, but include another one into several chunks

Status: RESOLVED FIXED

(Testing Graveyard :: JSMarionette, defect)

Description

[Oleg Zasypkin [:azasypkin]]

See Gij 15 and Gij 16 at [1].

Gij 15:
------- Will run the following test files:
/home/worker/git_checkout/apps/settings/test/marionette/tests/ime_uninstallation_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/improve_settings_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/keyboard_settings_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/language_settings_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/media_storage_settings_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/notifications_settings_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/root_settings_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/screen_lock_settings_test.js <---- Will also be in Gij 16

Gij 16 (first run):
------- Will run the following test files:
/home/worker/git_checkout/apps/settings/test/marionette/tests/screen_lock_settings_test.js <---- Hey, I has been run in Gij 15 already :/
/home/worker/git_checkout/apps/settings/test/marionette/tests/sound_settings_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/support_settings_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/usb_storage_settings_test.js
/home/worker/git_checkout/apps/sms/test/marionette/attachments_test.js
/home/worker/git_checkout/apps/sms/test/marionette/composer_test.js
/home/worker/git_checkout/apps/sms/test/marionette/conversation_test.js
/home/worker/git_checkout/apps/sms/test/marionette/conversion_banner_test.js
/home/worker/git_checkout/apps/sms/test/marionette/drafts_test.js

Gij 16 (second run):
------- Will run the following test files:
/home/worker/git_checkout/apps/settings/test/marionette/tests/sound_settings_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/support_settings_test.js
/home/worker/git_checkout/apps/settings/test/marionette/tests/usb_storage_settings_test.js
/home/worker/git_checkout/apps/sms/test/marionette/attachments_test.js
/home/worker/git_checkout/apps/sms/test/marionette/composer_test.js
/home/worker/git_checkout/apps/sms/test/marionette/conversation_test.js
/home/worker/git_checkout/apps/sms/test/marionette/conversion_banner_test.js
/home/worker/git_checkout/apps/sms/test/marionette/drafts_test.js
/home/worker/git_checkout/apps/sms/test/marionette/inbox_test.js <------ I was missed in Gij 16 first run (and in any other chunk as well) !!!
/home/worker/git_checkout/apps/sms/test/marionette/message_received_test.js <---- me too :/


[1] https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=a0167b8f7f362ad8d0dde0600411f6317d4d554c

Comment 1

[GitHub Autolander Bot]

Attachment: [gaia] julienw:1227219-force-locale > mozilla-b2g:master

Comment 2

[Julien Wajsberg [:julienw]]

Comment on attachment 8690920 [details] [review]
[gaia] julienw:1227219-force-locale > mozilla-b2g:master

I checked how sorting is done for unit tests: it's done with JavaScript's sort which does not depend on the locale. (I hope this is still true in node !)

You can see [1] before the patch and [2] with the patch. In [2] Gij16 is _almost_ always concerns the same set of files, but in [1] it can be different.

[1] https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=a0167b8f7f362ad8d0dde0600411f6317d4d554c
[2] https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=ab60032d54683b2a8fd7cc4320ad4037f68ef4b9

With the patch I still had a case ([3]) where it was different though. I want to dig a little deeper then...

[3] https://public-artifacts.taskcluster.net/Wj4Q8mieTnSGf8IApiuFTQ/0/public/logs/live_backing.log

Comment 3

[Julien Wajsberg [:julienw]]

Comment on attachment 8690920 [details] [review]
[gaia] julienw:1227219-force-locale > mozilla-b2g:master

Hey Aus,

I think this is now better :)

The additional issue was that we were appending 2 sets of files using a space instead of a new line, and so the sort was not sorting anymore for these files that were on the same line...

Tell me what you think !

Comment 4

[Aus Lacroix [:aus]]

Comment on attachment 8690920 [details] [review]
[gaia] julienw:1227219-force-locale > mozilla-b2g:master

Nice! Love the simplification as well. Thanks!

Comment 5

[Julien Wajsberg [:julienw]]

master: https://github.com/mozilla-b2g/gaia/commit/fbe3db3dea80c5f2d2f8d2a9e4d60fbbb94f01ed

Comment 6

[Julien Wajsberg [:julienw]]

Can we uplift this to v2.5 with a=npotb ?

Comment 7

[Carsten Book [:Tomcat]]

(In reply to Julien Wajsberg [:julienw] from comment #6)
> Can we uplift this to v2.5 with a=npotb ?

yeah landed this on 2.5 as https://github.com/mozilla-b2g/gaia/commit/34ccc2c8f17b87a1fab95a4186b0019ec78c7f75