Taskcluster's Firefox OS integration job is running incorrect Gaia UI Tests

RESOLVED WORKSFORME
(NeedInfo from)

Status

RESOLVED WORKSFORME
4 years ago
4 years ago

People

(Reporter: gmealer, Unassigned, NeedInfo)

Tracking

Details

https://github.com/mozilla-b2g/gaia/pull/28933 failed on integration test in an attempted merge to master by Autolander.

Looking at http://docs.taskcluster.net/tools/task-graph-inspector/#ASeWi2ltQx-N-JZs5-cOWg/RsbToel-TBOAK05aWCgnFQ/3 shows taskcluster running Gaia UI Tests that are either skip-if = device == "desktop" or marked external="True"

Examples are Gaia ui test 16 and Gaia ui test 19.

Former runs test_music_change_rating.py, which is marked for skipping here:

https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/music/manifest.ini

Latter runs test_findmydevice.py, which is marked external here:

https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/settings/manifest.ini

Spotchecking other Gaia ui test jobs shows similar issues.

The confusing thing is that the HTML reports on the tasks mention the device is desktop, but the skip-ifs don't seem to be honored, and neither is the --type=b2g-external we should be supplying on CLI.

If we changed how the tests are invoked, that might be responsible; unfortunately, the logs seem to only include stdout, not the commands sent, so I can't doublecheck that. The task definitions are referring to an opaque wrapper script.

If no other problem is obvious, I can eyeball the gaiatest invocation string if you copy it here, as can Dave Hunt or Johan Lorenzo (NI them as necessary).
Flags: needinfo?(jlal)
I can't understand why skip-if is not being followed for some tests. The only thing I noticed is that the mentioned test_music_change_rating test was recently disabled in bug 1146279 without any input from the B2G QA team - were we aware of this loss of coverage? I believe the practice is to at least raise a bug for investigation and ultimate restoration of the test after any intermittent failures have been resolved.

It looks like --type is not being passed on the command line, this is from one of the logs:

./bin/gaia-ui-tests \
    --gecko-log artifacts/gaia_ui_tests_gecko/ \
    --html-output artifacts/gaia_ui_tests_report.html \
    --this-chunk=${THIS_CHUNK:=1} \
    --total-chunks=${TOTAL_CHUNKS:=1} \
    tests/python/gaia-ui-tests/gaiatest/tests/$SUITE/manifest.ini

I'm not sure where this is configured, but we should be using --type=b2g-external
(In reply to Dave Hunt (:davehunt) from comment #1)
> I can't understand why skip-if is not being followed for some tests. The
> only thing I noticed is that the mentioned test_music_change_rating test was
> recently disabled in bug 1146279 without any input from the B2G QA team -
> were we aware of this loss of coverage? I believe the practice is to at
> least raise a bug for investigation and ultimate restoration of the test
> after any intermittent failures have been resolved.

Er, yeah. I agree that it's not awesome that the test was disabled without a note to the team. Heads up to James--no issue with emergency changes but you should at least NI me on stuff like that, since I'm module owner on the manifest file.

That said, we don't proactively manage the Gip coverage for the most part. IMO, "more integration coverage = better," but otherwise I don't expect full coverage out of the integration tests--especially Gip--so this wouldn't worry me much. It's still running on-device.

Not passing the --type makes sense for that part, but I still don't get the not skipping desktop bit. The HTML report and the manifest parser use the same device strings, don't they?
Dave pointed out, looking at https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/58w8b4Z5TdK8vJrt7nrWLg/0/public/logs/live_backing.log , that the branch we're testing against might be out of sync with master.

That possibly corresponds with the tests I found that should be skipped only being skip-if'ed a couple of days ago in bug 1146279. If we were out of sync, the manifest being run against might not have had those changes included.

=== LOG SNIPPETS ===

[tc-vcs] 0 run start : (cwd: /home/tester/git_checkout) git fetch https://github.com/mozilla-b2g/gaia master:refs/remotes/origin/master
From https://github.com/mozilla-b2g/gaia
   d6506bd..2ce9883  master     -> origin/master
[tc-vcs] run end : git fetch https://github.com/mozilla-b2g/gaia master:refs/remotes/origin/master (0) in 6354 ms
[tc-vcs] 0 run start : (cwd: /home/tester/git_checkout) git reset --hard
...progress...
HEAD is now at d6506bd Merge pull request #28945 from crh0716/1127718
[tc-vcs] run end : git reset --hard (0) in 31236 ms
[tc-vcs] 0 run start : (cwd: /home/tester/git_checkout) git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 206 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
[tc-vcs] run end : git checkout master (0) in 824 ms

...later...

From https://github.com/mozilla-b2g/gaia
 * branch            integration-master -> FETCH_HEAD
 * [new branch]      integration-master -> citarget/integration-master
+ git fetch citarget 33fd55f991483b44d42010329d3c05ff1b53472b
From https://github.com/mozilla-b2g/gaia
 * branch            33fd55f991483b44d42010329d3c05ff1b53472b -> FETCH_HEAD
+ git checkout -qf FETCH_HEAD

real	0m4.955s
user	0m1.138s
sys	0m0.792s
========= Finished git checkout (results: 0, elapsed: 5 secs) (at 2015-03-24 21:27:19.109555) =========
========= Started git revision... (results: 0, elapsed: 0 secs) (at 2015-03-24 21:27:19.114660) =========
[33mcommit 33fd55f991483b44d42010329d3c05ff1b53472b[m
Author: John Dorlus <jdorlus@mozilla.com>
Date:   Tue Mar 17 17:16:36 2015 -0400

    Bug 1139083 - Implement Brick Verifcation Test in Python
========= Finished git revision... (results: 0, elapsed: 0 secs) (at 2015-03-24 21:27:19.122684) =========
I think I got hit by this in bug 1146877, comment 17. The fix I provided shouldn't have anything to do with the failure mentioned there in taskcluster.
This seems to affect me in bug 1145955, too.
Is this still an issue?
Flags: needinfo?(gmealer)
No, I think it's resolved now. Closing the bug out.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Flags: needinfo?(gmealer)
Resolution: --- → WORKSFORME
Component: TaskCluster → General
Product: Testing → Taskcluster
Target Milestone: --- → mozilla41
Version: unspecified → Trunk
Resetting Version and Target Milestone that accidentally got changed...
Target Milestone: mozilla41 → ---
Version: Trunk → unspecified
You need to log in before you can comment on or make changes to this bug.