Closed Bug 1298233 Opened 8 years ago Closed 5 years ago

Intermittent test_appinfo.py TestAppInfo.test_valid_properties | AssertionError: u'20160825215758' != '20160825223519'

Categories

(Testing :: Firefox UI Tests, defect, P2)

Version 3
defect

Tracking

(firefox50 disabled, firefox51 disabled, firefox59 disabled, firefox60 disabled, firefox61 disabled, firefox63 disabled, firefox64 disabled, firefox65 fixed)

RESOLVED FIXED
mozilla65
Tracking Status
firefox50 --- disabled
firefox51 --- disabled
firefox59 --- disabled
firefox60 --- disabled
firefox61 --- disabled
firefox63 --- disabled
firefox64 --- disabled
firefox65 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: whimboo)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files)

Here the exact failure:

23:23:05     INFO - TEST-UNEXPECTED-FAIL | test_appinfo.py TestAppInfo.test_valid_properties | AssertionError: u'20160825215758' != '20160825223519'
23:23:05     INFO - Traceback (most recent call last):
23:23:05     INFO -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/marionette/marionette_test.py", line 344, in run
23:23:05     INFO -     testMethod()
23:23:05     INFO -   File "/home/worker/workspace/build/tests/firefox-ui/tests/puppeteer/test_appinfo.py", line 19, in test_valid_properties
23:23:05     INFO -     self.assertEqual(self.appinfo.platformBuildID, version_info['platform_buildid'])

Failure happens at this line:

https://dxr.mozilla.org/mozilla-central/rev/7963ebdd52b93f96b812eff2eab8d94097147b9c/testing/firefox-ui/tests/puppeteer/test_appinfo.py#19

That means the buildid we retrieve via mozinfo is not equal to the one we get via Services.appinfo. So one of those two components is wrongly reporting then. This would also explain why we have update test failures like bug 1285340.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
The TC task of the  Linux64 Opt build is:
https://tools.taskcluster.net/task-inspector/#UQoBONAuT-uHXGqytBhX_Q/0

The build id can be found in:
https://public-artifacts.taskcluster.net/UQoBONAuT-uHXGqytBhX_Q/0/public/build/target.txt

Which is:
> 20160825223519

So it's Services.appinfo which is wrong!
For testing I downloaded:
https://queue.taskcluster.net/v1/task/UQoBONAuT-uHXGqytBhX_Q/runs/0/artifacts/public%2Fbuild%2Ftarget.tar.bz2

Here the script which I have executed in Scratchpad:

> Components.utils.import("resource://gre/modules/Services.jsm")
> alert(Services.appinfo['appBuildID'] + ' ' + Services.appinfo['platformBuildID'])

The build ids I get displayed are: "20160825223519 20160825215758"

application.ini:

> [App]
> BuildID=20160825223519

platform.ini:

> [Build]
> BuildID=20160825223519

So we just found a bug in appinfo. I will file a new bug for Firefox.
Depends on: 1298328
Whiteboard: [blocked by bug 1298328]
Until we have a solution for bug 1298328 we should skip the specific assertion check for the platform buildid.
Attachment #8786945 - Flags: review?(mjzffr)
Keywords: leave-open
Comment on attachment 8786945 [details]
Bug 1298233 - Skip test for platform buildid in test_appinfo.py

https://reviewboard.mozilla.org/r/75794/#review73812
Attachment #8786945 - Flags: review?(mjzffr) → review+
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6795225dc65f
Skip test for platform buildid in test_appinfo.py r=maja_zf
Marking status as disabled so we can remember for which version we did it.
Assignee: hskupin → nobody
Status: ASSIGNED → NEW
Looks like this test failure also started to happen on Beta now. We should get it disabled on that branch too until we have our debug builds fixed for this issue.
Whiteboard: [blocked by bug 1298328] → [blocked by bug 1298328][checkin-needed-beta]
https://hg.mozilla.org/releases/mozilla-beta/rev/64bffdab3d1e
Whiteboard: [blocked by bug 1298328][checkin-needed-beta] → [blocked by bug 1298328]
We should be able to unskip this test given that bug 1298328 has been fixed now.
(In reply to Henrik Skupin (:whimboo) from comment #18)
> Try push is here:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=f1dab9cc5a447d9d51b813498ecc813c0bfa9da6

Tests are still failing due to the same reason. Maybe it's because this try build produced artifact builds?
Flags: needinfo?(cmanchester)
Yes, artifact builds will pretty much always have this mismatch, because the build job that produces the artifact build will set a build id that's different than the build id that was compiled into the originating build.
Flags: needinfo?(cmanchester)
Oh, I see. Is there a way to detect if the build is an artifact build or a full build when running the test? I would have to only enable the tests for a full build. If we don't have a chance to differentiate we might want to kill those tests for platform version and buildid.
Flags: needinfo?(cmanchester)
Yes I believe mozinfo.json has an 'artifact' key.
Flags: needinfo?(cmanchester)
The leave-open keyword is there and there is no activity for 6 months.
:whimboo, maybe it's time to close this bug?
Flags: needinfo?(hskupin)
Let me just fix it by skipping the test for artifact builds. The flag is available in mozinfo.json as Chris mentioned a while ago.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Flags: needinfo?(hskupin)
Priority: -- → P2
Whiteboard: [blocked by bug 1298328]
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f2de4c978901
[firefox-ui-tests] Skip TestAppInfo.test_valid_properties for artifact builds. r=ato
Keywords: leave-open
https://hg.mozilla.org/mozilla-central/rev/f2de4c978901
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
You need to log in before you can comment on or make changes to this bug.