1623134 - Update android host utils

Status: RESOLVED FIXED

(Testing :: General, task, P2)

Description

[Geoff Brown [:gbrown]]

+++ This bug was initially created as a clone of Bug #1585422 +++

Please update host-utils again.

See the discussion in bug 1622701: We want to make sure that this archive contains the latest fix_stacks.py, from the bin directory of the Linux common.tests.

Comment 1

[Andrew Erickson [:aerickson]]

Attachment: Bug 1623134: update mac hostutils

Comment 2

[Andrew Erickson [:aerickson]]

Attachment: Bug 1623134: update linux hostutils

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b96f40ed2773
update linux hostutils r=gbrown

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c55b247ef4df
update mac hostutils r=gbrown

Comment 5

[Narcis Beleuzu [:NarcisB]]

Backed out for mochitest failures on test_fetch_cors_sw_reroute.html

Backout link: https://hg.mozilla.org/integration/autoland/rev/cf75cef9d1f669e03b918451e2ec8e8e6d8e7718
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=293976170&repo=autoland&lineNumber=5486

Comment 6

[Nicholas Nethercote [inactive]]

Aw nuts, this backout means that D66918 in bug 1622701 will need to be backed out too.

Comment 7

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/c55b247ef4df

Comment 8

[Andrew Erickson [:aerickson]]

Something has changed in hostutils that's caused the test to fail. We'll need to land a test fix with this change.

Comment 9

[Nicholas Nethercote [inactive]]

aerickson: Do you know what the required test fix will be? Do you have an idea how long it might take? Thanks.

Comment 10

[Geoff Brown [:gbrown]]

test_fetch_cors_sw_reroute.html uses .sjs; we have had trouble in the past with host-utils updates and .sjs tests, as in bug 1514075 -- not sure if that's helpful.

Comment 11

[Geoff Brown [:gbrown]]

:snorp - Any idea why test_fetch_cors_sw_reroute.html might fail on geckoview when we update host-utils? Alternately, any objection to skipping the test on geckoview? (It is currently skipped on Linux and Windows 10.)

Comment 12

[Geoff Brown [:gbrown]]

:evilpie - I notice you made some changes to fetch tests in bug 1605854 and had a little trouble with test_fetch_cors_sw_reroute.html; do you have any idea what might be going wrong here? (That test fails on Android/geckoview when we update host-utils, which updates xpcshell and the test server support files for Android. See logs in comment 5.)

Comment 13

[Tom S [:evilpie]]

I really just did enough to keep the test running without toSource/uneva;, you should find someone who actually understands ServiceWorkers.

Comment 14

[Nicholas Nethercote [inactive]]

To prevent this from dragging out, I suggest we simply disable that test on Android.

Comment 15

[Andrew Erickson [:aerickson]]

:gbrown and :snorp, thoughts on disabling? I did a quick look at the tests and I'm not sure how to disable them.

Comment 16

[James Willcox (:snorp) ([e-mail]) (he/him)]

Yeah, I guess disabling is fine with me.

Comment 17

[James Willcox (:snorp) ([e-mail]) (he/him)]

Attachment: Bug 1623134 - Disable test_fetch_cors_sw_reroute.html on Android

This broke when hostutils was updated and we don't know why.

Comment 18

[Kartikaya Gupta (email:[e-mail])]

Right now running ./mach reftest <some-reftest> with an Android (emulator) build fails with a Python error about not finding the fix_stacks module. Does anybody know if this bug is supposed to address that, or should I file something new?

Comment 19

[Kartikaya Gupta (email:[e-mail])]

I filed bug 1626945 for it anyway, since it doesn't seem like the patches on this bug would fix what I'm seeing.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9cf0341dd964
Disable test_fetch_cors_sw_reroute.html on Android r=asuth

Comment 21

[Andrew Erickson [:aerickson]]

I'm rerunning the tests on my change now that :snorp's change to disable the tests has landed.

Comment 22

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/9cf0341dd964

Comment 23

[Geoff Brown [:gbrown]]

Still need to actually update host utils of course.

Comment 24

[Nicholas Nethercote [inactive]]

gbrown: how can I see what is within the host-utils tarball?

I ask because in bug 1626272 I had to make a change to fix_stacks.py, which necessitated a corresponding change in utils.py. I'm hoping that utils.py is also in the host-utils tarball, otherwise we'll end up using an old version of fix_stacks.py with a new version of utils.py and they won't be happy together. (Using the old version for both will be fine, because the change win bug 1626272 was to fix a problem on Windows.)

Comment 25

[Andrew Erickson [:aerickson]]

:njn, you can use tooltool to download the package and expand the archive.

arc patch D67236
python testing/mozharness/external_tools/tooltool.py fetch -m testing/config/tooltool-manifests/linux64/hostutils.manifest

I built the linux hostutils from this build/commit: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=5100681103a8c27f67a1acc7f657422b6b79e23b

Comment 26

[Nicholas Nethercote [inactive]]

Thanks for the info, aerickson.

Alas, utils.py is not in that tarball, so AFAICT I am faced with the prospect of an up-to-date, in-tree utils.py running in tandem with an old, host-util'd fix_stacks.py. That's going to be a big problem for bug 1626272, because it's difficult to fix that bug without being able to change fix_stacks.py and utils.py in tandem, sigh.

Comment 27

[Nicholas Nethercote [inactive]]

Hmm, I see a path forward. The fix for bug 1626272 can land, because it only affects opt builds, while the change in bug 1622701 only affects debug builds. And Android testing only runs on debug builds on CI. So, we can:

• Land bug 1626272, which changes fix_stacks.py and utils.py, but only for opt builds, which means it doesn't affect Android tests on CI.
• Update host-utils again, to incorporate the changes to fix_stacks.py in bug 1626272.
• Land bug 1622701, which affects debug builds, and thus Android tests on try.

Goodness. If Android tests ran on opt builds as well as debug builds on CI, bug 1626272 would be impossible to fix without temporarily having some kind of fallback code to handle things if an old fix_stack.py is present.

This whole "Android tests use old versions of stuff" is a real time waster :(

Comment 28

[Geoff Brown [:gbrown]]

(In reply to Nicholas Nethercote [:njn] from comment #27)

Goodness. If Android tests ran on opt builds as well as debug builds on CI
But they do! Both Android tests run against both opt and debug in CI:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=test-android

Usually the host-utils update is straight-forward and the host-utils patch is short -- easy enough to land in the same push as whatever requires it.

Comment 29

[Nicholas Nethercote [inactive]]

But they do! Both Android tests run against both opt and debug in CI:

You are right. I keep forgetting that stack-fixing doesn't occur via utils.py on opt builds.

Anyway, the path forward in comment 27 will still work.

Comment 30

[Nicholas Nethercote [inactive]]

aerickson: I just auto-landed the last change in bug 1626272, called "Make fix-stacks initialization eager". Assuming it sticks, once it merges to mozilla-central could you update the Linux hostutils update patch to include its revision? The Mac hostutils doesn't need updating, but if you like to keep Linux and Mac hostutils on the same revision then you could re-update it too.

Thank you, and sorry for the complications!

Comment 31

[Andrew Erickson [:aerickson]]

:njn, no worries. :) Will do.

Comment 32

[Nicholas Nethercote [inactive]]

aerickson: I just auto-landed the last change in bug 1626272, called "Make fix-stacks initialization eager". Assuming it sticks

It didn't, so I'm taking a different approach in that bug. Which means that the Linux hostutils update here can now land. Apologies for all the back and forth.

Comment 33

[Andrew Erickson [:aerickson]]

OK, landing. No worries. :)

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f7fdf8013efe
update linux hostutils r=gbrown

Comment 35

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/f7fdf8013efe