1521967 - Streamline `./mach raptor-test --test=raptor-tp6m-* ...` for local testers

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

This is just not yet handled. Bug 1506912 stood up the initial testing infrastructure but it's specific to linux64 'cuz that's what bitbar runs; however we want to be able to run these tests more generally, including on macOS (and potentially, Windows!) hosts.

Comment 1

[Robert Wood [:rwood]]

You can run raptor tp6m-1 on android using a host other than Linux x64 (I develop on OSX and it works). Raptor gets certutuil from your local build (obj.../dist/). The cerutuil built with android doesn't work. The current work-around for this is to do a Firefox desktop build first before running Raptor tp6m-1 on android, then it will pull in the certutil built as part of the Firefox desktop build instead.

One way to fix this is to pull in host utils from production when running locally - however there is no tooltool manifest for pulling in hostutils on Windows.

Ideally this would all be avoided if geckoview supported an API that would allow importing CA certificates, or a policy file like Firefox desktop uses. Then we could use that, instead of having to use the certutil tool to create an nss cert db, and import the mitmproxy certificate manually.

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Robert Wood [:rwood] from comment #1)

You can run raptor tp6m-1 on android using a host other than Linux x64 (I develop on OSX and it works). Raptor gets certutuil from your local build (obj.../dist/). The cerutuil built with android doesn't work. The current work-around for this is to do a Firefox desktop build first before running Raptor tp6m-1 on android, then it will pull in the certutil built as part of the Firefox desktop build instead.

I discovered this, but it's very unintuitive. Building two different targets into one objdir makes no sense -- I'm surprised you were able to convince the build system to do that at all!

There's a MOZ_HOST_BIN mechanism that we use for this, and indeed the Raptor script invokes the helper at https://searchfox.org/mozilla-central/rev/6c784c93cfbd5119ed07773a170b59fbce1377ea/testing/raptor/mach_commands.py#186 -- but does not ask for the "xre=True" flag, which would set it up. Is there a reason not to do that?

One way to fix this is to pull in host utils from production when running locally - however there is no tooltool manifest for pulling in hostutils on Windows.

Aye. We could arrange one easily. It's a little nutty that we do all this hostutils packaging at all, since all this stuff is packed into artifact builds out of the box. But that's a different problem.

Ideally this would all be avoided if geckoview supported an API that would allow importing CA certificates, or a policy file like Firefox desktop uses. Then we could use that, instead of having to use the certutil tool to create an nss cert db, and import the mitmproxy certificate manually.

Aye, I see your follow-up. I read earlier discussion about this and the sec team was strongly against; I do not really know anything more.

Comment 3

[Nick Alexander :nalexander [he/him]]

Aye, I see your follow-up. I read earlier discussion about this and the sec team was strongly against; I do not really know anything more.

Re-reading https://bugzilla.mozilla.org/show_bug.cgi?id=1506912#c4, I realize that the security team is not against this API, it was that they were against disabling cert checking in general. So the ticket you filed sounds like a good approach for making this better more generally.

Comment 4

[Robert Wood [:rwood]]

(In reply to Nick Alexander :nalexander [he/him] from comment #2)

There's a MOZ_HOST_BIN mechanism that we use for this, and indeed the Raptor script invokes the helper at https://searchfox.org/mozilla-central/rev/6c784c93cfbd5119ed07773a170b59fbce1377ea/testing/raptor/mach_commands.py#186 -- but does not ask for the "xre=True" flag, which would set it up. Is there a reason not to do that?

Thanks. I've never heard of that, and not sure what you mean / what adding that would do (will it pull in a working certutil even when building android?). I only build Firefox desktop; I just grab the geckoview example app from builds on treeherder.

Adding :gbrown, as he has more experience with the android build/tooling than I do.

Comment 5

[Nick Alexander :nalexander [he/him]]

I'll morph this ticket into something a little more specific.

Comment 6

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1521967 - Part 1: Use MOZ_HOST_BIN to find certutil in `mach raptor-test` for Android. r?rwood

Comment 7

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1521967 - Part 2: Don't crash in `mach raptor-test` on Android. r?rwood

This was just oversight: we don't have a runner under Android.

Depends on D17396

Comment 8

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1521967 - Part 2: Drop sleeps in `mach raptor-test` when running locally. r?rwood

I don't know why we're sleeping around certutil invocations at all,
but let's at least not spend 30s doing nothing.

Depends on D17397

Comment 9

[Robert Wood [:rwood]]

(In reply to Nick Alexander :nalexander [he/him] from comment #8)

Created attachment 9038611 [details]
Bug 1521967 - Drop sleeps in mach raptor-test when running locally. r?rwood

I don't know why we're sleeping around certutil invocations at all,
but let's at least not spend 30s doing nothing.

Depends on D17397

Without the sleeps after certutil calls we saw tons of intermittent failures in production; the sleep is definitely needed.

Comment 10

[Geoff Brown [:gbrown]]

Nick's approach looks right to me.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:nalexander are you still working on this or is it ready to land?

Comment 12

[Nick Alexander :nalexander [he/him]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #11)

:nalexander are you still working on this or is it ready to land?

This should land: I've rebased (and noted that rwood fixed one of the issues, yay!); try build percolating at

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6c60a59f9532036556a9c9b997b47d22235afc53

Comment 13

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Alexander :nalexander [he/him] from comment #12)

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #11)

:nalexander are you still working on this or is it ready to land?

This should land: I've rebased (and noted that rwood fixed one of the issues, yay!); try build percolating at

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6c60a59f9532036556a9c9b997b47d22235afc53

This try build has many tp6m-3 failures, but it looks like they're known. And in fact, one log looks pretty successful:

...
03-15 20:07:14.389 5932 5948 E GeckoConsole: [JavaScript Error: "NaN"]
03-15 20:07:14.391 5932 5948 I GeckoConsole: posting to control server
03-15 20:07:14.392 5932 5948 I GeckoConsole: raptor-tp6m-google-maps-geckoview,https://www.google.com/maps?force=pwa,[object Object]
03-15 20:07:14.406 5932 5948 I GeckoConsole: closed tab 0
03-15 20:07:14.409 5932 5948 I GeckoConsole: pageloader test finished
03-15 20:07:14.411 5932 5948 I GeckoConsole:
03-15 20:07:14.411 5932 5948 I GeckoConsole: __raptor_shutdownBrowser
03-15 20:07:14.412 5932 5948 I GeckoConsole: posting to control server
03-15 20:07:14.413 5932 5948 I GeckoConsole: __raptor_shutdownBrowser
03-15 20:07:14.421 5932 5948 I GeckoConsole: post success
03-15 20:07:14.564 1196 1677 I ActivityManager: Force stopping org.mozilla.geckoview_example appid=10128 user=0: from pid 6411
03-15 20:07:14.564 1196 1677 I ActivityManager: Killing 5953:org.mozilla.geckoview_example:tab/u0a128 (adj 100): stop org.mozilla.geckoview_example
03-15 20:07:14.566 1196 1677 W ActivityManager: Scheduling restart of crashed service org.mozilla.geckoview_example/org.mozilla.gecko.process.GeckoServiceChildProcess$tab in 1000ms
03-15 20:07:14.431 5932 5948 I GeckoConsole: post success
03-15 20:07:14.567 1196 1240 W zygote64: kill(-5953, 9) failed: No such process
03-15 20:07:14.570 1196 1677 I ActivityManager: Killing 5932:org.mozilla.geckoview_example/u0a128 (adj 0): stop org.mozilla.geckoview_example
03-15 20:07:14.572 1196 1677 W ActivityManager: Force removing ActivityRecord{c427652 u0 org.mozilla.geckoview_example/.GeckoViewActivity t5846}: app died, no saved state
03-15 20:07:14.575 1196 1677 I WindowManager: Failed to capture screenshot of Token{dc58d23 ActivityRecord{c427652 u0 org.mozilla.geckoview_example/.GeckoViewActivity t5846 f}} appWin=Window{9d58105 u0 org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity} drawState=4

Why that's not good enough isn't clear to me.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7ac6d4f7327e
Part 1: Use MOZ_HOST_BIN to find certutil in `mach raptor-test` for Android. r=rwood
https://hg.mozilla.org/integration/autoland/rev/b010581c27e8
Part 2: Drop sleeps in `mach raptor-test` when running locally. r=rwood

Comment 15

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/7ac6d4f7327e
https://hg.mozilla.org/mozilla-central/rev/b010581c27e8