1386996 - Mochitests not starting on OSX artifact builds after rev 509b327a4a10

Status: NEW

(Firefox Build System :: General, defect)

Description

[Julian Descottes [:jdescottes]]

Bug 1326520 [1] seems to be break mochitests for artifact builds, at least on OSX.

Mochitests are stuck on startup with the following error message: 

> JavaScript error: resource:///modules/E10SUtils.jsm, line 27: TypeError: aURL.pathQueryRef is undefined

No issue with a regular non-artifact build.

Here is an extract from my artifact build logs
> 0:21.96 Attempting to find a pushhead containing 509b327a4a101f685f9a1239a6f0512581aec058 on mozilla-central.
> 0:21.96 Attempting to find a pushhead containing 509b327a4a101f685f9a1239a6f0512581aec058 on integration/mozilla-inbound.
> 0:21.96 Attempting to find a pushhead containing 509b327a4a101f685f9a1239a6f0512581aec058 on releases/mozilla-beta.
> 0:23.39 Retrieving the last 50 pushheads starting with id 93313 on integration/mozilla-inbound
> 0:23.39 Retrieving the last 50 pushheads starting with id 32273 on mozilla-central
> 0:23.41 Installing from remote pushhead 320642944e42a889db13c6c55b404e32319d4de6 on mozilla-central

[1] https://hg.mozilla.org/mozilla-central/rev/509b327a4a10

Comment 1

[Julian Descottes [:jdescottes]]

Nick: looks like I am retrieving artifacts for https://hg.mozilla.org/mozilla-central/rev/320642944e42 which are incompatible with https://hg.mozilla.org/mozilla-central/rev/509b327a4a10, any idea ?

Comment 2

[Julian Descottes [:jdescottes]]

FWIW, the latest central now retrieves an artifact from revision: https://hg.mozilla.org/mozilla-central/rev/fa1da3c0b200abbd9cfab3cab19962824314044e

This one works fine and allows to start mochitests.

Would still be nice to know why we don't have a valid artifact for rev 509b327a4a101f685f9a1239a6f0512581aec058

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Julian Descottes [:jdescottes] from comment #1)
> Nick: looks like I am retrieving artifacts for
> https://hg.mozilla.org/mozilla-central/rev/320642944e42 which are
> incompatible with https://hg.mozilla.org/mozilla-central/rev/509b327a4a10,
> any idea ?

I'm going to redirect to chmanchester.  Chris, could you take a look?

Comment 4

[Chris Manchester (limited bugmail, email directly)]

Looking at 509b327a4a101f685f9a1239a6f0512581aec058 on central, it's in the middle of a merge from autoland. The artifact code always takes the working revision and works backwards to find the most recent push head that has artifacts for the target platform, so we end up with compiled artifacts that don't incorporate bug 1326520 (from the push at 320642944e42a889db13c6c55b404e32319d4de6), while JavaScript files in the local repo will, so they'll attempt to call the not-yet-existent api.

In this particular case we end up with a slightly older push head from central instead of any of the push heads from autoland that might include bug 1326520 because we don't look for artifacts from autoland for artifact builds. Due to the design of the autoland repo we're not generally expecting people to have revisions checked out locally that only have relevant artifacts available on autoland.

Comment 5

[Nick Alexander :nalexander [he/him]]

(In reply to Chris Manchester (:chmanchester) from comment #4)
> Looking at 509b327a4a101f685f9a1239a6f0512581aec058 on central, it's in the
> middle of a merge from autoland. The artifact code always takes the working
> revision and works backwards to find the most recent push head that has
> artifacts for the target platform, so we end up with compiled artifacts that
> don't incorporate bug 1326520 (from the push at
> 320642944e42a889db13c6c55b404e32319d4de6), while JavaScript files in the
> local repo will, so they'll attempt to call the not-yet-existent api.
> 
> In this particular case we end up with a slightly older push head from
> central instead of any of the push heads from autoland that might include
> bug 1326520 because we don't look for artifacts from autoland for artifact
> builds. Due to the design of the autoland repo we're not generally expecting
> people to have revisions checked out locally that only have relevant
> artifacts available on autoland.

Thanks Chris, I learned something.  Do you think it would be expensive to check for autoland pushes, not use them by default, and give a canned version of the explanation above?

Comment 6

[Julian Descottes [:jdescottes]]

(In reply to Chris Manchester (:chmanchester) from comment #4)
> Due to the design of the autoland repo we're not generally expecting
> people to have revisions checked out locally that only have relevant
> artifacts available on autoland.

Thanks for having a look! 

Not sure to understand exactly what you mean here, but when I first experienced this issue I was on the latest mozilla-central revision at the time, which was https://hg.mozilla.org/mozilla-central/rev/d942ef54fdf7 .

I had identified https://hg.mozilla.org/mozilla-central/rev/509b327a4a10 as being the breaking changeset which is why I mentioned it here. But right now, if I checkout d942ef54fdf7 and try to build, I still get the same issue with mochitests.

Comment 7

[Chris Manchester (limited bugmail, email directly)]

When I check out d942ef54fdf7 and do an artifact build on my macbook I get artifacts from d942ef54fdf7 and mochitests run fine. Is it possible this happened before builds for that push finished in automation?

Comment 8

[Julian Descottes [:jdescottes]]

(In reply to Chris Manchester (:chmanchester) from comment #7)
> When I check out d942ef54fdf7 and do an artifact build on my macbook I get
> artifacts from d942ef54fdf7 and mochitests run fine. Is it possible this
> happened before builds for that push finished in automation?

Yes, that's probably what happened! 

After running ./mach artifact clear-cache that's also what I get (and mochitests then run fine of course). I think I tested this exactly 24hours ago, so less than 2 hours after the changeset reached central.

I assume we are interested in this build: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=d942ef54fdf726840a698d2ddcaf989d2c00edbc ? I tested at 8am local time, and the OSX cross compiled opt build finished at 6:42 am. Maybe we are waiting for tests to be green before marking the changeset as valid for artifact builds?

A warning/error message of some sort would be amazing in this case, but not sure how feasible this is. Thanks for the explanation, feel free to close the bug if there is no follow up possible on your side.