1102974 - Can't build ESR31 on Try

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[[e-mail]]

About a month ago I was pushing ESR31.2.0 to Try without any problems. However I am no longer able to do so.

Initially I was running into bug 1036093 but even after rebasing the ESR31.3pre checkin onto my ESR31.2.0 queue the try push still failed. This time the error is as follows:
07:11:18 INFO - We know it took a while, but your build finally finished successfully!
07:11:18 INFO - Return code: 0
07:11:18 INFO - setting properties set by mach build. Looking in path: c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\obj-firefox\mach_build_properties.json
07:11:18 FATAL - Could not determine path for build properties. Does this exist: `c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\obj-firefox\mach_build_properties.json` ?
07:11:18 FATAL - Running post_fatal callback...
07:11:18 ERROR - setting return code to 2 because fatal was called
07:11:18 FATAL - Exiting -1

I also tried the reverse, rebasing my patch onto ESR31.3pre, but that try push failed too. This time the error is as follows:
08:28:14 INFO - gmake[2]: Entering directory `/builds/slave/try-lx-00000000000000000000000/build/src'
08:28:14 INFO - # Terminate sccache server. This prints sccache stats.
08:28:14 INFO - python2.7 /builds/slave/try-lx-00000000000000000000000/build/src/sccache/sccache.py
08:28:14 INFO - sccache: Terminated sccache server
08:28:14 INFO - sccache: Cache hits: 808
08:28:14 INFO - sccache: Cache misses: 5809
08:28:14 INFO - sccache: Failures: 0
08:28:14 INFO - sccache: Non-cachable calls: 198
08:28:14 INFO - sccache: Non-compilation calls: 818
08:28:14 INFO - sccache: Max processes used: 10
08:28:16 INFO - sccache: get_time_stats: <JSON output>
08:28:16 INFO - gmake[2]: Leaving directory `/builds/slave/try-lx-00000000000000000000000/build/src'
08:28:16 INFO - gmake[1]: Leaving directory `/builds/slave/try-lx-00000000000000000000000/build/src'
08:28:16 INFO - /usr/bin/gmake -C /builds/slave/try-lx-00000000000000000000000/build/src/obj-i686-pc-linux -j4 automation/build
08:28:17 INFO - gmake: Entering directory `/builds/slave/try-lx-00000000000000000000000/build/src/obj-i686-pc-linux'
08:28:17 INFO - gmake: *** No rule to make target `automation/build'. Stop.
08:28:17 INFO - gmake: Leaving directory `/builds/slave/try-lx-00000000000000000000000/build/src/obj-i686-pc-linux'
08:28:17 INFO - 260 compiler warnings present.
08:28:19 ERROR - Return code: 2
08:28:19 FATAL - 'mach build' did not run successfully. Please check log for errors.
08:28:19 FATAL - Running post_fatal callback...
08:28:19 FATAL - Exiting -1

Comment 1

[Michael Shal [:mshal]]

I believe this is fallout from the mach/mozharness work. The required patches should have been uplifted as part of bug 1093897, but it sounds like something was missed.

Comment 2

[Michael Shal [:mshal]]

For the "No rule to make target 'automation/build'" error, it looks like I missed uplifting bug 832112 to force mach to obey MOZ_OBJDIR (it's trying to run the target in a non-existent objdir). I'll get that sorted out...

Comment 3

[Michael Shal [:mshal]]

And by 832112 I mean bug 762358, since 832112 was backed out for c-c. I'm currently running it on try.

Comment 4

[Michael Shal [:mshal]]

Ahh, I was also missing bug 1031180, without which mach wouldn't find the objdir in some cases. I uplifted these to esr31 - assuming it sticks you should be able to rebase against rev 7e37a85a056d and then push to try. Sorry for the mess :/

remote:   https://hg.mozilla.org/releases/mozilla-esr31/rev/18290cdb7152
remote:   https://hg.mozilla.org/releases/mozilla-esr31/rev/a9fc0ee99b72
remote:   https://hg.mozilla.org/releases/mozilla-esr31/rev/0a98654ebabe
remote:   https://hg.mozilla.org/releases/mozilla-esr31/rev/1d25e9dbfa89
remote:   https://hg.mozilla.org/releases/mozilla-esr31/rev/7e37a85a056d

Comment 5

[[e-mail]]

(In reply to Michael Shal from comment #3)
> And by 832112 I mean bug 762358, since 832112 was backed out for c-c. I'm
> currently running it on try.

I tried rebasing my patches onto your try push. Some of them worked, some didn't. I was able to get the ones that didn't work to work by setting MOZ_AUTOMATION_L10N_CHECK=0 in the mozconfig. I don't know whether this relates to your later comment.

Comment 6

[[e-mail]]

(In reply to comment #5)
> I tried rebasing my patches onto your try push. Some of them worked, some
> didn't. I was able to get the ones that didn't work to work by setting
> MOZ_AUTOMATION_L10N_CHECK=0 in the mozconfig. I don't know whether this
> relates to your later comment.

e.g. https://treeherder.mozilla.org/#/jobs?repo=try&revision=4e82406e3aac

Comment 7

[Michael Shal [:mshal]]

The last set is also in b2g32:

remote:   https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/7d48aee15106
remote:   https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/fa79733768c5
remote:   https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/9d49015d2ae7
remote:   https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/a17e01a85356
remote:   https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/f91abc6127d9

I'm still trying to sort out the l10n-check issue that happens on try. Curiously, l10n-check runs fine on esr31 when it is run as a separate make command.

Comment 8

[Michael Shal [:mshal]]

Hmm, I don't seem to get the l10n-check error with the latest on esr31:

https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=89d16b981b96

Is there something I'm missing? I also haven't had any luck reproducing it locally :/

Comment 9

[Mike Hommey [:glandium]]

From #build:

<rkent> One of the checkins to mozilla-esr31 in the last couple of days seems to have broken my comm-esr31 builds, at the configure steps, getting confused about MOZCONFIG. Is this a known issue?
<rkent> It's the push to esr31 from bug 762358 that is causing the failure
<rkent> Now Treeherder comm-esr31 builds are failing from that bug, affecting our ability to build the latest Thunderbird 31 which was supposed to happen today

Comment 10

[[e-mail]]

(In reply to Michael Shal from comment #8)
> Hmm, I don't seem to get the l10n-check error with the latest on esr31:

No, I'm not seeing it with esr31 tip, but I was seeing it against an earlier rebase.

Comment 11

[Michael Shal [:mshal]]

Is there anything else you need here? Or can we mark it closed?

Comment 12

[[e-mail]]

I think I'm fine, go ahead, if I run into more problems I'll let you know ;-)

Comment 13

[Michael Shal [:mshal]]

Ok, feel free to file a new bug & assign it to me for anything you hit :)