Closed Bug 731617 Opened 12 years ago Closed 12 years ago

No nightly builds on maple branch since 27 Feb

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: joe, Assigned: armenzg)

Details

(Whiteboard: [buildduty])

Attachments

(2 files)

For some reason, there have been no nighty builds on the maple branch since 27 Feb. I'm not sure if nightlies are turned on for maple, but they *should* be turned on. :)
I was told that there would be a nightly if all platforms successfully built. Maybe it's related to Win64 opt being pending on all jobs.
I am looking at http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-maple/?C=M;O=D and I can only see win64 nightlies on the 28th and then everything else on the 27th.

I see "enable_nightly" builds on project_branches.py

While we figure it out, on self-serve you can trigger new nightly builds if you were to need them.
I fired off a nightly build on current maple tip, under the assumption that no nightly will otherwise be triggered tonight.
Oh ye of little faith - you actually got a Win64 build on https://hg.mozilla.org/projects/maple/rev/d137bbd5fe5b (which must have coalesced up about 15 pending ones), so if that was the problem, you might have actually gotten one on that. Maybe. Or not.
It seems we have a second batch coming out today.
https://tbpl.mozilla.org/?tree=Maple&jobname=nightly
Component: Release Engineering → Release Engineering: Automation
OS: Mac OS X → All
QA Contact: release → catlee
Hardware: x86 → All
Whiteboard: [buildduty]
Assignee: nobody → armenzg
I triggered today's nightly too.
I believe the problem is revealed in the faithful sayings of philor which go far beyond our earthly understandings.

We don't have enough win64 machines and the priority of this branch is low compared to other branches. As we speak there are 5 build changes pending. If there is no good known revision then there is no nightly IIUC [1][2].

I say we remove win64 and see if it starts working again.

On another side, if I am right we probably want to modify the behavior of lastGoodFunc()[3].

Looking into writing a patch.

[1]
https://tbpl.mozilla.org/?tree=Maple&jobname=WINNT%206.1%20x86-64
[2]
http://hg.mozilla.org/build/buildbotcustom/file/default/misc.py#l886
[3]
http://mxr.mozilla.org/build/source/buildbotcustom/misc_scheduler.py#252
+1 on removing Win64 for maple.
Attachment #602468 - Flags: review?(rail)
Can I assume this can wait to land in production until Monday since self-serve allows you to do it manually?
Could you also *not* trigger nightly builds on Tuesday? I would like to see if they get triggered naturally.
Priority: -- → P2
Attachment #602468 - Flags: review?(rail) → review+
Your wish is my command.
Removing Win64 is probably the wrong thing to do.

We shouldn't be solving the twin problems of "only 4 of the Win64 build slaves are running" and "we set the priority of disposable branches to 4 back when we were only going to use them as temporary personal try servers, not as project branches for several-month-long projects" by just disabling some build which Maple will still be required to not break when it merges to mozilla-central. If Maple will either never touch anything which might affect Windows (unlikely), or it can just figure out what it was with a push to try just before merging, then we should be disabling every waste-of-resources build and test on every platform other than Android, not just the one platform where we don't want to restart the build slaves.

While I think my patch is the right thing to do (pushes to Maple are not less important than pushes to services-central, and are certainly not less important than every single test job on try), I don't think either thing is going to actually work, since the current tip of Maple got a Win64 build last night (partly low load, partly because I killed several inbound Win64 builds while they were pending, to free up a slave to do it), and it still didn't trigger the Sunday morning nightly it should have.
Attachment #602756 - Flags: review?(armenzg)
Attachment #602756 - Flags: review?(armenzg) → review+
Attachment #602468 - Flags: checked-in+
This change went live in a reconfiguration around 9AM.
Comment on attachment 602756 [details] [diff] [review]
better fix, don't underprioritize Maple

http://hg.mozilla.org/build/buildbot-configs/rev/8de384bb9997
Attachment #602756 - Flags: checked-in+
Since I'm a little slow on the uptake, I didn't realize what we were doing here (partly because I think this was the one where we both thought you had already pushed your patch, and then both decided that you had not).

My patch was intended to be an alternative to yours, that instead of not doing Win64 because it never gets run because the priority of Maple is too low, we would continue to do Win64 but increase the priority of Maple high enough that it would run.

With your patch landed, there's no actual point to my patch: we don't need to help Maple fight for Win64 build slaves if it isn't fighting for Win64 build slaves.
Leftover pending jobs for maple have been deleted from the db (2 win64, 3 desktop for nightly forcing).
Reconfiged x2.
Do you guys still need the maple branch? I don't see any more commits since the merge to m-c in the 14th.

https://wiki.mozilla.org/ReleaseEngineering/DisposableProjectBranches#BOOKING_SCHEDULE
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
No, we're done with it now. Thanks!
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: