Last Comment Bug 791492 - Do automated blocklist updates on mozilla-release and mozilla-esr10
: Do automated blocklist updates on mozilla-release and mozilla-esr10
Status: RESOLVED FIXED
:
Product: Release Engineering
Classification: Other
Component: General Automation (show other bugs)
: other
: All All
: -- normal (vote)
: ---
Assigned To: Phil Ringnalda (:philor, back in August)
: Chris AtLee [:catlee]
Mentors:
Depends on: 806255
Blocks: 785636
  Show dependency treegraph
 
Reported: 2012-09-15 15:45 PDT by Phil Ringnalda (:philor, back in August)
Modified: 2013-08-12 21:54 PDT (History)
4 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
+
fixed


Attachments
blocklist updates for both mozilla-release and esr10 (2.57 KB, patch)
2012-09-15 15:45 PDT, Phil Ringnalda (:philor, back in August)
coop: review+
lukasblakk+bugs: review+
philringnalda: checked‑in+
Details | Diff | Splinter Review

Description Phil Ringnalda (:philor, back in August) 2012-09-15 15:45:35 PDT
Created attachment 661535 [details] [diff] [review]
blocklist updates for both mozilla-release and esr10

In bug 663820 comment 8, we decided not to do blocklist updates on m-r because there aren't any nightlies. Bug 717106 created esr10 as a copy of m-r, thus without nightlies or blocklist updates, then bug 725852 added nightlies.

Because of bug 785636 comment 2 talking about how we blocklist things because they cause startup crashes, I think we want to either do automated updates on mozilla-release and esr-10 or do manual updates immediately before any go-to-build, but even if the "no nightlies means no reason to do blocklist updates" reasoning holds, esr10 does have nightlies, and thus doesn't have m-r's reason to not update.

As I said in bug 785636 comment 5, a Saturday morning update for things which go to build on Friday isn't especially well timed, but the blocklist from the Saturday before a release is still more up to date than the blocklist from the last Saturday that a mozilla-release was on mozilla-beta, or the last Saturday before 10.0 left mozilla-central (apparently 2011-10-15).
Comment 1 Chris Cooper [:coop] 2012-09-18 08:34:07 PDT
Comment on attachment 661535 [details] [diff] [review]
blocklist updates for both mozilla-release and esr10

Review of attachment 661535 [details] [diff] [review]:
-----------------------------------------------------------------

Patch is syntactically sound. I'll leave it to Alex to confirm that we want this on release and ESR.
Comment 2 Alex Keybl [:akeybl] 2012-10-12 14:22:31 PDT
(In reply to Phil Ringnalda (:philor) from comment #0)
> Created attachment 661535 [details] [diff] [review]
> blocklist updates for both mozilla-release and esr10
> 
> In bug 663820 comment 8, we decided not to do blocklist updates on m-r
> because there aren't any nightlies. Bug 717106 created esr10 as a copy of
> m-r, thus without nightlies or blocklist updates, then bug 725852 added
> nightlies.
> 
> Because of bug 785636 comment 2 talking about how we blocklist things
> because they cause startup crashes, I think we want to either do automated
> updates on mozilla-release and esr-10 or do manual updates immediately
> before any go-to-build, but even if the "no nightlies means no reason to do
> blocklist updates" reasoning holds, esr10 does have nightlies, and thus
> doesn't have m-r's reason to not update.
> 
> As I said in bug 785636 comment 5, a Saturday morning update for things
> which go to build on Friday isn't especially well timed, but the blocklist
> from the Saturday before a release is still more up to date than the
> blocklist from the last Saturday that a mozilla-release was on mozilla-beta,
> or the last Saturday before 10.0 left mozilla-central (apparently
> 2011-10-15).

Sorry for seeing this late, I don't normally have r? and missed this bugmail. Can you explain the before and after here? "Before, we updated the blocklist every X, now we'll blocklist every Y". Also, are we ensuring that we pull the final blocklist with our final build?
Comment 3 Phil Ringnalda (:philor, back in August) 2012-10-14 23:43:16 PDT
Actual results vary, because of bug 799160, but assuming reliable automated updates, there are three cases:

Before: a .0 on mozilla-release has the blocklist from 10 days before it ships (it gets updated on beta on a Saturday, the following Friday it merges from beta to release and gets tagged, so the Saturday before the Tuesday release is too late to make the build)

After: the exact same thing as before

----

Before: a .0.1 and a .0.9 on mozilla-release have that same blocklist from 10 days before the .0 shipped

After: they have the blocklist from the Saturday before they were tagged

----

Before: an esr build, any esr build, has the blocklist from when it was last on an updated tree, before the esr branch was created - in the case of esr10, that's 2011-10-15

After: like .0 builds on mozilla-release, the blocklist from 10 days before it ships, since it's tagged on a Friday before the Saturday update

I can't think of any meaning of "are we ensuring that we pull the final blocklist with our final build?" to which the answer would be "yes." The only way we would have an absolute final blocklist would be if someone (that would be you, I'm afraid) at the time of the go to build decision looks at the hg log for the blocklist file, looks at what AMO is serving, and if they aren't the same manually pushes an update, waits for it to build and test, and then goes to build on that cset. (There are a couple of other alternatives that occur to me, either making releng manually run their automatic script (because you can't have a system which relies on releng's person in charge of a release having push access themselves) and determine whether or not tests still passed, after they are given the go, or, go back to the bad old days when we did DONTBUILD updates, and update as part of tagging without actually seeing whether or not it broke anything, but neither of them appeals even as much as "make relman push a manual update," despite that also not being very appealing.)
Comment 4 Lukas Blakk [:lsblakk] use ?needinfo 2012-10-15 15:52:23 PDT
This doesn't sound like a proper fix for this bug then.  Relman having to do manual updates isn't particularly automated and I'm not sure I see why we can't move the blocklist updates themselves to fit better with our release cycles.  Who should be called upon to explain why it's done on Saturdays instead of Friday?
Comment 5 Phil Ringnalda (:philor, back in August) 2012-10-15 23:29:16 PDT
Well, there are multiple bugs (in both the sense of bugs, and the sense of things which should be separate bug reports, fixed by different people, at different times, with different priorities, on different timescales).

One bug is that we have esr nightlies, and so the part of this patch which says "whatever happens with automatic updates, that thing should happen on esr10 where it currently does not, because the nightlies should ship with a blocklist that is updated at least weekly, not once every six weeks, and not never."

Another bug is that Saturday updates are ill-timed for Friday go to builds. Alas, that bug won't be fixed by just changing http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#911 to Friday, because as bug 426214 comment 29 says, we deal with the update script losing push races, where someone else pushes between the time that the machine starts to clone the repo and the time it pushes, so that its push fails, by just saying "oh, well, maybe it'll go through next week" and by scheduling it for 3:02am Saturday, when we don't really expect anyone to be pushing. 3:02am Friday, in the heart of the last European day after we've said "approved for esr10, but push it right away, before Friday morning PDT!"? We'll lose push races, it won't go through, and someone will have to have a checklist item saying "check whether a blocklist update should have landed, check whether it did land, ask releng to manually run the script until it works if it should have landed but didn't." I only had to go back to September 14th to find https://tbpl.mozilla.org/?tree=Mozilla-Esr10&rev=ee8351424c56 which would have raced a builder that started at 03:02 in the current scheme of things, where they just do a slow full clone over http. Even if someone rewrites the script to do something that would horrify me, like try to automatically merge and push again so that push race failures aren't an issue, it's still possible for the script to fail, and if we actually want to be sure that it has always worked before we release, someone's going to have to check that it did work.

Another bug is that chemspill releases will not get an up to the minute blocklist with any automated updates - the only way to ensure that we ship them with an updated blocklist, rather than a blocklist from between 0 and 6 days old (again, assuming reliable automatic updates which we don't have) is to have a checklist item saying "get releng to manually-automatically update the blocklist."
Comment 6 Lukas Blakk [:lsblakk] use ?needinfo 2012-10-17 11:49:50 PDT
Comment on attachment 661535 [details] [diff] [review]
blocklist updates for both mozilla-release and esr10

How about this?  We'll approve this patch, blocklist updates will happen on Saturdays, please also add whatever manual steps would need to be run on merge days to https://wiki.mozilla.org/Release_Management/Merge_Documentation so that it's clear what RelMan needs to do every merge day to ensure that the most up to date blocklist goes into the mozilla-release repo at merge time.  For chemspills, if we're chemspilling for something that's related to the blocklist (startup crash for example) we'd obviously be updating the blocklist prior to building, so we're fine there and for other chemspills, we'd just keep shipping the current version of the release blocklist.
Comment 7 Lukas Blakk [:lsblakk] use ?needinfo 2012-10-17 11:50:43 PDT
Not going to track this for 17 though, it's independent of a particular release as far as I'm concerned.  When this patch is landed & the merge documentation is updated we'll be good to go the following merge day.
Comment 8 Phil Ringnalda (:philor, back in August) 2012-10-17 15:32:14 PDT
Comment on attachment 661535 [details] [diff] [review]
blocklist updates for both mozilla-release and esr10

http://hg.mozilla.org/build/buildbot-configs/rev/3aef6931bbef
Comment 9 Phil Ringnalda (:philor, back in August) 2012-10-17 15:36:44 PDT
I'll have to do a trial run or two, since I'm neither of the people who have ever run it, but the docs should wind up as simple as "save a copy of http://hg.mozilla.org/build/tools/raw-file/default/scripts/blocklist/sync-hg-blocklist.sh, chmod, pass it this commandline with your username and the path to your key, profit."
Comment 10 Lukas Blakk [:lsblakk] use ?needinfo 2012-10-17 16:01:33 PDT
Talked more about this more with Alex, it's something we'll want to be sure we have in place by the time 17 merge comes around so I'll track.
Comment 11 Aki Sasaki [:aki] 2012-10-18 15:31:58 PDT
This is now live.
Comment 12 Phil Ringnalda (:philor, back in August) 2012-10-21 17:15:35 PDT
How bad a thing will it be to have the instructions for manually-automatically updating start with

mkdir ~/m-r-blocklist/
cd ~/m-r-blocklist/
(do a thing which is going to do a fresh clone of m-r)
cd ~
mkdir ~/m-esr-blocklist/
cd ~/m-esr-blocklist/
(do a thing which is going to do a fresh clone of m-esr10)?

I avoid ever under any circumstances doing a fresh clone, so I don't know how bad a thing it is for people with a decent connection.

Also, I presume it has to work on OS X? The current commit message assumes both that `/bin/hostname -s` will work (it won't) and that it will produce something you want to commit in "from host ___" (you probably won't).
Comment 13 Phil Ringnalda (:philor, back in August) 2012-10-28 15:55:38 PDT
(In reply to Phil Ringnalda (:philor) from comment #12)
> Also, I presume it has to work on OS X? The current commit message assumes
> both that `/bin/hostname -s` will work (it won't) and that it will produce
> something you want to commit in "from host ___" (you probably won't).

It's not entirely impossible that I drink a bit - /bin/hostname -s has worked since long before the dawn of time, and the only problem I had with https://hg.mozilla.org/mozilla-central/rev/76a12989a99c was the need to add "No bug, " since the commitmessage hook whitelists ffxbld and lets through that message that otherwise is blocked.
Comment 14 Lukas Blakk [:lsblakk] use ?needinfo 2012-10-31 16:05:37 PDT
marking fixed, given comment 11, and when there are any startup crashes fixed in the final beta we will follow the steps in comment 12 to update the list again.
Comment 15 Phil Ringnalda (:philor, back in August) 2012-11-01 22:22:51 PDT
comment 12 wasn't actually instructions, it was handwaving about what the instructions might look like. With bug 806255 landed, what they look like is:

https://wiki.mozilla.org/Release_Management/Merge_Documentation#Get_all_of_the_Repos
[[[
$ hg clone http://hg.mozilla.org/releases/mozilla-esr10/ mozilla-esr10
$ hg clone http://hg.mozilla.org/build/tools/ tools
]]]

https://wiki.mozilla.org/Release_Management/Merge_Documentation#Finishing_up

[[[
After the merge appears to be successfully building, do dry-run blocklist updates to see if you need to update:

$ cd ~/Mozilla/
$ tools/scripts/blocklist/sync-hg-blocklist.sh -n -r mozilla-release -b releases/mozilla-release
(You'll either be told "Blocklist files differ, but not updating hg in dry-run mode" or "Blocklist files are identical. Nothing to update.")
$ tools/scripts/blocklist/sync-hg-blocklist.sh -n -r mozilla-esr10 -b releases/mozilla-esr10
(You'll either be told "Blocklist files differ, but not updating hg in dry-run mode" or "Blocklist files are identical. Nothing to update.")

If either one said "Blocklist files differ," rerun that one without -n, and with -c -a -u your_username -k path_to_key, e.g.

$ tools/scripts/blocklist/sync-hg-blocklist.sh -c -a -u philringnalda@gmail.com -k ~/.ssh/id_dsa -r mozilla-esr10 -b releases/mozilla-esr10
]]]

Note You need to log in before you can comment on or make changes to this bug.