Closed Bug 1172560 Opened 9 years ago Closed 1 year ago

Make sure we include current add-on blocklist in beta/release builds.


(Release Engineering :: General, defect)

Not set


(Not tracked)



(Reporter: kairo, Unassigned)



The block we did in bug 1170633 almost a week ago was supposed to fix startup crashes (which are top priority) by not letting add-ons be activated that crash Firefox when they get loaded. We knew this would only take effect when people get updated to that blocklist either before they update to the new version, in safe mode, or - most importantly - by having the preloaded blocklist in the build itself.
Unfortunately, only landed the update to the preloaded blocklist 4 days later and 39.0b3 was shipped without it in the mean time.

Because we do blocks like that which need to take effect on startup (and otherwise people can't launch Firefox and get updated online), it's  important that we ship the current version of the blocklist when we build betas or releases. Can we update how the automated check-ins on beta (and release?) are made so that we can ensure that (or at least get closer than with the current once-a-week update)?
From discussion in the channel meeting I think this currently runs on Saturdays.  It would be good to at least do the update when we do beta releases.  But maybe daily would be reasonable if it can be automated and isn't difficult or resource-heavy.
Because I love ancient history, was the culmination of what I think was the first instance of this discussion (I used those instructions quite a lot, when updates on various other trees failed, though I don't think anyone in relman ever actually used them).
Though in the interim the blocklist changed locations, so you wouldn't actually want to follow those instructions now.
When we start doing release promotion, these updates will need to happen prior to the changeset that we want to ship being checked in. KaiRo, catlee, and I talked a bit on IRC today and it sounds like the ideal thing is to check-in a new blocklist whenever it changes.

Based on, it looks like these will cause a full set of CI builds which is probably good, since we may want to promote one of them, but it might become infeasible if it changes too often.

Jorge, do you know how often the blocklist changes?
Flags: needinfo?(jorge)
We have broken tests with blocklist updates (not frequently, but we have, both by having buggy tests that depended on the contents of the blocklist and when we've blocklisted our own ancient versions of plugins), so a DONTBUILD solution won't please sheriffs any more than it did in 2011 with bug 637444.
It varies, since it depends mainly on plugin vulnerabilities and me having time to look into extension blocklist requests. At least a couple of times a month:
Flags: needinfo?(jorge)
(In reply to Jorge Villalobos [:jorgev] from comment #6)
> It varies, since it depends mainly on plugin vulnerabilities and me having
> time to look into extension blocklist requests. At least a couple of times a
> month:

OK, it's clear we're not talking about multiple updates per day on a regular basis (assuming the ones on the same day on that page are batched). If I'm right about that I don't see any reason we shouldn't just update the in tree blocklists whenever they change.

I recognize that there is some potential to cause test failures, so we definitely shouldn't set DONTBUILD for them. The risk of that vs. the reward of shipping the most up-to-date blocklist to our users every time makes this pretty clear cut to me.
Nick, I have a couple of questions here:

1) how the script handles the noop operations? Are we safe to run it if no updates are available?

2) Any objections to run it daily instead of weekly, so we can close this bug?
Flags: needinfo?(nthomas)
No updates, no problem,

What is a problem is - the reason we run it once a week at 3am Saturday is that its solution to push races is "crap, well, maybe next week nobody will push between the time I start and the time I finish." Maybe, possibly, since only --blocklist is a whole lot faster than --hsts --hpkp --blocklist, with a little negotiation with Tomcat for a clear landing space you could mostly avoid races.
Phil makes good points. Do we uplift the blocklist though ? Guessing not, so rebase should be very likely to succeed. may be of some use.

Ideally there is a push notification when the blocklist changes, so that we can trigger the script.
Flags: needinfo?(nthomas)
Aiui, we update the blocklist in-tree. Reso fixed?
Not really. We updated it then, just like we do now. The too-little that he asked for, daily updates instead of once a week, is fixed because we switched to daily, pretending it would fix something else it didn't fix, but what this bug was actually about was "don't ship anything without first checking that the in-tree blocklist is identical to the AMO blocklist." Running updates daily doesn't do anything to solve that, since there is no indication to anyone at all if they don't land. We don't land them CLOSED TREE, and we don't land them on relbranches, and although the mozilla-central updates run on taskcluster and show a decision task and a periodic-update job on treeherder, if you polled everyone who might star a job on m-c or m-b or m-r or esr52 about what it might mean if the pfu job failed, you would not get reso fixed answers, and near as I can tell we aren't (yet) even showing jobs on beta/release/esr52, so it might be only central that has switched to running them through taskcluster. There certainly isn't a stream of red jobs from our inability to land a blocklist update on mozilla-beta since it was closed on the 11th.

Fixed would be a relpro task which runs --blocklist (or the equivalent) on the revision being promoted, and puts a stop to shipping if the in-tree blocklist doesn't match the AMO blocklist.
Component: General Automation → General
I think having the check inside release promotion will be a bit late, as it means the revision's been selected, a lot of tasks have been run, and then we have the risk of throwing it away, and wasting money & time.

Some of the options here depend on knowing 'How old is too old?' If the file is up to date at the start of the build, but out of date a day later when it's time to publish the release, is that a show-stopper? If it is, it could be an endless race, between starting a build and trying to get it out before the list is updated again.

The earliest we can know if the file is out of date in-tree is when the revision is selected, so we could check as early as setting up the new release in ship-it. A warning at that point would mean it could either be ignored (if RelMan decide it's not too old) or an uplift and a new revision being chosen.

We could start the periodic file updates running against m-b, m-r and -esr* as well, or a subset of them. We'd need to answer questions about merges, and what happens if it only fails on some of them.
I'm not deeply involved any more in this and so can't completely speak for current requirements, but when I filed this, I would have been very happy to have the current-at-build-time version included there.

To the extent that this still matters at all, I think it's been fixed. added support for remote settings updates (which is where the addon blocklist is now, I believe).

Closed: 1 year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.