1172560 - Make sure we include current add-on blocklist in beta/release builds.

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Robert Kaiser]

The block we did in bug 1170633 almost a week ago was supposed to fix startup crashes (which are top priority) by not letting add-ons be activated that crash Firefox when they get loaded. We knew this would only take effect when people get updated to that blocklist either before they update to the new version, in safe mode, or - most importantly - by having the preloaded blocklist in the build itself.
Unfortunately, http://hg.mozilla.org/releases/mozilla-beta/rev/15c42675b6de4c0fc0a4ebbce42cf4ba61333f57 only landed the update to the preloaded blocklist 4 days later and 39.0b3 was shipped without it in the mean time.

Because we do blocks like that which need to take effect on startup (and otherwise people can't launch Firefox and get updated online), it's  important that we ship the current version of the blocklist when we build betas or releases. Can we update how the automated check-ins on beta (and release?) are made so that we can ensure that (or at least get closer than with the current once-a-week update)?

Comment 1

[Liz Henry (:lizzard) (relman/hg->git project)]

From discussion in the channel meeting I think this currently runs on Saturdays.  It would be good to at least do the update when we do beta releases.  But maybe daily would be reasonable if it can be automated and isn't difficult or resource-heavy.

Comment 2

[Phil Ringnalda (:philor)]

Because I love ancient history, https://bugzilla.mozilla.org/show_bug.cgi?id=791492#c15 was the culmination of what I think was the first instance of this discussion (I used those instructions quite a lot, when updates on various other trees failed, though I don't think anyone in relman ever actually used them).

Comment 3

[Phil Ringnalda (:philor)]

Though in the interim the blocklist changed locations, so you wouldn't actually want to follow those instructions now.

Comment 4

[[e-mail] (:bhearsum)]

When we start doing release promotion, these updates will need to happen prior to the changeset that we want to ship being checked in. KaiRo, catlee, and I talked a bit on IRC today and it sounds like the ideal thing is to check-in a new blocklist whenever it changes.

Based on http://hg.mozilla.org/releases/mozilla-beta/rev/530b85a091bf, it looks like these will cause a full set of CI builds which is probably good, since we may want to promote one of them, but it might become infeasible if it changes too often.

Jorge, do you know how often the blocklist changes?

Comment 5

[Phil Ringnalda (:philor)]

We have broken tests with blocklist updates (not frequently, but we have, both by having buggy tests that depended on the contents of the blocklist and when we've blocklisted our own ancient versions of plugins), so a DONTBUILD solution won't please sheriffs any more than it did in 2011 with bug 637444.

Comment 6

[Jorge Villalobos [:jorgev] (he/him)]

It varies, since it depends mainly on plugin vulnerabilities and me having time to look into extension blocklist requests. At least a couple of times a month: https://addons.mozilla.org/en-US/firefox/blocked/

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Jorge Villalobos [:jorgev] from comment #6)
> It varies, since it depends mainly on plugin vulnerabilities and me having
> time to look into extension blocklist requests. At least a couple of times a
> month: https://addons.mozilla.org/en-US/firefox/blocked/

OK, it's clear we're not talking about multiple updates per day on a regular basis (assuming the ones on the same day on that page are batched). If I'm right about that I don't see any reason we shouldn't just update the in tree blocklists whenever they change.

I recognize that there is some potential to cause test failures, so we definitely shouldn't set DONTBUILD for them. The risk of that vs. the reward of shipping the most up-to-date blocklist to our users every time makes this pretty clear cut to me.

Comment 8

[Rail Aliiev [:rail]]

Nick, I have a couple of questions here:

1) how the script handles the noop operations? Are we safe to run it if no updates are available?

2) Any objections to run it daily instead of weekly, so we can close this bug?

Comment 9

[Phil Ringnalda (:philor)]

No updates, no problem, https://dxr.mozilla.org/build-central/source/tools/scripts/periodic_file_updates/periodic_file_updates.sh#660

What is a problem is https://dxr.mozilla.org/build-central/source/tools/scripts/periodic_file_updates/periodic_file_updates.sh#524 - the reason we run it once a week at 3am Saturday is that its solution to push races is "crap, well, maybe next week nobody will push between the time I start and the time I finish." Maybe, possibly, since only --blocklist is a whole lot faster than --hsts --hpkp --blocklist, with a little negotiation with Tomcat for a clear landing space you could mostly avoid races.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Phil makes good points. Do we uplift the blocklist though ? Guessing not, so rebase should be very likely to succeed. http://hg.mozilla.org/build/tools/file/default/lib/python/util/hg.py#l608 may be of some use.

Ideally there is a push notification when the blocklist changes, so that we can trigger the script.

Comment 11

[Aki Sasaki (not active)]

Aiui, we update the blocklist in-tree. Reso fixed?

Comment 12

[Phil Ringnalda (:philor)]

Not really. We updated it then, just like we do now. The too-little that he asked for, daily updates instead of once a week, is fixed because we switched to daily, pretending it would fix something else it didn't fix, but what this bug was actually about was "don't ship anything without first checking that the in-tree blocklist is identical to the AMO blocklist." Running updates daily doesn't do anything to solve that, since there is no indication to anyone at all if they don't land. We don't land them CLOSED TREE, and we don't land them on relbranches, and although the mozilla-central updates run on taskcluster and show a decision task and a periodic-update job on treeherder, if you polled everyone who might star a job on m-c or m-b or m-r or esr52 about what it might mean if the pfu job failed, you would not get reso fixed answers, and near as I can tell we aren't (yet) even showing jobs on beta/release/esr52, so it might be only central that has switched to running them through taskcluster. There certainly isn't a stream of red jobs from our inability to land a blocklist update on mozilla-beta since it was closed on the 11th.

Fixed would be a relpro task which runs https://dxr.mozilla.org/build-central/source/tools/scripts/periodic_file_updates/periodic_file_updates.sh --blocklist (or the equivalent) on the revision being promoted, and puts a stop to shipping if the in-tree blocklist doesn't match the AMO blocklist.

Comment 13

[Simon Fraser [:sfraser] ⌚️GMT]

I think having the check inside release promotion will be a bit late, as it means the revision's been selected, a lot of tasks have been run, and then we have the risk of throwing it away, and wasting money & time.

Some of the options here depend on knowing 'How old is too old?' If the file is up to date at the start of the build, but out of date a day later when it's time to publish the release, is that a show-stopper? If it is, it could be an endless race, between starting a build and trying to get it out before the list is updated again.

The earliest we can know if the file is out of date in-tree is when the revision is selected, so we could check as early as setting up the new release in ship-it. A warning at that point would mean it could either be ignored (if RelMan decide it's not too old) or an uplift and a new revision being chosen.

We could start the periodic file updates running against m-b, m-r and -esr* as well, or a subset of them. We'd need to answer questions about merges, and what happens if it only fails on some of them.

Comment 14

[Robert Kaiser]

I'm not deeply involved any more in this and so can't completely speak for current requirements, but when I filed this, I would have been very happy to have the current-at-build-time version included there.

Comment 15

[[e-mail] (:bhearsum)]

To the extent that this still matters at all, I think it's been fixed. https://bugzilla.mozilla.org/show_bug.cgi?id=1436369 added support for remote settings updates (which is where the addon blocklist is now, I believe).