The default bug view has changed. See this FAQ.

Modify test channel names to support releases on Balrog

RESOLVED FIXED

Status

Release Engineering
Releases
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: nthomas, Assigned: bhearsum)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(8 attachments)

(Reporter)

Description

3 years ago
There's some history in bug 407435, but the requirements have shifted a little bit for Balrog. 

Whenever we use betatest/releasetest for beta *and* release builds we'll have trouble with Balrog:
* RC builds - release builds go on beta channel for stability checking, the meaning of betatest depends on what you start with (in a way which can't be written compactly in Balrog rules)
* OS blocking - we will want to block deprecated OS in beta test-channels before release

We could do something like this:
   release       -->   release
   betatest      -->   release-localtest
   releasetest   -->   release-cdntest

   beta          -->   beta
   betatest      -->
   releasetest   -->   beta-cdntest

   esr           -->   esr
   esrtest       -->  
   releasetest   -->   esr-cdntest

betatest & esrtest aren't useful if we've automatically got the updates available via download.m.o and CDN. beta-cdntest would be used for both regular betas and the RC situation, we're going via bouncer & CDN either way.

We don't need this before beta on Balrog, in fact much more convenient after that transition. We do need it for two or more of (beta,release,esr) on Balrog though.
Another proposal could be to use releasetest-ftp for updates served via our ftp server, and add releasetest-cdn for release builds. Similar for the other build types. That way we emphasize the test nature of the channel while making it clear where the updates are coming from.
(Reporter)

Comment 2

3 years ago
That would be fine too. 

I didn't say it directly in comment #0, but I don't want to use any variation on releasetest for the beta or esr channels.
Sure. That was clear for me after reading comment 0! :) It's the same for me, given that the current usage is kinda confusing.
(Assignee)

Comment 4

3 years ago
Created attachment 8489419 [details] [diff] [review]
update rule ids for all release configs

I went through Balrog dev and prod today and updated all of the rules to use the new-style channel names. I didn't add rules for the live release and esr channels (though one existed for release in dev already) for now. This patch updates all of the rule ids in the release config templates to reflect the changes. Here's a couple of queries that might help review:
Dev:
+---------+-------------------+-------------+------------------------------------------+---------+
| rule_id | channel           | product     | mapping                                  | comment |
+---------+-------------------+-------------+------------------------------------------+---------+
|      32 | NULL              | Firefox     | Firefox-32.0b1-build2                    | NULL    |
|      17 | NULL              | FirefoxOS   | FirefoxOS-mozilla-central-nightly-latest | NULL    |
|      12 | aurora            | NULL        | No-Update                                | NULL    |
|       7 | aurora            | Fennec      | Fennec-mozilla-aurora-nightly-latest     | NULL    |
|       2 | aurora            | Firefox     | Firefox-mozilla-aurora-nightly-latest    | NULL    |
|      14 | aurora            | Thunderbird | Thunderbird-comm-aurora-nightly-latest   | NULL    |
|      34 | beta              | Fennec      |                                          | NULL    |
|      26 | beta              | Firefox     | Firefox-32.0b1-build1                    | NULL    |
|      33 | beta              | Thunderbird | No-Update                                | NULL    |
|      25 | beta*             | NULL        | No-Update                                | NULL    |
|      36 | beta*             | Firefox     | Firefox-29.0b8-build1-schema2            | NULL    |
|      37 | beta-cdntest      | Fennec      |                                          | NULL    |
|      41 | beta-cdntest      | Firefox     | Firefox-32.0b1-build1                    | NULL    |
|      43 | beta-cdntest      | Thunderbird | No-Update                                | NULL    |
|      40 | beta-localtest    | Firefox     | Firefox-32.0b1-build1                    | NULL    |
|      21 | beta-localtest    | Thunderbird | No-Update                                | NULL    |
|       3 | elm               | Firefox     | Firefox-elm-nightly-latest               | NULL    |
|      42 | esr-cdntest       | Firefox     | No-Update                                | NULL    |
|      29 | esr-localtest     | Firefox     | No-Update                                | NULL    |
|      28 | nightly           | B2G         | B2G-mozilla-central-nightly-latest       | NULL    |
|       6 | nightly           | Fennec      | Fennec-mozilla-central-nightly-latest    | NULL    |
|       1 | nightly           | Firefox     | Firefox-mozilla-central-nightly-latest   | NULL    |
|      13 | nightly           | Thunderbird | Thunderbird-comm-central-nightly-latest  | NULL    |
|       9 | nightly*          | NULL        | No-Update                                | NULL    |
|       8 | nightly-cedar     | Fennec      | Fennec-cedar-nightly-latest              | NULL    |
|      15 | nightly-cert-test | Firefox     | Firefox-mozilla-central-newcert-test-1   | NULL    |
|      16 | nightly-cert-test | Firefox     | Firefox-mozilla-central-newcert-test-2   | NULL    |
|      18 | nightly-holly     | Firefox     | Firefox-31.0.0esr-build1                 | ueue    |
|       5 | nightly-ux        | Firefox     | Firefox-ux-nightly-latest                | NULL    |
|      35 | release           | Fennec      |                                          | NULL    |
|      31 | release           | Firefox     | Firefox-30.0-build2                      | NULL    |
|      24 | release*          | NULL        | No-Update                                | NULL    |
|      39 | release-cdntest   | Fennec      |                                          | NULL    |
|      20 | release-cdntest   | Firefox     | Firefox-30.0-build2                      | NULL    |
|      22 | release-cdntest   | Thunderbird | No-Update                                | NULL    |
|      38 | release-localtest | Fennec      |                                          | NULL    |
|      19 | release-localtest | Firefox     | Firefox-30.0-build2                      | NULL    |
|      44 | release-localtest | Thunderbird | No-Update                                | NULL    |
+---------+-------------------+-------------+------------------------------------------+---------+

Prod:
mysql> select rule_id, channel, product, mapping, substring(comment, 1, 75) as comment from rules order by channel, product;
+---------+-------------------+--------------+-----------------------------------------+-----------------------------------------------------------------------------+
| rule_id | channel           | product      | mapping                                 | comment                                                                     |
+---------+-------------------+--------------+-----------------------------------------+-----------------------------------------------------------------------------+
|       2 | aurora            | NULL         | No-Update                               | OS Blocking rule for de-supported OS' on Aurora.                            |
|      41 | aurora            | B2G          | B2G-mozilla-aurora-nightly-latest       | NULL                                                                        |
|       8 | aurora            | Fennec       | Fennec-mozilla-aurora-nightly-latest    | NULL                                                                        |
|      10 | aurora            | Firefox      | Firefox-mozilla-aurora-nightly-latest   | NULL                                                                        |
|      18 | aurora            | MetroFirefox | Firefox-mozilla-aurora-nightly-latest   | NULL                                                                        |
|      13 | aurora            | Thunderbird  | Thunderbird-comm-aurora-nightly-latest  | NULL                                                                        |
|      15 | auroratest        | Fennec       | Fennec-mozilla-aurora-nightly-latest    | NULL                                                                        |
|      16 | auroratest        | Firefox      | Firefox-mozilla-aurora-nightly-latest   | NULL                                                                        |
|      32 | beta              | Firefox      | Firefox-33.0b3-build1                   | NULL                                                                        |
|      43 | beta              | Thunderbird  | Thunderbird-32.0b1-build1               | NULL                                                                        |
|      31 | beta*             | NULL         | No-Update                               | OS Blocking rule for de-supported OS' on Beta channel                       |
|      51 | beta*             | Firefox      | Firefox-11.0b5-build1                   | watershed                                                                   |
|      52 | beta*             | Firefox      | Firefox-29.0b8-build1-schema2           | watershed                                                                   |
|      45 | beta-cdntest      | Firefox      | Firefox-33.0b3-build1                   | NULL                                                                        |
|      60 | beta-cdntest      | Thunderbird  | Thunderbird-32.0b1-build1               | NULL                                                                        |
|      25 | beta-localtest    | Firefox      | Firefox-33.0b3-build1                   | NULL                                                                        |
|      27 | beta-localtest    | Thunderbird  | Thunderbird-32.0b1-build1               | NULL                                                                        |
|      34 | esr               | Firefox      | No-Update                               | NULL                                                                        |
|      36 | esr               | Thunderbird  | No-Update                               | NULL                                                                        |
|      59 | esr-cdntest       | NULL         | Firefox-31.1.0esr-build1                | NULL                                                                        |
|      58 | esr-localtest     | Firefox      | Firefox-31.1.0esr-build1                | NULL                                                                        |
|      19 | nightly           | NULL         | No-Update                               | Global shut off rule. Bump priority of this rule to 100 to shut off all upd |
|      39 | nightly           | B2G          | B2G-mozilla-central-nightly-latest      | disabled because of busted patch making it into latest nightly              |
|       4 | nightly           | Fennec       | Fennec-mozilla-central-nightly-latest   | NULL                                                                        |
|       3 | nightly           | Firefox      | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      17 | nightly           | MetroFirefox | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      14 | nightly           | Thunderbird  | Thunderbird-comm-central-nightly-latest | NULL                                                                        |
|       1 | nightly*          | NULL         | No-Update                               | OS Blocking rule for de-supported OS' on all nightly channels.              |
|      42 | nightly-b2g30     | B2G          | B2G-mozilla-b2g30_v1_4-nightly-latest   | NULL                                                                        |
|      47 | nightly-b2g32     | B2G          | B2G-mozilla-b2g32_v2_0-nightly-latest   | NULL                                                                        |
|       6 | nightly-cedar     | Fennec       | Fennec-cedar-nightly-20131107040204     | NULL                                                                        |
|       9 | nightly-elm       | Firefox      | Firefox-elm-nightly-20131221040201      | NULL                                                                        |
|      12 | nightly-esr24     | Firefox      | Firefox-mozilla-esr24-nightly-latest    | NULL                                                                        |
|      44 | nightly-esr31     | Firefox      | Firefox-mozilla-esr31-nightly-latest    | NULL                                                                        |
|      48 | nightly-esr31     | Thunderbird  | Thunderbird-comm-esr31-nightly-latest   | NULL                                                                        |
|      20 | nightly-holly     | Firefox      | Firefox-holly-nightly-20140321040202    | NULL                                                                        |
|       5 | nightly-oak       | Firefox      | Firefox-oak-nightly-latest              | NULL                                                                        |
|       7 | nightly-ux        | Firefox      | Firefox-ux-nightly-20140612040202       | NULL                                                                        |
|      40 | nightlytest       | B2G          | B2G-mozilla-central-nightly-latest      | NULL                                                                        |
|      24 | nightlytest       | Fennec       | Fennec-mozilla-central-nightly-latest   | NULL                                                                        |
|      22 | nightlytest       | Firefox      | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      23 | nightlytest       | MetroFirefox | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      33 | release           | Firefox      | No-Update                               | NULL                                                                        |
|      35 | release           | Thunderbird  | No-Update                               | NULL                                                                        |
|      57 | release-cdntest   | Firefox      | Firefox-32.0.1-build2                   | NULL                                                                        |
|      62 | release-cdntest   | Thunderbird  | Thunderbird-31.1.1-build1               | NULL                                                                        |
|      56 | release-localtest | Firefox      | Firefox-32.0.1-build2                   | NULL                                                                        |
|      61 | release-localtest | Thunderbird  | Thunderbird-31.1.1-build1               | NULL                                                                        |
+---------+-------------------+--------------+-----------------------------------------+-----------------------------------------------------------------------------+
Attachment #8489419 - Flags: review?(nthomas)
(Assignee)

Comment 5

3 years ago
Created attachment 8489488 [details] [diff] [review]
make final verify support new channel names

Pete, I think comment #0 should give you enough background, but the tl;dr is that we need to support *-{localtest,cdntest} as the respective internal and external testing channels (in addition to betatest/releasetest for awhile). This tweak to final verify's regex should give it support for doing that.
Attachment #8489488 - Flags: review?(pmoore)
(Assignee)

Comment 6

3 years ago
Created attachment 8489492 [details] [diff] [review]
support new channel names in balrog submitter

This block is used to determine whether we should produce direct-to-ftp or bouncer URLs. localtest is the new direct-to-ftp channel, so we need to include it here.
Attachment #8489492 - Flags: review?(rail)
Attachment #8489492 - Flags: review?(rail) → review+
(Assignee)

Updated

3 years ago
Attachment #8489492 - Flags: checked-in+
(Reporter)

Comment 7

3 years ago
Comment on attachment 8489419 [details] [diff] [review]
update rule ids for all release configs

Review of attachment 8489419 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for the db info, very helpful.

::: mozilla/release-firefox-mozilla-esr31.py.template
@@ +119,5 @@
>  }
>  releaseConfig['releaseChannel']        = 'esr'
>  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3
>  releaseConfig['testChannels']          = ['releasetest', 'esrtest']
> +releaseConfig['testChannelRuleIds']    = [58,59]

59 needs product set to Firefox, it's NULL at the moment.

::: mozilla/staging_release-thunderbird-comm-esr31.py.template
@@ +131,5 @@
>      'macosx64': 'mail/config/mozconfigs/macosx-universal/release',
>      'win32': 'mail/config/mozconfigs/win32/release',
>  }
>  releaseConfig['releaseChannel']        = 'release'
>  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3

The other staging configs have a release rule, would be worth adding one to keep things consistent.
Attachment #8489419 - Flags: review?(nthomas) → review+
(Assignee)

Updated

3 years ago
Blocks: 933414
(Assignee)

Comment 8

3 years ago
Comment on attachment 8489419 [details] [diff] [review]
update rule ids for all release configs

(In reply to Nick Thomas [:nthomas] from comment #7)
> Comment on attachment 8489419 [details] [diff] [review]
> update rule ids for all release configs
> 
> Review of attachment 8489419 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Thanks for the db info, very helpful.
> 
> ::: mozilla/release-firefox-mozilla-esr31.py.template
> @@ +119,5 @@
> >  }
> >  releaseConfig['releaseChannel']        = 'esr'
> >  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3
> >  releaseConfig['testChannels']          = ['releasetest', 'esrtest']
> > +releaseConfig['testChannelRuleIds']    = [58,59]
> 
> 59 needs product set to Firefox, it's NULL at the moment.

Done.

> ::: mozilla/staging_release-thunderbird-comm-esr31.py.template
> @@ +131,5 @@
> >      'macosx64': 'mail/config/mozconfigs/macosx-universal/release',
> >      'win32': 'mail/config/mozconfigs/win32/release',
> >  }
> >  releaseConfig['releaseChannel']        = 'release'
> >  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3
> 
> The other staging configs have a release rule, would be worth adding one to
> keep things consistent.

Sure thing. I added this and landed the patch as https://hg.mozilla.org/build/buildbot-configs/rev/0fa6f60cfa9f
Attachment #8489419 - Flags: checked-in+
Comment on attachment 8489488 [details] [diff] [review]
make final verify support new channel names

Review of attachment 8489488 [details] [diff] [review]:
-----------------------------------------------------------------

Hi Ben,

Thanks for the overview. I can't claim to completely understand how the channels work, but looking at the table, and seeing the *localtest and *cdntest entries, I see that when there is a *localtest entry, there seems to be a corresponding *cdntest entry, and based on your explanation it seems intuitive that one would be for internal (localtest) and the other for external (cdntest) testing. Regarding the validity of the sed expression, I can confirm that it works, and plays well on both mac and linux (which I am happy about - because it keeps final verification completely mac-compatible).

The only thing I see missing is maybe a wiki page explaining the channel names, which I think could be important given the number of channel names and complexity of the setup, especially as this has been a little revised now.

Of course I don't want to block on this, but do you think you could doc something up as an extra artefact for this bug? Maybe a page which just gives a high level explanation about the new channel name strategy? Sorry for the extra work! I would write it myself, but I am one of the people that would need to read it, to understand how it works. :)

Last point was rule_id 18 seems to have an incorrect comment - probably also worth fixing.

Thanks!
Pete
Attachment #8489488 - Flags: review?(pmoore) → review+
(Assignee)

Comment 10

3 years ago
Comment on attachment 8489488 [details] [diff] [review]
make final verify support new channel names

(In reply to Pete Moore [:pete][:pmoore] from comment #9)
> Comment on attachment 8489488 [details] [diff] [review]
> make final verify support new channel names
> 
> Review of attachment 8489488 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Hi Ben,
> 
> Thanks for the overview. I can't claim to completely understand how the
> channels work, but looking at the table, and seeing the *localtest and
> *cdntest entries, I see that when there is a *localtest entry, there seems
> to be a corresponding *cdntest entry, and based on your explanation it seems
> intuitive that one would be for internal (localtest) and the other for
> external (cdntest) testing. Regarding the validity of the sed expression, I
> can confirm that it works, and plays well on both mac and linux (which I am
> happy about - because it keeps final verification completely mac-compatible).
> 
> The only thing I see missing is maybe a wiki page explaining the channel
> names, which I think could be important given the number of channel names
> and complexity of the setup, especially as this has been a little revised
> now.

I created https://wiki.mozilla.org/Releases/Update_Channels just a few days ago - is that the kind of thing you're looking for?

> Last point was rule_id 18 seems to have an incorrect comment - probably also
> worth fixing.

That's a rule from the dev server, I'm not going to worry about it...
Attachment #8489488 - Flags: checked-in+
(Assignee)

Comment 11

3 years ago
Created attachment 8490204 [details] [diff] [review]
support new test channel variables in balrog pusher
Attachment #8490204 - Flags: review?(nthomas)
(Assignee)

Comment 12

3 years ago
Created attachment 8490206 [details] [diff] [review]
support new test channel substitutions in release mail; use local test channel for update verify

Because we have different channels for different release streams, some scripts and tools (final verify, update verify config bumper, etc.) can't easily figure out which channels to use. Rather than adding hardcodes to all of those tools, changing our release configs like this lets us do the right thing in all places without hardcodes.

I did a bunch of staging tests with this (and the other) patches and everything looked fine. Specifically, I ran the updates, final verify, and ready for releasetest builders of Firefox+Thunderbird Beta+Release - everything seemed fine there (modulo some staging specific issues).
Attachment #8490206 - Flags: review?(nthomas)
(Assignee)

Comment 13

3 years ago
Created attachment 8490207 [details] [diff] [review]
switch to new test channel variables
Attachment #8490207 - Flags: review?(nthomas)
(Assignee)

Updated

3 years ago
Attachment #8490206 - Attachment description: split out test channels in release configs; use local test channel for update verify → support new test channel substitutions in release mail; use local test channel for update verify
(Reporter)

Updated

3 years ago
Attachment #8490204 - Flags: review?(nthomas) → review+
(Reporter)

Comment 14

3 years ago
Comment on attachment 8490207 [details] [diff] [review]
switch to new test channel variables

Review of attachment 8490207 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with minor tweak.

::: mozilla/staging_release-thunderbird-comm-esr24.py.template
@@ +138,5 @@
>      'win32': 'mail/config/mozconfigs/win32/release',
>  }
>  releaseConfig['releaseChannel']      = 'release'
> +releaseConfig['localTestChannel']    = 'betatest'
> +releaseConfig['cdnTestChannel']      = 'releasetest'

These are empty in the production config, because updates are gone for esr24 ?
Attachment #8490207 - Flags: review?(nthomas) → review+
(Reporter)

Comment 15

3 years ago
Comment on attachment 8490206 [details] [diff] [review]
support new test channel substitutions in release mail; use local test channel for update verify

Review of attachment 8490206 [details] [diff] [review]:
-----------------------------------------------------------------

Scratch everything I said on IRC about snippet comparison, got the wrong end of the stick.
Attachment #8490206 - Flags: review?(nthomas) → review+
(Assignee)

Comment 16

3 years ago
(In reply to Nick Thomas [:nthomas] from comment #14)
> Comment on attachment 8490207 [details] [diff] [review]
> switch to new test channel variables
> 
> Review of attachment 8490207 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r+ with minor tweak.
> 
> ::: mozilla/staging_release-thunderbird-comm-esr24.py.template
> @@ +138,5 @@
> >      'win32': 'mail/config/mozconfigs/win32/release',
> >  }
> >  releaseConfig['releaseChannel']      = 'release'
> > +releaseConfig['localTestChannel']    = 'betatest'
> > +releaseConfig['cdnTestChannel']      = 'releasetest'
> 
> These are empty in the production config, because updates are gone for esr24
> ?

I'll change this for staging on check-in - thanks!
(Assignee)

Comment 17

3 years ago
(In reply to Nick Thomas [:nthomas] from comment #15)
> Comment on attachment 8490206 [details] [diff] [review]
> support new test channel substitutions in release mail; use local test
> channel for update verify
> 
> Review of attachment 8490206 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Scratch everything I said on IRC about snippet comparison, got the wrong end
> of the stick.

Just to be safe I ended up doing full runs of the updates builder for mozilla-beta and mozilla-release on my dev-master -- they both went totally fine.
In production
(Assignee)

Updated

3 years ago
Attachment #8490204 - Flags: checked-in+
(Assignee)

Comment 19

3 years ago
Comment on attachment 8490206 [details] [diff] [review]
support new test channel substitutions in release mail; use local test channel for update verify

Landed this, including a transplant to the production-0.8 branch so that it will get picked up with 33.0b5 today.
Attachment #8490206 - Flags: checked-in+
(Assignee)

Comment 20

3 years ago
Comment on attachment 8490207 [details] [diff] [review]
switch to new test channel variables

Landed this, including a transplant to the production branch so that it's picked up with 33.0b5 today.
Attachment #8490207 - Flags: checked-in+
(Assignee)

Comment 21

3 years ago
I had to land a follow-up to put the new variables in the actual release configs (instead of just the templates), otherwise the reconfig would've busted:
https://hg.mozilla.org/build/buildbot-configs/rev/86f1fc6aff10
In prod with reconfig on 2014-09-18 08:14 PT
(Assignee)

Comment 23

3 years ago
Everything we need to do for Beta is done now. I'm going to leave this bug open to continue to track the migration when we put Release and ESR on Balrog.
(Assignee)

Comment 24

2 years ago
Created attachment 8546710 [details] [diff] [review]
switch release configs to new rules/channels
Assignee: nobody → bhearsum
Status: NEW → ASSIGNED
Attachment #8546710 - Flags: review?(nthomas)
(Reporter)

Updated

2 years ago
Attachment #8546710 - Flags: review?(nthomas) → review+
(Assignee)

Comment 25

2 years ago
Comment on attachment 8546710 [details] [diff] [review]
switch release configs to new rules/channels

Review of attachment 8546710 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/staging_release-fennec-mozilla-release.py.template
@@ +163,3 @@
>  releaseConfig['partialUpdates']      = {}
>  releaseConfig['bouncerServer']       = 'download.mozilla.org'
>  releaseConfig['releaseChannelRuleIds'] = []

I updated my local branch to include a release channel rule id, which now exists.
(Assignee)

Updated

2 years ago
Attachment #8546710 - Flags: checked-in+
(Assignee)

Comment 26

2 years ago
Only thing left to do here is mozilla-esr31, which will happen at the same time as bug 1117959 (hopefully in the next week or two).
Blocks: 735184
(Assignee)

Comment 27

2 years ago
Created attachment 8551905 [details] [diff] [review]
switch esr templates to new channels, add rule ids

The last ones!
Attachment #8551905 - Flags: review?(nthomas)
In production: https://hg.mozilla.org/build/buildbot-configs/rev/70a8291cfe60
(Reporter)

Updated

2 years ago
Attachment #8551905 - Flags: review?(nthomas) → review+
(Assignee)

Updated

2 years ago
Attachment #8551905 - Flags: checked-in+
(Assignee)

Comment 29

2 years ago
With the ESR changes landed we're all done here!
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
In production: https://hg.mozilla.org/build/buildbot-configs/rev/8830cc5285d1
You need to log in before you can comment on or make changes to this bug.