1282959 - implement release sanity from release-runner as unit testing

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

Rail suggested today that it would be really nice to have the release sanitizer implemented as a unit test script rather than a normal one. The upsides include already existing error handling, neater code, decoupled logic. In particular, it will be easier for anyone to add another check/case of release sanity by just adding a new test logic. Neat, clean, easy to understand.

The logic to be moved lies in http://hg.mozilla.org/build/tools/file/old-release-runner/buildfarm/release/release-runner.py#l423 but also in other scenarios we've encoutered lately.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Status update on this:
I'm building an initial draft of this, will push it on my Github by EOD.

Note to self:
* I believe the release sanitizer unit testing should lie under kickoff folder in tools to be accessible for both graphs task generation. Existing lib functions and helper methods under the lib/python is also another benefit
* the sanitizer should also include the existing en-US binary sanity and the version sanity that is already implemented in the release-runner
* not sure though what's the best option to pass down the kwargs from release runner here but there's a certain amount of possibilities to opt from
* ideally, release runner should be clean straight-forward graph-generation unit

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: https://github.com/mozilla/build-tools/pull/20

Just some preliminary stub for the release sanititzer as unittests:
* I added some preliminary revision and version check
* l10n sanity

I've got advanced WIP for partials as well but I'm stuck on some module configs. Will follow-up next with some patches but I thought it would be a good idea to throw this up for an initial opinion.

We still need to figure out what's the best way to pass the kwargs from release runner to the release sanitizer unit tests.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self - don't forget to double check these are covered as well:
* we should block the build if one of the ship-it partials build did not ship successfully to prevent situations like this 
(48.0b2build1 failed on one particular win64 chunk, we had a follow-up build2, but for 48.0b3 we used 48.0b2build1 which ended up serbing complete mars for users, instead of partial mars
* maybe consider blocking the build if there is follow-up for one of the specified partial builds (if used 48.0b2build1, fail and suggest there is a follow-up build2 or build3 to be used)
* ensure all previous mar files exist for each platform based on partial list submitted in ship-it

Also, keep in mind this:
* Would be also great to check if RC has both beta and release versions in partials, see bug 1265579

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: First stub to move sanity under kickoff in tools

* Some preliminary stub with sanity under kickoff in tools.

* There are still things to add/change/improve but am throwing this for a first glance to get some feedback.

* since it's not final form, I preferred to attach a diff via old-way rather than using MozReview

* would be happy to discuss if the module structure / class/functions structure fits our needs

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8770331 [details] [diff] [review]
First stub to move sanity under kickoff in tools

Review of attachment 8770331 [details] [diff] [review]:
-----------------------------------------------------------------

In overall it looks very close to what we want! \o/

I turned my nitpicking mode on, be aware. :P

The only major thing that we need to address is how to accumulate SanityExceptions and not halting execution in the middle of a test with multiple Exceptions (in loops for example).

::: lib/python/kickoff/sanity.py
@@ +15,5 @@
> +    def _get():
> +        req = requests.head(page_url, timeout=60)
> +        req.raise_for_status()
> +
> +    return retry(_get, attempts=5, sleeptime=1)

A nit: return is redundant here, because there is no return in _get()

@@ +23,5 @@
> +    """Make a GET request to retrieve page content"""
> +    def _get():
> +        req = requests.get(page_url, timeout=60)
> +        req.raise_for_status()
> +        return req.content.strip()

Even though you need to strip() everywhere when you use this function, it doesn't sound right to have it here... I'd probably use make_generic_get_request(url).strip() explicitly.

@@ +36,5 @@
> +    return make_generic_get_request(url)
> +
> +
> +def get_l10_dashboard_changeset(version, product,
> +                                dash_base="https://l10n.mozilla.org/shipping/l10n-changesets?ms={milestone}"):

you don't pass dash_base anywhere, I'd rather move it to module level L10N_DASHBOARD_URL_TEMPLATE to make sure .format() doesn't barf if you pass something without {milestone}.

@@ +54,5 @@
> +
> +def get_single_locale_config(repo_path,
> +                             revision,
> +                             branch,
> +                             filename='testing/mozharness/configs/single_locale/{branch}.py'):

Probably the same here?

@@ +82,5 @@
> +        ],
> +        ...
> +    }
> +    """
> +    assert branch

Do you need this? ^

@@ +91,5 @@
> +
> +def get_releases_bouncer_config(repo_path,
> +                                revision,
> +                                branch,
> +                                filename='testing/mozharness/configs/releases/bouncer_{ff_branch}.py'):

And here?

@@ +94,5 @@
> +                                branch,
> +                                filename='testing/mozharness/configs/releases/bouncer_{ff_branch}.py'):
> +    """TODO"""
> +    assert branch
> +    _ = branch.replace("-", "_")

hmm, maybe instead of this magic use a dictionary of branch-to-config mappings?

@@ +123,5 @@
> +                                       self.kwargs["branch"])
> +        locals_dict = dict()
> +        exec(ret, {}, locals_dict)
> +        single_locale_config = locals_dict.get('config')
> +        self.kwargs.update(single_locale_config)

Wouldn't this override some variables if they have the same name? Do we want this? Maybe leave it as self.single_locale_config instead? I feel that it'd be safer and more explicit.

@@ +125,5 @@
> +        exec(ret, {}, locals_dict)
> +        single_locale_config = locals_dict.get('config')
> +        self.kwargs.update(single_locale_config)
> +
> +    def sanitize(self):

Assuming this is the main method to be called from somewhere else. One thing to consider. If I call this method, it will run until it hits an exception. It would be great to have a helper which would run *all* sanity checks, accumulates exceptions and only after checking all of them would raise another exception. This way you run all checks every time and there is no need to incrementally fix the issues. Something similar to unittests. :) Does it make sense?

@@ +135,5 @@
> +        """TODO"""
> +        repo_path = self.kwargs["repo_path"]
> +        revision = self.kwargs["revision"]
> +        version = self.kwargs["version"].replace("esr", "")
> +        _assertEqual = self.assertEqual

I'm not sure why you needed this? ^

@@ +155,5 @@
> +                        tree_version=display_version, version=version)
> +            _assertEqual(version, display_version, err_msg)
> +
> +        _test_branch_and_revision_validation_in_tree()
> +        _test_display_version_validation_in_tree()

Hmmm. I'd probably have these 2 calls as 2 different methods and call them explicitly from self.sanitize(). Otherwise you won't be able to get to the second check if you fail the first.

@@ +158,5 @@
> +        _test_branch_and_revision_validation_in_tree()
> +        _test_display_version_validation_in_tree()
> +
> +    def test_partials(self):
> +        """TODO"""

Can you make sure to document this method and how we use SHA512SUMS files to validate things? It's not obvious, because it's an indirect method. :)

@@ +163,5 @@
> +        candidate_url = ("https://archive.mozilla.org/pub/firefox/candidates/"
> +                         "{version}-candidates/build{build_number}/SHA512SUMS")
> +        releases_url = ("https://archive.mozilla.org/pub/firefox/releases/"
> +                        "{version}/SHA512SUMS")
> +        _assertEqual = self.assertEqual

the same here ^

@@ +175,5 @@
> +                    raise SanityException(err_msg)
> +
> +                return sha_sum
> +
> +            for partial_version, cnt in self.kwargs["partial_updates"].iteritems():

Sorry for nitpicking :) I spent a couple of minutes trying to understand what mnemonics is used behind "cnt" and failed. :) Feel free to use a longer name ;)

@@ +191,5 @@
> +                err_msg = ("{version}-build{build_number} is a good candidate"
> +                        " build but not the one we shipped!").format(
> +                                version=partial_version,
> +                                build_number=build_number)
> +                _assertEqual(releases_sha, candidate_sha, err_msg)

I'd probably accumulate all errors before raising, so you go over all partials.

@@ +193,5 @@
> +                                version=partial_version,
> +                                build_number=build_number)
> +                _assertEqual(releases_sha, candidate_sha, err_msg)
> +
> +        _test_partials_validity()

This can be removed to dedent the code ;)

@@ +199,5 @@
> +    def test_l10n(self):
> +        """TODO"""
> +        repo_path = self.kwargs["repo_path"]
> +        revision = self.kwargs["revision"]
> +        locales = self.kwargs["l10n_changesets"]

some of these are used everywhere, maybe convert them to self.repo_path etc?

@@ +200,5 @@
> +        """TODO"""
> +        repo_path = self.kwargs["repo_path"]
> +        revision = self.kwargs["revision"]
> +        locales = self.kwargs["l10n_changesets"]
> +        _assertEqual = self.assertEqual

here too

@@ +204,5 @@
> +        _assertEqual = self.assertEqual
> +
> +        def _test_shipped_locales():
> +            r = make_hg_get_request(repo_path, revision,
> +                                    filename='browser/locales/shipped-locales')

Just keep in mind that we will need something similar for fennec.

@@ +210,5 @@
> +            current_l10n = set(self.kwargs["l10n_changesets"].keys())
> +
> +            err_msg = "Shipped locales are different than ones from Ship it!"
> +            self.assertEqual(shipped_l10n.difference(current_l10n),
> +                             set(['en-US']),

Can you add a comment why en-US is ignored?

@@ +233,5 @@
> +                    err_msg = "{locale} locale with revision {revision} not found".format(
> +                        locale=locale,
> +                        revision=revision
> +                    )
> +                    raise SanityException(err_msg)

Something needs to accumulate these errors in the loop, so you avoid incremental fixes.

@@ +246,5 @@
> +            _assertEqual(shipit_l10n_changesets, dash_l10n_changesets)
> +
> +        _test_verify_l10n_changesets()
> +        _test_shipped_locales()
> +        _test_l10n_dashboard()

Similar to other one, I'd have these as separate methods, so you can run all of them without failing early.

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Second iteration on moving release sanity under kickoff

Second iteration on moving sanity under kickoff in tools
> addressed rail's earlier comments
> more refactoring

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Second iteration on moving sanity under kickoff

Addressed rail's comments + some more refactoring

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8771022 [details] [diff] [review]
Second iteration on moving release sanity under kickoff

Review of attachment 8771022 [details] [diff] [review]:
-----------------------------------------------------------------

Wheee! this one is almost ready to go! Just test it and address the TODO comments. \o/

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1282959 - increase verbosity in new relpro sanity.

Review commit: https://reviewboard.mozilla.org/r/64464/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/64464/

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

I left 1-2 TODOs in the patch to be addressed a little bit later, once the integration is tested.

The output looks something like this, in case of exceptions aggregation:

===
(relsanitizer) mihaitabara@MacBook-Pro-7:[sanitizer]~/work/mozilla/clones/git/build-tools/buildfarm/release$ python release-runner.py
Traceback (most recent call last):
  File "release-runner.py", line 543, in <module>
    main()
  File "release-runner.py", line 338, in main
    validate_graph_kwargs(None, None, **kwargs)
  File "release-runner.py", line 269, in validate_graph_kwargs
    raise SanityException("Issues on release sanity %s" % errors)
kickoff.sanity.SanityException: Issues on release sanity [('Shipped locale file not found in releases/mozilla-beta repo under rev2f6f69a19150e03ad68062f2ac92342afb1ef788',
  HTTPError(u'404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/raw-file/2f6f69a19150e03ad68062f2ac92342afb1ef788/browser/locales/shipped-locales',)),
 ('Failed to retrieve single locale config file forreleases/mozilla-beta, revision 2f6f69a19150e03ad68062f2ac92342afb1ef788, mozilla-beta branch',
  HTTPError(u'404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/raw-file/2f6f69a19150e03ad68062f2ac92342afb1ef788/testing/mozharness/configs/single_locale/mozilla-beta.py',)),
 ('Broken build - hash https://archive.mozilla.org/pub/firefox/candidates/46.0b2-candidates/build4/SHA512SUMS not found',
  HTTPError(u'404 Client Error: Not Found for url: https://archive.mozilla.org/pub/firefox/candidates/46.0b2-candidates/build4/SHA512SUMS',)),
 ('46.0b2-build4 is a good candidate build but not the one we shipped!',
  'No more details to show'),
 ('Broken build - hash https://archive.mozilla.org/pub/firefox/candidates/46.0-candidates/build6/SHA512SUMS not found',
  HTTPError(u'404 Client Error: Not Found for url: https://archive.mozilla.org/pub/firefox/candidates/46.0-candidates/build6/SHA512SUMS',)),
 ('46.0-build6 is a good candidate build but not the one we shipped!',
  'No more details to show'),
 ('display version config file not found in releases/mozilla-beta under2f6f69a19150e03ad68062f2ac92342afb1ef788 revision',
  HTTPError(u'404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/raw-file/2f6f69a19150e03ad68062f2ac92342afb1ef788/browser/config/version_display.txt',)),
 ('releases/mozilla-beta repo does not exist with 2f6f69a19150e03ad68062f2ac92342afb1ef788 revision',
  HTTPError(u'404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/rev/2f6f69a19150e03ad68062f2ac92342afb1ef788',))]
===

If you need me to tweak the output anyhow, please let me know.

Comment 12

[Rail Aliiev [:rail]]

Yeah, the output is a bit cryptic for a human eye. It'd be better to have each error be distinctive, maybe have an empty line between them?

Comment 13

[Rail Aliiev [:rail]]

https://reviewboard.mozilla.org/r/64464/#review61592

Almost there! We can survive with the current implementation, but can you address the following minor issues as well, please? :)

::: buildfarm/release/release-runner.py:252
(Diff revision 1)
>      return h.hexdigest()
>  
>  
>  def validate_graph_kwargs(queue, gpg_key_path, **kwargs):
>      # We don't export "esr" in the version
> +    # TODO: redundant, to be removed soon, once new relpro sanity is in place

I think it's safe to delete the call and the implementation. Do eet! :)

::: lib/python/kickoff/sanity.py:225
(Diff revision 1)
> +            This check prevents one possible fail-scenario in which the build
> +            under canidates is good and valid, but a follow-up build was
> +            actually shipped under releases. Since shipping to releases
> +            implies an actual copy of the files, for that particular reason we
> +            make sure that SHA512 checksums of the build under candidates is
> +            bitwise the same as the one from releases.

Wow! O_O

::: lib/python/kickoff/sanity.py:331
(Diff revision 1)
> +
> +    def _test_l10n_dashboard(self, result):
> +        """test_l10n method
> +        Tests if l10n dashboard changesets match the current l10n changesets
> +        """
> +        # TODO: this should be turned on for all betas and turned off for all

Hmm, it would be great to have it enabled...

Can you add something like:
```python
if not self.kwargs["dashsboard_check"]:
    log.warn("Skipping l10n dashboard check")
    return
```
I don't recall the exact variable name, but should be something like that.

::: lib/python/kickoff/sanity.py:400
(Diff revision 1)
> +
> +    def get_errors(self):
> +        """Retrieves the list of errors from the result objecti
> +        in a nicely-formatted string
> +        """
> +        return pprint.pformat(self.result.errors)

I think it would be better to have a custom representation. The structure is not that complicated. ;)

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

https://reviewboard.mozilla.org/r/64464/#review61592

> Wow! O_O

Not sure if this is a good "wow" or a bad one. Did I misunderstand something here?

> I think it would be better to have a custom representation. The structure is not that complicated. ;)

Not sure what you mean by custom representation.

Comment 15

[Rail Aliiev [:rail]]

https://reviewboard.mozilla.org/r/64464/#review61592

> Not sure if this is a good "wow" or a bad one. Did I misunderstand something here?

It was a GREAT WOW! :)

> Not sure what you mean by custom representation.

I mean something different than pprint. Maybe something like:

Release sanity failed:

* Shipped locale file not found in releases/mozilla-beta repo under rev2f6f69a19150e03ad68062f2ac92342afb1ef788:
  404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/raw-file/2f6f69a19150e03ad68062f2ac92342afb1ef788/browser/locales/shipped-locales

* Failed to retrieve single locale config file forreleases/mozilla-beta, revision 2f6f69a19150e03ad68062f2ac92342afb1ef788, mozilla-beta branch:
  404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/raw-file/2f6f69a19150e03ad68062f2ac92342afb1ef788/testing/mozharness/configs/single_locale/mozilla-beta.py

* Etc

Does it make sense?

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

https://reviewboard.mozilla.org/r/64464/#review61592

> I mean something different than pprint. Maybe something like:
> 
> Release sanity failed:
> 
> * Shipped locale file not found in releases/mozilla-beta repo under rev2f6f69a19150e03ad68062f2ac92342afb1ef788:
>   404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/raw-file/2f6f69a19150e03ad68062f2ac92342afb1ef788/browser/locales/shipped-locales
> 
> * Failed to retrieve single locale config file forreleases/mozilla-beta, revision 2f6f69a19150e03ad68062f2ac92342afb1ef788, mozilla-beta branch:
>   404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/raw-file/2f6f69a19150e03ad68062f2ac92342afb1ef788/testing/mozharness/configs/single_locale/mozilla-beta.py
> 
> * Etc
> 
> Does it make sense?

Yeah, it makes perfect sense. I gotta prepare the string beforehand though prior to feeding it to the Sanity Exception.
Or you want to exit() the call and before doing that print this?

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1282959 - implement new relpro release sanity. refactoring upon review.

Review commit: https://reviewboard.mozilla.org/r/64808/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/64808/

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

The output looks something like this now:

====
(relsanitizer) mihaitabara@MacBook-Pro-7:[sanitizer]~/work/mozilla/clones/git/build-tools/lib/python/kickoff$ python tester.py 
Traceback (most recent call last):
  File "tester.py", line 81, in <module>
    main()
  File "tester.py", line 77, in main
    raise SanityException("Issues on release sanity %s" % errors)
sanity.SanityException: Issues on release sanity 

* Current ship-it changesets are not the same as the l10n dashboard changesets:
Result of assertion, no exception stacktrace available

* Shipped locale file not found in releases/mozilla-beta repo under rev2f6f69a19150e03ad68062f2ac92342afb1ef788:
404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/raw-file/2f6f69a19150e03ad68062f2ac92342afb1ef788/browser/locales/shipped-locales

* Failed to retrieve single locale config file forreleases/mozilla-beta, revision 2f6f69a19150e03ad68062f2ac92342afb1ef788, mozilla-beta branch:
404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/raw-file/2f6f69a19150e03ad68062f2ac92342afb1ef788/testing/mozharness/configs/single_locale/mozilla-beta.py

* Broken build - hash https://archive.mozilla.org/pub/firefox/candidates/46.0b2-candidates/build4/SHA512SUMS not found:
404 Client Error: Not Found for url: https://archive.mozilla.org/pub/firefox/candidates/46.0b2-candidates/build4/SHA512SUMS

* 46.0b2-build4 is a good candidate build but not the one we shipped!:
Result of assertion, no exception stacktrace available

* Broken build - hash https://archive.mozilla.org/pub/firefox/candidates/46.0-candidates/build6/SHA512SUMS not found:
404 Client Error: Not Found for url: https://archive.mozilla.org/pub/firefox/candidates/46.0-candidates/build6/SHA512SUMS

* 46.0-build6 is a good candidate build but not the one we shipped!:
Result of assertion, no exception stacktrace available

* display version config file not found in releases/mozilla-beta under2f6f69a19150e03ad68062f2ac92342afb1ef788 revision:
404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/raw-file/2f6f69a19150e03ad68062f2ac92342afb1ef788/browser/config/version_display.txt

* releases/mozilla-beta repo does not exist with 2f6f69a19150e03ad68062f2ac92342afb1ef788 revision:
404 Client Error: Not Found for url: https://hg.mozilla.org/releases/mozilla-beta/rev/2f6f69a19150e03ad68062f2ac92342afb1ef788

====

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8771808 [details]
Bug 1282959 - implement new relpro release sanity. refactoring upon review.

https://reviewboard.mozilla.org/r/64808/#review61858

LGTM! 

I mentioned this in IRC, we also need to add the new argument to the graph 2 generator script:
https://dxr.mozilla.org/build-central/source/tools/buildfarm/release/releasetasks_graph_gen.py#107

Comment 20

[Rail Aliiev [:rail]]

Comment on attachment 8771197 [details]
Bug 1282959 - increase verbosity in new relpro sanity.

https://reviewboard.mozilla.org/r/64464/#review61860

This gets auto r+ wiht the fixes in the following patch

Comment 21

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1282959 - implement new relpro release sanity. fix new argument in graph2.

Review commit: https://reviewboard.mozilla.org/r/64914/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/64914/

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 8771947 [details]
Bug 1282959 - implement new relpro release sanity. fix new argument in graph2.

https://reviewboard.mozilla.org/r/64914/#review61870

::: buildfarm/release/releasetasks_graph_gen.py:52
(Diff revision 1)
>          "version": release_config["version"],
>          "revision": release_config["mozilla_revision"],
>          "mozharness_changeset": release_config["mozharness_changeset"] or release_config["mozilla_revision"],
>          "buildNumber": release_config["build_number"],
>          "l10n_changesets": release_config["l10n_changesets"],
> +        "dashboard_check": release_config["dashboardCheck"],

I don't think we have this in the release configs. Can you set it to False instead? It's kind of irrelevent to check this here..

Comment 23

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1282959 - bugfix argument in graph2.

Review commit: https://reviewboard.mozilla.org/r/64928/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/64928/

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 8771964 [details]
Bug 1282959 - bugfix argument in graph2.

https://reviewboard.mozilla.org/r/64928/#review61886

ship it!

Comment 25

[Mihai Tabara [:mtabara]⌚️GMT]

Shipped - 
https://hg.mozilla.org/build/tools/rev/c0e9a28a0548
https://hg.mozilla.org/build/tools/rev/53eab5ff4978
https://hg.mozilla.org/build/tools/rev/82fcd04ef489
https://hg.mozilla.org/build/tools/rev/d2b4d5a0dbe4

Comment 26

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8771197 [details]
Bug 1282959 - increase verbosity in new relpro sanity.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/64464/diff/1-2/

Comment 27

[Rail Aliiev [:rail]]

https://reviewboard.mozilla.org/r/64464/#review61946

Comment 28

[Mihai Tabara [:mtabara]⌚️GMT]

Landed - http://hg.mozilla.org/build/tools/rev/2c012bef149a

Comment 29

[Mihai Tabara [:mtabara]⌚️GMT]

Changes tested against Firefox-48.0b9-build1 worked as expected.
Closing this!