1192167 - [pulsetranslator] Fix handling of locale repacks for nightly builds based on new mozharness code

Status: VERIFIED FIXED

(Webtools :: Pulse, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The way how repacks are done has been changed. Now everything is based on mozharness. As result of that the routing key for notifications of l10n repacks have been changed too. Sadly we were not made aware of it. 

As Rail pointed out on bug 1177917 the routing keys are named like this:

build.{branch}-{platform}-l10n-nightly-1.*.finished
build.{branch}-{platform}-l10n-nightly-2.*.finished
build.{branch}-{platform}-l10n-nightly-3.*.finished
build.{branch}-{platform}-l10n-nightly-4.*.finished
build.{branch}-{platform}-l10n-nightly-5.*.finished
build.{branch}-{platform}-l10n-nightly-6.*.finished
build.{branch}-{platform}-l10n-nightly-7.*.finished
build.{branch}-{platform}-l10n-nightly-8.*.finished
build.{branch}-{platform}-l10n-nightly-9.*.finished
build.{branch}-{platform}-l10n-nightly-10.*.finished

Marking this bug as a blocker for us given that no locales are getting tested at the moment.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Example pulse message

This is an example pulse message for Aurora on Win64. Pushing it through pulsetranslator results in a single message being translated with the locale en-US. I assume that we should iterate through the locales array and issue separate normalized notifications similar to the repack builds for releases.

Minimized data:

{'locale': 'en-US', 'testsurl': None, 'previous_buildid': None, 'job_number': 11, 'build_number': None, 'builddate': 1438900808, 'buildername': u'Firefox mozilla-aurora win64 l10n nightly-6', 'platform': u'win64', 'version': None, 'revision': u'250b48c714a0', 'status': 0, 'buildtype': 'opt', 'product': u'firefox', 'slave': u'b-2008-ix-0147', 'test_packages_url': None, 'buildid': u'20150807004008', 'timestamp': '2015-08-07T15:20:53Z', 'key': u'build.mozilla-aurora-win64-l10n-nightly-6.11.finished', 'locales': u'{"xh": "Success", "ur": "Success", "uk": "Success", "uz": "Success", "sr": "Success", "sq": "Success", "tr": "Success", "son": "Success", "zh-CN": "Success", "zh-TW": "Success", "th": "Success", "vi": "Success", "sv-SE": "Success", "te": "Success", "ta": "Success"}', 'logurl': None, 'repack': None, 'tree': u'mozilla-aurora', 'buildurl': None, 'release': u'production'}

Sadly I do not see any previous_buildid anymore. Not sure if this is on purpose or simply missed. Ben do you know? Without that information we can no longer run our Firefox UI update tests for locale builds.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) from comment #1)
> Created attachment 8644987 [details]
> Example pulse message
> 
> This is an example pulse message for Aurora on Win64. Pushing it through
> pulsetranslator results in a single message being translated with the locale
> en-US. I assume that we should iterate through the locales array and issue
> separate normalized notifications similar to the repack builds for releases.
> 
> Minimized data:
> 
> {'locale': 'en-US', 'testsurl': None, 'previous_buildid': None,
> 'job_number': 11, 'build_number': None, 'builddate': 1438900808,
> 'buildername': u'Firefox mozilla-aurora win64 l10n nightly-6', 'platform':
> u'win64', 'version': None, 'revision': u'250b48c714a0', 'status': 0,
> 'buildtype': 'opt', 'product': u'firefox', 'slave': u'b-2008-ix-0147',
> 'test_packages_url': None, 'buildid': u'20150807004008', 'timestamp':
> '2015-08-07T15:20:53Z', 'key':
> u'build.mozilla-aurora-win64-l10n-nightly-6.11.finished', 'locales':
> u'{"xh": "Success", "ur": "Success", "uk": "Success", "uz": "Success", "sr":
> "Success", "sq": "Success", "tr": "Success", "son": "Success", "zh-CN":
> "Success", "zh-TW": "Success", "th": "Success", "vi": "Success", "sv-SE":
> "Success", "te": "Success", "ta": "Success"}', 'logurl': None, 'repack':
> None, 'tree': u'mozilla-aurora', 'buildurl': None, 'release': u'production'}
> 
> Sadly I do not see any previous_buildid anymore. Not sure if this is on
> purpose or simply missed. Ben do you know? Without that information we can
> no longer run our Firefox UI update tests for locale builds.

It was probably an oversight.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Rail, would it be possible for you to send me original pulse notifications for the following type of builds?

1. Nightly en-US
2. Nightly l10n repack

I assume for builds of type 2 we will always have the locales property and 'locale' should not be obeyed anymore? Even it contains a value?

Thanks

Comment 4

[Rail Aliiev [:rail]]

(In reply to Henrik Skupin (:whimboo) from comment #3)
> Rail, would it be possible for you to send me original pulse notifications
> for the following type of builds?
> 
> 1. Nightly en-US
> 2. Nightly l10n repack


I don't have any of those messages handy. You can use Pulse Inspector at https://tools.taskcluster.net/pulse-inspector/ to listen and inspect.

 
> I assume for builds of type 2 we will always have the locales property and
> 'locale' should not be obeyed anymore? Even it contains a value?

Correct, 'locale' shouldn't be set.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, I wrote a little script and will fetch all messages from today for nightlies. I will check later today for the properties we get and which we actually miss.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: github_pull_request.txt

This PR will enable us to process l10n nightly builds again. It will also update the locale specific status from the repack task.

I'm not sure if the update for the routing key is fine with you, but given that we publish multiple notifications we need a variation in the key. Otherwise we would use the same key multiple times.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh, and I also added the symbolUrl property which we were missing until now, and which we need for mozharness and processing crash data.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Merged as:
https://github.com/mozilla/pulsetranslator/commit/15415df5eac85e377d8043a122f5d8988d39f921

Jonathan, please publish it when you have the time. Thanks.

Comment 9

[Jonathan Griffin (:jgriffin)]

Deployed.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Works perfectly again. Thanks a lot for deploying that quickly.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=96e619dc79f0&filter-job_group_symbol=Ff&filter-job_group_symbol=Fr&filter-job_group_symbol=Fu&exclusion_profile=false