[pulsetranslator] Fix handling of locale repacks for nightly builds based on new mozharness code

VERIFIED FIXED

Status

--
major
VERIFIED FIXED
3 years ago
3 years ago

People

(Reporter: whimboo, Assigned: whimboo)

Tracking

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Assignee)

Description

3 years ago
The way how repacks are done has been changed. Now everything is based on mozharness. As result of that the routing key for notifications of l10n repacks have been changed too. Sadly we were not made aware of it. 

As Rail pointed out on bug 1177917 the routing keys are named like this:

build.{branch}-{platform}-l10n-nightly-1.*.finished
build.{branch}-{platform}-l10n-nightly-2.*.finished
build.{branch}-{platform}-l10n-nightly-3.*.finished
build.{branch}-{platform}-l10n-nightly-4.*.finished
build.{branch}-{platform}-l10n-nightly-5.*.finished
build.{branch}-{platform}-l10n-nightly-6.*.finished
build.{branch}-{platform}-l10n-nightly-7.*.finished
build.{branch}-{platform}-l10n-nightly-8.*.finished
build.{branch}-{platform}-l10n-nightly-9.*.finished
build.{branch}-{platform}-l10n-nightly-10.*.finished

Marking this bug as a blocker for us given that no locales are getting tested at the moment.
(Assignee)

Comment 1

3 years ago
Created attachment 8644987 [details]
Example pulse message

This is an example pulse message for Aurora on Win64. Pushing it through pulsetranslator results in a single message being translated with the locale en-US. I assume that we should iterate through the locales array and issue separate normalized notifications similar to the repack builds for releases.

Minimized data:

{'locale': 'en-US', 'testsurl': None, 'previous_buildid': None, 'job_number': 11, 'build_number': None, 'builddate': 1438900808, 'buildername': u'Firefox mozilla-aurora win64 l10n nightly-6', 'platform': u'win64', 'version': None, 'revision': u'250b48c714a0', 'status': 0, 'buildtype': 'opt', 'product': u'firefox', 'slave': u'b-2008-ix-0147', 'test_packages_url': None, 'buildid': u'20150807004008', 'timestamp': '2015-08-07T15:20:53Z', 'key': u'build.mozilla-aurora-win64-l10n-nightly-6.11.finished', 'locales': u'{"xh": "Success", "ur": "Success", "uk": "Success", "uz": "Success", "sr": "Success", "sq": "Success", "tr": "Success", "son": "Success", "zh-CN": "Success", "zh-TW": "Success", "th": "Success", "vi": "Success", "sv-SE": "Success", "te": "Success", "ta": "Success"}', 'logurl': None, 'repack': None, 'tree': u'mozilla-aurora', 'buildurl': None, 'release': u'production'}

Sadly I do not see any previous_buildid anymore. Not sure if this is on purpose or simply missed. Ben do you know? Without that information we can no longer run our Firefox UI update tests for locale builds.
Flags: needinfo?(bhearsum)
(Assignee)

Updated

3 years ago
Depends on: 1192334
(In reply to Henrik Skupin (:whimboo) from comment #1)
> Created attachment 8644987 [details]
> Example pulse message
> 
> This is an example pulse message for Aurora on Win64. Pushing it through
> pulsetranslator results in a single message being translated with the locale
> en-US. I assume that we should iterate through the locales array and issue
> separate normalized notifications similar to the repack builds for releases.
> 
> Minimized data:
> 
> {'locale': 'en-US', 'testsurl': None, 'previous_buildid': None,
> 'job_number': 11, 'build_number': None, 'builddate': 1438900808,
> 'buildername': u'Firefox mozilla-aurora win64 l10n nightly-6', 'platform':
> u'win64', 'version': None, 'revision': u'250b48c714a0', 'status': 0,
> 'buildtype': 'opt', 'product': u'firefox', 'slave': u'b-2008-ix-0147',
> 'test_packages_url': None, 'buildid': u'20150807004008', 'timestamp':
> '2015-08-07T15:20:53Z', 'key':
> u'build.mozilla-aurora-win64-l10n-nightly-6.11.finished', 'locales':
> u'{"xh": "Success", "ur": "Success", "uk": "Success", "uz": "Success", "sr":
> "Success", "sq": "Success", "tr": "Success", "son": "Success", "zh-CN":
> "Success", "zh-TW": "Success", "th": "Success", "vi": "Success", "sv-SE":
> "Success", "te": "Success", "ta": "Success"}', 'logurl': None, 'repack':
> None, 'tree': u'mozilla-aurora', 'buildurl': None, 'release': u'production'}
> 
> Sadly I do not see any previous_buildid anymore. Not sure if this is on
> purpose or simply missed. Ben do you know? Without that information we can
> no longer run our Firefox UI update tests for locale builds.

It was probably an oversight.
Flags: needinfo?(bhearsum)
(Assignee)

Comment 3

3 years ago
Rail, would it be possible for you to send me original pulse notifications for the following type of builds?

1. Nightly en-US
2. Nightly l10n repack

I assume for builds of type 2 we will always have the locales property and 'locale' should not be obeyed anymore? Even it contains a value?

Thanks
Flags: needinfo?(rail)
Summary: [pulsetranslator] mozharness based l10n repacks are using a different routing key → [pulsetranslator] Fix handling of locale repacks for nightly builds based on new mozharness code
(In reply to Henrik Skupin (:whimboo) from comment #3)
> Rail, would it be possible for you to send me original pulse notifications
> for the following type of builds?
> 
> 1. Nightly en-US
> 2. Nightly l10n repack


I don't have any of those messages handy. You can use Pulse Inspector at https://tools.taskcluster.net/pulse-inspector/ to listen and inspect.

 
> I assume for builds of type 2 we will always have the locales property and
> 'locale' should not be obeyed anymore? Even it contains a value?

Correct, 'locale' shouldn't be set.
Flags: needinfo?(rail)
(Assignee)

Comment 5

3 years ago
Ok, I wrote a little script and will fetch all messages from today for nightlies. I will check later today for the properties we get and which we actually miss.
(Assignee)

Updated

3 years ago
Depends on: 1194207
(Assignee)

Comment 6

3 years ago
Created attachment 8647523 [details] [review]
github_pull_request.txt

This PR will enable us to process l10n nightly builds again. It will also update the locale specific status from the repack task.

I'm not sure if the update for the routing key is fine with you, but given that we publish multiple notifications we need a variation in the key. Otherwise we would use the same key multiple times.
Attachment #8647523 - Flags: review?(jgriffin)
(Assignee)

Comment 7

3 years ago
Oh, and I also added the symbolUrl property which we were missing until now, and which we need for mozharness and processing crash data.
Attachment #8647523 - Flags: review?(jgriffin) → review+
(Assignee)

Comment 8

3 years ago
Merged as:
https://github.com/mozilla/pulsetranslator/commit/15415df5eac85e377d8043a122f5d8988d39f921

Jonathan, please publish it when you have the time. Thanks.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
status-firefox42: affected → ---
Flags: needinfo?(jgriffin)
Resolution: --- → FIXED
(Assignee)

Updated

3 years ago
Whiteboard: [qa-automation-blocked]
Deployed.
Flags: needinfo?(jgriffin)
You need to log in before you can comment on or make changes to this bug.