Closed Bug 745975 Opened 10 years ago Closed 9 years ago

include list of locales in properties of release repack builders

Categories

(Release Engineering :: Release Automation: Other, defect, P3)

defect

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: whimboo, Assigned: bhearsum)

References

Details

Attachments

(1 file)

As Chris told me on IRC the build machines for repacks of beta and release candidate builds are sending out messages for a bulk of locale repacks.

To allow tools to react on individual l10n repacks, notifications for each locale have to be sent out. Those should also include the locale as property. That's something which I'm still missing in the current notification which makes it kinda useless for consumers and our testing.
As Chris mentioned on IRC it's probably easier for now to add a 'locales' property which contains all the locales which have been processed by the repack. But at the end individual notifications would be highly wanted.
Here some steps from Chris via IRC how to get the multiple locales property included in teh notification:

15:24 catlee-away: our release repacks use ScriptFactory 
15:24 catlee-away: http://hg.mozilla.org/build/buildbotcustom/file/f40fec804a5e/process/factory.py#l8541
15:25 catlee-away: which basically means that the buildbot code checks out the tools repo and runs a script
15:25 catlee-away: the nice part is that afterwards it looks in the 'properties' directory for properties
15:26 catlee-away: so if your script outputs files into the properties dir with contents like
15:26 catlee-away: locales:ab-CD,ed-GH
15:26 catlee-away: then you'll get that added as a buildbot property
15:27 catlee-away: the factory is created here: http://hg.mozilla.org/build/buildbotcustom/file/f40fec804a5e/process/release.py#l742
15:27 catlee-away: so you can see that it's running the scripts/l10n/release_repacks.sh script
15:27 catlee-away: which is
15:27 catlee-away: http://hg.mozilla.org/build/tools/file/3a5375ef6c58/scripts/l10n/release_repacks.sh
15:27 catlee-away: whcih does some stuff and then calls http://hg.mozilla.org/build/tools/file/3a5375ef6c58/scripts/l10n/create-release-repacks.py
15:28 catlee-away: so I think all you need to do is make http://hg.mozilla.org/build/tools/file/3a5375ef6c58/scripts/l10n/create-release-repacks.py create a properties file
15:29 catlee-away: right here: http://hg.mozilla.org/build/tools/file/3a5375ef6c58/scripts/l10n/create-release-repacks.py#l190
Not a high priority.
Priority: -- → P3
We are blocked on this for our testing of locales for beta and release candidate builds of Firefox. Can we get some action from the RelEng side on it?
Whiteboard: [automation-blocked]
The bug title and previous comments seem to be asking for different things. We could set things up such that the locales for each repack chunk are known at the end - basically what's in comment #2. Having per-locale notifications would require a lot more work though. Is the former good enough, Henrik?
My comment 1 still stands. So yes, it would be a workaround and we would have to process that on our side. We may be able to use pulsetranslator to create the 1:m mapping for each notification for now. But at the end I would love to see individual notifications.

If getting a locales property added is simple enough, lets get this done. Thanks!
OK, I think I can make this happen before the next beta...
Assignee: nobody → bhearsum
I tested this in staging w/ the actual repack work commented out to save time. I ended up with a "locales" property with the value "de,gh,id,zh-TW,zu" when I tested in a standalone repack builder.
Attachment #724909 - Flags: review?(rail)
Comment on attachment 724909 [details] [diff] [review]
add 'locales' property to repack jobs

ScriptFactory ftw! :)
Attachment #724909 - Flags: review?(rail) → review+
Summary: Repack builders for Firefox candidate builds (beta/release) should send out notifications for individual locales → include list of locales in properties of release repack builders
Attachment #724909 - Flags: checked-in+
Henrik, this should be fixed for 20.0b6. For the sake of organization, can we file a new bug to track individual locale notifications?
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
(In reply to Ben Hearsum [:bhearsum] from comment #10)
> Henrik, this should be fixed for 20.0b6. For the sake of organization, can
> we file a new bug to track individual locale notifications?

Thanks a lot for the quick fix. I will be away the next week but will implement it as soon as I'm back. Regarding the new bug, can't we make use of bug 745968 which was the other half of our issue?
(In reply to Henrik Skupin (:whimboo) [away 03/15 - 03/22] from comment #11)
> (In reply to Ben Hearsum [:bhearsum] from comment #10)
> > Henrik, this should be fixed for 20.0b6. For the sake of organization, can
> > we file a new bug to track individual locale notifications?
> 
> Thanks a lot for the quick fix. I will be away the next week but will
> implement it as soon as I'm back. Regarding the new bug, can't we make use
> of bug 745968 which was the other half of our issue?

Bug 745968 is a completely different issue. Adding individual locale notifications is a major refactor, bug 745968 is a pretty simple rename, per my comment in that bug.
Had to land a bustage fix for this. The "properties" dir was getting clobbered before we could write to it: https://hg.mozilla.org/build/tools/rev/d8992a4ed2b2
Blocks: 849351
Whiteboard: [automation-blocked]
Blocks: 857971
Ben, is there a way for you to create such a notification for me? I would kinda appreciate that, so that I can implement the logic in pulsetranslator. Thanks.
Flags: needinfo?(bhearsum)
Blocks: 857980
(In reply to Henrik Skupin (:whimboo) from comment #14)
> Ben, is there a way for you to create such a notification for me? I would
> kinda appreciate that, so that I can implement the logic in pulsetranslator.
> Thanks.

Create what kind of notification? "locales" is a property in the release repack jobs since this bug was fixed. That's what we agreed to do in comments #5 and 6. I see you filed bug 857971 for individual locale notifications, but that's not trivial to fix and unlikely to get addressed soon.
Flags: needinfo?(bhearsum)
(In reply to Ben Hearsum [:bhearsum] from comment #15)
> Create what kind of notification? "locales" is a property in the release
> repack jobs since this bug was fixed. That's what we agreed to do in
> comments #5 and 6.

A pulse message which contains the data added by this patch.
(In reply to Henrik Skupin (:whimboo) from comment #16)
> (In reply to Ben Hearsum [:bhearsum] from comment #15)
> > Create what kind of notification? "locales" is a property in the release
> > repack jobs since this bug was fixed. That's what we agreed to do in
> > comments #5 and 6.
> 
> A pulse message which contains the data added by this patch.

The existing pulse messages when the release repack builders finish contain this information in their properties AFAIK.
(In reply to Ben Hearsum [:bhearsum] from comment #17)
> The existing pulse messages when the release repack builders finish contain
> this information in their properties AFAIK.

Correct. But I would like to have an example. I don't want to wait until the next messages will be send out for the next beta release candidate next week. Reading your code it should look like:

locales: [it,de,fr]

But given that I never stored such a pulse message I also wonder if the routing key is different to the en-US version. I'm not sure at the moment for what I have to look for. That's why I would really appreciate an pulse message as example. Or aren't you able to force the creation of one? If that's the case are such messages stored somewhere?
(In reply to Henrik Skupin (:whimboo) from comment #18)
> (In reply to Ben Hearsum [:bhearsum] from comment #17)
> > The existing pulse messages when the release repack builders finish contain
> > this information in their properties AFAIK.
> 
> Correct. But I would like to have an example. I don't want to wait until the
> next messages will be send out for the next beta release candidate next
> week. Reading your code it should look like:
> 
> locales: [it,de,fr]
> 
> But given that I never stored such a pulse message I also wonder if the
> routing key is different to the en-US version. I'm not sure at the moment
> for what I have to look for. That's why I would really appreciate an pulse
> message as example. Or aren't you able to force the creation of one? If
> that's the case are such messages stored somewhere?

You probably want to watch any routing keys that match:
build.release-*_repack*.*.finished

For example, release-mozilla-beta-linux_repack_4/6
Thanks. As discussed on IRC I will let the client run for 'build\.release-.*repack.*\.finished' until Monday when the next beta candidate builds are expected.
Verified fixed by the latest notifications for beta candidate builds which contain:

["locales", "fi,fr,fy-NL,ga-IE,gd,gl,gu-IN,he,hi-IN,hr,hu,hy-AM,id,is,it", "SetProperty Step"]
Status: RESOLVED → VERIFIED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.