1176828 - Build messages for l10n repacks are missing buildid

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P5)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The last repack notifications we got for a beta/release build which contained the build id is from Dec 19th. Since then the appropriate property is empty at least in the normalized message as sent out by pulsetranslator.

I don't know if that is a pulsetranslator regression or from buildbot_custom. At least for the former I was not able to see a related change around that time. But not sure if anything else happened at this time for buildbot. So maybe Ben or Armen might know.

Jonathan, do have an archive of received messages? If yes it would be good to have an example of the original repack notification. Given that I do not have access to the pulsetranslator production machine or even don't know where it is located those days, could you have a look at it please? Thanks.

Comment 1

[Jonathan Griffin (:jgriffin)]

Pusletranslator doesn't keep an archive of old messages.  I think catlee was doing that at some point; catlee, does this pulse message archive still exist?

Henrik, what's the routing key for the relevant messages?  I can look and see what we're getting from buildbot now.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The routing key is e.g. "build.release-mozilla-release-linux64_repack_3.0.log_uploaded". Here an example of its content in the normalized view:

{"locale": "en-GB", "testsurl": null, "previous_buildid": null, "job_number": 0, "build_number": 1, "builddate": null, "buildername": "release-mozilla-release-linux64_repack_3/10", "platform": "linux64", "version": "38.0.6", "revision": null, "status": 0, "buildtype": "opt", "product": "firefox", "slave": "bld-linux64-spot-314", "tags": [], "buildid": null, "timestamp": "2015-06-05T19:06:41Z", "key": "build.release-mozilla-release-linux64_repack_3.0.log_uploaded", "locales": "dsb,el,en-GB,en-ZA,eo,es-AR,es-CL,es-ES,es-MX", "logurl": "http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/38.0.6-candidates/build1/logs/release-mozilla-release-linux64_repack_3-bm74-build1-build0.txt.gz", "repack": true, "tree": "release-mozilla-release", "buildurl": null, "release": "805560a3fe33"}

Chris doesn't seem to run this archive anymore. At least no builds are shown anymore:
http://people.mozilla.org/~catlee/mozilla-releng-pulsedata/

Comment 3

[[e-mail] (:bhearsum)]

I'm pretty sure this is a regression from bug 607392.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #3)
> I'm pretty sure this is a regression from bug 607392.

To expand on this: I don't see buildid as property in any repack jobs from this year. bug 607392 landed very shortly after Dec 19th, and it touched the way l10n repacks for releases are triggered. IIRC, we had to do a follow-up fix for it too because the buildid was no longer passed as a property...we might be able to get the repack script to set the property instead to fix this.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Ben Hearsum [:bhearsum] from comment #4)
> To expand on this: I don't see buildid as property in any repack jobs from
> this year. bug 607392 landed very shortly after Dec 19th, and it touched the

That's right. I haven't seen this earlier because there was no need for me all the last months to investigate a problem with pulse. Now that I had to do it, I have seen this glitch because we cache notifications locally and all file names started with None instead of the buildid.

Thanks Ben for taking care of it!

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I think we can remove the ni? from Chris.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) [away 07/01 - 07/31] from comment #5)
> (In reply to Ben Hearsum [:bhearsum] from comment #4)
> > To expand on this: I don't see buildid as property in any repack jobs from
> > this year. bug 607392 landed very shortly after Dec 19th, and it touched the
> 
> That's right. I haven't seen this earlier because there was no need for me
> all the last months to investigate a problem with pulse. Now that I had to
> do it, I have seen this glitch because we cache notifications locally and
> all file names started with None instead of the buildid.
> 
> Thanks Ben for taking care of it!

To be clear: I didn't say that I was necessarily going to find a fix, I just dug into what caused it.

Why wasn't this reported 6 months ago when the regression was introduced?

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For testing release builds we do not need the build id right now, and given what I said above it simply wasn't observed earlier.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) [away 07/01 - 07/31] from comment #8)
> For testing release builds we do not need the build id right now, and given
> what I said above it simply wasn't observed earlier.

OK, so what I'm reading is that this isn't actively hurting anything? If that's the case, I don't think we should do anything about this at the moment. We've got a big release automation project going on this quarter (bug 1118794), and that work is likely to fix this as a side effect.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Jepp, that's totally fine.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Rail, do you know if that issue has been fixed meanwhile in release promotion?

Comment 12

[Rail Aliiev [:rail]]

No, we don't set that property. It shouldn't be hard to add if needed.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It would help us a lot in case we have to retrigger all the tests for a specific buildid. We had this situation a couple of times but weren't able to do it due to the missing information. The current workaround is to use the timestamp but that gives us different values for each locale repack. :(

Comment 14

[Rail Aliiev [:rail]]

Do we still care about the build ids? It's not hard to expose them via buildbot properties, but considering TC it's not obvious the value of this in the future...

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I think once our tests are part of the big nightly graph and we don't have to trigger them via mozmill-ci, it would be less a problem. It's mainly used for logging purposes right now I think.

Comment 16

[Aki Sasaki (not active)]

Release l10n is now in TC for Fx59+.