1105356 - [pulsetranslator] Missing normalized build notifications for Firefox 34.0.5 release across platforms and locales

Status: RESOLVED DUPLICATE of bug 1094272

(Webtools :: Pulse, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We got the Firefox 34.0 and 34.0.5 candidate builds today. While for the 34.0 release, the build notifications have been sent to the pulse exchange, we haven't gotten any notification for the 34.0.5 release across all platforms and locales. I talked to Mark and there were even no listed failures for pulsetranslator.

So could there be any code path we hit here, which caused the notifications not to be sent out?

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks like we are getting those notifications. But something is wrong with those. I fetched one while being connected to the pulse inspector. I will follow-up with more details in a bit.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As seen by the Pulse inspector the notifications have been sent out. I noticed at least two of them for repacks. Thing is that they were not processed by pulsetranslator! Using such a message locally and pushing it through pulsetranslator works fine. Also the normalized message is correctly handled by pulsebuildmonitor and mozmill-ci.

So this is clearly a problem with pulsetranslator. Moving bug into the right component.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So Mark has a contributor who could create a tool, which listens on both exchanges and compares the build messages. That way we can find out where those message get lost or corrupted. Also he is setting up a little script locally to store all the received messages from the build exchange. We could check later if we missed builds.

Btw. this also affects the 34.1 builds! So two complete releases are missing!

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mark, maybe this is also network related again and depend on bug 1083166? Probably it was not only SSL, which is causing this behavior?

Sadly I cannot find anything helpful for those releases in the uploaded logs:
http://people.mozilla.org/~mcote/pulsetranslator_logs/

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Actually 34.1 was a special configuration in releng they did before the build, and didn't wanted to change this back to 34.0.5 to cause bustage. So it's really "only" the complete 34.0.5 release we are missing.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mark, any update for the script you wanted to get started to listen for the build messages from the build exchange?

Comment 7

[Mark Côté [:mcote]]

I have a version of pulsetranslator now publishing to pulse-dev.  It's not using SSL at the moment, since I didn't want the queue to grow too big if it got stuck, but at least it's set up now.  It's easy to flip over to SSL, so I'll coordinate some testing with cyliang, probably the week after next.

Follow bug 1104911 for the test app.  Probably won't take Ricardo too long, as long as I have time to help and do reviews next week at all. :)

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We disabled SSL and all is working fine for now. So no need to keep this bug as blocker for us.

Comment 9

[Mark Côté [:mcote]]

We determined that SSL was, for some reason, the culprit a long ago, so duping.  This should be fixed with the Pulse and pulsetranslator migrations anyway.