978854 - exception in getLocaleOrTopLevelParam since latest balrog push to prod

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

Description

[[e-mail] (:bhearsum)]

I noticed this on Sentry this morning:
https://errormill.mozilla.org/Releng/balrog-prod/group/165917/

These exceptions are being hit when clients are making requests to URLs like:
https://aus4.mozilla.org/update/3/Firefox/27.0a2/20131103004005/Darwin_x86_64-gcc3/sv-SE/aurora/Darwin%2013.1.0/default/default/update.xml

getLocaleOrTopLevelParam hasn't changed in ages, but https://github.com/mozilla/balrog/commit/b0f9f909567f481289e59eaf706c07bb347a4924 added a new call to it.

I suspect that this is happening because some older blobs don't have extv set, but I need to dig into it more still.

Comment 1

[[e-mail] (:bhearsum)]

For the Mac requests we're definitely hitting this because Darwin_x86_64-gcc3 is the build target - which doesn't exist in the blob (should be Darwin_x86{,_64}-gcc3-u-i386-x86_64. I don't understand why we'd be getting requests for that.

There's also some Windows requests that are failing for a reason I don't understand. The ones I've looked at have an extv defined for the build target/locale in the request.

We should probably handle the case where we can't find the thing we want in the locale or top level better, but I'd also like to debug this a bit more to see if there's something else going wrong.

Comment 2

[[e-mail] (:bhearsum)]

Not really a surprise, but we're also seeing this for some old Fennec URLs:
https://aus4.mozilla.org/update/4/Fennec/28.0a2/20140131004003/Android_arm-eabi-gcc3/de/aurora/4.2.2/default/default/28.0a2/update.xml

Comment 3

[[e-mail] (:bhearsum)]

So, the Windows update that I was looking at yesterday was https://aus4.mozilla.org/update/3/Firefox/28.0a1/20131107030200/WINNT_x86-msvc/en-US/nightly/Windows_NT%205.1.0.0%20%28x86%29/default/default/update.xml

"Windows_NT 5.1.0" is one of the versions that we block, so the error in this case is coming from not being able to find an "extv" in the No-Update blob.

And looking at the Fennec one I see that it's for a 'de' build. We only serve updates for 'en-US' (which is really the multilocale build), so that's why we're getting a traceback.

So, all of the cases I've looked at are cases where we should be serving an empty update (which we are). The only thing to do here is stop the tracebacks from happening.

I am still puzzled about what's generating the original Darwin with ("Darwin_x86_64-gcc3" as the build target), but I don't think that needs to stop us from fixing this.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: catch KeyErrors

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8385383 [details] [diff] [review]
catch KeyErrors

Looks like malformed blobs sidestep the new safeties, lets not offer an update instead.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #5)
> Comment on attachment 8385383 [details] [diff] [review]
> catch KeyErrors
> 
> Looks like malformed blobs sidestep the new safeties, lets not offer an
> update instead.

Oh, good point. My brain was broken and thought that the release was the incoming one, which may not exist. Deeeerp.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: don't serve an update when mapping is missing extv; add test

Comment 8

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/ecd6b83679165cb869445017b22474a5746869ac
bug 978854: exception in getLocaleOrTopLevelParam since latest balrog push to prod. r=nthomas

Comment 9

[[e-mail] (:bhearsum)]

Should be fixed now. I'm going to resolve the exception on Sentry and see if it comes back.