1440671 - Please deploy buildhub lambda function 1.1.5

Status: RESOLVED FIXED

(Cloud Services :: Operations: Kinto, task)

Description

[Mathieu Leplatre [:leplatrem]]

Could you please update the lambda function for Buildhub with the following one?

Bug fixes:

- Now pick Windows .exe archives only (fixes mozilla-services/buildhub#338)

https://github.com/mozilla-services/buildhub/releases/tag/v1.1.5

https://github.com/mozilla-services/buildhub/releases/download/v1.1.5/buildhub-lambda-v1.1.5.zip

Thanks!

Comment 1

[Mathieu Leplatre [:leplatrem]]

Hi Wei! Any chance we can deploy this please? I'd like to reduce the lambda alerts to the strict minimum and this fixes the conflict error for Windows archives.
Let me know if I can help you :)
Thanks!

Comment 2

[:wezhou]

Yep, it's deployed.

Comment 3

[Mathieu Leplatre [:leplatrem]]

Wei, we received lot of lambda errors last night (and no Sentry). Could you please share some logs please?

Comment 4

[:wezhou]

Hi Mathieu, do you have a time range and keyword you'd like me to check in particular?

Between 2:00 and 4:00am PST, I see errors like this in the log,

> [ERROR] 2018-02-28T10:48:41.827Z e36f5b76-1c74-11e8-af7d-81c48254c370 Backing off fetch_json(...) for 1.7s (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found')

In fact, there were so many that at a point it even reached Sentry's rate limit.

For example there were tens of thousands of the below exception during the above said hours,

https://sentry.prod.mozaws.net/operations/buildhub-prod/issues/1594362/


Also, when you say you received lot of lambda errors, where did you receive them, can you forward one of them for me to check as well?

Comment 5

[:wezhou]

Mathieu, I just emailed you a link to download the lambda logs between 2:00 and 4:00 am PST.

Comment 6

[Ethan Glasser-Camp (:glasserc)]

I tried to poke through these logs but I'm not really sure what I'm looking for. It seems like the only actual error comes from getting 404s from S3. In particular, these all fail:

1649c055e48f496eb4eb5783749df6f7-6.log:1519815826664 GET 'https://archive.mozilla.org/pub/mobile/nightly/2018/02/2018-02-28-10-03-14-mozilla-central-android-x86/fennec-60.0a1.multi.android-i386.json'

8255c79373884cc58da5841c9f8d30c0-6.log:1519815396912 GET 'https://archive.mozilla.org/pub/mobile/nightly/2018/02/2018-02-28-10-03-14-mozilla-central-android-aarch64-l10n/'

91b1fd54bf70451bbc4d286faae35c28-1.log:1519827151933 GET 'https://archive.mozilla.org/pub/thunderbird/candidates/59.0b2-candidates/build2/linux-i686/en-US/thunderbird-59.0b2.json'

982787b2155243e2bf9ac13a1870e383-1.log:1519814938068 GET 'https://archive.mozilla.org/pub/mobile/nightly/2018/02/2018-02-28-10-03-14-mozilla-central-android-api-16-old-id-l10n/'

a640482872494697b755ecdf03b17dd0-296.log:1519815840619 GET 'https://archive.mozilla.org/pub/mobile/nightly/2018/02/2018-02-28-10-03-14-mozilla-central-android-x86/fennec-60.0a1.multi.android-i386.json'

be0b8cb301f1442cb6baeeead8d949c8-12.log:1519816517961 GET 'https://archive.mozilla.org/pub/mobile/nightly/2018/02/2018-02-28-10-03-14-mozilla-central-android-x86-old-id-l10n/'

e3a26c049bd64b00a3921684562e0532-96.log:1519815687040 GET 'https://archive.mozilla.org/pub/mobile/nightly/2018/02/2018-02-28-10-03-14-mozilla-central-android-x86/fennec-60.0a1.multi.android-i386.json'

e52f0c6e09c54098affaafef84561778-2.log:1519815967799 GET 'https://archive.mozilla.org/pub/firefox/nightly/2018/02/2018-02-28-10-01-10-mozilla-central-l10n/'

f58286f0da044304821a70ea8e629d2a-40.log:1519815791348 GET 'https://archive.mozilla.org/pub/mobile/nightly/2018/02/2018-02-28-10-03-14-mozilla-central-android-x86/en-US/fennec-60.0a1.en-US.android-i386.json'

The timestamps range from around 10:45 UTC to around 11:15 UTC. Except for the Thunderbird file, it seems like we're mostly looking at Android/Fennec stuff? What to make of this?

Comment 7

[:wezhou]

Looking at https://sentry.prod.mozaws.net/operations/buildhub-prod/ there does look like there were excessive amount of 404 errors. Because there are so many, I bet there were lots of them got rate limited by Sentry too.

However I can wget at least one of the urls in comment #6, so I wonder if it is because the lambda tried to download the files when they had not been existing or something.

Current lambda version running on -prod is v1.1.5 if that makes any difference.