1415268 - Internal Server Error while trying to set up an ESR build

Status: RESOLVED FIXED

(Release Engineering :: Applications: Shipit, defect)

Description

[Liz Henry (:lizzard) (relman/hg->git project)]

I tried twice to set up a build for ESR 52.5.0 (build 2) but got an internal server error. 

Here's the error message: "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application."

Comment 1

[Liz Henry (:lizzard) (relman/hg->git project)]

jlorenzo pointed out this is from a very long list of suggested partial builds.

Comment 2

[Johan Lorenzo [:jlorenzo]]

:jcristau and I investigated this issue earlier today. Leaving a NI to myself so I write down the results and I track this issue for ESR 52.6.0 (in 6 weeks)

Comment 3

[Johan Lorenzo [:jlorenzo]]

There are 2 issues here:
1. The server should refuse input that contain more than X partials. X to be determined.
2. The string comparison done at [1] is not valid anymore. It compares 52.1.0esr to 52.1.0. I don't understand how the behavior has changed since ESR 52.4.0.

Marking it as blocking the next merge day (bug 1412962).

[1] https://github.com/mozilla-releng/ship-it/blob/23042ec72ccf4242e31545b99800ca311c2af93a/kickoff/static/suggestions.js#L120

Comment 4

[Johan Lorenzo [:jlorenzo]]

Attachment: Ship-it PR - Make error explicit on the server side

Fixes issue #1.

Comment 5

[Johan Lorenzo [:jlorenzo]]

Attachment: Ship-it PR - Prevent front end to add to many partials

Fixes issue #2.

Comment 6

[Johan Lorenzo [:jlorenzo]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #3)
> I don't understand how the behavior has changed since ESR 52.4.0.

I now do \o/. TL;DR: That's because we shipped 52.4.1 to MacOS users only.


All partials used here[1] are taken from partial.json[2]. This file is overwritten from times to times. Most importantly, it's ordered by number of ADIs. We shipped 52.4.1 to Mac users only[3], leaving the rest of the population on 52.4.0. Hence, partial.json was ordered this way:

> {
>   "esr": [{
>     "version": "52.4.0esr",
>     "ADI": 40000
>   }, {
>     "version": "52.3.0esr",
>     "ADI": 30000
>   }, {
>     "version": "52.4.1esr",
>     "ADI": 20000
>   }]
> }

(Don't mind the numbers, I replaced them by dummy ones)

So populatePartial() first added 52.4.1 because it's the last shipped release[4], then added the topmost release in partial.json (which was 52.4.0), but couldn't exit the loop, because of this strict equal condition.

Attachment 8929492 [details] will fix this. I added a new test scenario [7] to validate the theory and the fix.

[1] https://github.com/mozilla-releng/ship-it/blob/9c6ecbac45fe7f5e8f1a4f5bbde43361954a3847/kickoff/static/suggestions.js#L174
[2] https://ship-it.mozilla.org/cron/partial.json
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1404481#c7
[4] https://github.com/mozilla-releng/ship-it/blob/9c6ecbac45fe7f5e8f1a4f5bbde43361954a3847/kickoff/static/suggestions.js#L205
[5] https://github.com/mozilla-releng/ship-it/blob/9c6ecbac45fe7f5e8f1a4f5bbde43361954a3847/kickoff/static/suggestions.js#L228
[6] https://github.com/mozilla-releng/ship-it/blob/9c6ecbac45fe7f5e8f1a4f5bbde43361954a3847/kickoff/static/suggestions.js#L232
[7] https://github.com/mozilla-releng/ship-it/pull/207/commits/1dc08a10be6a018819927c22dec2167a489dc1aa

Comment 7

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8929492 [details] [review]
Ship-it PR - Prevent front end to add to many partials

Was r+'d at https://github.com/mozilla-releng/ship-it/pull/207#pullrequestreview-77993670.

Comment 8

[Johan Lorenzo [:jlorenzo]]

Landed both fixes at:
* https://github.com/mozilla-releng/ship-it/commit/18dc35c511c2d38a1f7472c34c435176f3807212
* https://github.com/mozilla-releng/ship-it/commit/c62574340c1688f659dadc22f9b0a73beaa35854

I deployed and manually tested them to prod. Crafting 52.5.0 (build3) was still an issue in prod, and it's no more.