994767 - Problem parsing generator response - stage review instance

Status: RESOLVED FIXED

(Marketplace Graveyard :: Integration, defect)

Description

[Austin King [:ozten]]

Running some load on https://apk-controller-review.stage.mozaws.net

I'm seeing this error in the kibana logs:

    Problem parsing generator response

Comment 1

[Austin King [:ozten]]

I've run 5 apps that are 504 on stage review through my local step and they are all 200.

I've run 5 apps that are { [Error: socket hang up] code: 'ECONNRESET' }   on stage review through my local step and those are also 200.

Comment 2

[Austin King [:ozten]]

I've disabled the distributed build queue fix from Bug#976170.

I believe builds were getting stuck in a wait state, which was causing errors.

Now only seeing occasional 400s and mostly 200s on review.
Almost all 200s on release.

Comment 3

[Austin King [:ozten]]

too busy module was triggering some of our errors, bumped settings on that.

Overhauled how we log exceptions and errors to fix the empty error logged to Kibana.

Put in an explicit JSON.stringify instead of relying on the express framework.

Deploying to stage (takes about 17 minutes).

Comment 4

[Austin King [:ozten]]

The too busy middleware was returning plain text instead of JSON.

Fixed in 9e91ac2.