1546698 - Content of text attachments sometimes truncated

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect, P2)

Description

[Alex Thayer [:alexical] (she/her)]

STR:

• Load the attachment at https://bug1544298.bmoattachments.org/attachment.cgi?id=9058642
• Scroll to the bottom. It should end with three lines containing "Thread Exit"
• If it doesn't, stop - that's the bug. If it does, refresh until it doesn't

Expected results:

The file should always end with the three "Thread Exit" lines

Actual results:

The file occasionally ends in seemingly random places

Comment 1

[Kohei Yoshino [:kohei]]

Seems a variant of Bug 1538047, but the file in question is served as text/plain; charset=UTF-8. Maybe different.

Comment 2

[Dylan Hardison [:dylan] (he/him)]

The encoding of this is screwy. I'm going to make this directly use the mojolicious APIs which should fix the problem

Comment 3

[Dylan Hardison [:dylan] (he/him)]

Attachment: junk test

Comment 4

[Dylan Hardison [:dylan] (he/him)]

Attachment: more junk

Comment 5

[Dylan Hardison [:dylan] (he/him)]

This was fixed via an nginx configuration change, and it seems to have been a low-level kernel or network problem. The nginx fix is to enabling proxy buffering so that it is able to write the response fast enough to the ELB before the ELB "gets bored" and closes the connection.

Comment 6

[Kohei Yoshino [:kohei]]

This is still happening to me.

Comment 8

[Dylan Hardison [:dylan] (he/him)]

I bet the configuration on the nginx side got reverted.

The thing the buffers requests must be turned on in nginx, or else gzip must be disabled. Either one will work.

Comment 9

[Kohei Yoshino [:kohei]]

So :ckolos can help configure nginx once again?

Comment 10

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

buffers are turned on.

grep proxy_buffer nginx_conf.erb 
    proxy_buffer_size 16k;
    proxy_buffers 4 16k;
        proxy_buffering on;
    proxy_buffer_size 16k;
    proxy_buffers 4 16k;
        proxy_buffering on;

:dylan, I seem to recall something about the proxy_http_version influencing this. It's currently set to 1.1; could that be the cause?

Comment 11

[Dylan Hardison [:dylan] (he/him)]

http 1.1 is fine. I would suggest turning off gzip until you can switch to something that actually properly implements HTTP, since ELBs clearly do not. :-)

Comment 12

[Dylan Hardison [:dylan] (he/him)]

well, a little less snarky. Basically this happens if you take too long writing chunked encoding to ELBs. Previously the buffering was enough of a speed gain to make that work, and I suspect now it's not. What the ELBs are supposed to do when not given a final chunk is to insert one themselves, but they're not doing this either. Note my chunky-dumps script (check slack history, it's on dockerhub) can help debug this a bit. nginx itself behaves fine, but the ELB "gets bored" and never sends the 0\r\n chunk

Comment 13

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

:kohei, can you confirm that this is no longer an issue?

Comment 14

[Kohei Yoshino [:kohei]]

Yeah, I can normally load the attachments linked from the Comment 0 and Bug 1559970 as well as dylan’s 2 test attachments here. Seems solved.