485584 - moz2-master on production master hanging hanging when /waterfall is attempted to be loaded

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Starting just over an hour ago every time someone attempted to load http://production-master.build.mozilla.org:8010/waterfall, the master would seize up completely: no output in twistd.log, no new builds starting, nothing. Every 12 minutes or so we get a bit of output in the log, and then it seizes again (presumably because somebody refreshed /waterfall).

Comment 1

[[e-mail] (:bhearsum)]

catlee attached strace to Buildbot and noticed a pattern of mmap2() calls with an increasing size...don't know what it's mmap2'ing but presumably one of the logs or other resources the waterfall needs...

Comment 2

[[e-mail] (:bhearsum)]

We're going to try a fresh Buildbot master without _any_ of the build or change history to test the /waterfall theory, and hopefully get things going again.

Comment 3

[[e-mail] (:bhearsum)]

Strating anew fixed the problem for the time being...this will bite us again though, possibly as soon as the next round of l10n builds come out.

Comment 4

[Chris AtLee [:catlee]]

I suspect this is due to how buildbot constucts the html for the waterfall.  It uses a series of "body += newstring" instead of constructing a list of strings and using "".join() afterwards.

The old production waterfall is gone right now, but the staging waterfall is sitting around 600k in size.  This means that when buildbot is close to finishing constructing the html for the waterfall, it's copying the same 600k of data over and over.

I'll come up with a patch to reduce the number of string concatenations in the waterfall to see if it helps any.

Comment 5

[[e-mail] (:bhearsum)]

I've saved the old moz2-master files in /builds/buildbot/moz2-master-old, we can have a look at them if we want to try and reproduce (probably want to move them to a different VM, though).

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to comment #4) 
> The old production waterfall is gone right now, but the staging waterfall is
> sitting around 600k in size.  This means that when buildbot is close to
> finishing constructing the html for the waterfall, it's copying the same 600k
> of data over and over.

Man, the sheer quantity of l10n builds must be absolutely hosing this. Here's hoping the "".join() helps.

Comment 7

[Chris AtLee [:catlee]]

Attachment: Make html generation more efficient

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 369755 [details] [diff] [review]
Make html generation more efficient

hope this helps!

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 369755 [details] [diff] [review]
Make html generation more efficient

changeset:   11:eccc76210c86

Comment 10

[alice nodelman [:alice] [:anode]]

So, several people seem to be working on this - can somebody grab the bug and assign it to themselves for book keeping purposes?

Comment 11

[alice nodelman [:alice] [:anode]]

Nobody has any current plans for this bug, though the problem is not entirely fixed.  Moving to Future.

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

If it becomes a blocker again, we'd obviously move it from Future and work on it immediately. Meanwhile, reducing priority.

Comment 13

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

afaict, this was really a bug about excessive load on production-master. Is this something we can close, now that we have split load across production-master, production-master02?

Comment 14

[[e-mail] (:bhearsum)]

(In reply to comment #13)
> afaict, this was really a bug about excessive load on production-master. Is
> this something we can close, now that we have split load across
> production-master, production-master02?
Yes.

Comment 15

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.