756532 - Clobberer fails to work if you try to clobber more than 1024 things at once, like all of mozilla-inbound

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

Using either:
https://build.mozilla.org/clobberer/?branch=mozilla-central
or
https://build.mozilla.org/clobberer/?branch=mozilla-inbound

If you try to clobber everything at once (the master check box in the left hand column) and submit the form; the clobberer page that is returned, shows a blank "last clobbered" column, rather than the current date/time and LDAP login used.

Presumably due to the POST data exceeding a certain size.

Comment 1

[Ed Morley [:emorley]]

Sorry, intended to add:
Works fine if you only check a subset of the boxes on each submit.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

heh, not sure why I'm copied here -- I'm definitely not the one who will be rewriting clobberer to be smarter :)

Comment 3

[Phil Ringnalda (:philor)]

Oh, failed to actually type the comment which went with the cc, which was suspicion that it was something in front of clobberer in its new home, which thinks that n bytes of postdata is fine, but m bytes is an attack that must be thwarted.

Comment 4

[Phil Ringnalda (:philor)]

Comment 0 misleadingly does cast the blame on clobberer itself, talking about returning a blank last clobbered column, when in fact that was just what the tree we were clobbering at the time had. The actual "Actual Result" is that the return from the post is exactly what the page looked like before, just like you didn't send a thing.

Comment 5

[Ed Morley [:emorley]]

Sorry, I should have made that more explicit.

Comment 6

[Ed Morley [:emorley]]

Dustin, any ideas given comment 3? :-)

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, yes - but I thought clobberer used POST?  Could we change it to POST instead?

Comment 8

[Phil Ringnalda (:philor)]

It does POST - I wasn't worried about GET URL length so much as the multiple layers of things limiting total request size, starting with PHP's post_max_size out through Apache's LimitRequestBody and all the firewalls and load balancers in front of Apache that might also limit request size.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

I don't see any such settings in Zeus.  Jake, is there such a thing?

Comment 10

[Jake Maul [:jakem]]

I'm not aware of any such setting in Zeus. It might be possible to do something like this in a Service Protection Class (I don't know), but there are none enabled for the virtual servers handling this domain (build.mozilla.org).

How big is the incoming POST data?

post_max_size = 8M

LimitRequestBody is unset, so it should default to 0 (unlimited)

Comment 11

[Phil Ringnalda (:philor)]

Sigh. It's tiny, just a pair of "slave-100125: on" for each builder listed, so I was going to estimate how big it was by counting the rows. That would be (currently, not at the time this was filed, because things get pruned out) 761 pairs for mozilla-central, which currently doesn't hit this (sorry, we'll be having clobbered builds all day), and in a very obvious pointer to the nature of the problem in clobberer's PHP somewhere, 1031 pairs for mozilla-inbound, which still hits this.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Smells like running out of parameters in the mysql query to me.

Comment 13

[Ed Morley [:emorley]]

Is there anyone who can drive this forwards? This is impacting me (and others) pretty often.

Comment 14

[Ed Morley [:emorley]]

Ping?

Something like bug 614262 would also fix this.

Comment 15

[Armen [:armenzg]]

Setting to "major" as it does not "block" any trees or releases.

I have no idea about clobberer so I will do my best to figure this out.

Comment 16

[Armen [:armenzg]]

jhopkins volunteered for this.

Comment 17

[John Hopkins (:jhopkins)]

Attachment: proposed patch

Test URL: https://build.mozilla.org/stage-clobberer/index-test.php

Comment 18

[Ed Morley [:emorley]]

<3 jhopkins :-D

Comment 19

[John Hopkins (:jhopkins)]

Comment on attachment 637546 [details] [diff] [review]
proposed patch

armenzg r+'d via IRC

Comment 20

[John Hopkins (:jhopkins)]

Comment on attachment 637546 [details] [diff] [review]
proposed patch

Landed in http://hg.mozilla.org/build/tools/rev/c80a6efb18bb and deployed to production.

Comment 21

[John Hopkins (:jhopkins)]

mozilla-inbound clobbered ok.

Comment 22

[Ed Morley [:emorley]]

Thank you :-D