652151 - clobberer form submission is broken

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

I tried setting the clobberer and when it reloaded, it didn't list me as having set the clobberer.  I am on build-vpn, so i will try again on a non-buildvpn connected machine.

This is blocking 652127 because the alternative is untenable.

Comment 1

[Phil Ringnalda (:philor)]

FWIW, I'm now up to nine tries on a two non-buildvpn connections, but you make it sound like you only tried once, which surprises me: when it is actually working, it always takes me either two or three tries before it shows anything. I think the only time it ever worked first time was right after bhearsum completely wiped out the database.

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #1)
> FWIW, I'm now up to nine tries on a two non-buildvpn connections, but you make
> it sound like you only tried once, which surprises me: when it is actually
> working, it always takes me either two or three tries before it shows anything.
> I think the only time it ever worked first time was right after bhearsum
> completely wiped out the database.

Phil, don't worry for now.  There are errors from the php page that aren't getting through to the UI :(

The error on form submission is:

[Fri Apr 22 11:07:24 2011] [error] [client <snip>] PHP Notice:  Object of class PDO to string conversion in /var/www/html/build/clobberer/index.php on line 261, referer: https://build.mozilla.org/clobberer/
[Fri Apr 22 11:07:24 2011] [error] [client <snip>] PHP Notice:  Undefined variable: Object in /var/www/html/build/clobberer/index.php on line 261, referer: https://build.mozilla.org/clobberer/
[Fri Apr 22 11:07:24 2011] [error] [client <snip>] PHP Fatal error:  Call to a member function errorInfo() on a non-object in /var/www/html/build/clobberer/index.php on line 261, referer: https://build.mozilla.org/clobberer/

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

Zandr, any idea when the tools (specifically php) on this host were last updated?

Comment 4

[Zandr Milewski [:zandr]]

There's nothing even vaguely recent in yum.log. (Aug 11 2010)

You're running PHP 5.1.6.

My guess is that this is a corner case in the clobberer code.  Let me know if there's anything I can do to help, but nothing's been updated on this box in waaaay too long.

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

Thanks Zandr,  I guess that is good and bad at the same time.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I don't think the last-updated date of tools is useful information.

261                 ."(NULL, $branch, $builddir, $slave, $e_user, $now)") or die(print_r($$dbh->errorInfo(), TRUE));

$$foo works like perl - interpolate $foo as a variable name.  This is a simple typo, and should be $dbh.  That answers the first and third error messages.

If you fix that locally, you'll probably see an error message, which will help to figure out what the underlying problem is.

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

Thanks Dustin,
now i get 

Array ( [0] => HY000 [1] => 8 [2] => attempt to write a readonly database )

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #7)
> Thanks Dustin,
> now i get 
> 
> Array ( [0] => HY000 [1] => 8 [2] => attempt to write a readonly database )

ignore that, i didn't have the correct settings on a test db

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

I fixed the $$, and here's what I see from my own attempt to clobber:

[Fri Apr 22 11:58:17 2011] [error] [client 99.89.149.20] PHP Notice:  Undefined index:  buildername in /var/www/html/build/clobberer/index.php on line 272, referer: https://build.mozilla.org/clobberer/
[Fri Apr 22 11:58:17 2011] [error] [client 99.89.149.20] PHP Notice:  Undefined index:  builddir in /var/www/html/build/clobberer/index.php on line 273, referer: https://build.mozilla.org/clobberer/
[Fri Apr 22 11:58:17 2011] [error] [client 99.89.149.20] PHP Notice:  Undefined index:  slave in /var/www/html/build/clobberer/index.php on line 274, referer: https://build.mozilla.org/clobberer/
[Fri Apr 22 11:58:17 2011] [error] [client 99.89.149.20] PHP Notice:  Undefined index:  branch in /var/www/html/build/clobberer/index.php on line 275, referer: https://build.mozilla.org/clobberer/
[Fri Apr 22 11:58:17 2011] [error] [client 99.89.149.20] PHP Notice:  Undefined index:  master in /var/www/html/build/clobberer/index.php on line 276, referer: https://build.mozilla.org/clobberer/
[Fri Apr 22 11:59:11 2011] [error] [client 99.89.149.20] PHP Notice:  Use of undefined constant sort_func - assumed 'sort_func' in /var/www/html/build/clobberer/index.php on line 423, referer: https://build.mozilla.org/clobberer/

The stuff in lines 272-276 is harmless PHP lazniess.  The error on line 423 is also harmless, although it's bad programming practice and should be fixed.  I'll have a patch for those in a moment.

In a common PHP error, the form submission code sends a "Location" header to redirect the user back to the non-POST page, but then proceeds to render the page anyway - an expensive operation in this case.  I'll add an exit(0) after the header() call to fix this.

As for the errors, I suspect these are failures to acquire a db lock - the script does an absolutely *huge* number of queries for each form submission, and they require an exclusive lock, so if that gets blocked behind a series of long-running read locks for the display select, it would eventually time out and produce an error.  Due to the $$ bug, that error would never be seen.

Note that lots of plain-old page loads also timeout, and the page is not rendered completely.

I think this needs a complete rewrite, which is not a short-term thing.  I'll file a bug for that.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m652151-tools-r1.patch

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 527834 [details] [diff] [review]
m652151-tools-r1.patch

>   // This prevents accidentally resubmitting the form if the user reloads the 
>   // page
>   header("Location: " . $_SERVER['REQUEST_URI']);
>+  print("Done.");
>+  exit(0);
> }

Instead of print("Done.");, maybe we could print("Redirecting to " . $_SERVER['REQUEST_URI']) in case the redirect fails for whatever reason.

r+ with or without that change.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 527834 [details] [diff] [review]
m652151-tools-r1.patch

In trying to test this I'm seeing stuff like

[Fri Apr 22 12:47:25 2011] [error] [client 99.89.149.20] PHP Fatal error:  Maximum execution time of 60 seconds exceeded in /var/www/html/build/clobberer/index.php on line 177, referer: https://build.mozilla.org/clobberer/

But .. what can you do.

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 527834 [details] [diff] [review]
m652151-tools-r1.patch

          What    |Removed                     |Added
----------------------------------------------------------------------------
Attachment #527834 [details] [diff]|review+                     |review?, checked-in+
             Flags|                            |

Still r+ :P