635452 - Add debug output to Email Preferences test script

Status: RESOLVED WONTFIX

(Bugzilla :: QA Test Scripts, enhancement)

Description

[Teemu Mannermaa (:wicked)]

Attachment: Email Preferences test debug output, V1

Here's a patch that add some debug output for the problematic Email Preference test script. It makes the following changes:
 1) Switches parameters to is_deeply() around to have a more logical output of expected and seen diagnostics if the arrays differ.
 2) Add a separate dump of the arrays to see their full content if an error is detected for a is_deeply() test.
 3) Also adds a dump of the full body text as seen by Selenium.

BTW, this change is actually live for the 3.6 MySQL QA tests to help debug the burning.

Comment 1

[Frédéric Buclin]

Comment on attachment 513705 [details] [diff] [review]
Email Preferences test debug output, V1

>+is_deeply(\@email_sentto, \@email_both, "Admin and normal user got bugmail")
>+  or diag(Dumper(\@email_sentto, \@email_both, $sel->get_body_text()));

I'm not a fan to display the whole page. This won't bring any value IMO, besides maybe the CC, assignee and QA fields. I would prefer if you limit the output to these fields only.

Comment 2

[Frédéric Buclin]

Comment on attachment 513705 [details] [diff] [review]
Email Preferences test debug output, V1

Per my previous comment.

Comment 3

[Teemu Mannermaa (:wicked)]

Attachment: Email Preferences test debug output, V2

I refactored the bugmail recipient checks to a sub, like you requested on IRC. This also extracts roles from the page in case an error happens.

Screens after setting prefs and error are also stored locally for reference. This change can be left out if you don't want it in repo.

Comment 4

[Frédéric Buclin]

Comment on attachment 515855 [details] [diff] [review]
Email Preferences test debug output, V2

As discussed on IRC, this code currently doesn't help us find what's wrong with this QA script on Tinderbox (which is the only place where we found a problem). So I would prefer to limit this patch to Tinderbox for now. You can re-request review later if it really helps us track what's wrong. But for now, I keep it out of my radar.

Comment 5

[Frédéric Buclin]

I suspect a race condition could be the reason of the random failures in this script (e.g. Selenium doesn't have time to check a checkbox before changes are submitted). Debugging code won't help as it really depends on the version of Selenium RC and of Firefox you have. So marking as wontfix.