683680 - Fix QA scripts to work with Bugzilla 4.2

Status: RESOLVED FIXED

(Bugzilla :: QA Test Scripts, defect)

Description

[Frédéric Buclin]

Test Summary Report
-------------------
test_bug_edit.t                         (Wstat: 1280 Tests: 504 Failed: 5)
  Failed tests:  62, 300, 311, 347, 357
  Non-zero exit status: 5
test_classifications.t                  (Wstat: 256 Tests: 108 Failed: 1)
  Failed test:  44
  Non-zero exit status: 1
test_custom_fields.t                    (Wstat: 1792 Tests: 363 Failed: 7)
  Failed tests:  14, 138, 186-188, 192, 323
  Non-zero exit status: 7
test_edit_products_properties.t         (Wstat: 65280 Tests: 171 Failed: 10)
  Failed tests:  123, 136, 147-152, 157, 171
  Non-zero exit status: 255
test_email_preferences.t                (Wstat: 2816 Tests: 309 Failed: 11)
  Failed tests:  174, 185, 192, 199, 207, 226, 233, 242
                249, 280, 287
  Non-zero exit status: 11
test_enter_new_bug.t                    (Wstat: 768 Tests: 51 Failed: 3)
  Failed tests:  14, 31, 48
  Non-zero exit status: 3
test_flags2.t                           (Wstat: 2816 Tests: 283 Failed: 11)
  Failed tests:  63-64, 108, 123, 133, 154, 183-184, 199
                221, 233
  Non-zero exit status: 11
test_flags.t                            (Wstat: 65280 Tests: 49 Failed: 2)
  Failed tests:  48-49
  Non-zero exit status: 255
test_groups.t                           (Wstat: 3840 Tests: 233 Failed: 15)
  Failed tests:  42, 53-55, 58, 65, 88, 91, 93, 107, 109
                139, 166, 172, 227
  Non-zero exit status: 15
test_keywords.t                         (Wstat: 65280 Tests: 80 Failed: 5)
  Failed tests:  50, 58, 78-80
  Non-zero exit status: 255
test_long_list_redirection.t            (Wstat: 512 Tests: 3 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
test_milestones.t                       (Wstat: 1536 Tests: 137 Failed: 6)
  Failed tests:  49, 60, 71, 74, 120, 123
  Non-zero exit status: 6
test_private_attachments.t              (Wstat: 1536 Tests: 208 Failed: 6)
  Failed tests:  38, 53, 91, 106, 131, 186
  Non-zero exit status: 6
test_qa_contact.t                       (Wstat: 2304 Tests: 153 Failed: 9)
  Failed tests:  27, 34-36, 66, 130-132, 136
  Non-zero exit status: 9
test_require_login.t                    (Wstat: 1024 Tests: 149 Failed: 4)
  Failed tests:  69, 95, 101, 109
  Non-zero exit status: 4
test_shutdown.t                         (Wstat: 1024 Tests: 144 Failed: 4)
  Failed tests:  70, 98, 104, 114
  Non-zero exit status: 4
test_status_whiteboard.t                (Wstat: 512 Tests: 89 Failed: 2)
  Failed tests:  23, 29
  Non-zero exit status: 2
test_target_milestones.t                (Wstat: 256 Tests: 114 Failed: 1)
  Failed test:  39
  Non-zero exit status: 1
test_time_summary.t                     (Wstat: 1024 Tests: 78 Failed: 4)
  Failed tests:  25, 38-40
  Non-zero exit status: 4
test_user_matching.t                    (Wstat: 512 Tests: 206 Failed: 2)
  Failed tests:  43, 91
  Non-zero exit status: 2
test_user_preferences.t                 (Wstat: 65280 Tests: 87 Failed: 11)
  Failed tests:  41, 45, 53-54, 56, 68, 74-75, 77, 83, 87
  Non-zero exit status: 255
test_user_privs.t                       (Wstat: 256 Tests: 49 Failed: 1)
  Failed test:  25
  Non-zero exit status: 1
webservice_bug_attachments.t            (Wstat: 3072 Tests: 337 Failed: 12)
  Failed tests:  108, 133, 159, 173, 188, 213, 239, 253
                268, 291, 314, 328
  Non-zero exit status: 12
webservice_bug_fields.t                 (Wstat: 3840 Tests: 1329 Failed: 15)
  Failed tests:  14, 189, 222, 397, 430, 605, 625, 667-668
                861, 903-904, 1097, 1136-1137
  Non-zero exit status: 15
  Parse errors: Bad plan.  You planned 1338 tests but ran 1329.
webservice_bug_update_see_also.t        (Wstat: 512 Tests: 115 Failed: 2)
  Failed tests:  38, 94
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 117 tests but ran 115.
webservice_bug_update.t                 (Wstat: 5632 Tests: 965 Failed: 22)
  Failed tests:  208, 230-231, 428, 434, 437, 473, 475-477
                482, 689, 711-712, 909, 915, 918, 954, 956-958
                963
  Non-zero exit status: 22
webservice_product.t                    (Wstat: 65280 Tests: 14 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 134 tests but ran 14.
Files=55, Tests=11014, 2902 wallclock secs ( 2.21 usr  0.23 sys + 21.66 cusr  1.73 csys = 25.83 CPU)
Result: FAIL

Comment 1

[Frédéric Buclin]

Attachment: part 1 (test_bug_edit.t, QA::Util, generate_test_data.pl)

Most scripts are failing due to the usage of history.ReplaceState which replaces "Bug XXX processed" by "Bug XXX - bug summary". So I had to refactor this code. This first part fixes generate_test_data.pl to set the post_bug_submit_action user pref to 'nothing' for everybody, so that we can correctly predict the behavior of process_bug.cgi and the title of the page (see e.g. bug 685607). It also fixes test_bug_edit.t to use this new code.

Committing to: bzr+ssh://[e-mail]/bugzilla/qa/4.2/
modified config/generate_test_data.pl
modified t/test_bug_edit.t
modified t/lib/QA/Util.pm
Committed revision 195.

Comment 2

[Frédéric Buclin]

test_edit_products_properties.t still fails due to bug 686133.

Comment 3

[Frédéric Buclin]

Attachment: part 2

4 additional scripts fixed.

Committing to: bzr+ssh://[e-mail]/bugzilla/qa/4.2/
modified t/test_classifications.t
modified t/test_custom_fields.t
modified t/test_edit_products_properties.t
modified t/test_email_preferences.t
Committed revision 196.

Comment 4

[Frédéric Buclin]

Attachment: part 3

3 additional scripts fixed. test_flags.t still fails due to bug 686227.

Committing to: bzr+ssh://[e-mail]/bugzilla/qa/4.2/
modified t/test_enter_new_bug.t
modified t/test_flags.t
modified t/test_flags2.t
Committed revision 197.

Comment 5

[Frédéric Buclin]

Attachment: part 4

6 additional scripts fixed + test_long_list_redirection.t removed as long_list.cgi no longer exists:

Committing to: bzr+ssh://[e-mail]/bugzilla/qa/4.2/
deleted t/test_long_list_redirection.t
Committed revision 198.

Committing to: bzr+ssh://[e-mail]/bugzilla/qa/4.2/
modified t/test_groups.t
modified t/test_keywords.t
modified t/test_milestones.t
modified t/test_private_attachments.t
modified t/test_qa_contact.t
modified t/test_require_login.t
Committed revision 199.

Comment 6

[Frédéric Buclin]

Attachment: part 5

6 more scripts fixed:

Committing to: bzr+ssh://[e-mail]/bugzilla/qa/4.2/
modified t/test_shutdown.t
modified t/test_status_whiteboard.t
modified t/test_target_milestones.t
modified t/test_time_summary.t
modified t/test_user_matching.t
modified t/test_user_privs.t
Committed revision 200.

Comment 7

[Frédéric Buclin]

Note to self: test_user_preferences.t and test_votes.t are the last 2 Selenium scripts to fix.

Comment 8

[Frédéric Buclin]

Attachment: part 6

This patch fully fixes test_user_preferences.t, and test_votes.t as much as possible. I cannot fully test test_votes.t because Bugzilla crashes due to bug 686786 and the fix is not trivial. It also fails due to bug 686780, but I already attached a patch for this one.

All test_*.t Selenium scripts now pass when extensions are disabled. When the Voting extension is enabled, two scripts still fail: test_votes.t and test_edit_product_properties.t, both for the reason mentioned above, i.e. due to bug 686786.

Committing to: bzr+ssh://[e-mail]/bugzilla/qa/4.2/
modified t/test_user_preferences.t
modified t/test_votes.t
Committed revision 204.

Comment 9

[Frédéric Buclin]

Attachment: part 7

This patch fixes two WebService QA scripts. There are still several WS QA scripts which fail, but they all seem due to real bugs in Bugzilla.

Committing to: bzr+ssh://[e-mail]/bugzilla/qa/4.2/
modified t/webservice_bug_attachments.t
modified t/webservice_bug_fields.t
Committed revision 205.

Comment 10

[Frédéric Buclin]

There are 3 WS QA scripts which still fail:

webservice_bug_update_see_also.t
webservice_bug_update.t
webservice_product.t

They all look to fail due to real bugs in Bugzilla:

Can't call method "isa" on an undefined value at Bugzilla/Bug.pm line 2892. (bug 686904)

The function Bugzilla::Comment->create requires a thetext argument, and that argument was not set. (no bug filed yet)


The remaining errors may be due to the scripts themselves; I don't know yet.

Comment 11

[Frédéric Buclin]

(In reply to Frédéric Buclin from comment #10)
> The function Bugzilla::Comment->create requires a thetext argument, and that
> argument was not set. (no bug filed yet)

I filed bug 686963

Comment 12

[Frédéric Buclin]

Attachment: part 8

Takes the fix from bug 686971 into account about See Also.

Committing to: bzr+ssh://[e-mail]/bugzilla/qa/4.2/
modified t/webservice_bug_update.t
Committed revision 213.

Comment 13

[Frédéric Buclin]

Attachment: part 9

Here is the final patch. Now all tests pass with and without extensions enabled.

Committing to: bzr+ssh://[e-mail]/bugzilla/qa/4.2/
modified t/test_bug_edit.t
modified t/webservice_bug_get_bugs.t
modified t/webservice_bug_update.t
Committed revision 214.

Comment 14

[Frédéric Buclin]

Should turn green now.