668763 - Assertions in l10n testrun do not contain a message

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[[:Aleksej]]

Mozmill 1.5.3 and 1.5.4b5.

Russian Nightly Mozilla/5.0 (X11; Linux x86_64; rv:7.0a1) Gecko/20110630 Firefox/7.0a1

The Information column for the testCropped/test1.js failure is a bunch of list bullets without text.

Comment 1

[[:Aleksej]]

Same happens with at least /testAccessKeys/test1.js

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Does that only happen for 7.0 builds of Firefox or are 6.0 and 5.0 builds also affected?

Comment 3

[[:Aleksej]]

Happens for a 6.0 build: http://mozmill-crowd.brasstacks.mozilla.com/#/l10n/report/3e673a01042add46ed7599eaba5cc3d8
but not for 5.0: http://mozmill-crowd.brasstacks.mozilla.com/#/l10n/report/3e673a01042add46ed7599eaba5d2ac4

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So it should have been caused by a regression in Firefox 6. Would be great if someone could check for a regression range.

Comment 5

[[:Aleksej]]

Firefox 6.0a1 (6.0a1, en-US, 20110413030543)
Firefox 6.0a1 (6.0a1, en-US, 20110414030535)

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=6bcaec19d09e&tochange=aa200a803e07

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I assume it's also related to bug 637207 and bug 650646. Will check later what has to be changed to make it work again. Thanks for finding the regression range!

Comment 7

[u415141]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/15312549

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I will check this today

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1

We should simply get rid of the old jumlib assertions and use our new soft assertions (expect). That will solve our problems here.

Comment 10

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Comment on attachment 545895 [details] [diff] [review]
Patch v1

r+, looks fine. We finally found a real-life use for pass()/fail()!

I'm not a big fan of having one-sided assertions though--that is, ones where it fails when bad but does nothing when good. It'll give an inconsistent number of test results depending on whether things fail or not.

Would it be worth it to loop back at some point and add elses to each if that do an expect.pass?

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I would punt this for now, but yes we should. I'm happy that those scripts are working for now but we have to investigate more time to get everything updated. It's really something for the future.

Landed on all supported branches:
http://hg.mozilla.org/qa/mozmill-tests/rev/d8d0f9531d31 (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/92cea4f5f594 (aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/fd665a90c424 (beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/e21319c10521 (release)
http://hg.mozilla.org/qa/mozmill-tests/rev/3330137a61bd (2.0)
http://hg.mozilla.org/qa/mozmill-tests/rev/efb11a741a54 (1.9.2)