904610 - Test failure "'false' should equal 'true'" in testAddons_changeTheme/test3.js

Status: RESOLVED INVALID

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Cosmin Malutan, [:cosmin-malutan]]

Happened today on Windows 7 x64 with Nightly de
I couldn't reproduce it locally.

http://mozmill-daily.blargon7.com/#/functional/report/b7ef1fb3d9703aeaf2c46e07d281e719

Comment 1

[Andrei Eftimie]

The URL field should point to a list of all failures if it is a functional test.
Also remember to set the OS/tracking flags.

Comment 2

[Andrei Eftimie]

This fails in test3 without the mentioned failure in test2 from bug 905170

Comment 3

[mario garbi]

Happened again today on Mac OS X 10.8.4 (x86_64) with Firefox 26.0a2 'it' using Mozmill 2.0:
http://mozmill-daily.blargon7.com/#/functional/report/6ec6776efe900da3fd2b64a750c0152d

Comment 4

[Daniel Gherasim]

Happened today on MAC OS X 10.7 with Aurora zh-CN.
http://mozmill-daily.blargon7.com/#/functional/report/c90eeeb0a5511695efc7462553aa0fe1

Comment 5

[Cosmin Malutan, [:cosmin-malutan]]

Failed again with Beta  ja-JP-mac on mac 10.8 (mm-osx-108-4)
http://mozmill-release.blargon7.com/#/functional/report/44e0169acb64b68a44c1ad880b35e38d

Comment 6

[Andrei Eftimie]

This failure is just fallout from bug 972912.

Comment 7

[Liz Henry (:lizzard) (relman/hg->git project)]

Test failure for Firefox 31.0 (31.0, he, 20140616143923), Mac OS X 10.7.5 (x86_64).

/restartTests/testAddons_changeTheme/test3.js 	testChangedThemeToDefault 	

    'false' should equal 'true' 

http://mozmill-release.blargon7.com/#/functional/report/6959f9a8236961096763389083a7a3d7

Comment 8

[Andreea Matei [:AndreeaMatei]]

Failed on Beta ru, on OS X 10.7.5:
http://mozmill-release.blargon7.com/#/functional/report/883d8406dab55a771a00934158ae8cac

Comment 9

[Andrei Eftimie]

(In reply to Andreea Matei [:AndreeaMatei] from comment #8)
> Failed on Beta ru, on OS X 10.7.5:
> http://mozmill-release.blargon7.com/#/functional/report/
> 883d8406dab55a771a00934158ae8cac

Also fallout from bug 972912

Comment 10

[u279076]

This has failed 5 times in the last week on Beta:
http://mozmill-release.blargon7.com/#/functional/failure?app=Firefox&branch=31.0&platform=Mac&from=2014-07-01&to=2014-07-08&test=%2FrestartTests%2FtestAddons_changeTheme%2Ftest3.js&func=testChangedThemeToDefault

Can we get this test disabled for Firefox 31?

Comment 11

[Andrei Eftimie]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #10)
> This has failed 5 times in the last week on Beta:
> http://mozmill-release.blargon7.com/#/functional/
> failure?app=Firefox&branch=31.0&platform=Mac&from=2014-07-01&to=2014-07-
> 08&test=%2FrestartTests%2FtestAddons_changeTheme%2Ftest3.
> js&func=testChangedThemeToDefault
> 
> Can we get this test disabled for Firefox 31?

Unfortunately this is not this tests fault, but the harness itself.
It's the same issue as mentioned before. We have 3 test files and test2 was not run!
This is again mozmill bug 972912.

One thing we might do is to refactor all restart tests to be just 1 file, we should be able to avoid these kind of failures where 1 testfile is not running.

Comment 12

[Teodor Druta]

Attachment: skipchangetheme.patch

Multiple failures on OSX beta/aurora/default branches on multiple locales

http://mozmill-daily.blargon7.com/#/functional/failure?app=Firefox&branch=37&platform=Mac&from=2014-11-25&to=&test=%2FrestartTests%2FtestAddons_changeTheme%2Ftest3.js&func=testChangedThemeToDefault
http://mozmill-release.blargon7.com/#/functional/failure?app=Firefox&branch=35&platform=Mac&from=2014-11-25&to=&test=%2FrestartTests%2FtestAddons_changeTheme%2Ftest3.js&func=testChangedThemeToDefault

I say we should skip this test.

Comment 13

[Teodor Druta]

(In reply to Teodor Druta from comment #12)
> Created attachment 8531198 [details] [diff] [review]
> skipchangetheme.patch
> 
> Multiple failures on OSX beta/aurora/default branches on multiple locales
> 
> http://mozmill-daily.blargon7.com/#/functional/
> failure?app=Firefox&branch=37&platform=Mac&from=2014-11-
> 25&to=&test=%2FrestartTests%2FtestAddons_changeTheme%2Ftest3.
> js&func=testChangedThemeToDefault
> http://mozmill-release.blargon7.com/#/functional/
> failure?app=Firefox&branch=35&platform=Mac&from=2014-11-
> 25&to=&test=%2FrestartTests%2FtestAddons_changeTheme%2Ftest3.
> js&func=testChangedThemeToDefault
> 
> I say we should skip this test.

Actually it seems that this is the failure reported in Bug 972912.

Comment 14

[Andreea Matei [:AndreeaMatei]]

So this can only happen with release and esr31 branches, as the other have the test in a single file.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozmill tests have been superseded by Marionette tests.