1066573 - Failures in mozmill tests due to e10s on nightly

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Andreea Matei [:AndreeaMatei]]

e10s being included in Nightly will break some of our tests, until we make use of Mozmill 2.0.7. 
Keeping track here of the failing tests, so far:
Notification popup state has been opened - restartTests/testAddons_InstallAddonWithoutEULA/test1.js
http://mozmill-daily.blargon7.com/#/remote/report/b509c8df3c5f32f8b1b39e28bee35173

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sadly we have only been informed a couple of hours before this change went life in Nightly. There was no way to react that quickly, to not see lots of our tests failing due to the changes in bug 1064885. :( I really would appreciate to get a heads-up information 2 or more days before. Actually this is again a downside not being on TBPL.

So as for the status update, mozmill 2.0.7 got released. We are working on the mozmill-automation 2.0.7 release (https://github.com/mozilla/mozmill-automation/issues/170) now. Once this is done new environments will be created and mozmill-ci staging be updated. I don't want to upgrade production, given that QA has to run beta update tests today. So not earlier than Monday we will be able to run Mozmill tests successfully against Nightly.

Comment 2

[Chris Peterson [:cpeterson]]

Sorry again, Henrik. I was not familiar with how Mozmill works.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Chris Peterson (:cpeterson) from comment #2)
> Sorry again, Henrik. I was not familiar with how Mozmill works.

I understand. Looks like we simply faced a bad combination of situations. With also having our QA lead of this feature out last week, we simply got this info a bit too late through other channels. I would say at least we got it! So thanks to Gavin and Marc, who both pointed that out to me.


Ok, so after kinda hard work on Friday, I haven't seen any failures for recent Nightly builds anymore. So I pushed the upgrade to version 2.0.7 to our production instance. As seen right now, all tests are passing again, and none is showing the issues as we have seen here for e10s.

Lets try to make Mozmill ready for e10s at latest early October.

Other than that everything else on this bug is done.