784305 - Mozmill test failure /restartTests/testAddons_RestartlessExtensionWorksAfterRestart/test2.js | Current URL should match expected URL

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P1)

Description

[Maniac Vlad Florin (:vladmaniac)]

This started failing today with Mozmill 1.5.17 on the default branch for mac os x 10.6.x and Ubuntu 11.10

http://mozmill-ci.blargon7.com/#/functional/report/d87d47fd1034f072b9bece6ee66aeee1
http://mozmill-ci.blargon7.com/#/functional/report/d87d47fd1034f072b9bece6ee66ae644

Comment 1

[Maniac Vlad Florin (:vladmaniac)]

This is reproducible locally

Comment 2

[Maniac Vlad Florin (:vladmaniac)]

So this is failing on default, and I bet it has something to do with landing of Bug 782639 in litmus-data. 

Andreea and Henrik can you confirm this?

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Maniac Vlad Florin (:vladmaniac) from comment #2)
> So this is failing on default, and I bet it has something to do with landing
> of Bug 782639 in litmus-data. 

Why should this be related? This test doesn't make use of the blocklisting feature. What happens exactly when you can reproduce it locally?

Comment 4

[Maniac Vlad Florin (:vladmaniac)]

(In reply to Henrik Skupin (:whimboo) from comment #3)
> (In reply to Maniac Vlad Florin (:vladmaniac) from comment #2)
> > So this is failing on default, and I bet it has something to do with landing
> > of Bug 782639 in litmus-data. 
> 
> Why should this be related? This test doesn't make use of the blocklisting
> feature. What happens exactly when you can reproduce it locally?

Tested and it seems that our extension does not work with Nightly anymore. 
Just a reminder of how it works: 

Normally it inserts a context menu entry with the label "Open mozqa.com" and we click that to check its working. On Fx Nightly, it stopped working for reasons unknown to me at the moment. 
We have the context menu entry, but clicking on it does not open our page. This can be also tested manually, leading to the same result.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Is it based on the addons sdk? If yes we should probably make it not so but a simple bootstrapped addon. Can you check for a regression range? It could be an Addon SDK regression if we base on it.

Comment 6

[Maniac Vlad Florin (:vladmaniac)]

(In reply to Henrik Skupin (:whimboo) from comment #5)
> Is it based on the addons sdk? If yes we should probably make it not so but
> a simple bootstrapped addon. Can you check for a regression range? It could
> be an Addon SDK regression if we base on it.

Not sure what you mean with "simple bootstrapped addon" because all add-ons build with SDK or add-on Builder are bootstrapped [1]

[1] https://developer.mozilla.org/en-US/docs/Extensions/Bootstrapped_extensions

Comment 7

[Maniac Vlad Florin (:vladmaniac)]

The extension is not SDK based because: 

* it has no directory structure to match the one created with sdk using "cfx init" 
* it does not include package.json which SDK uses to package the extensions 
* running a simple "cfx test" would trigger errors 

This is enough for me to say its not SDK based. Likely its made with Add-on Builder. 
Next step is finding the regression range which I think its easy since we have seen failures only since yesterday.

Comment 8

[Maniac Vlad Florin (:vladmaniac)]

Regression range: 

Build worked: http://hg.mozilla.org/mozilla-central/rev/812ea773f166
Build failed: http://hg.mozilla.org/mozilla-central/rev/8c85c83068e7

Pushlog: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=812ea773f166&tochange=8c85c83068e7

Comment 9

[Maniac Vlad Florin (:vladmaniac)]

Extension logging pref echoes the following error and I think I'm pretty close in finding a fix: 

error: An exception occurred.
Traceback (most recent call last):
  File "javascript:self.on("click", function (node, data) { window.location.href = "http://mozqa.com/data/firefox/layout/mozilla.html";});", line 1, in 
TypeError: self.on is not a function

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Maniac Vlad Florin (:vladmaniac) from comment #7)
> The extension is not SDK based because: 

Not sure where you get this assumption from. If you check harness-options.js in the root folder it states: "sdkVersion": "1.2.1". So it is a very old version of the SDK, which could indeed cause problems. We probably should rebuild and also put its source into the litmus-data repository.

(In reply to Maniac Vlad Florin (:vladmaniac) from comment #6)
> (In reply to Henrik Skupin (:whimboo) from comment #5)
> > Is it based on the addons sdk? If yes we should probably make it not so but
> > a simple bootstrapped addon. Can you check for a regression range? It could
> > be an Addon SDK regression if we base on it.
> 
> Not sure what you mean with "simple bootstrapped addon" because all add-ons
> build with SDK or add-on Builder are bootstrapped [1]

A simple bootstrapped addon does *NOT* make use of the Addon SDK. So we would not be affected by SDK incompatibilities and would have a much smaller code base.

Comment 11

[Maniac Vlad Florin (:vladmaniac)]

Re-building with SDK 1.9 does not fix the problem. 
Currently looking at bug 783497 to see if this is the cause

Comment 12

[Maniac Vlad Florin (:vladmaniac)]

Attachment: proposed fix v1.0

* This patch is replacing data/addons/extensions/restartless.xpi with another add-on which is 1.9 SDK based, but with simple code base 
Looks like it fixes the problem
The patch was tested with default branch, this is where we get the error only. 

Most certainly, when we are going to land this, we'll need an infrastructure bug to put the new version of restartless.xpi in litmus-data

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Lets get this test disabled on default ASAP until the underlying extension issue has been fixed.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: skip patch [checked-in]

Landed this before todays builds are up so we are not swamped with test failures.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 654160 [details] [diff] [review]
skip patch [checked-in]

Landed skip patch on default as:
http://hg.mozilla.org/qa/mozmill-tests/rev/ce8193bf5289

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 654151 [details] [diff] [review]
proposed fix v1.0

This will not work because the addon will not be compatible with older versions of Firefox like ESR10. I really would like to know what has been changed in the code so that it is working now.

Comment 17

[Maniac Vlad Florin (:vladmaniac)]

(In reply to Henrik Skupin (:whimboo) from comment #16)
> Comment on attachment 654151 [details] [diff] [review]
> proposed fix v1.0
> 
> This will not work because the addon will not be compatible with older
> versions of Firefox like ESR10. I really would like to know what has been
> changed in the code so that it is working now.

aurora, beta, release and esr work fine with the old version of the add-on. 
I just build it with the latest released sdk: What I did exactly was: 

* cfx init so I can get the folder structure
* copy paste main.js from the old add-on
* updated package.json to reflect the data from the old install.rdf
* cfx test
* cfx xpi

..and worked. did nothing more or less.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I might work at the moment but as you can see in the addons manager the extension is marked to be incompatible. I would suggest you file a new bug so we can update the extension in the litmus-data repository, which should in any case always happen first. Then we can grab the extension for Mozmill's usage.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

When will we get an update here? It's on the list of tasks since 3 weeks now without an update. Please take care of this bug so we can re-enable the failure ASAP.

Comment 20

[Andreea Matei [:AndreeaMatei]]

While I was waiting for the new addon, I've checked this on OS X 10.6.8 and 10.7.5, Ubuntu 11.10 where the two reports were, single run and functional testruns, but it's not reproducible anymore. 
Could be that with the new releases on nightly, the current addon works fine and the behavior from comment 4 is not happening now.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Andrea, were you able to reproduce it with one of the builds which showed this failure?

Comment 22

[Andreea Matei [:AndreeaMatei]]

Henrik, yes, the build 20120821 from August does not open mozqa.com when clicking it on the context menu.

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So it could have been that a patch for Firefox fixed that. Would you mind to do a quick regression test since when it is working?

Comment 24

[Andreea Matei [:AndreeaMatei]]

It was fixed in the build from the next day - August 22nd. I see some failure reports in that day, but they were with the build from 21st.

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Please provide a pushlog as usual so we can hopefully figure out what has fixed it.

Comment 26

[Andreea Matei [:AndreeaMatei]]

Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=688a80cdf39f&tochange=abc17059522b

Build that works: 
ftp://ftp.mozilla.org/pub/firefox/nightly/2012/08/2012-08-22-03-05-58-mozilla-central/

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The only I could find in the pushlog which affects restartless extensions is bug 784233. Bobby, do you think that this one has fixed our problem here which caused that our test extension hasn't shown a context menu when the user did a right click in a webpage?

Andreea, if the test works now lets get it re-enabled. There is no need to wait longer. Good to hear that!

Comment 28

[Andreea Matei [:AndreeaMatei]]

Attachment: enable patch

Patch that enables the test.

Comment 29

[Bobby Holley (:bholley)]

(In reply to Henrik Skupin (:whimboo) from comment #27)
> The only I could find in the pushlog which affects restartless extensions is
> bug 784233. Bobby, do you think that this one has fixed our problem here
> which caused that our test extension hasn't shown a context menu when the
> user did a right click in a webpage?

Not unless that problem only existed for a few days. Bug 784233 was effectively a partial backout of bug 553102, which landed a few days earlier.

Comment 30

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The failures first appeared on 2012-08-16 and were intermittent at this time. Starting by 2012-08-21 they were permanent failures. Given by the work from Andreea they were gone on 2012-08-22.

Comment 31

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 675076 [details] [diff] [review]
enable patch

We don't need a new patch for that. We can simply backout the skip patch.

Comment 32

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Backed-out skip patch:
http://hg.mozilla.org/qa/mozmill-tests/rev/b178f3346289 (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/38eb00a470f5 (aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/867fe223027a (beta)

Given that it will be hard to find the bug which fixed it just call it WFM.

Comment 33

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This is failing again on mozilla-aurora:

http://mozmill-ci.blargon7.com/#/functional/report/190d57cf64e3c5c3996c0c1059017bd7
http://mozmill-ci.blargon7.com/#/functional/report/2e7486a8c08b5541b9074117e0f65aa9

Let's get this skipped again and investigate the failure on Aurora.

Comment 34

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Re-skipped on aurora:
http://hg.mozilla.org/qa/mozmill-tests/rev/05b0c885ab09

Comment 35

[Andreea Matei [:AndreeaMatei]]

I will investigate this more on Aurora. I wonder if it's not a fix that has been applied to Nightly and not also on Aurora, from the pushlog.

Comment 36

[Andreea Matei [:AndreeaMatei]]

I was working on a regression for Aurora and I saw this report:
http://mozmill-ci.blargon7.com/#/functional/report/190d57cf64e3c5c3996c0c10593796ca

which is for Aurora, with a build from yesterday and the test it's not skipped there. Neither on 28th.
Although when I check an aurora repo the test is skipped.

Comment 37

[Andreea Matei [:AndreeaMatei]]

We had no reports failing on August 21st for Aurora, just Nightly was affected back then. 
I ran the test over several builds, from August but also now in October (on Windows cause there was happening) and I can't reproduce.
We will have to check on the machine where it did fail for debugging.

Comment 38

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andreea Matei [:AndreeaMatei] from comment #36)
> which is for Aurora, with a build from yesterday and the test it's not
> skipped there. Neither on 28th.
> Although when I check an aurora repo the test is skipped.

That was because Mozmill-CI hasn't updated itself to use the latest version of the mozilla-aurora branch. As you can see it's skipped now:

http://hg.mozilla.org/qa/mozmill-tests/file/mozilla-aurora/tests/functional/restartTests/testAddons_RestartlessExtensionWorksAfterRestart/test2.js

Comment 39

[Andreea Matei [:AndreeaMatei]]

It appears on the latest Aurora from today this is not reproducible anymore. I will look back some builds to see when it started to work exactly.

Comment 40

[Andreea Matei [:AndreeaMatei]]

So I've checked older builds where it used to fail and it's not reproducing on them either. 
From August until now, like on default, it's hard to find what has fixed this. I will monitor this on mozmill-ci after unskipping.

Comment 41

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Backed-out skip patch:
http://hg.mozilla.org/qa/mozmill-tests/rev/5adb83935f30 (aurora)

If it sticks in the next couple of days we can mark the bug as fixed.

Comment 42

[Andreea Matei [:AndreeaMatei]]

Hasn't failed on Aurora these days. Yey!

Comment 43

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We haven't landed a patch here, so it has to be WFM.

Comment 44

[Andrei Eftimie]

Failed on Linux and OSX, Nightly, en-US and fr:
http://mozmill-ci.blargon7.com/#/functional/failure?branch=All&platform=All&from=2013-03-26&to=2013-03-28&test=%2FrestartTests%2FtestAddons_RestartlessExtensionWorksAfterRestart%2Ftest2.js&func=test2.js%3A%3AtestRestartlessExtensionWorksAfterRestart

Comment 45

[Daniela Petrovici]

Attachment: skip patch v1.1

The previous skip patch did not cleanly apply. Updated skip patch.

Comment 46

[Henrik Skupin [:whimboo][⌚️UTC+2]]

How often did that fail? If it's a one off I don't think we should disable the test.

Comment 47

[Andrei Eftimie]

18 times today until now.
So a skip might be good.

Comment 48

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It's failing a lot. So please work on this bug next so we get it fixed. Thanks.

Comment 49

[Andreea Matei [:AndreeaMatei]]

This is the error I get before the one in the title, several times (on each test):

error: An exception occurred.
Traceback (most recent call last):
  File "javascript:self.on("click", function (node, data) { window.location.href = "http://mozqa.com/data/firefox/layout/mozilla.html";});", line 1, in 
TypeError: self.on is not a function

With a build from yesterday it works, I'm looking at today's pushlog.

Comment 50

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 730636 [details] [diff] [review]
skip patch v1.1

Review of attachment 730636 [details] [diff] [review]:
-----------------------------------------------------------------

Landed skip patch as:
http://hg.mozilla.org/qa/mozmill-tests/rev/ac4ea323a447

Comment 51

[Andreea Matei [:AndreeaMatei]]

I can reproduce manually with today's nightly, after installing the addon and clicking Open mozqa.com in context menu, nothing happens.

Could be this changeset? I'm building now to confirm:
https://hg.mozilla.org/mozilla-central/rev/f46bbb876415

Comment 52

[Andreea Matei [:AndreeaMatei]]

Nope. This is the pushlog, a few merges again, but I'm getting good at this :)
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=178a4a770bb1&tochange=962f5293f87f

Comment 53

[Andreea Matei [:AndreeaMatei]]

With hg bisect I found this as the responsible changeset:
http://hg.mozilla.org/mozilla-central/rev/478ef0049c1a

Henrik mentioned some time ago in comment 27 the bug that at that time fixed our problem. Now with the removal of that code, it reappeared. 
So I think it's time to update our addon, in bug 785014.

Comment 54

[Henrik Skupin [:whimboo][⌚️UTC+2]]

If it's unclear how to fix our add-on we should ask the jetpack guys over on bug 785014. :Ochameau might be able to help us.

I would have loved if we wouldn't have reopened this bug but created a new one. Anyway, lets get this fixed soon.

Comment 55

[Gilles Leblanc]

I don't know if it helps but I have had the same error message mentioned in the bug while working on bug 992137.

error: An exception occurred.
Traceback (most recent call last):
  File "javascript:self.on("click", function (node, data) { window.location.href = "http://mozqa.com/data/firefox/layout/mozilla.html";});", line 1, in
TypeError: self.on is not a function


On Firefox Beta 29.0 on Linux Mint running mozmill-tests/firefox/lib/tests/manifest.ini with the most recent version of MozMill.

Comment 56

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Gilles, I think that warrants a new bug. When you file it please make sure to check which test in details fails here. Thanks.

Comment 57

[Daniel Gherasim]

This is still reproducible manually & we are blocked here by bug 785014 where we need to update our restartless.xpi addon to work. 

The test will still need some updates after the addon fix, mostly because we are not correctly checking if the addon is installed now (most likely some pages in the about:addons page)  & our current style guide.

We still have a skipped test here so this stays as P1.

Comment 58

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Teodor, you said that this problem is not reproducible anymore with the current restartless extension. Now I see that this test is disabled. Can you please check what's going on here, and if we indeed need a bootstrapped extension?

Comment 59

[Teodor Druta]

(In reply to Henrik Skupin (:whimboo) from comment #58)
> Teodor, you said that this problem is not reproducible anymore with the
> current restartless extension. Now I see that this test is disabled. Can you
> please check what's going on here, and if we indeed need a bootstrapped
> extension?

Yes, I haven't seen that this test is actually skipped for a really long time, I'll take a look at the reported failures, to see if this really requires a new extension.

Comment 60

[Teodor Druta]

Yeah, I think we need a new restartless extension, this one doesn't seem to work anymore. Reopening Bug 785014.

Comment 61

[Teodor Druta]

Attachment: fixrestartless.patch

This is a fixed version of the RestartlessExtensionWorksAfterRestart test.
It makes use of the new restartless addon from Bug 785014.

Comment 62

[Mihaela Velimiroviciu (:mihaelav)]

Comment on attachment 8554507 [details] [diff] [review]
fixrestartless.patch

Review of attachment 8554507 [details] [diff] [review]:
-----------------------------------------------------------------

It looks fine to me.

Comment 63

[Teodor Druta]

Attachment: fixrestartless.patch

Added the new restartless.xpi addon to data.

Comment 64

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8557837 [details] [diff] [review]
fixrestartless.patch

Review of attachment 8557837 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/qa/mozmill-tests/rev/4a56e147df51 (default)

Comment 65

[Teodor Druta]

Attachment: unskiprestartless.patch

This test is skipped, we should unskip it since it was fixed.
This is the unskip patch for default.

Comment 66

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8559662 [details] [diff] [review]
unskiprestartless.patch

Review of attachment 8559662 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/qa/mozmill-tests/rev/f3081462ebae (default)

Comment 67

[Teodor Druta]

Attachment: fixrestartless_aurora.patch

Here is the patch for aurora.

Comment 68

[Teodor Druta]

Attachment: fixrestartless_beta_release.patch

And here is the patch for beta and release.

Comment 69

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8559694 [details] [diff] [review]
fixrestartless_aurora.patch

Review of attachment 8559694 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/qa/mozmill-tests/rev/f23e56eeae33 (aurora)

Comment 70

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8559695 [details] [diff] [review]
fixrestartless_beta_release.patch

Review of attachment 8559695 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/qa/mozmill-tests/rev/9714853ffb47 (beta)
https://hg.mozilla.org/qa/mozmill-tests/rev/c3f9a625f24f (release)