Closed Bug 784305 Opened 10 years ago Closed 8 years ago

Mozmill test failure /restartTests/testAddons_RestartlessExtensionWorksAfterRestart/test2.js | Current URL should match expected URL

Categories

(Mozilla QA Graveyard :: Mozmill Tests, defect, P1)

defect

Tracking

(firefox35 fixed, firefox36 fixed, firefox37 fixed, firefox38 fixed)

RESOLVED FIXED
Tracking Status
firefox35 --- fixed
firefox36 --- fixed
firefox37 --- fixed
firefox38 --- fixed

People

(Reporter: vladmaniac, Assigned: teodruta)

References

()

Details

(Keywords: regression, Whiteboard: [mozmill-test-failure])

Attachments

(5 files, 4 obsolete files)

Assignee: nobody → vlad.mozbugs
Status: NEW → ASSIGNED
Whiteboard: [mozmill-test-failure]
This is reproducible locally
So this is failing on default, and I bet it has something to do with landing of Bug 782639 in litmus-data. 

Andreea and Henrik can you confirm this?
(In reply to Maniac Vlad Florin (:vladmaniac) from comment #2)
> So this is failing on default, and I bet it has something to do with landing
> of Bug 782639 in litmus-data. 

Why should this be related? This test doesn't make use of the blocklisting feature. What happens exactly when you can reproduce it locally?
Severity: normal → major
Severity: major → normal
Priority: -- → P2
(In reply to Henrik Skupin (:whimboo) from comment #3)
> (In reply to Maniac Vlad Florin (:vladmaniac) from comment #2)
> > So this is failing on default, and I bet it has something to do with landing
> > of Bug 782639 in litmus-data. 
> 
> Why should this be related? This test doesn't make use of the blocklisting
> feature. What happens exactly when you can reproduce it locally?

Tested and it seems that our extension does not work with Nightly anymore. 
Just a reminder of how it works: 

Normally it inserts a context menu entry with the label "Open mozqa.com" and we click that to check its working. On Fx Nightly, it stopped working for reasons unknown to me at the moment. 
We have the context menu entry, but clicking on it does not open our page. This can be also tested manually, leading to the same result.
Is it based on the addons sdk? If yes we should probably make it not so but a simple bootstrapped addon. Can you check for a regression range? It could be an Addon SDK regression if we base on it.
(In reply to Henrik Skupin (:whimboo) from comment #5)
> Is it based on the addons sdk? If yes we should probably make it not so but
> a simple bootstrapped addon. Can you check for a regression range? It could
> be an Addon SDK regression if we base on it.

Not sure what you mean with "simple bootstrapped addon" because all add-ons build with SDK or add-on Builder are bootstrapped [1]

[1] https://developer.mozilla.org/en-US/docs/Extensions/Bootstrapped_extensions
The extension is not SDK based because: 

* it has no directory structure to match the one created with sdk using "cfx init" 
* it does not include package.json which SDK uses to package the extensions 
* running a simple "cfx test" would trigger errors 

This is enough for me to say its not SDK based. Likely its made with Add-on Builder. 
Next step is finding the regression range which I think its easy since we have seen failures only since yesterday.
Extension logging pref echoes the following error and I think I'm pretty close in finding a fix: 

error: An exception occurred.
Traceback (most recent call last):
  File "javascript:self.on("click", function (node, data) { window.location.href = "http://mozqa.com/data/firefox/layout/mozilla.html";});", line 1, in 
TypeError: self.on is not a function
(In reply to Maniac Vlad Florin (:vladmaniac) from comment #7)
> The extension is not SDK based because: 

Not sure where you get this assumption from. If you check harness-options.js in the root folder it states: "sdkVersion": "1.2.1". So it is a very old version of the SDK, which could indeed cause problems. We probably should rebuild and also put its source into the litmus-data repository.

(In reply to Maniac Vlad Florin (:vladmaniac) from comment #6)
> (In reply to Henrik Skupin (:whimboo) from comment #5)
> > Is it based on the addons sdk? If yes we should probably make it not so but
> > a simple bootstrapped addon. Can you check for a regression range? It could
> > be an Addon SDK regression if we base on it.
> 
> Not sure what you mean with "simple bootstrapped addon" because all add-ons
> build with SDK or add-on Builder are bootstrapped [1]

A simple bootstrapped addon does *NOT* make use of the Addon SDK. So we would not be affected by SDK incompatibilities and would have a much smaller code base.
Re-building with SDK 1.9 does not fix the problem. 
Currently looking at bug 783497 to see if this is the cause
Attached patch proposed fix v1.0 (obsolete) — Splinter Review
* This patch is replacing data/addons/extensions/restartless.xpi with another add-on which is 1.9 SDK based, but with simple code base 
Looks like it fixes the problem
The patch was tested with default branch, this is where we get the error only. 

Most certainly, when we are going to land this, we'll need an infrastructure bug to put the new version of restartless.xpi in litmus-data
Attachment #654151 - Flags: review?(hskupin)
Attachment #654151 - Flags: review?(dave.hunt)
Lets get this test disabled on default ASAP until the underlying extension issue has been fixed.
Attached patch skip patch [checked-in] (obsolete) — Splinter Review
Landed this before todays builds are up so we are not swamped with test failures.
Whiteboard: [mozmill-test-failure] → [mozmill-test-failure][mozmill-test-skipped]
Comment on attachment 654160 [details] [diff] [review]
skip patch [checked-in]

Landed skip patch on default as:
http://hg.mozilla.org/qa/mozmill-tests/rev/ce8193bf5289
Attachment #654160 - Attachment description: skip patch → skip patch [checked-in]
Comment on attachment 654151 [details] [diff] [review]
proposed fix v1.0

This will not work because the addon will not be compatible with older versions of Firefox like ESR10. I really would like to know what has been changed in the code so that it is working now.
Attachment #654151 - Flags: review?(hskupin)
Attachment #654151 - Flags: review?(dave.hunt)
Attachment #654151 - Flags: review-
(In reply to Henrik Skupin (:whimboo) from comment #16)
> Comment on attachment 654151 [details] [diff] [review]
> proposed fix v1.0
> 
> This will not work because the addon will not be compatible with older
> versions of Firefox like ESR10. I really would like to know what has been
> changed in the code so that it is working now.

aurora, beta, release and esr work fine with the old version of the add-on. 
I just build it with the latest released sdk: What I did exactly was: 

* cfx init so I can get the folder structure
* copy paste main.js from the old add-on
* updated package.json to reflect the data from the old install.rdf
* cfx test
* cfx xpi

..and worked. did nothing more or less.
I might work at the moment but as you can see in the addons manager the extension is marked to be incompatible. I would suggest you file a new bug so we can update the extension in the litmus-data repository, which should in any case always happen first. Then we can grab the extension for Mozmill's usage.
Depends on: 785014
Whiteboard: [mozmill-test-failure][mozmill-test-skipped] → [mozmill-test-failure][mozmill-test-skipped] s=2012-8-27 u=failure c=addons p=1
Whiteboard: [mozmill-test-failure][mozmill-test-skipped] s=2012-8-27 u=failure c=addons p=1 → [mozmill-test-failure][mozmill-test-skipped] s=q3 u=failure c=addons p=1
When will we get an update here? It's on the list of tasks since 3 weeks now without an update. Please take care of this bug so we can re-enable the failure ASAP.
Assignee: vlad.mozbugs → nobody
Assignee: nobody → andreea.matei
While I was waiting for the new addon, I've checked this on OS X 10.6.8 and 10.7.5, Ubuntu 11.10 where the two reports were, single run and functional testruns, but it's not reproducible anymore. 
Could be that with the new releases on nightly, the current addon works fine and the behavior from comment 4 is not happening now.
Andrea, were you able to reproduce it with one of the builds which showed this failure?
Henrik, yes, the build 20120821 from August does not open mozqa.com when clicking it on the context menu.
So it could have been that a patch for Firefox fixed that. Would you mind to do a quick regression test since when it is working?
It was fixed in the build from the next day - August 22nd. I see some failure reports in that day, but they were with the build from 21st.
Please provide a pushlog as usual so we can hopefully figure out what has fixed it.
The only I could find in the pushlog which affects restartless extensions is bug 784233. Bobby, do you think that this one has fixed our problem here which caused that our test extension hasn't shown a context menu when the user did a right click in a webpage?

Andreea, if the test works now lets get it re-enabled. There is no need to wait longer. Good to hear that!
Attached patch enable patch (obsolete) — Splinter Review
Patch that enables the test.
Attachment #675076 - Flags: review?(hskupin)
Attachment #675076 - Flags: review?(dave.hunt)
(In reply to Henrik Skupin (:whimboo) from comment #27)
> The only I could find in the pushlog which affects restartless extensions is
> bug 784233. Bobby, do you think that this one has fixed our problem here
> which caused that our test extension hasn't shown a context menu when the
> user did a right click in a webpage?

Not unless that problem only existed for a few days. Bug 784233 was effectively a partial backout of bug 553102, which landed a few days earlier.
The failures first appeared on 2012-08-16 and were intermittent at this time. Starting by 2012-08-21 they were permanent failures. Given by the work from Andreea they were gone on 2012-08-22.
Comment on attachment 675076 [details] [diff] [review]
enable patch

We don't need a new patch for that. We can simply backout the skip patch.
Attachment #675076 - Attachment is obsolete: true
Attachment #675076 - Flags: review?(hskupin)
Attachment #675076 - Flags: review?(dave.hunt)
Backed-out skip patch:
http://hg.mozilla.org/qa/mozmill-tests/rev/b178f3346289 (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/38eb00a470f5 (aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/867fe223027a (beta)

Given that it will be hard to find the bug which fixed it just call it WFM.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → WORKSFORME
Whiteboard: [mozmill-test-failure][mozmill-test-skipped] s=q3 u=failure c=addons p=1 → [mozmill-test-failure] s=q3 u=failure c=addons p=1
This is failing again on mozilla-aurora:

http://mozmill-ci.blargon7.com/#/functional/report/190d57cf64e3c5c3996c0c1059017bd7
http://mozmill-ci.blargon7.com/#/functional/report/2e7486a8c08b5541b9074117e0f65aa9

Let's get this skipped again and investigate the failure on Aurora.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Whiteboard: [mozmill-test-failure] s=q3 u=failure c=addons p=1 → [mozmill-test-failure] s=121029 u=failure c=addons p=1
Re-skipped on aurora:
http://hg.mozilla.org/qa/mozmill-tests/rev/05b0c885ab09
Whiteboard: [mozmill-test-failure] s=121029 u=failure c=addons p=1 → [mozmill-test-failure][mozmill-test-skipped] s=121029 u=failure c=addons p=1
I will investigate this more on Aurora. I wonder if it's not a fix that has been applied to Nightly and not also on Aurora, from the pushlog.
I was working on a regression for Aurora and I saw this report:
http://mozmill-ci.blargon7.com/#/functional/report/190d57cf64e3c5c3996c0c10593796ca

which is for Aurora, with a build from yesterday and the test it's not skipped there. Neither on 28th.
Although when I check an aurora repo the test is skipped.
We had no reports failing on August 21st for Aurora, just Nightly was affected back then. 
I ran the test over several builds, from August but also now in October (on Windows cause there was happening) and I can't reproduce.
We will have to check on the machine where it did fail for debugging.
(In reply to Andreea Matei [:AndreeaMatei] from comment #36)
> which is for Aurora, with a build from yesterday and the test it's not
> skipped there. Neither on 28th.
> Although when I check an aurora repo the test is skipped.

That was because Mozmill-CI hasn't updated itself to use the latest version of the mozilla-aurora branch. As you can see it's skipped now:

http://hg.mozilla.org/qa/mozmill-tests/file/mozilla-aurora/tests/functional/restartTests/testAddons_RestartlessExtensionWorksAfterRestart/test2.js
Status: REOPENED → ASSIGNED
It appears on the latest Aurora from today this is not reproducible anymore. I will look back some builds to see when it started to work exactly.
So I've checked older builds where it used to fail and it's not reproducing on them either. 
From August until now, like on default, it's hard to find what has fixed this. I will monitor this on mozmill-ci after unskipping.
Backed-out skip patch:
http://hg.mozilla.org/qa/mozmill-tests/rev/5adb83935f30 (aurora)

If it sticks in the next couple of days we can mark the bug as fixed.
Whiteboard: [mozmill-test-failure][mozmill-test-skipped] s=121029 u=failure c=addons p=1 → [mozmill-test-failure] s=121029 u=failure c=addons p=1
Hasn't failed on Aurora these days. Yey!
Status: ASSIGNED → RESOLVED
Closed: 10 years ago10 years ago
Resolution: --- → FIXED
We haven't landed a patch here, so it has to be WFM.
Resolution: FIXED → WORKSFORME
Attached patch skip patch v1.1Splinter Review
The previous skip patch did not cleanly apply. Updated skip patch.
Attachment #654160 - Attachment is obsolete: true
Attachment #730636 - Flags: review?(andreea.matei)
How often did that fail? If it's a one off I don't think we should disable the test.
18 times today until now.
So a skip might be good.
It's failing a lot. So please work on this bug next so we get it fixed. Thanks.
Status: REOPENED → ASSIGNED
Priority: P2 → P1
Attachment #730636 - Flags: review?(dave.hunt)
This is the error I get before the one in the title, several times (on each test):

error: An exception occurred.
Traceback (most recent call last):
  File "javascript:self.on("click", function (node, data) { window.location.href = "http://mozqa.com/data/firefox/layout/mozilla.html";});", line 1, in 
TypeError: self.on is not a function

With a build from yesterday it works, I'm looking at today's pushlog.
Comment on attachment 730636 [details] [diff] [review]
skip patch v1.1

Review of attachment 730636 [details] [diff] [review]:
-----------------------------------------------------------------

Landed skip patch as:
http://hg.mozilla.org/qa/mozmill-tests/rev/ac4ea323a447
Attachment #730636 - Flags: review?(dave.hunt)
Attachment #730636 - Flags: review?(andreea.matei)
Attachment #730636 - Flags: review+
I can reproduce manually with today's nightly, after installing the addon and clicking Open mozqa.com in context menu, nothing happens.

Could be this changeset? I'm building now to confirm:
https://hg.mozilla.org/mozilla-central/rev/f46bbb876415
Nope. This is the pushlog, a few merges again, but I'm getting good at this :)
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=178a4a770bb1&tochange=962f5293f87f
With hg bisect I found this as the responsible changeset:
http://hg.mozilla.org/mozilla-central/rev/478ef0049c1a

Henrik mentioned some time ago in comment 27 the bug that at that time fixed our problem. Now with the removal of that code, it reappeared. 
So I think it's time to update our addon, in bug 785014.
If it's unclear how to fix our add-on we should ask the jetpack guys over on bug 785014. :Ochameau might be able to help us.

I would have loved if we wouldn't have reopened this bug but created a new one. Anyway, lets get this fixed soon.
Assignee: andreea.matei → nobody
I don't know if it helps but I have had the same error message mentioned in the bug while working on bug 992137.

error: An exception occurred.
Traceback (most recent call last):
  File "javascript:self.on("click", function (node, data) { window.location.href = "http://mozqa.com/data/firefox/layout/mozilla.html";});", line 1, in
TypeError: self.on is not a function


On Firefox Beta 29.0 on Linux Mint running mozmill-tests/firefox/lib/tests/manifest.ini with the most recent version of MozMill.
Gilles, I think that warrants a new bug. When you file it please make sure to check which test in details fails here. Thanks.
This is still reproducible manually & we are blocked here by bug 785014 where we need to update our restartless.xpi addon to work. 

The test will still need some updates after the addon fix, mostly because we are not correctly checking if the addon is installed now (most likely some pages in the about:addons page)  & our current style guide.

We still have a skipped test here so this stays as P1.
Whiteboard: [mozmill-test-failure] s=121029 u=failure c=addons p=1 → [mozmill-test-failure][Blocked by bug 785014]
Whiteboard: [mozmill-test-failure][Blocked by bug 785014] → [mozmill-test-failure][mozmill-test-skipped][blocked by bug 785014]
Teodor, you said that this problem is not reproducible anymore with the current restartless extension. Now I see that this test is disabled. Can you please check what's going on here, and if we indeed need a bootstrapped extension?
Flags: needinfo?(teodor.druta)
(In reply to Henrik Skupin (:whimboo) from comment #58)
> Teodor, you said that this problem is not reproducible anymore with the
> current restartless extension. Now I see that this test is disabled. Can you
> please check what's going on here, and if we indeed need a bootstrapped
> extension?

Yes, I haven't seen that this test is actually skipped for a really long time, I'll take a look at the reported failures, to see if this really requires a new extension.
Flags: needinfo?(teodor.druta)
Yeah, I think we need a new restartless extension, this one doesn't seem to work anymore. Reopening Bug 785014.
Attached patch fixrestartless.patch (obsolete) — Splinter Review
This is a fixed version of the RestartlessExtensionWorksAfterRestart test.
It makes use of the new restartless addon from Bug 785014.
Assignee: nobody → teodor.druta
Attachment #8554507 - Flags: review?(mihaela.velimiroviciu)
Attachment #8554507 - Flags: review?(andreea.matei)
QA Contact: teodor.druta → hskupin
Comment on attachment 8554507 [details] [diff] [review]
fixrestartless.patch

Review of attachment 8554507 [details] [diff] [review]:
-----------------------------------------------------------------

It looks fine to me.
Attachment #8554507 - Flags: review?(mihaela.velimiroviciu) → review+
Added the new restartless.xpi addon to data.
Attachment #654151 - Attachment is obsolete: true
Attachment #8554507 - Attachment is obsolete: true
Attachment #8554507 - Flags: review?(andreea.matei)
Attachment #8557837 - Flags: review?(mihaela.velimiroviciu)
Attachment #8557837 - Flags: review?(andreea.matei)
Attachment #8557837 - Flags: review?(mihaela.velimiroviciu) → review+
Comment on attachment 8557837 [details] [diff] [review]
fixrestartless.patch

Review of attachment 8557837 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/qa/mozmill-tests/rev/4a56e147df51 (default)
Attachment #8557837 - Flags: review?(andreea.matei) → review+
This test is skipped, we should unskip it since it was fixed.
This is the unskip patch for default.
Attachment #8559662 - Flags: review?(andreea.matei)
Comment on attachment 8559662 [details] [diff] [review]
unskiprestartless.patch

Review of attachment 8559662 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/qa/mozmill-tests/rev/f3081462ebae (default)
Attachment #8559662 - Flags: review?(andreea.matei) → review+
Here is the patch for aurora.
Attachment #8559694 - Flags: review?(andreea.matei)
And here is the patch for beta and release.
Attachment #8559695 - Flags: review?(andreea.matei)
Comment on attachment 8559694 [details] [diff] [review]
fixrestartless_aurora.patch

Review of attachment 8559694 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/qa/mozmill-tests/rev/f23e56eeae33 (aurora)
Attachment #8559694 - Flags: review?(andreea.matei) → review+
Whiteboard: [mozmill-test-failure][mozmill-test-skipped][blocked by bug 785014] → [mozmill-test-failure][mozmill-test-skipped]
Comment on attachment 8559695 [details] [diff] [review]
fixrestartless_beta_release.patch

Review of attachment 8559695 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/qa/mozmill-tests/rev/9714853ffb47 (beta)
https://hg.mozilla.org/qa/mozmill-tests/rev/c3f9a625f24f (release)
Attachment #8559695 - Flags: review?(andreea.matei) → review+
Status: ASSIGNED → RESOLVED
Closed: 10 years ago8 years ago
Resolution: --- → FIXED
Whiteboard: [mozmill-test-failure][mozmill-test-skipped] → [mozmill-test-failure]
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.