Closed Bug 1017305 Opened 6 years ago Closed 6 years ago

Several TPS tests are failing due to password manager changes on bug 853549

Categories

(Testing Graveyard :: TPS, defect)

defect
Not set
normal

Tracking

(firefox32 affected)

RESOLVED WORKSFORME
Tracking Status
firefox32 --- affected

People

(Reporter: whimboo, Unassigned)

References

()

Details

(Keywords: regression)

Attachments

(1 file)

A couple of tests for TPS-CI started to fail for password manager related tests:

test_sync.js 	TEST-UNEXPECTED-FAIL 	[phase2] Exception caught: ASSERTION FAILED! password not found No traceback available
test_passwords.js 	TEST-UNEXPECTED-FAIL 	[phase2] Exception caught: ASSERTION FAILED! password not found No traceback available
test_bug501528.js 	TEST-UNEXPECTED-FAIL 	[phase3] Exception caught: ASSERTION FAILED! password not found No traceback available
test_privbrw_passwords.js 	TEST-UNEXPECTED-FAIL 	[phase2] Exception caught: ASSERTION FAILED! password not found No traceback available
test_client_wipe.js 	TEST-UNEXPECTED-FAIL 	[phase2] Exception caught: ASSERTION FAILED! password not found No traceback available

This happened with the new build based off 1ff71bef9c99.

Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=4b59c0435b6e&tochange=1ff71bef9c99

Paolo, I highly assume it is related to your patch on bug 853549. Is that correct? 

If that is the case we have to get our password manager handling fixed:
http://mxr.mozilla.org/mozilla-central/source/services/sync/tps/extensions/tps/resource/modules/passwords.jsm
(In reply to Henrik Skupin (:whimboo) from comment #0)
> Paolo, I highly assume it is related to your patch on bug 853549. Is that
> correct?

I confirm!

> If that is the case we have to get our password manager handling fixed:
> http://mxr.mozilla.org/mozilla-central/source/services/sync/tps/extensions/
> tps/resource/modules/passwords.jsm

The patch there is not supposed to change the interface behavior - we have to understand whether we're observing a real regression, in which case we may have to add tests in the Login Manager for the scenario that is not currently covered.

Local Sync tests work correctly with the patch applied.

Can you provide more details on the root cause of the issue?
Duplicate of this bug: 1017913
I tried to reproduce this with the latest tinderbox build but I couldn't. 

(In reply to :Paolo Amadini from comment #1)(In reply to :Paolo Amadini from comment #1)
> The patch there is not supposed to change the interface behavior - we have
> to understand whether we're observing a real regression, in which case we
> may have to add tests in the Login Manager for the scenario that is not
> currently covered.
Which scenario is that, I can work on this, but I'm not aware of what was failing.
(In reply to Cosmin Malutan from comment #3)
> Which scenario is that, I can work on this, but I'm not aware of what was
> failing.

Heh, this is what I was asking to the experts here :-) I'm not familiar with TPS tests.
Whimboo, is this still a problem?  I just ran TPS from a recent build and got:

Test Summary

TEST-PASS | test_sync.js | 
TEST-PASS | test_prefs.js | 
TEST-PASS | test_tabs.js | 
TEST-PASS | test_passwords.js | 
TEST-PASS | test_history.js | 
TEST-PASS | test_formdata.js | 
TEST-PASS | test_bug530717.js | 
TEST-PASS | test_bug531489.js | 
TEST-PASS | test_bug538298.js | 
TEST-PASS | test_bug556509.js | 
TEST-PASS | test_bug562515.js | 
TEST-PASS | test_bug563989.js | 
TEST-PASS | test_bug535326.js | 
TEST-PASS | test_bug501528.js | 
TEST-PASS | test_bug575423.js | 
TEST-PASS | test_bug546807.js | 
TEST-PASS | test_history_collision.js | 
TEST-PASS | test_privbrw_formdata.js | 
TEST-PASS | test_privbrw_passwords.js | 
TEST-PASS | test_privbrw_tabs.js | 
TEST-PASS | test_bookmarks_in_same_named_folder.js | 
TEST-PASS | test_client_wipe.js | 
TEST-PASS | test_special_tabs.js | 
TEST-PASS | test_addon_sanity.js | 
TEST-PASS | test_addon_restartless_xpi.js | 
TEST-PASS | test_addon_nonrestartless_xpi.js | 
TEST-PASS | test_addon_reconciling.js | 
TEST-PASS | test_addon_wipe.js | 

If it is still a problem for you, any insights into how I might be able to repro it?
Flags: needinfo?(hskupin)
I already wanted to comment yesterday, but missed it. So we had a mystic decrease of test failures on June 10th. Not sure from where this is coming. We didn't make any changes to our tests. I only setup two more machines for TPS testing (OS X and Windows - those do not send reports yet). But I doubt that has something to do with. 

A raw pushlog of that range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=4bf5e8d7c440&tochange=b9f589c4d2e7

I cannot give a more detailed one given that the numbers are going a bit up and down:
https://mail.mozilla.org/pipermail/tps-reports/2014-June/thread.html

But a test this morning failed again on one of the TPS-machines:
test_passwords.js 	TEST-UNEXPECTED-FAIL 	[phase2] Exception caught: ASSERTION FAILED! password not found No traceback available

Maybe all that is related to the machine in SCL3, and the existence of the Squid proxy? I haven't run those tests lately on my own machine, so I might have to do it again.
Flags: needinfo?(hskupin)
(In reply to Henrik Skupin (:whimboo) from comment #6)
> A raw pushlog of that range is:
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=4bf5e8d7c440&tochange=b9f589c4d2e7

Nothing leaps out there...

> But a test this morning failed again on one of the TPS-machines:
> test_passwords.js 	TEST-UNEXPECTED-FAIL 	[phase2] Exception caught:
> ASSERTION FAILED! password not found No traceback available

It makes sense that this problem might be intermittent.  Is TPS able to capture logs for failed runs?  They might be insightful...
I have enabled sending reports by OS X and Windows now. Lets see if that works. That way we get better ideas of permanent failures, and those only happen on specific boxes.

(In reply to Mark Hammond [:markh] from comment #7)
> > But a test this morning failed again on one of the TPS-machines:
> > test_passwords.js 	TEST-UNEXPECTED-FAIL 	[phase2] Exception caught:
> > ASSERTION FAILED! password not found No traceback available
> 
> It makes sense that this problem might be intermittent.  Is TPS able to
> capture logs for failed runs?  They might be insightful...

We do not run in --debug mode yet, but that's something I could change for the sake of more detailed information. But we can only store the most recent log. So it will be overwritten each time a new testrun starts. Means we would have to closely follow running tests, and save off the log file immediately. I could try to do that today on the Linux box.
Attached file log.zip
Mark, for now I simply copied the existent log output from the console window. The attached file contains all the logs since June 2nd. It's a lot of content, but may be it helps us to identify the issue.
The exception message we currently throw are far away from being helpful. It would be good if those would contain the entries, which are getting compared.
Unfortunately it will be the sync logs we need.  To get them, tps will probably need to set a number of log related prefs and perform some logging magic to capture them for each test.  I'll try and look a little more into this tomorrow...
Blocks: 1024100
We haven't seen this failure at least in the last two weeks. I will reopen if we will see it again with the new TPS CI system being active.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.