Closed Bug 708494 Opened 13 years ago Closed 12 years ago

testUntrustedConnectionErrorPage.js fails to find element getMeOutOfHereButton

Categories

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Product:
Mozilla QA Graveyard
Component:
Mozmill Tests
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox13 fixed, firefox14 fixed, firefox15 fixed, firefox16 fixed, firefox-esr10 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox13 --- fixed
firefox14 --- fixed
firefox15 --- fixed
firefox16 --- fixed
firefox-esr10 --- fixed

People

(Reporter: u279076, Assigned: remus.pop)

References

(
URL
)

Details

(Whiteboard: [mozmill-test-failure][qa-])

Attachments

(2 files, 2 obsolete files)

disable test (all) [checked-in]
2.11 KB, patch
u279076
: review+
Details | Diff | Splinter Review
patch v2 (all branches)
1.72 KB, patch
whimboo
: review+
Details | Diff | Splinter Review 
The following test fails today during the 9.0b5 testrun:

File: /testSecurity/testUntrustedConnectionErrorPage.js
Test: testUntrustedPageGetMeOutOfHereButton()
Fail: could not find element ID getMeOutOfHereButton
Assignee: nobody → remus.pop
Attached patch patch v1 (obsolete) — — Splinter Review 
Changed the webpage which we access and used the one stated in the litmus testcase https://litmus.mozilla.org/show_test.cgi?id=8581
Attachment #579990 - Flags: review?(vlad.mozbugs)
Al, do we already have an invalid SSL cert on mozqa.com which matches the one from https://overstock.com (Error code: ssl_error_bad_cert_domain)?
(In reply to Remus Pop (:RemusPop) from comment #1)
> Created attachment 579990 [details] [diff] [review]
> patch v1
> 
> Changed the webpage which we access and used the one stated in the litmus
> testcase https://litmus.mozilla.org/show_test.cgi?id=8581

Can't we do this with a local test page? Cancelling review until we decide that
Attachment #579990 - Flags: review?(vlad.mozbugs)
Attached patch skip test (all branches) (obsolete) — — Splinter Review 
Skips the test until we figure out which webpage to use in the test.
Attachment #580043 - Flags: review?(vlad.mozbugs)
Comment on attachment 580043 [details] [diff] [review]
skip test (all branches)

We can skip the test for now - please put it in your queue because this is still a P1 fix needed
Attachment #580043 - Flags: review?(vlad.mozbugs) → review+
Vlad, please always remember to r? patches to me when you r+ it, otherwise I might miss it for check-in.
Your commit message was backwards, here is the patch I have checked in.
Attachment #580043 - Attachment is obsolete: true
Attachment #580081 - Flags: review+
Comment on attachment 580081 [details] [diff] [review]
disable test (all) [checked-in]

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/067f20985b4d (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/f97711232805 (mozilla-aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/b8e4bf0d6b7a (mozilla-beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/d36f65c96e82 (mozilla-release)
http://hg.mozilla.org/qa/mozmill-tests/rev/70ba5b2392fe (mozilla-1.9.2)
Attachment #580081 - Attachment description: disable test (all) → disable test (all) [checked-in]
Note in previous comment the landing on 1.9.2 as this is failing across ALL versions of Firefox.
Status: NEW → ASSIGNED
Whiteboard: [mozmill-test-failure] → [mozmill-test-failure][mozmill-test-skipped]
(In reply to Henrik Skupin (:whimboo) from comment #2)
> Al, do we already have an invalid SSL cert on mozqa.com which matches the
> one from https://overstock.com (Error code: ssl_error_bad_cert_domain)?

I know that Al mentioned before that we had all the valid cert types installed. I'm not sure if any "invalid" certs were installed though.
We have an expired cert installed, as far as invalid types. I haven't installed any for another domain that doesn't match mozqa.com (well, the expired one is for another domain but we're hosting that domain now too).
Thanks Al. Do you think one of those certs will work for this test? If not, should I open a dependency bug to get said invalid cert?
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #12)
> Thanks Al. Do you think one of those certs will work for this test? If not,
> should I open a dependency bug to get said invalid cert?

The litmus test is just looking for an untrusted connection when it expects one. Overstock.com versus www.overstock.com.

If you use https://summitbook.mozilla.org, we'll get the same sort of warning, because the cert is expired. That site is hosted on mozqa.com so it would seem to fit the bill.
Thanks Al.

Remus, please refactor this test to use the URL given in comment 13.
(In reply to Al Billings [:abillings] from comment #13)
> If you use https://summitbook.mozilla.org, we'll get the same sort of
> warning, because the cert is expired. That site is hosted on mozqa.com so it
> would seem to fit the bill.

No, that's a different thing. Please check the detailed SSL warning. This is different for summitbook.
Well, the problem here is we have a bunch of legal certs for *.mozqa.com, a self-signed cert for the same, and the expired cert for summitbook. 

We don't have a simple "something.mozqa.com" cert that is a mismatch for the domain you load.
I'm not a security expert and can't tell how different those test cases will be. So I would propose that Anthony(?) gets in contact with the security guys and find out if we really need all those different invalid SSL tests, or if most of those are already covered by automated tests.

For the time being we can use the proposed external URL.
(In reply to Henrik Skupin (:whimboo) from comment #17)
> I'm not a security expert and can't tell how different those test cases will
> be. So I would propose that Anthony(?) gets in contact with the security
> guys and find out if we really need all those different invalid SSL tests,
> or if most of those are already covered by automated tests.

I can certainly get in touch with some devs. Which "different invalid SSL tests" do you refer to? All the tests we have already automated in Mozmill? The ones in Litmus?

> For the time being we can use the proposed external URL.
Maybe I missed it, can you please specify the URL?
The url is in the litmus case in comment 1.
Just some clarifications here...

https://overstock.com: Untrusted Connection due to ssl_error_bad_cert_domain
https://summitbook.mozilla.org: Untrusted Connection due to sec_error_expired_certificate

The Litmus testcase is not specific about the reason "Untrusted Connection" appears, only that it appears and that the "Get Me Out Of Here" button works. If this were a FFT for the particular symptom of the failure I would argue in favour of comment 2; but we aren't.

I am personally okay if we use summitbook.mozilla.org for this Mozmill test. Though, I would give Henrik a chance for rebuttal before moving forward with this refactor.
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #20)
> https://overstock.com: Untrusted Connection due to ssl_error_bad_cert_domain
> https://summitbook.mozilla.org: Untrusted Connection due to
> sec_error_expired_certificate

As you can see here both failures are different. So please get in contact with the security guys before making an assumption that we should not differentiate between both. This can but doesn't have to be wrong.

Personally I think having separate tests for any failure makes sense and we should not mix different SSL cert failures and websites.
While I agree Henrik (I will reach out to security team about your question), but I think I didn't make my last question clear.

The purpose of this Litmus / Mozmill test is to check that the Untrusted Connection error page appears and the Get Me Out Of Here button works. It is not a test for the specific error messages. We should have a separate Litmus / Mozmill FFT for those in my opinion.

testUntrustedConnectionErrorPage.js
testSSLErrorBadCertDomain.js
testSSLErrorExpiredCertification.js

Hopefully that clarifies the point I was trying to make.
It depends on how elements on this pages are getting created. There is still the risk that for a special SSL failure the button will not exist. It's a black box for us.
Okay, so I guess the first step here is to figure out from the security team the general workflow for generating these pages under various conditions.
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #24)
> Okay, so I guess the first step here is to figure out from the security team
> the general workflow for generating these pages under various conditions.

Anthony, can you take this so that we get some progress on this bug?
(In reply to Henrik Skupin (:whimboo) from comment #25)
> Anthony, can you take this so that we get some progress on this bug?

I've emailed Sid Stamm who should be able to shed some light on this for us. I'll report back when I hear something from him.
Is your test racing the page load?  The button is in the XUL document, so it's not dynamically generated as far as I know.  

http://mxr.mozilla.org/mozilla-central/source/browser/components/certerror/content/aboutCertError.xhtml#217
Is it possible that the URL load and TLS negotiation is succeeding sometimes (not showing the error) and failing (showing error) some other times? The live mozilla.org works fine for me without www, and I get no cert error.  What happens if you change it to https://irs.gov or one of the sites in comment 20?
I guess it's possible. Considering we are talking about an automated test, I'm wondering what we can do to simulate the conditions required for these error pages to appear on our own mozqa.com server.
The original page has been changed. So we cannot make use of mozilla.org anytime longer. I have checked mozqa.com and as it looks like we have to get bug 666966 fully fixed first. Once that happened we could make use of it.

Anthony, or would that be the wrong cert for this test?
Depends on: 666966
bug 666966 was fixed last August.
No longer depends on: 666966
Remus, can you please update the page to use https://ssl-selfsigned.mozqa.com/ ? That's the one we own and which should not change. Thanks.
Depends on: 639939
No, that was a failure. This test is about an untrusted connection and not an expired certificate. It got mixed-up.
Modified the visited page to the one stated in comment 33
Attachment #579990 - Attachment is obsolete: true
Attachment #631890 - Flags: review?(hskupin)
Attachment #631890 - Flags: review?(hskupin) → review+
Pushed:
http://hg.mozilla.org/qa/mozmill-tests/rev/561c9fb10b6d (default)

If we pass we can backport the patch. Remus, on which branches we have to land? Can you please mark those in the status flags list?
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
status-firefox16: --- → fixed
Resolution: --- → FIXED
All branches are affected and work with this fix.
status-firefox-esr10: --- → affected
status-firefox13: --- → affected
status-firefox14: --- → affected
status-firefox15: --- → affected
Flags: in-litmus?(remus.pop)
Whiteboard: [mozmill-test-failure][mozmill-test-skipped] → [mozmill-test-failure]
Flags: in-litmus?(remus.pop) → in-litmus+
Which Litmus tests have been updated?
The weren't disabled.
Whiteboard: [mozmill-test-failure] → [mozmill-test-failure][qa-]
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: