Last Comment Bug 708494 - testUntrustedConnectionErrorPage.js fails to find element getMeOutOfHereButton
: testUntrustedConnectionErrorPage.js fails to find element getMeOutOfHereButton
Status: RESOLVED FIXED
[mozmill-test-failure][qa-]
:
Product: Mozilla QA
Classification: Other
Component: Mozmill Tests (show other bugs)
: unspecified
: All All
: -- normal (vote)
: ---
Assigned To: Remus Pop (:RemusPop)
:
Mentors:
http://mozmill-release.brasstacks.moz...
Depends on: 639939
Blocks:
  Show dependency treegraph
 
Reported: 2011-12-07 17:41 PST by Anthony Hughes (:ashughes) [GFX][QA][Mentor]
Modified: 2012-08-14 14:31 PDT (History)
5 users (show)
remus.pop: in‑litmus+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---
fixed
fixed
fixed
fixed
fixed


Attachments
patch v1 (1.21 KB, patch)
2011-12-08 02:36 PST, Remus Pop (:RemusPop)
no flags Details | Diff | Splinter Review
skip test (all branches) (2.08 KB, patch)
2011-12-08 07:45 PST, Remus Pop (:RemusPop)
vlad.mozbugs: review+
Details | Diff | Splinter Review
disable test (all) [checked-in] (2.11 KB, patch)
2011-12-08 09:52 PST, Anthony Hughes (:ashughes) [GFX][QA][Mentor]
anthony.s.hughes: review+
Details | Diff | Splinter Review
patch v2 (all branches) (1.72 KB, patch)
2012-06-11 07:22 PDT, Remus Pop (:RemusPop)
hskupin: review+
Details | Diff | Splinter Review

Description Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-07 17:41:32 PST
The following test fails today during the 9.0b5 testrun:

File: /testSecurity/testUntrustedConnectionErrorPage.js
Test: testUntrustedPageGetMeOutOfHereButton()
Fail: could not find element ID getMeOutOfHereButton
Comment 1 Remus Pop (:RemusPop) 2011-12-08 02:36:20 PST
Created attachment 579990 [details] [diff] [review]
patch v1

Changed the webpage which we access and used the one stated in the litmus testcase https://litmus.mozilla.org/show_test.cgi?id=8581
Comment 2 Henrik Skupin (:whimboo) 2011-12-08 02:59:37 PST
Al, do we already have an invalid SSL cert on mozqa.com which matches the one from https://overstock.com (Error code: ssl_error_bad_cert_domain)?
Comment 3 Maniac Vlad Florin (:vladmaniac) 2011-12-08 04:15:38 PST
(In reply to Remus Pop (:RemusPop) from comment #1)
> Created attachment 579990 [details] [diff] [review]
> patch v1
> 
> Changed the webpage which we access and used the one stated in the litmus
> testcase https://litmus.mozilla.org/show_test.cgi?id=8581

Can't we do this with a local test page? Cancelling review until we decide that
Comment 4 Remus Pop (:RemusPop) 2011-12-08 07:45:21 PST
Created attachment 580043 [details] [diff] [review]
skip test (all branches)

Skips the test until we figure out which webpage to use in the test.
Comment 5 Maniac Vlad Florin (:vladmaniac) 2011-12-08 07:55:51 PST
Comment on attachment 580043 [details] [diff] [review]
skip test (all branches)

We can skip the test for now - please put it in your queue because this is still a P1 fix needed
Comment 6 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-08 09:43:12 PST
Vlad, please always remember to r? patches to me when you r+ it, otherwise I might miss it for check-in.
Comment 7 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-08 09:52:15 PST
Created attachment 580081 [details] [diff] [review]
disable test (all) [checked-in]

Your commit message was backwards, here is the patch I have checked in.
Comment 9 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-08 09:57:47 PST
Note in previous comment the landing on 1.9.2 as this is failing across ALL versions of Firefox.
Comment 10 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-08 10:00:33 PST
(In reply to Henrik Skupin (:whimboo) from comment #2)
> Al, do we already have an invalid SSL cert on mozqa.com which matches the
> one from https://overstock.com (Error code: ssl_error_bad_cert_domain)?

I know that Al mentioned before that we had all the valid cert types installed. I'm not sure if any "invalid" certs were installed though.
Comment 11 Al Billings [:abillings] 2011-12-08 10:39:39 PST
We have an expired cert installed, as far as invalid types. I haven't installed any for another domain that doesn't match mozqa.com (well, the expired one is for another domain but we're hosting that domain now too).
Comment 12 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-08 10:44:40 PST
Thanks Al. Do you think one of those certs will work for this test? If not, should I open a dependency bug to get said invalid cert?
Comment 13 Al Billings [:abillings] 2011-12-08 11:00:34 PST
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #12)
> Thanks Al. Do you think one of those certs will work for this test? If not,
> should I open a dependency bug to get said invalid cert?

The litmus test is just looking for an untrusted connection when it expects one. Overstock.com versus www.overstock.com.

If you use https://summitbook.mozilla.org, we'll get the same sort of warning, because the cert is expired. That site is hosted on mozqa.com so it would seem to fit the bill.
Comment 14 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-08 11:39:05 PST
Thanks Al.

Remus, please refactor this test to use the URL given in comment 13.
Comment 15 Henrik Skupin (:whimboo) 2011-12-08 12:24:17 PST
(In reply to Al Billings [:abillings] from comment #13)
> If you use https://summitbook.mozilla.org, we'll get the same sort of
> warning, because the cert is expired. That site is hosted on mozqa.com so it
> would seem to fit the bill.

No, that's a different thing. Please check the detailed SSL warning. This is different for summitbook.
Comment 16 Al Billings [:abillings] 2011-12-08 12:36:57 PST
Well, the problem here is we have a bunch of legal certs for *.mozqa.com, a self-signed cert for the same, and the expired cert for summitbook. 

We don't have a simple "something.mozqa.com" cert that is a mismatch for the domain you load.
Comment 17 Henrik Skupin (:whimboo) 2011-12-08 15:15:39 PST
I'm not a security expert and can't tell how different those test cases will be. So I would propose that Anthony(?) gets in contact with the security guys and find out if we really need all those different invalid SSL tests, or if most of those are already covered by automated tests.

For the time being we can use the proposed external URL.
Comment 18 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-08 16:31:29 PST
(In reply to Henrik Skupin (:whimboo) from comment #17)
> I'm not a security expert and can't tell how different those test cases will
> be. So I would propose that Anthony(?) gets in contact with the security
> guys and find out if we really need all those different invalid SSL tests,
> or if most of those are already covered by automated tests.

I can certainly get in touch with some devs. Which "different invalid SSL tests" do you refer to? All the tests we have already automated in Mozmill? The ones in Litmus?

> For the time being we can use the proposed external URL.
Maybe I missed it, can you please specify the URL?
Comment 19 Al Billings [:abillings] 2011-12-08 17:08:12 PST
The url is in the litmus case in comment 1.
Comment 20 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-08 21:39:20 PST
Just some clarifications here...

https://overstock.com: Untrusted Connection due to ssl_error_bad_cert_domain
https://summitbook.mozilla.org: Untrusted Connection due to sec_error_expired_certificate

The Litmus testcase is not specific about the reason "Untrusted Connection" appears, only that it appears and that the "Get Me Out Of Here" button works. If this were a FFT for the particular symptom of the failure I would argue in favour of comment 2; but we aren't.

I am personally okay if we use summitbook.mozilla.org for this Mozmill test. Though, I would give Henrik a chance for rebuttal before moving forward with this refactor.
Comment 21 Henrik Skupin (:whimboo) 2011-12-09 02:25:54 PST
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #20)
> https://overstock.com: Untrusted Connection due to ssl_error_bad_cert_domain
> https://summitbook.mozilla.org: Untrusted Connection due to
> sec_error_expired_certificate

As you can see here both failures are different. So please get in contact with the security guys before making an assumption that we should not differentiate between both. This can but doesn't have to be wrong.

Personally I think having separate tests for any failure makes sense and we should not mix different SSL cert failures and websites.
Comment 22 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-09 06:21:03 PST
While I agree Henrik (I will reach out to security team about your question), but I think I didn't make my last question clear.

The purpose of this Litmus / Mozmill test is to check that the Untrusted Connection error page appears and the Get Me Out Of Here button works. It is not a test for the specific error messages. We should have a separate Litmus / Mozmill FFT for those in my opinion.

testUntrustedConnectionErrorPage.js
testSSLErrorBadCertDomain.js
testSSLErrorExpiredCertification.js

Hopefully that clarifies the point I was trying to make.
Comment 23 Henrik Skupin (:whimboo) 2011-12-09 06:25:04 PST
It depends on how elements on this pages are getting created. There is still the risk that for a special SSL failure the button will not exist. It's a black box for us.
Comment 24 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-12-09 06:28:55 PST
Okay, so I guess the first step here is to figure out from the security team the general workflow for generating these pages under various conditions.
Comment 25 Henrik Skupin (:whimboo) 2012-06-05 04:12:53 PDT
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #24)
> Okay, so I guess the first step here is to figure out from the security team
> the general workflow for generating these pages under various conditions.

Anthony, can you take this so that we get some progress on this bug?
Comment 26 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-06-05 12:24:55 PDT
(In reply to Henrik Skupin (:whimboo) from comment #25)
> Anthony, can you take this so that we get some progress on this bug?

I've emailed Sid Stamm who should be able to shed some light on this for us. I'll report back when I hear something from him.
Comment 27 Sid Stamm [:geekboy or :sstamm] 2012-06-05 13:12:38 PDT
Is your test racing the page load?  The button is in the XUL document, so it's not dynamically generated as far as I know.  

http://mxr.mozilla.org/mozilla-central/source/browser/components/certerror/content/aboutCertError.xhtml#217
Comment 28 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-06-05 14:40:05 PDT
I don't think it's racing the page load. Here is the test:
http://hg.mozilla.org/qa/mozmill-tests/file/tip/tests/functional/testSecurity/testUntrustedConnectionErrorPage.js
Comment 29 Sid Stamm [:geekboy or :sstamm] 2012-06-05 15:26:06 PDT
Is it possible that the URL load and TLS negotiation is succeeding sometimes (not showing the error) and failing (showing error) some other times? The live mozilla.org works fine for me without www, and I get no cert error.  What happens if you change it to https://irs.gov or one of the sites in comment 20?
Comment 30 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-06-05 15:45:11 PDT
I guess it's possible. Considering we are talking about an automated test, I'm wondering what we can do to simulate the conditions required for these error pages to appear on our own mozqa.com server.
Comment 31 Henrik Skupin (:whimboo) 2012-06-06 15:29:38 PDT
The original page has been changed. So we cannot make use of mozilla.org anytime longer. I have checked mozqa.com and as it looks like we have to get bug 666966 fully fixed first. Once that happened we could make use of it.

Anthony, or would that be the wrong cert for this test?
Comment 32 Al Billings [:abillings] 2012-06-06 16:24:11 PDT
bug 666966 was fixed last August.
Comment 33 Henrik Skupin (:whimboo) 2012-06-06 17:31:21 PDT
Remus, can you please update the page to use https://ssl-selfsigned.mozqa.com/ ? That's the one we own and which should not change. Thanks.
Comment 34 Al Billings [:abillings] 2012-06-06 17:32:45 PDT
I think you meant https://summitbook.mozilla.org.
Comment 35 Henrik Skupin (:whimboo) 2012-06-06 17:35:17 PDT
No, that was a failure. This test is about an untrusted connection and not an expired certificate. It got mixed-up.
Comment 36 Remus Pop (:RemusPop) 2012-06-11 07:22:03 PDT
Created attachment 631890 [details] [diff] [review]
patch v2 (all branches)

Modified the visited page to the one stated in comment 33
Comment 37 Henrik Skupin (:whimboo) 2012-06-11 14:04:29 PDT
Pushed:
http://hg.mozilla.org/qa/mozmill-tests/rev/561c9fb10b6d (default)

If we pass we can backport the patch. Remus, on which branches we have to land? Can you please mark those in the status flags list?
Comment 38 Remus Pop (:RemusPop) 2012-06-12 02:08:20 PDT
All branches are affected and work with this fix.
Comment 40 Henrik Skupin (:whimboo) 2012-06-14 05:21:27 PDT
Which Litmus tests have been updated?
Comment 41 Remus Pop (:RemusPop) 2012-06-14 05:24:21 PDT
The weren't disabled.

Note You need to log in before you can comment on or make changes to this bug.