Last Comment Bug 875142 - Webapp manifest validator does not support Server Name Indication
: Webapp manifest validator does not support Server Name Indication
Status: VERIFIED FIXED
:
Product: Marketplace
Classification: Server Software
Component: Validation (show other bugs)
: 1.5
: All All
: P2 normal (vote)
: 2013-11-19
Assigned To: Mathieu Pillard [:mat]
:
Mentors:
: 933755 936564 (view as bug list)
Depends on: 892654
Blocks: 826960
  Show dependency treegraph
 
Reported: 2013-05-22 16:40 PDT by Fabian Köster
Modified: 2013-11-15 07:30 PST (History)
7 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
The failing webapp manifest (590 bytes, text/plain)
2013-05-22 16:45 PDT, Fabian Köster
no flags Details
Wget succeeds downloading files (1.50 KB, text/plain)
2013-05-22 16:47 PDT, Fabian Köster
no flags Details

Description Fabian Köster 2013-05-22 16:40:06 PDT
User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31

Steps to reproduce:

I created a simple manifest file https://www.bonner-nacht.de/manifest.webapp
which is delivered with the correct mime-type and defines a launch-path '/' and some icons which are all perfectly accessable.

I tried to submit the webapp into marketplace.


Actual results:

The validator fails with the following output:

Error while requesting icon
Fehler: A remote resource was requested, but an error prevented the request from completing. This may include connection, DNS, or HTTP issues.
Requested resource: https://www.bonner-nacht.de/img/logos/BonnerNacht48.png


Expected results:

I do not see why it fails. The servers log show only one access for the manifest, but no following requests. I therefore think the validator is broken.

Even if there is a problem with my manifest or webserver I cannot fix it because the validator does not give detailed information about the problem.
Comment 1 Fabian Köster 2013-05-22 16:45:01 PDT
Created attachment 753031 [details]
The failing webapp manifest
Comment 2 Fabian Köster 2013-05-22 16:47:50 PDT
Created attachment 753034 [details]
Wget succeeds downloading files

I also attach a log showing that my client can retrieve the files using wget without problems.
Comment 3 Kumar McMillan [:kumar] (needinfo all the things) 2013-06-19 13:06:43 PDT
I tried this out and it's failing the same way for me. I'm not sure why though. The icon is accessible.
Comment 4 Matt Basta [:basta] 2013-07-10 18:19:23 PDT
requests.exceptions.SSLError: hostname 'www.bonner-nacht.de' doesn't match either of 'wiki.bonner-nacht.de', 'bonner-nacht.de'

The SSL errors with the www.bonner-nacht.de domain are causing this error. This error is seen with python-requests 1.2.3.

> import requests
> requests.get('https://www.bonner-nacht.de/img/logos/BonnerNacht48.png')

At first, the request hung but now it consistently fails with SSL verification errors.
Comment 5 Fabian Köster 2013-07-10 23:57:21 PDT
Matt, thank you very much for explaining the cause of this error!

I am not okay with closing this bug as 'invalid', though. For two reasons:

1.) The cause of the problem seems to be that python-requests does not correctly work with VirtualHosts (multiple domain names on same IP). Other http clients (like Chromium, Firefox, wget, ...) do not have this problem and use the correct ssl certificate for this site. I think the webapp validator should also work with vhosts and the bug in python-requests should be fixed (i will search for an upstream fix).

2.) The webapp validator should forward the error message you gave me to the user and until the python-requests bug is fixed should say "vhosts are not supported yet".

Kind regards,
Fabian
Comment 6 Fabian Köster 2013-07-11 00:00:41 PDT
I just noticed the assignment of ssl certificates to virtual host domain names is called "Server Name Indication".
Comment 7 Fabian Köster 2013-07-11 00:03:02 PDT
I just found the issue upstream in python-requests:

https://github.com/kennethreitz/requests/issues/749
Comment 8 Fabian Köster 2013-07-11 00:46:06 PDT
As a workaround I changed the default certificate to the one for www.bonner-nacht.de, now the validation succeeds.
Comment 9 Kumar McMillan [:kumar] (needinfo all the things) 2013-07-11 11:01:37 PDT
Thanks for digging up that commit. It looks like the validator actually uses the version of requests that support this but we seem to hit this error (which is caught and swallowed by requests):

>>> from requests.packages.urllib3.contrib import pyopenssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kumar/tmp/app-validator-env2/lib/python2.6/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 23, in <module>
    from ndg.httpsclient.ssl_peer_verification import (ServerSSLCertVerification,
ImportError: No module named ndg.httpsclient.ssl_peer_verification

We just need some more modules. Basta, r? https://github.com/mozilla/app-validator/pull/10

Zamboni's prod/compiled requirements will need to be fixed up too. 

We should fix this but P2 since there is a workaround.
Comment 10 Kumar McMillan [:kumar] (needinfo all the things) 2013-07-11 12:45:28 PDT
Fixed in the validator:
https://github.com/mozilla/app-validator/commit/9190ff79244f521b0d53a873da2904687f9ddf4b
https://github.com/mozilla/app-validator/commit/f604ed588d0fca628e3b7bdfb01bde6329e6cc10

Zamboni patches coming next...
Comment 11 Kumar McMillan [:kumar] (needinfo all the things) 2013-07-11 12:46:21 PDT
we'll need to wait on bug 892654 until we can deploy this
Comment 12 Kumar McMillan [:kumar] (needinfo all the things) 2013-07-11 15:31:04 PDT
hmm, this introduces a regression when trying to connect to the BrowserID verifier service.

15:16:40 django_browserid.base:INFO Verification URL: https://verifier.login.persona.org/verify :/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/django_browserid/base.py:143
15:16:40 requests.packages.urllib3.connectionpool:INFO Starting new HTTPS connection (1): verifier.login.persona.org :/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py:549
Traceback (most recent call last):
  File "/usr/local/Cellar/python26/2.6.8/lib/python2.6/wsgiref/handlers.py", line 93, in run
    self.result = application(self.environ, self.start_response)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/django/contrib/staticfiles/handlers.py", line 67, in __call__
    return self.application(environ, start_response)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 241, in __call__
    response = self.get_response(request)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/django/core/handlers/base.py", line 179, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/django/views/decorators/csrf.py", line 77, in wrapped_view
    return view_func(*args, **kwargs)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/tastypie/resources.py", line 192, in wrapper
    response = callback(request, *args, **kwargs)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/tastypie/resources.py", line 397, in dispatch_list
    return self.dispatch('list', request, **kwargs)
  File "/Users/kumar/dev/zamboni/mkt/api/base.py", line 96, in dispatch
    .dispatch(request_type, request, **kwargs))
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/tastypie/resources.py", line 427, in dispatch
    response = method(request, **kwargs)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/tastypie/resources.py", line 1165, in post_list
    updated_bundle = self.obj_create(bundle, request=request, **self.remove_api_resource_names(kwargs))
  File "/Users/kumar/dev/zamboni/mkt/account/api.py", line 132, in obj_create
    is_native=bundle.data.get('is_native', False)
  File "/Users/kumar/dev/zamboni/apps/users/views.py", line 334, in browserid_authenticate
    url=url, extra_params=extra_params)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/django_browserid/base.py", line 149, in verify
    result = _verify_http_request(url, args)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/django_browserid/base.py", line 86, in _verify_http_request
    r = requests.post(url, **parameters)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/requests/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/Users/kumar/.virtualenvs/zamboni/lib/python2.6/site-packages/requests/adapters.py", line 331, in send
    raise SSLError(e)
SSLError: [Errno bad handshake]


I don't understand why. OpenSSL on Mac is outdated (0.98) but even after I rebuilt PyOpenSSL with the 1.01 lib I saw the same problem.
Comment 13 Kumar McMillan [:kumar] (needinfo all the things) 2013-07-11 15:35:28 PDT
work in progress: https://github.com/mozilla/zamboni/pull/888
Comment 14 Kumar McMillan [:kumar] (needinfo all the things) 2013-08-26 08:54:20 PDT
ashort says this may be a bug in requests. If so, we may need to get a patch upstream.
Comment 15 Allen Short [:ashort] 2013-08-26 09:50:51 PDT
Testing this locally, I encountered the issue described here: https://github.com/kennethreitz/requests/issues/749#issuecomment-19284753

Changes to django-browserid result in a slightly different traceback than the above, but the requests.post() call it makes results in a SSLError(SSLError(SSLError('bad handshake', WantReadError()),),) exception.

This will require fixing PyOpenSSL support in urllib3 and requests.
Comment 16 Matt Basta [:basta] 2013-11-01 12:49:02 PDT
*** Bug 933755 has been marked as a duplicate of this bug. ***
Comment 17 Matt Basta [:basta] 2013-11-05 13:04:27 PST
Any update on this? Can we get this done now?
Comment 18 Mathieu Pillard [:mat] 2013-11-06 04:16:53 PST
According to the requests issue mentioned in #comment 15:
>> The changes in shazow/urllib3#233 (now pulled into urllib3 master) fix the issue for me (SSLError('bad handshake', WantReadError())) which occurred for some urls, some times, in some configurations. Can we get this pulled into requests master?
> We'll pull in an up-to-date version into Requests when we release 2.0. =)

requests 2.0.1 is out now, so it's worth upgrading locally and check if we can still reproduce the issue we had with 1.2.3.
Comment 19 Harald Kirschner :digitarald 2013-11-11 08:27:31 PST
Any update on this? This seems to block a few submissions including partners.
Comment 20 Mathieu Pillard [:mat] 2013-11-12 05:20:56 PST
requests 2.0.1 is broken in an horrible way (can't even install) : https://github.com/kennethreitz/requests/issues/1732

2.0.0 seems to work though, but I'd need a manifest hosted on a server needing SNI to be sure. Does anyone have a manifest URL that doesn't currently work in production ?
Comment 21 classicning 2013-11-12 18:45:49 PST
Please use this manifest to test:

https://delicious.com/delicious.webapp
Comment 22 Mathieu Pillard [:mat] 2013-11-13 04:54:37 PST
classicning@ : that server doesn't seem to be using SNI ? At first glance it looks like there is an unrelated icon problem (icon size mismatch)

:digitarald : can you give me a manifest from a partner that is blocked by this bug ?
Comment 23 Harald Kirschner :digitarald 2013-11-13 09:38:27 PST
The delicious URL reflects the issue discussed here. The validation doesn't fail with the icon mismatch but with the bug related issue: https://marketplace.firefox.com/developers/upload/aca5e24292b44491ad19175f846cda5f . On a related note, I wrote the partner to fix the icons.
Comment 24 Mathieu Pillard [:mat] 2013-11-14 09:16:47 PST
Fixed in https://github.com/mozilla/zamboni/commit/4b53b93b6b239875df0cdd04383fd90035ca8609

STR:
- Go to https://marketplace-dev.allizom.org/developers/validator 
- Log out, log back in, you shouldn't get any errors
- Test that https://delicious.com/delicious.webapp validates
- Test a couple extra hosted apps that you know should validate just in case
Comment 25 Matt Basta [:basta] 2013-11-14 09:49:20 PST
*** Bug 936564 has been marked as a duplicate of this bug. ***
Comment 26 Victor Carciu 2013-11-15 07:30:05 PST
Verified as fixed : http://screencast.com/t/acsgI3K1

Note You need to log in before you can comment on or make changes to this bug.