888085 - Send user-agent with manifest request

Status: RESOLVED WONTFIX

(Marketplace Graveyard :: Validation, defect)

Description

[Louis Stowasser [:louisstow]]

When the validator requests a hosted app manifest file, no user-agent is sent in the request header. This causes some servers to respond with 403 Forbidden (assuming the request a bot or spam).

Can we add the Firefox OS user-agent to the request?

Comment 1

[Christopher Van Wiemeersch [:cvan]]

Which servers are doing this? I'm hesitant to do this for reasons as described in bug 891654.

If we do want to patch our manifest crawler, this will do it:

	diff --git a/mkt/developers/tasks.py b/mkt/developers/tasks.py
	index 613a9f6..53afd3a 100644
	--- a/mkt/developers/tasks.py
	+++ b/mkt/developers/tasks.py
	@@ -343,7 +343,10 @@ def get_preview_sizes(ids, **kw):
	 def _fetch_content(url):
	     with statsd.timer('developers.tasks.fetch_content'):
	         try:
	-            stream = urllib2.urlopen(url, timeout=30)
	+            req = urllib2.Request(url)
	+            req.add_header('User-Agent',
	+                'Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0')
	+            stream = urllib2.urlopen(req, timeout=30)
	             if not 200 <= stream.getcode() < 300:
	                 raise Exception(
	                     'An invalid HTTP status code was returned.')

But we still need to take care of the validator, as it pulls image resources independently.

Comment 2

[Louis Stowasser [:louisstow]]

The server in particular is http://findthebest.com and the manifest to test is http://www.findthebest.com/publishers/upload/other/ftb_manifest.webapp

It will send a 403 if no user agent is supplied so the content of the user agent isn't an issue.

Comment 3

[Louis Stowasser [:louisstow]]

Actually it looks like it's failing with the default Python user-agent (Python-urllib/2.6), my bad.

Comment 4

[Wil Clouser [:clouserw]]

bug 891654 comment 10 summarizes why we shouldn't be changing this UA from the actual agent making the request.  I definitely don't want to hardcode Firefox 18 into the UA and I don't want to update the version number as each release reaches its end of life.

Wontfixing this.