871209 - Scraper is failing to see 21.0build3 builds

Status: RESOLVED INVALID

(Mozilla QA Graveyard :: Mozmill Automation, defect)

Description

[u279076]

We just got 21.0build3 candidates and I tried running the ondemand_functional tests but it's failing to find the builds.

The URL the script returns is:
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/21.0-candidates/

If I load this URL I can see the build3 folder and I can see that it contains builds so I don't know why the scraper is failing. This *might* be unintentional fallout from bug 869249 but I'm filing this bug anyway.

This blocks QA from qualifying Firefox 21.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Works manually for me too, and no .htaccess in the build3/ dir.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

This is fallout from build2 not appearing in the directory listing. I've added a fake directory to work around this.

https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#L462 assumes there will be N members in the list if you're up to buildN, and that's not true with build2 hidden. Why does it not do something like 
        regex = r'%(PREFIX)s%(BUILD)s/%(UNSIGNED)s%(PLATFORM)s/%(LOCALE)s'
        return regex % {'PREFIX': self.candidate_build_list_regex,
                        'BUILD': "build%s" % self.build_number,
                        'LOCALE': self.locale,
                        'PLATFORM': self.platform_regex,
                        'UNSIGNED': "unsigned/" if self.unsigned else ""}
at
 https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#L504
(plus assign that in __init__).

Comment 3

[u279076]

(In reply to Nick Thomas [:nthomas] from comment #2)
> This is fallout from build2 not appearing in the directory listing. I've
> added a fake directory to work around this.

Thanks Nick. I've retriggered one of the failed testruns and it passed:
http://mozmill-ondemand.blargon7.com/#/functional/report/ea82256a8ae9808d91b7e8145ea3217a
http://mozmill-ondemand.blargon7.com/#/functional/report/ea82256a8ae9808d91b7e8145ea32f3b

I'm going to keep this bug open because I think we should try to improve the scraper code to better handle this type of situation. However, this no longer blocks release.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This is a failure in mozdownload which has its own repository at:
https://github.com/mozilla/mozdownload

Can you please file an issue on github so we can get this fixed? Thanks.

Comment 5

[u279076]

Reported:
https://github.com/mozilla/mozdownload/issues/77

Thank you, Henrik.