1216883 - mozregression broken since ftp.m.o -> S3 migration

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Cameron McCormack (:heycam)]

Since the ftp.m.o -> S3 migration in bug 1213721, mozregression no longer works.

$ ./mach mozregression --launch 2015-03-02
 0:00.59 LOG: MainThread ERROR Unable to find build info for 2015-03-02

Comment 1

[Julien Pagès (:parkouss)]

Oh well - this is not since bug 1213721, I am pretty sure about that (we would have seen it). I can reproduce.

For nightlies (using dates as parameters) we are scrapping some web pages, like:

https://archive.mozilla.org/pub/firefox/nightly/2015/03/2015-03-02-03-02-04-mozilla-central/

And I can see that this is no more served by apache, or at least the pages now have a completely different look - and code -, breaking our scraping methods. Looking into that.

I suspect that mozdownload is impacted in the same way ? Whimboo, is that right ?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Yes, it is. And I have https://github.com/mozilla/mozdownload/pull/330 to get a workaround in place. As you said the HTML structure has been changed, so we also have to update our parser for mozdownload.

Comment 3

[Julien Pagès (:parkouss)]

Attachment: fix amo crawling

Comment 4

[Julien Pagès (:parkouss)]

So I still landed the patch in https://github.com/mozilla/mozregression/commit/781363ed1e4397645494d07ce45f3e2ad0d766d6, because we need a new release now.

Comment 5

[Julien Pagès (:parkouss)]

grr. There is some missing folders on a.m.o currently: https://archive.mozilla.org/pub/firefox/nightly/2015/

Dir 	.. 		
Dir 	01/ 		
Dir 	02/ 		
Dir 	03/ 		
Dir 	04/ 		
Dir 	10/ 	

So this is not fixed. I'm going to use 'http://ftp-origin-scl3.mozilla.org/pub' instead of a.m.o as Henrik mentioned (Thanks!)

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

(In reply to Julien Pagès (:parkouss) from comment #5)
> grr. There is some missing folders on a.m.o currently:
> https://archive.mozilla.org/pub/firefox/nightly/2015/
> 
> Dir 	.. 		
> Dir 	01/ 		
> Dir 	02/ 		
> Dir 	03/ 		
> Dir 	04/ 		
> Dir 	10/ 	

Nick, have the remaining folder not been synced yet or was there a problem?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Sync is still going, it's a lot of data. https://groups.google.com/d/msg/mozilla.dev.tree-management/FnAn_4a7Ohc/92-udZOuDQAJ

Comment 8

[Julien Pagès (:parkouss)]

Oh, looks like there is now all the data in https://archive.mozilla.org/pub/firefox/nightly/2015/.

Nick, is the sync done ?

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

oremj has been doing the syncing, passing the request on.

Comment 10

[William Lachance (:wlach)]

Comment on attachment 8676756 [details] [review]
fix amo crawling

This is obviously fine. :)

Comment 11

[Jeremy Orem [:oremj]]

The firefox/nightly is now synced up.

Comment 12

[Julien Pagès (:parkouss)]

(In reply to Jeremy Orem [:oremj] from comment #11)
> The firefox/nightly is now synced up.

Thanks!

Sitched back to archive.m.o: https://github.com/mozilla/mozregression/commit/c6579337eb712590a803d5ffcd1f23e14cc394ff

Going to do a minor patch release.