413572 - nightly.mozilla.org

Status: RESOLVED FIXED

(Websites :: other.mozilla.org, enhancement)

Description

[Jeff Walden [:Waldo]]

A one-stop shop for downloading the latest nightly, with small links to FTP to find previous nightlies, would be awesome for making downloading nightlies easy.  See the URL for an example of how this would work.

Comment 1

[Reed Loden [:reed]]

Nick, think you could make something like this from your hourly-archive stuff? I could set up nightly.localgho.st for now to let you try it...

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Actually, this is closer to update-watch. That already scrapes ftp for nightly builds. 

Comment 5

[Robert Sayre]

Attachment: nightly web page contents

please please please don't bikeshed this to death. we should get something up and running. it's always possible file followups to change it. Most people are looking for Firefox, so that's all this does.

morgamic asked whether it should it be .com or .org... I said "both!"

Comment 6

[Robert Sayre]

(In reply to comment #5)
> Created an attachment (id=408443) [details]
> nightly web page contents
> 
> please please please don't bikeshed this to death. 

To clarify, it obviously needs graphic design. But I claim the page contents are good enough to start with. Examples of things I don't think we should address right now are: N.N.N branch builds, Firefox ARM builds, tracemonkey builds, thunderbird builds, fennec builds, shark builds, "what's a nightly?", mozilla.com navbars, the fact that the binary is called Minefield, etc, etc.

All of those things might be good to fix later. Most could be added to a page that the "more stuff" link goes to, as a separate enhancement.

I think this page should be as simple as possible to start, and any changes we make should be done with download metrics in hand.

Comment 7

[Jeff Walden [:Waldo]]

+Infinity, agree that pretty is relatively easy once it's up and running

Comment 8

[Robert Sayre]

(In reply to comment #7)
> +Infinity, agree that pretty is relatively easy once it's up and running

Oh, we can even make it pretty to start... let's just make the thing I attached pretty, not debate the content of the page.

Comment 9

[John Slater]

(In reply to comment #5)
> Created an attachment (id=408443) [details]
> nightly web page contents

> morgamic asked whether it should it be .com or .org... I said "both!"

In terms of visual design, we should pick one and then have the other redirect to it. Meaning, if the URL is nightly.mozilla.org then it should be a page in the mozilla.org theme, and then nightly.mozilla.com could just redirect to that (or vice versa).

Mozilla.com is generally the home of Firefox and Firefox-related content, so nightly.mozilla.com makes sense on that level. But, I know there's a history with mozilla.org here. If we were starting from scratch I'd vote for mozilla.com and that design theme, but I think either could work.

Comment 10

[Robert Sayre]

(In reply to comment #9)
> (In reply to comment #5)
> > Created an attachment (id=408443) [details] [details]
> > nightly web page contents
> 
> > morgamic asked whether it should it be .com or .org... I said "both!"
> 
> In terms of visual design, we should pick one and then have the other redirect
> to it. Meaning, if the URL is nightly.mozilla.org then it should be a page in
> the mozilla.org theme, and then nightly.mozilla.com could just redirect to that
> (or vice versa)

Sounds good. A search for "firefox nightly" shows only mozilla.org pages, so I think we should go with that, with the mozilla.com redirect.

Comment 11

[David Boswell]

Agreed about comment #10.  Unless there's a really good reason, there shouldn't be any sites that are on *.mozilla.com that don't redirect to *.mozilla.org (or vice versa).  As for which of the domains to default to, my personal opinion is that things should default to mozilla.org unless there's a good reason not to.

Comment 12

[Robert Sayre]

(In reply to comment #2)
> Actually, this is closer to update-watch. That already scrapes ftp for nightly
> builds. 

OK, so is the way forward here to modify update-watch, or what?

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

If what you want to do is scrape some timestamps and filenames out of 
  http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/
and generate some html then update-watch is not well suited. That's best done in [your favorite scripting language]. 

Any reason you're linking to the zip and not the installer for windows ? We should promote the build types we ship as releases.

Comment 14

[Samuel Sidler (old account; do not CC)]

(If we link to the installer, it would be good to add a quick note along the lines of... "Already using a nightly? Use Help -> Check for Updates to download the latest nightly", but we can add that later.)

Comment 15

[Robert Sayre]

(In reply to comment #13)
> If what you want to do is scrape some timestamps and filenames out of 
>   http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/

I want the thing that uploads to that directory to generate an HTML page. Are you telling me that's actually a bunch of distinct processes, so scraping is the way to go?

Comment 16

[Mike Connor [:mconnor]]

Do you want an HTML page, or some formatted data we can use to generate an HTML page? I'd think the latter is easier for everyone, and then page layout changes don't get tied up with build infra...

Comment 17

[Robert Sayre]

Attachment: generate the HTML page

this scrapes the mod_autoindex-generated apache page from ftp.mozilla.org and then builds something like the HTML page I attached.

Needs a sort function to put things in the right order, and maybe some date formatting.

Comment 18

[Robert Sayre]

Attachment: order the builds

this uses the order the builds are defined in, avoiding the sorting issue. I'll make the list members a class I guess, once we figure out exactly what's needed.

Comment 19

[Robert Sayre]

Attachment: make the HTML a little easier to read

Comment 20

[Robert Sayre]

Attachment: sample output

Comment 21

[Robert Sayre]

ok, do I need to add a bunch of error handling to this? or can I count on some cron infrastructure to send out alerts if it starts exiting with an error code?

seems like we could install the thing I attached in here and get started setting up the redirects, without advertising that the page exists.

Comment 22

[Reed Loden [:reed]]

Need to get this into version control somewhere. Maybe a new repo under http://hg.mozilla.org/webtools/?

Comment 23

[Robert Sayre]

Attachment: scrape the build server, make a nice HTML page

I bet morgamic will know where to put this file, too.

Comment 24

[Michael Morgan [:morgamic]]

Comment on attachment 408650 [details] [diff] [review]
scrape the build server, make a nice HTML page

Thoughts:
- bug 405088 has an idea about creating a generic feed for all product information (not just nightlies); this could be extended to meet that need at some point
- missing doctype, just use 4.0.1 trans or somethin
- style; do we care?
- if we made the output JSON we could just syndicate it and slurp it/inject it anywhere else; any feelings about that?
- patch seems simple enough and can't get xss'd unless someone MITM's the ftp url...

Next:
- add doctype
- consider writing JSOn output and HTML counterpart that eats JSON and spits out HTML output (and that won't have to change via cron, only the JSON will)
- set up the domain
- file IT bug to host it

Overall, I think it's fine as is but we could expand it to be easier to integrate with other sites.  Do you think it's worth doing?

Comment 25

[Michael Morgan [:morgamic]]

Regarding my JSON comments in comment #24, just creating a buildJSON(builds) method would work and you could add that to main()...

Comment 26

[Robert Sayre]

(In reply to comment #24)
> (From update of attachment 408650 [details] [diff] [review])
> Thoughts:
> - bug 405088 has an idea about creating a generic feed for all product
> information (not just nightlies); this could be extended to meet that need at
> some point

A fair point, but that bug is even older than this one. ;)

> - missing doctype, just use 4.0.1 trans or somethin

I'll add an HTML5 doctype.

> - style; do we care?

Of course we care. I'll file a follow-up for jslater or his delegate to style it. I'll tweak the markup as needed then.

> - if we made the output JSON we could just syndicate it and slurp it/inject it
> anywhere else; any feelings about that?

I can spit out JSON no problem, assuming there's a json module I can use, or just copy yet another copy of simplejson. This means the file would need some config information since it would be spitting out two files. Right now, it can just write to stdout.

> - patch seems simple enough and can't get xss'd unless someone MITM's the ftp
> url...

If it gets fancier, a templating system that actually prevents xss (genshi etc) might be good.

> - set up the domain
> - file IT bug to host it

ok, so three bugs. these two, plus the styling bug.

Comment 27

[Reed Loden [:reed]]

(In reply to comment #26)
> > - set up the domain
> > - file IT bug to host it

That should just be one bug, not two... Also, I'll repeat comment #22: This needs to be in version control somewhere.

Comment 28

[Robert Sayre]

dang it, reed is right. hate it when that happens.

where should we put it?

Comment 29

[Michael Morgan [:morgamic]]

IT: bug 524879
Make sexy: bug 524881

Umm - hg is fine, or svn.  Up to you, you're the author!

Comment 30

[Robert Sayre]

Filed bug 524962 to create the repository.

Comment 31

[Robert Sayre]

Attachment: address review comments

dumps to /tmp by default, makes two files: index.html and index.json

Comment 32

[Michael Morgan [:morgamic]]

Comment on attachment 408884 [details]
address review comments

Valid html, JSON looks good.  Hhave to remind IT about the output_dir, simplejson and python 2.6.  Should be easy to add the mozilla.org header/footer crap -- I can do that in the other bug.

Comment 33

[Robert Sayre]

http://hg.mozilla.org/webtools/nightly/rev/4b18c73bd660