677449 - Use new git repository for nightly.mozilla.org website

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Webdev, task)

Description

[tofumatt [:tofumatt]]

Webdev traditionally develops in Git, not Mercurial, so I've moved the nightly.mozilla.org scraper script and resources to GitHub. This was discussed in Bug 668476.

The new repository is at git://github.com/mozilla/nocturnal.git. The codebase is the same as http://hg.mozilla.org/webtools/nightly/.

Any script/repository config IT has pulling from hg should be changed to point to the new git repo, as it's where development on nightly.mozilla.org will happen henceforth.

Please let me know (either on IRC or in this bug) if any clarification is needed.

Comment 1

[Shyam Mani [:fox2mike]]

tofumatt,

We (at IT) are kinda swamped right now, but I'll look into this. Should be fairly straightforward to switch. I'll ask if I have any questions.

Comment 3

[Jake Maul [:jakem]]

This is completed, and I made a Mana page for updating it in the future.

For the record, the actual code here does not auto-update. However, there is a cron job that invokes a script in the code, which rebuilds the displayed content in public_html. So what's displayed is auto-updated hourly, but the code that generates it is not... that's the part IT is involved in. Hopefully that makes sense.

Comment 4

[Shyam Mani [:fox2mike]]

Thanks for grabbing this, Jake :)

Comment 5

[Jake Maul [:jakem]]

This is causing the following to be emailed to us every hour:

Traceback (most recent call last):
 File "/data/static/www/nightly.mozilla.org/nightly/scrape.py", line 241, in <module>
   main()
 File "/data/static/www/nightly.mozilla.org/nightly/scrape.py", line 235, in main
   os.path.join(CURRENT_PATH, OUTPUT_PATH, folder))
 File "/usr/lib64/python2.6/shutil.py", line 146, in copytree
   os.makedirs(dst)
 File "/usr/lib64/python2.6/os.py", line 157, in makedirs
   mkdir(name, mode)
OSError: [Errno 17] File exists: '/data/static/www/nightly.mozilla.org/public_html/css'


In /usr/lib64/python2.6/shutil.py, I found this in the copytree function:

    114 def copytree(src, dst, symlinks=False, ignore=None):
    115     """Recursively copy a directory tree using copy2().
    116 
    117     The destination directory must not already exist.
    118     If exception(s) occur, an Error is raised with a list of reasons.


Reopening and assigning to tofumatt to investigate. Thanks!

Comment 6

[Justin Dow [:jabba]]

We really need to get this failing cronjob taken care of. It looks like it tries to create the directories css, fonts, img, and js. If I mv those four out of the way and run the cron by hand, it puts them all back, which I assume it is supposed to do. However running it again fails because they exist already. I'm not sure how this is supposed to work. Should it be rm'ing the directories before running? Should it not be a cron job?

Since it's broken and not doing anything anyway right now, I've commented out the cronjob in order restore my sanity of getting this e-mail every hour. Once there is a fix, please have us re-enable it.

Comment 7

[Fred Wenzel [:wenzel]]

Tofumatt, will you help fix this?

Comment 8

[Anthony Ricaud (:rik)]

Here is my tentative fix : https://github.com/Rik/nocturnal/commit/985be8a6a83c4fe4a68d900ca8a93f46f2751602

I couldn't find how to create a pull request with only this commit so…

Comment 10

[tofumatt [:tofumatt]]

rik: Thanks buddy!

jabba: The code has been patched and should no longer throw errors. Could we get this running again so the nightly page is updated? Thanks.

Comment 11

[tofumatt [:tofumatt]]

Firefox 10 is on the Nightly page now. Thanks everyone.

Comment 12

[Anthony Ricaud (:rik)]

Just for confirmation, the cron job is now running again?

Comment 13

[Shyam Mani [:fox2mike]]

(In reply to Anthony Ricaud (:rik) from comment #12)
> Just for confirmation, the cron job is now running again?

It has been enabled, yes.