750184 - relengweb1.dmz.scl3 followups

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

1, please set up an rsync from cruncher.srv.releng.scl3:/var/www/html/builds to relengweb1.dmz.scl3:/var/www/html/build/builds

This is for coop's last-job-per-slave.html, and my pending graphs. We had a syncbld account on dm-wwwbuild01 and cruncher.sjc1 for this previously, and Dustin has previously expressed a preference for a pull task rather than push.

2, we should also track down what happened to the cron that updated the dirty talos profiles, and get them synced so that http://build.mozilla.org/talos/profiles/dirtyDBs.zip is fresher than April 02.

3, typo |MAILTO="[e-mail]"| in root's cron on relengweb1.dmz.scl3

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Nick Thomas [:nthomas] from comment #0)
> 1, please set up an rsync from cruncher.srv.releng.scl3:/var/www/html/builds
> to relengweb1.dmz.scl3:/var/www/html/build/builds

I set this up, but it's going to delete a lot of stuff, so I'll hold off until I get a r+ on the rsync log, which I'll attach shortly.

> 2, we should also track down .. dirty talos profiles

These were generated on cruncher and pulled from there:

==== /var/spool/cron/talos
0 0 * * * (/usr/local/bin/python /home/talos/public_html/updateGenerated.py) >> /home/talos/public_html/reporter.log && mv /home/talos/public_html/dirtyDBs.zip /var/www/html/talos_profiles/. &&  mv /home/talos/public_html/dirtyMaxDBs.zip /var/www/html/talos_profiles/. && cp /var/www/html/talos_profiles/dirtyMaxDBs.zip /home/talos/public_html/. && cp /var/www/html/talos_profiles/dirtyDBs.zip /home/talos/public_html/.
==== /var/spool/cron/root
#10 */2 * * * /usr/bin/rsync -a -e 'ssh -i /root/.ssh/id_dsa.syncbld' /var/www/html/talos_profiles/ [e-mail]:/var/www/html/build/talos/profiles/
#10 */2 * * * /usr/bin/rsync -a -e 'ssh -i /root/.ssh/id_dsa.syncbld' /var/www/html/talos_profiles/ [e-mail]:/var/www/html/build/talos/profiles/

The new cruncher looks way better (kudos, bear!):

[root@cruncher ~]# crontab -l -u talos
MAILTO=[e-mail]
@daily    /usr/sbin/logrotate -s $HOME/logrotate.state $HOME/logrotate.conf
@daily    $HOME/bin/update_talos_profiles.sh

I'm a little worried about copying a partially written zip.  updateGenerated.py is some *terrible* python, so I modified update_talos_profiles.sh instead to move the zips (atomically) to /var/www/html/talos_profiles as was done on the old cruncher. relengweb1 pulls from there.

> 3, typo |MAILTO="[e-mail]"| in root's cron on relengweb1.dmz.scl3

fixed

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: rsync-log.txt

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Revealed by 3,:

4, remove cronjobs no longer required from trybld@relengweb1
41 * * * * /usr/bin/rsync -e 'ssh -i .ssh/trybld_dsa' -rltD --delete --size-only dm-wwwbuild01.mozilla.org:/symbols/ /symbols/

Luckily this is failing to write to /symbols/lost+found and aborting any file deletion. We'd be losing any symbols uploaded since the switch otherwise.

This is also surplus now:
# Puppet Name: sync-tryserver-symbols
MAILTO="[e-mail]"
41 * * * * /usr/bin/rsync -e 'ssh -i .ssh/trybld_dsa' -rltD --size-only dm-wwwbuild01.mozilla.org:/symbols/ /symbols/

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 619605 [details]
rsync-log.txt

Looks like the same cleanup I did in sjc1 prior to the switch. Loading some content out of http://cruncher.srv.releng.scl3.mozilla.com/builds/ looks fine too.

catlee, what's the status of buildfaster.csv.gz ? Should we be generating that on new cruncher for ateam ?

For posterity,
$ cat .htaccess 
AddOutputFilterByType DEFLATE application/x-javascript
Header set Access-Control-Allow-Origin "*"
AddType text/plain .gz
AddEncoding x-gzip .gz
ExpiresActive on

ExpiresDefault "access plus 1 minute"
which is all for the json we used to have here but don't any more.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I fixed up the errant crontasks.  Puppet had gotten itself confused.  There's no more 'dm-wwwbuild01' in any crontask on relengweb1.

I activated the crontask for #1.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Some extra tweaks for the puppet jobs pretty please.

1, Remove the -v from the talos and cruncher rsyncs so we don't get cron mail; add -t or switch from -r to -a
2, Do the cruncher rsync every 5 minutes instead of once a day

Thanks!

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

--- a/modules/releng/manifests/web/build-mozilla-org.pp
+++ b/modules/releng/manifests/web/build-mozilla-org.pp
@@ -17,7 +17,7 @@ class releng::web::build-mozilla-org {
     # synchronizing dirty talos profiles - copy from cruncher per bug 750184
     cron {
         "pull-talos-profiles":
-            command => "rsync -vr [e-mail]:/var/www/html/talos_profiles/ /var/www/html/build/talos/profiles/",
+            command => "rsync -r [e-mail]:/var/www/html/talos_profiles/ /var/www/html/build/talos/profiles/",
             minute => '0',
             hour => '1',
             environment => ["MAILTO=\"[e-mail]\""];
@@ -26,9 +26,8 @@ class releng::web::build-mozilla-org {
     # synchronizing random releng stuff - this is pretty much a dumping-ground on cruncher
     cron {
         "pull-releng-builds-dir":
-            command => "rsync -vr --delete [e-mail]:/var/www/html/builds/ /var/www/html/build/builds/",
-            minute => '0',
-            hour => '1',
+            command => "rsync -r --delete [e-mail]:/var/www/html/builds/ /var/www/html/build/builds/",
+            minute => '*/5',
             environment => ["MAILTO=\"[e-mail]\""];
     }
 

done.