924389 - reportor has tons of rsync errors in cronmail

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Date: Tue,  8 Oct 2013 07:00:27 -0700 (PDT)
From: Cron Daemon <[e-mail]>
To: [e-mail]
Subject: Cron <root@relengwebadm> rsync -r --links --delete [e-mail]:/var/www/html/builds/ /mnt/netapp/relengweb/builddata/reports/

directory has vanished: "/var/www/html/builds/reportor/2013-10-05:13"
IO error encountered -- skipping file deletion
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1518) [generator=3.0.9]


Looks like this particular one started on September 28th.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I would guess that a cron to delete older reportor output (or equivalent in the code doing the generation) was added then. catlee ?

Comment 3

[Chris AtLee [:catlee]]

there's a cronjob on cruncher that also cleans up old reportor output. I suspect this gets run at the same time the rsync is running, and so stuff gets deleted after rsync has figured out which files to transfer

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Chris AtLee [:catlee] from comment #3)
> there's a cronjob on cruncher that also cleans up old reportor output. I
> suspect this gets run at the same time the rsync is running, and so stuff
> gets deleted after rsync has figured out which files to transfer

Dustin, can we adjust the timing on one of these jobs to reduce the possibility of hitting this? I don't think these machines are in PuppetAgain, but I'm happy to write the patch if it's in a repo I have access to.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Cruncher's managed by infra puppet.  I don't remember the name "reportor", and it seems to be in catlee's and buildduty's crontabs, so I don't think I have any involvement here.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Dustin J. Mitchell [:dustin] (I read my bugmail; don't needinfo me) from comment #5)
> Cruncher's managed by infra puppet.  I don't remember the name "reportor",
> and it seems to be in catlee's and buildduty's crontabs, so I don't think I
> have any involvement here.

This is relengweb's cron, actually. It's a bit confusing because the cronjob in question is running an rsync against cruncher.

> From: Cron Daemon <[e-mail]>

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Oh, I thought you were suggesting moving the cleanup cronjob.  How would you like the timing set up?  It's currently */5.

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Dustin J. Mitchell [:dustin] (I read my bugmail; don't needinfo me) from comment #7)
> Oh, I thought you were suggesting moving the cleanup cronjob.  How would you
> like the timing set up?  It's currently */5.

If the rsync and delete ones are currently both running every 5 minutes can we offset them so that one runs at 2min/7min/12min etc. and the other at 5/10/15/etc?

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

That could be tricky with cron syntax -- but why is a daily report getting cleaned up every 5m?

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Dustin J. Mitchell [:dustin] (I read my bugmail; don't needinfo me) from comment #9)
> That could be tricky with cron syntax -- but why is a daily report getting
> cleaned up every 5m?

Oh, looks like I was confused. The cleanup _does_ happen on cruncher...and different cleanups happen either @daily or @hourly by the looks of it. Seems I can just adjust those cronjobs to address this.

Comment 11

[[e-mail] (:bhearsum)]

I switched the @daily cronjobs to "2 0 * * *" and the hourly ones to "2 * * * *". Hopefully that helps reduce the frequency of this.

Comment 12

[[e-mail] (:bhearsum)]

Frequency is definitely reduced, not quite gone yet though...

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

We could also roll the rsync and cleanup into a single crontask, if you'd like.  I'll be happy to install a shell script on relengwebadm.