477768 - Add nagios monitoring for l10n builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Axel Hecht [:Pike]]

We had a few occasions where side effects and other issues made the l10n builds just fall off the planet.

We should get some nagios monitoring up to get notified when no builds go up.

At least for 1.9.0, we can check ftp for stale builds.

For 3.2a1pre probably too, as those build on cycle. Not all builds need to succeed here, though, but that should be a simple check.

For those two, I'd suggest to check if there are no new builds within the last 24 hours.

For 3.1, we're only building on push, we should either wait with monitoring that until we have the regular builds included, or check every second day. That might still generate some noise, but less, I guess. Poem.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: monitor l10n directories for 1.9/1.9.2

I think this is an OK stop-gap. This will make sure *something* in that directory has been modified in the past 28 hours.

This isn't per platform, but that will require more hacking than I have time for right now.

Comment 2

[Axel Hecht [:Pike]]

Comment on attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

The dirs are allright, opening a tentative second review request because I know jack about what those files actually do.

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

Did you mean to ask someone for review here?

Comment 4

[Axel Hecht [:Pike]]

As I have no idea whom to ask review from, no. I just wanted to denote that if you need a review on "this is doing the right thing at the right point in time", you should probably get someone knowledgeable on the monitor stuff to look at the patch.

Comment 5

[Chris Cooper [:coop] (he/him)]

Comment on attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

As a stop-gap, this is fine.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

Checking in Firefox_mozilla-central.txt;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/Firefox_mozilla-central.txt,v  <--  Firefox_mozilla-central.txt
new revision: 1.8; previous revision: 1.7
done
Checking in Firefox_mozilla1.9.0.txt;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/Firefox_mozilla1.9.0.txt,v  <--  Firefox_mozilla1.9.0.txt
new revision: 1.7; previous revision: 1.6
done

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

Turns out this doesn't work. I'm not entirely certain why.

Comment 8

[Chris Cooper [:coop] (he/him)]

Attachment: Don't fail on dirs in the filelist

Comment 9

[Chris Cooper [:coop] (he/him)]

Comment on attachment 362307 [details] [diff] [review]
Don't fail on dirs in the filelist

I tried this out on surf and it seems to work, assuming directory timestamps are properly updated when their contents change.

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 362307 [details] [diff] [review]
Don't fail on dirs in the filelist

\o/!

Comment 11

[Chris Cooper [:coop] (he/him)]

Comment on attachment 362307 [details] [diff] [review]
Don't fail on dirs in the filelist

Checking in Firefox_mozilla-central.txt;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/Firefox_mozilla-central.txt,v  <--  Firefox_mozilla-central.txt
new revision: 1.10; previous revision: 1.9
done
Checking in Firefox_mozilla1.9.0.txt;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/Firefox_mozilla1.9.0.txt,v  <--  Firefox_mozilla1.9.0.txt
new revision: 1.9; previous revision: 1.8
done
Checking in check_nightly_builds;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/check_nightly_builds,v  <--  check_nightly_builds
new revision: 1.2; previous revision: 1.1
done

Comment 12

[Chris Cooper [:coop] (he/him)]

Change is live on stage now.

Comment 13

[[e-mail] (:bhearsum)]

Looks like the directory timestamp isn't updated :(. Maybe it only gets updated when a *new* file appears? Might be fixed by touching the dir when we upload.

http://www.grabup.com/uploads/1362f15b47812050aed800e06d9bbc37.png?direct

Comment 14

[Chris Cooper [:coop] (he/him)]

Sigh. I'll work on updating the upload steps.

Comment 15

[Chris Cooper [:coop] (he/him)]

Attachment: Set the modified time on container directories when copying files with post_upload.py

This fixes the issue for m-c. Tried this out in the python shell on stage and it seems to work.

I will hopefully have a patch for 1.9.0 later today.

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 363886 [details] [diff] [review]
Set the modified time on container directories when copying files with post_upload.py

Tiny nit: place run os.utime() outside of the loop - no need to execute it multiple times. Sounds good otherwise! r=bhearsum with that change.

Comment 17

[Axel Hecht [:Pike]]

Actually, I'm not sure if we should move the touch outside the loop, as that would touch the dir (and silence nagios) even if no file would be uploaded. I'd really prefer to do that on done copy, and take the small hit.

Comment 18

[[e-mail] (:bhearsum)]

post_upload.py doesn't call ReleaseTo functions if it's not passed any files so I don't see that happening.

Anyways, I don't want to bikeshed so r=me either way.

Comment 19

[Chris Cooper [:coop] (he/him)]

Comment on attachment 363886 [details] [diff] [review]
Set the modified time on container directories when copying files with post_upload.py

changeset:   231:2ba74001ce81

Comment 20

[Chris Cooper [:coop] (he/him)]

Attachment: Adds build step to touch upload dir after file upload for 1.9.0 l10n builds

I've tested this on staging-1.9 and it works fine.

Comment 21

[Chris Cooper [:coop] (he/him)]

There was a small issue with paths and quoting so I checked in a bustage fix after testing again on all platforms on staging-1.9.