If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Add nagios monitoring for l10n builds

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
9 years ago
4 years ago

People

(Reporter: Pike, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 1 obsolete attachment)

(Reporter)

Description

9 years ago
We had a few occasions where side effects and other issues made the l10n builds just fall off the planet.

We should get some nagios monitoring up to get notified when no builds go up.

At least for 1.9.0, we can check ftp for stale builds.

For 3.2a1pre probably too, as those build on cycle. Not all builds need to succeed here, though, but that should be a simple check.

For those two, I'd suggest to check if there are no new builds within the last 24 hours.

For 3.1, we're only building on push, we should either wait with monitoring that until we have the regular builds included, or check every second day. That might still generate some noise, but less, I guess. Poem.
Created attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

I think this is an OK stop-gap. This will make sure *something* in that directory has been modified in the past 28 hours.

This isn't per platform, but that will require more hacking than I have time for right now.
Attachment #361539 - Flags: review?(l10n)
(Reporter)

Updated

9 years ago
Attachment #361539 - Flags: review?(l10n)
Attachment #361539 - Flags: review?
Attachment #361539 - Flags: review+
(Reporter)

Comment 2

9 years ago
Comment on attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

The dirs are allright, opening a tentative second review request because I know jack about what those files actually do.
Comment on attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

Did you mean to ask someone for review here?
(Reporter)

Comment 4

9 years ago
As I have no idea whom to ask review from, no. I just wanted to denote that if you need a review on "this is doing the right thing at the right point in time", you should probably get someone knowledgeable on the monitor stuff to look at the patch.
(Assignee)

Comment 5

9 years ago
Comment on attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

As a stop-gap, this is fine.
Attachment #361539 - Flags: review? → review+
Attachment #361539 - Flags: checked‑in+ checked‑in+
Comment on attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

Checking in Firefox_mozilla-central.txt;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/Firefox_mozilla-central.txt,v  <--  Firefox_mozilla-central.txt
new revision: 1.8; previous revision: 1.7
done
Checking in Firefox_mozilla1.9.0.txt;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/Firefox_mozilla1.9.0.txt,v  <--  Firefox_mozilla1.9.0.txt
new revision: 1.7; previous revision: 1.6
done
Comment on attachment 361539 [details] [diff] [review]
monitor l10n directories for 1.9/1.9.2

Turns out this doesn't work. I'm not entirely certain why.
Attachment #361539 - Flags: checked‑in+ checked‑in+ → checked‑in- checked‑in-
(Assignee)

Updated

9 years ago
Assignee: nobody → ccooper
Status: NEW → ASSIGNED
Priority: -- → P2
(Assignee)

Comment 8

9 years ago
Created attachment 362307 [details] [diff] [review]
Don't fail on dirs in the filelist
Attachment #362307 - Flags: review?(bhearsum)
(Assignee)

Updated

9 years ago
Attachment #361539 - Attachment is obsolete: true
(Assignee)

Comment 9

9 years ago
Comment on attachment 362307 [details] [diff] [review]
Don't fail on dirs in the filelist

I tried this out on surf and it seems to work, assuming directory timestamps are properly updated when their contents change.
Attachment #362307 - Flags: review?(bhearsum) → review+
Comment on attachment 362307 [details] [diff] [review]
Don't fail on dirs in the filelist

\o/!
(Assignee)

Comment 11

9 years ago
Comment on attachment 362307 [details] [diff] [review]
Don't fail on dirs in the filelist

Checking in Firefox_mozilla-central.txt;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/Firefox_mozilla-central.txt,v  <--  Firefox_mozilla-central.txt
new revision: 1.10; previous revision: 1.9
done
Checking in Firefox_mozilla1.9.0.txt;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/Firefox_mozilla1.9.0.txt,v  <--  Firefox_mozilla1.9.0.txt
new revision: 1.9; previous revision: 1.8
done
Checking in check_nightly_builds;
/cvsroot/mozilla/tools/tinderbox-configs/monitoring/check_nightly_builds,v  <--  check_nightly_builds
new revision: 1.2; previous revision: 1.1
done
Attachment #362307 - Flags: checked‑in+ checked‑in+
(Assignee)

Comment 12

9 years ago
Change is live on stage now.
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Looks like the directory timestamp isn't updated :(. Maybe it only gets updated when a *new* file appears? Might be fixed by touching the dir when we upload.

http://www.grabup.com/uploads/1362f15b47812050aed800e06d9bbc37.png?direct
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 14

9 years ago
Sigh. I'll work on updating the upload steps.
Status: REOPENED → ASSIGNED
(Assignee)

Comment 15

9 years ago
Created attachment 363886 [details] [diff] [review]
Set the modified time on container directories when copying files with post_upload.py

This fixes the issue for m-c. Tried this out in the python shell on stage and it seems to work.

I will hopefully have a patch for 1.9.0 later today.
Attachment #363886 - Flags: review?(bhearsum)
Attachment #363886 - Flags: review?(bhearsum) → review+
Comment on attachment 363886 [details] [diff] [review]
Set the modified time on container directories when copying files with post_upload.py

Tiny nit: place run os.utime() outside of the loop - no need to execute it multiple times. Sounds good otherwise! r=bhearsum with that change.
(Reporter)

Comment 17

9 years ago
Actually, I'm not sure if we should move the touch outside the loop, as that would touch the dir (and silence nagios) even if no file would be uploaded. I'd really prefer to do that on done copy, and take the small hit.
post_upload.py doesn't call ReleaseTo functions if it's not passed any files so I don't see that happening.

Anyways, I don't want to bikeshed so r=me either way.
(Assignee)

Comment 19

9 years ago
Comment on attachment 363886 [details] [diff] [review]
Set the modified time on container directories when copying files with post_upload.py

changeset:   231:2ba74001ce81
Attachment #363886 - Flags: checked‑in+ checked‑in+
(Assignee)

Comment 20

9 years ago
Created attachment 363924 [details] [diff] [review]
Adds build step to touch upload dir after file upload for 1.9.0 l10n builds

I've tested this on staging-1.9 and it works fine.
Attachment #363924 - Flags: review?(bhearsum)
Attachment #363924 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 21

9 years ago
There was a small issue with paths and quoting so I checked in a bustage fix after testing again on all platforms on staging-1.9.
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago9 years ago
Resolution: --- → FIXED
(Assignee)

Updated

9 years ago
Attachment #363924 - Flags: checked‑in+ checked‑in+
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.