Closed Bug 398148 Opened 14 years ago Closed 14 years ago

First nightly can set bad permissions on new YYYY/MM/ dirs and block other tinderboxes uploading

Categories

(mozilla.org Graveyard :: Server Operations, task, P2)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: tonymec, Assigned: justdave)

References

()

Details

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a9pre) Gecko/2007093001 SeaMonkey/2.0a1pre
Build Identifier: http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2007-10-01-02-trunk/

Details: See "Steps to Reproduce".


Reproducible: Always

Steps to Reproduce:
1. Browse to http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey
2. Scroll down to the box ending at 03:06:00 today (October 1st) in the first column, headed "Linux cb-sea-linux-tbox Dep release".
3. Click the "D" link in that box.

Actual Results:  
404 Not Found; The requested URL /pub/mozilla.org/seamonkey/nightly/2007-10-01-02-trunk/ was not found on this server.


Expected Results:  
Directory listing with downloadable Linux build


See also the log ("Success") at http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1191227640.1191233837.29395.gz
Working with KaiRo to fix this.
Assignee: server-ops → build
Status: UNCONFIRMED → NEW
Component: Server Operations → Build & Release
Ever confirmed: true
Priority: -- → P1
QA Contact: justin → preed
Assignee: build → nrthomas
The build has appeared (Linux .tar.gz, Windows .installer.exe, Windows .zip). Thanks! :-) :-) :-)
(In reply to comment #2)
> The build has appeared (Linux .tar.gz, Windows .installer.exe, Windows .zip).
> Thanks! :-) :-) :-)
> 

Oops; I meant .tar.bz2
The problem was that seamonkey/nightly/2007/10/ was created 755 kairo:users by one of the 1.8 tinderboxes that KaiRo runs. Then trunk community boxes using seabld can't write files there. That's been fixed, and KaiRo has pushed the trunk files up with upload_packages.sh. 

Leaving open so to fix this in time for next month, and to check on other apps.
Priority: P1 → P2
(In reply to comment #4)
[...]
> Leaving open so to fix this in time for next month, and to check on other apps.
> 

OK. I'll keep an eye on the next nightlies every day at about the same time, and add comments here if I can't get at them. :-)
> OK. I'll keep an eye on the next nightlies every day at about the same time,
> and add comments here if I can't get at them. :-)

Should just be an issue when a new month rolls around, but if you're downloading nightlies anyway then that'd be good.
(In reply to comment #6)
> > OK. I'll keep an eye on the next nightlies every day at about the same time,
> > and add comments here if I can't get at them. :-)
> 
> Should just be an issue when a new month rolls around, but if you're
> downloading nightlies anyway then that'd be good.
> 

Count on me. :-)
Depends on: 394069
OS: Linux → All
Hardware: PC → All
QA Contact: mozpreed → build
Not sure if we'll get this fixed before the end of the month, so I'll make the dirs ahead of time.
Summary: SeaMonkey Linux trunk nightly produced but not uploaded to ftp.m.o. today (01-Oct-2007) → First nightly can set bad permissions on new 2007/MM/ dirs and block other tinderboxes uploading
John points out that this'll also be a problem at the new year.
Summary: First nightly can set bad permissions on new 2007/MM/ dirs and block other tinderboxes uploading → First nightly can set bad permissions on new YYYY/MM/ dirs and block other tinderboxes uploading
(In reply to comment #8)
> Not sure if we'll get this fixed before the end of the month, so I'll make the
> dirs ahead of time.
> 

Well, we're on the first of the month and the bug has reverted:
- ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-mozilla1.8/ has no Linux entry for today, even though the Tinderbox page for Mozilla1.8 says that today's Tb Linux nightly has been produced
- on http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla1.8 , clicking the "D" link in the crazyhorse (Linux Tb-nightly) column gives 404 Not Found.

(In reply to comment #9)
> John points out that this'll also be a problem at the new year.
> 

New Year's day is the first of a month anyway.
Oh man, I dropped this ball. Will fix it up where I can. Kairo, can you check on seamonkey ? I've fixed the permissions on the 2007/11/ directories.
(In reply to comment #11)
> Oh man, I dropped this ball. Will fix it up where I can. Kairo, can you check
> on seamonkey ? I've fixed the permissions on the 2007/11/ directories.
> 

OK, I see the Tb builds now.

About the Sm 1.1.7pre builds, I see a Mac build (dated today 15:02) and a Linux build (today 14:15) but the "latest" W32 build on the FTP server is dated yesterday 10:59. Clicking the "D" link under "WINNT 5.1 tpol Dep (1.1)" at http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla1.8-SeaMonkey gives a 404 error.

Anything else you want checked?
Adding KaiRo to the Cc List: Robert, please see comment #11 by Nick Thomas.
Trunk is fine, branch builds are partly missing, I need to also check if those boxen (my private ones) have current tinderbox code
(In reply to comment #10)
> (In reply to comment #9)
> > John points out that this'll also be a problem at the new year.
> New Year's day is the first of a month anyway.
Agreed. :-)

My concern was that the code creating the YYYY dir would have same/similar permission problem as when creating the MM dir. Hence an MM dir might not be created in January. Either way, we would still have the same overall symptom (no nightlies), so not sure if this distinction matters, but its a slightly different wrinkle on the problem, hence pointing it out.
...so that whatever fix we do works for both scenarios.

(sorry for bugspam, clicked send too soon)
I've created the 2007/12 dirs for tomorrow's change of month.
(In reply to comment #17)
> I've created the 2007/12 dirs for tomorrow's change of month.
> 

Nick, I admire your dedication; but couldn't this directory creation be assigned to a cron-monthly job or something? I don't dare to imagine what would happen if you fell ill near the end of a month.
;-)
We can (and will) fix or automate this. In the meantime 2008/01 created with correct perms.
The assignment of group by LDAP was pushed out till later, so I'm wondering if we can make some the default group changes: 

  ffxbld          firefox
  tbirdbld        thunderbird
  xrbld           xulrunner
  seabld, kairo   seamonkey
  calbld          calendar
  caminobld       camino

How does that sound to you Dave ? Or we could write a script for cron to run each month, but that's a bit blech.
This is fixed on surf.m.o. justdave and I are figuring out how to solve it on the new staging server (dm-stage02).

Blocks: 394069
No longer depends on: 394069
For the record I don't like the way it was fixed on surf. :)

It seems that mkdir manages to avoid the setgid on the parent directory and fails to set permissions when it's creating more than one level of directories at once via the -p flag.  There's two ideas so far how to fix this:

1) Hack mkdir so that it propagates those permissions correctly.
2) Set up a cron job that fixes the permissions.

I'd been leaning towards #1, but I'm thinking #2 will be less likely to break when we get security updates for the OS and so forth.
So a quick find gives me the following directory trees that have date folders with this structure in them, is this correct?

camino/nightly/
firefox/nightly/
firefox/nightly/contrib/
firefox/nightly/experimental/18relauto/
firefox/nightly/experimental/leopard/
thunderbird/nightly/
thunderbird/nightly/contrib/
seamonkey/nightly/
seamonkey/nightly/contrib/
seamonkey/experimental/
xulrunner/nightly/
xulrunner/nightly/experimental/
calendar/lightning/nightly/
calendar/lightning/experimental/nightly/
calendar/sunbird/nightly/
calendar/sunbird/experimental/nightly/
Sounds right to me.
We need this by the end of the month, rather than before enabling the new system.
Blocks: 419978
No longer blocks: 394069
2008/04 dirs added for the nightly/ dirs.
Argh, lost sight of this during the stage migration and undo. 

justdave, is it ok to:
1) reassign this bug to you
2) have you setup cronjob that runs at the start of the month, creating next month's dir with the correct permissions (this avoids the gnarly end-of-month calculations we talked about).
(In reply to comment #27)
> justdave, is it ok to:
> 1) reassign this bug to you
> 2) have you setup cronjob that runs at the start of the month, creating next
> month's dir with the correct permissions (this avoids the gnarly end-of-month
> calculations we talked about).

Yep.
Assignee: nrthomas → justdave
Component: Release Engineering → Server Operations
QA Contact: build → justin
Updated directory list:

camino/nightly/
firefox/nightly/
firefox/nightly/contrib/
thunderbird/nightly/
thunderbird/nightly/contrib/
seamonkey/nightly/
seamonkey/nightly/contrib/
seamonkey/experimental/
xulrunner/nightly/
xulrunner/nightly/experimental/
calendar/lightning/nightly/
calendar/lightning/experimental/nightly/
calendar/sunbird/nightly/
calendar/sunbird/experimental/nightly/

Will you have a chance to get this before the end of the month ?
Yeah, the size of my buglist is shrinking finally so I should be on top of this really shortly.
ok, this script is now in place and runs daily on dm-stage01.  The script exits without doing anything if it's not the last day of the month (since cron has no way to specify "last day of the month" as the day to run).  Besides creating the directories if they don't exist (including the year directory if it's a new year), it also ensures they have 2775 permissions (meaning any new files/directories created in them inherit the parent group membership, regardless of the membership of the person posting the files).

It reads the list of parent directories to create dated directories in from /etc/stagemaint/stage-nightlydirs on dm-stage01.  I set up /etc/stagemaint with the intention of having that in version control somewhere so it's easy for people to edit and we can have it pulled by the script when it starts, but I haven't actually done that.  Not sure how worth it it is, but it's an option. :)

The monthly directories for 2008/05 have been created.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
OK, the script has been adjusted to run on the second-to-last day of the month.

Someone already created the 2008/05 directories for calendar before I ran the script, despite not being 2008/05 yet, and they already had the wrong perms.  Obviously the tinderbox in question is probably running in Europe.  I fixed the perms on those directories, too.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.