467721 - the l10n factory should run its steps inside of the "build" folder

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Armen [:armenzg]]

The "rm -rf *-nightly/build" that was added to other nightly factories removes the "build" folder inside every *-nightly folders on a slave

In the l10n steps on the RepackFactory, "mozilla-central" repo and the "l10n-central" repo are checked out into "." instead of "build" therefore not being deleted by other nightly builds runs.

Fixing this will ensure that when other nightly builds start we will be sure that anything checked out by the l10n steps gets cleaned out so we 1) keep the disk usage low and will ensure that 2) we start the l10n process from a fresh point every night

CURRENTLY
.
./build <- This guy is removed
./mozilla-central <- This guy is *not* being removed because it is not inside of the previous folder
./l10n-central

IT SHOULD BE:
.
./build/mozilla-central <- This guy and the next should now be removed since they live inside of  "*-nightly/build"
./build/l10n-central


Side note, the l10n nightly runs will not be deleting themselves (or other branches' l10n nightly runs) since their remove step is slightly different from other normal nightly runs:
 'find .. -maxdepth 2 -type d -path "*-nightly/build" -not -path "*-l10n-nightly/build" -print -exec rm -rf {} \;'

Comment 1

[Chris Cooper [:coop] (he/him)]

This will also entail removing the existing checkouts of l10n-central and mozilla-central from all the slaves that have already checked them out in the wrong place or we'll end up compounding the problem.

Comment 2

[Chris Cooper [:coop] (he/him)]

Attachment: Run l10n build steps in the build/ subdir to allow for proper cleanup.

Comment 3

[Chris Cooper [:coop] (he/him)]

Attachment: Run l10n build steps in the build/ subdir to allow for proper cleanup, v2.

Also checks to make sure the upload dir exists before trying to remove it.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 352411 [details] [diff] [review]
Run l10n build steps in the build/ subdir to allow for proper cleanup, v2.

Looks fine to me.

Comment 5

[Chris Cooper [:coop] (he/him)]

Ran fine in staging. I'll add this to the list of things to push during the downtime tomorrow.

Comment 6

[Axel Hecht [:Pike]]

Reading the patch this will remove and re-clone mozilla-central for each locale, i.e., some 70 times per platform?

Comment 7

[Chris Cooper [:coop] (he/him)]

Not ideal, I know, but space is really at a premium on these slaves.

Comment 8

[Armen [:armenzg]]

(In reply to comment #6)
> Reading the patch this will remove and re-clone mozilla-central for each
> locale, i.e., some 70 times per platform?
>
To make it clear (if I have not lost track of things)
Each locale's run *won't* be removing what the previous locale has checked out.
'find .. -maxdepth 2 -type d -path "*-nightly/build" -not -path
"*-l10n-nightly/build" -print -exec rm -rf {} \;'

The only thing that each new locale's run will be deleting is the nightly folders of other nightly runs and the 'build/'+BRANCH+'/dist/upload'

Other nightly runs will after this patch be able to remove what the l10n nightly runs have checked out.

Comment 9

[Chris Cooper [:coop] (he/him)]

Comment on attachment 352411 [details] [diff] [review]
Run l10n build steps in the build/ subdir to allow for proper cleanup, v2.

Checking in factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.57; previous revision: 1.56
done