694873 - Clean up job for nightly snippets can remove valid updates when nightlies more than 7 days apart

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

These days we only build nightlies when there are changes in a repository (and the compiles all work). Meanwhile, we clean up nightly update snippets using this cron job
56  22  *  *  *  find /opt/aus2/incoming/2 -mindepth 4 -maxdepth 4 -mtime +6 ! -wholename '*tracemonkey*' -exec rm -rf {} \;
So if we go more than 7 days without a nightly we'll end up with no update at all, instead of pointing to the most recent build. 

We'll probably have to script up something that only preserves the the last few directories for each platform & locale (at least the last two, more if we want debugging). That's complicated by the ordering of the directories being platform/buildid/locale/.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Is there any action planned on this? Our update tests on nightly builds currently depend on the actual installed build, and we got stuck with builds from 10-29. I could update the build on all of our platforms again and again, but it looks like that we will end-up in the same situation again shortly.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I've been intending to do this, but it keeps falling down the list compared to other things. Eventually the problem will go away when the new update server comes along, but it's an issue until then.

Here's an outline of what we need to do, if someone is motivated. For the snippet store,
* iterate over all branches  (eg directory named mozilla-1.9.2)
  * iterate over all platforms  (eg directory named WINNT_x86-msvc)
    * in the soup of buildID/locale directories, locate the latest two sets of 
      snippets for each locale
    * remove older snippets
    * clean up any empty directories, depth first (ie locales then buildIDs)

The most complicated case to deal with is like this, eg for a mozilla-1.9.2/WINNT_x86-msvc directory, with files like this:

 ./20111217031145/en-US
 ./20111217031145/en-US/complete.txt
 ./20111217031145/en-US/partial.txt
 ./20111217031145/zh-TW/complete.txt
 ./20111217031145/zh-TW/partial.txt

 ./20111218031140/en-US
 ./20111218031140/en-US/complete.txt
 ./20111218031140/en-US/partial.txt
 ./20111218031140/zh-TW/complete.txt
 ./20111218031140/zh-TW/partial.txt

 ./20111219031209/en-US
 ./20111219031209/en-US/complete.txt
 ./20111219031209/en-US/partial.txt

Here 20111219... is the latest en-US build, and 20111218... is the latest zh-TW build. In this case we should delete only the files/dirs
 ./20111217031145/en-US/complete.txt
 ./20111217031145/en-US/partial.txt
and clean up the dir
 ./20111217031145/en-US/
Once there are no locales left in ./20111217031145 it would get removed too.

Comment 4

[Axel Hecht [:Pike]]

Bug 657789 used to be high profile, if that's blocked now, should this get beefed up in priorities?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

I don't think this blocks bug 657789 actually. We can put the snippets for that work in some large buildID like 20610103000000 and exempt those from the existing cron job.

This is still nice to have but would be deprecated by deployment Balrog.

Comment 6

[[e-mail] (:bhearsum)]

WONTFIX'ing this because it's not worth fixing with Balrog taking over for nightly style updates.