Closed Bug 622566 Opened 13 years ago Closed 9 years ago

Add crontab entries to remove old partner repacks

Categories

(Release Engineering :: Release Automation: Other, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: coop, Unassigned)

References

Details

(Whiteboard: [partner-repacks])

This is an offshoot to Bug 616555.

Partner repacks are currently published to surf, but we have no plan in place yet to expire them.

Kev: what sort of retention policy do you want here?
(In reply to comment #0)
> Kev: what sort of retention policy do you want here?

Guess it would help if I explicitly cc-ed Kev.
Looks like everything under /mnt/netapp/stage/releases.mozilla.com/archive/partners is owned by kneedham anyway, so you might as well implement your own crontab. Feel free to ask me for feedback/review if needed.
Assignee: coop → kev
Retention has been, pretty much, keep everything but Yahoo! Yahoo! builds are too large to maintain any kind of large repo, and a full build is ~3.5GB (including others). We don't have a retention policy in place, but what I think I'd look at is maintaining .0 releases and the latest 2 point releases, if it came to that, for supported builds.
Found during triage. Its ~20months since last activity. 

Reassigning to tomcat who is working on that these types of partner repacks, as he might have best idea how to proceed.
Assignee: kev → cbook
We can probably stick these into S3 for long-term storage: bug 707843
Depends on: 707843
Or chuck 'em on tape ? I've been deleting the partner builds along with everything else in firefox/candidates/ on ftp, which now seems like not a great idea.
(In reply to Nick Thomas [:nthomas] from comment #6)
> Or chuck 'em on tape ? I've been deleting the partner builds along with
> everything else in firefox/candidates/ on ftp, which now seems like not a
> great idea.

Nick since we build today all these builds from hg, would there be a problem to move/copy partner builds also to http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/partners/ 

like http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/partners/{firefox Release number}/
Component: Release Engineering: Custom Builds → Release Engineering: Releases
QA Contact: bhearsum
Assignee: cbook → nobody
Component: Release Engineering: Releases → Release Engineering: Automation (Release Automation)
Product: mozilla.org → Release Engineering
(In reply to Carsten Book [:Tomcat] from comment #7)
> Nick since we build today all these builds from hg, would there be a problem
> to move/copy partner builds also to
> http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/partners/ 

I don't think that would be a good use of space on ftp.m.o. Basically IT don't want us to use expensive NetApp disk space (as found behind ftp.m.o) as a way of archiving files. We generate 12G of files every Firefox release on the release channel. Plus the same for two betas a week, which may be less valuable. That added up to 130G for the 24.0 release cycle.

We should work out what the use cases is for the files once they're more than one major version behind. AIUI we're not serving them to users (except for maybe Twitter + Bing at another location), so presumably they only have archival value at best. If we're uncomfortable about throwing them away we can backup to tape, or to Amazon S3 or Glacier, but someone who cares should make the case for that. Have there been any cases were we actually needed the old files ?
coop, I don't know what we're doing with release.m.c these days, but I'm trying to keep firefox/candidates/ under control on ftp.m.o. Should we be archiving those to S3/glacier ?  I'm inclined to not bother for beta builds, but we could to do for release channel. 

Also, it wouldn't be that hard to reproduce builds after-the-fact, given everything under version control these days.
Flags: needinfo?(coop)
(In reply to Nick Thomas [:nthomas] from comment #9)
> coop, I don't know what we're doing with release.m.c these days, but I'm
> trying to keep firefox/candidates/ under control on ftp.m.o. Should we be
> archiving those to S3/glacier ?

eg firefox/27.0-candidates/build1/partner-repacks/ --> s3://<some_bucket>/27.0/build1/
(In reply to Nick Thomas [:nthomas] from comment #9)
> coop, I don't know what we're doing with release.m.c these days, but I'm
> trying to keep firefox/candidates/ under control on ftp.m.o. Should we be
> archiving those to S3/glacier ?  I'm inclined to not bother for beta builds,
> but we could to do for release channel. 
> 
> Also, it wouldn't be that hard to reproduce builds after-the-fact, given
> everything under version control these days.

I'm fine with ditching the beta builds. As you say, they're easy enough to regenerate.

I'm also fine with switch storage to s3 provided it doesn't greatly impact how the partners access the builds. In almost every case, the builds are only going to be slurped down once by each partner, so some small decrease in access speed is OK though.
Flags: needinfo?(coop)
Do we still need this?
Flags: needinfo?(coop)
Status: NEW → RESOLVED
Closed: 9 years ago
Flags: needinfo?(coop)
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.