881369 - Automatically delete obsolete latest-* builds for calendar

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

There seems to be a script that removes obsolete Thunderbird nightly builds from latest-comm-central and latest-comm-aurora. Now that Lightning is being run from the Thunderbird build infra and the user is tbirdbld instead of calbld, I cannot remove these files manually.

Please extend that script to also remove all obsolete lightning- and gdata-provider- builds in:

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-central/

and

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-aurora/

Comment 1

[Stefan Sitter [:ssitter]]

The following folder should be considered too:
https://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/

Nightly builds are already archived therefore it seems redundant to me to keep every hourly build too.

Comment 2

[Stefan Sitter [:ssitter]]

Could someone confirm that this is the appropriate Bugzilla product/component for this request or move it to the correct product/component or add the right people that could decide this to CC?

Comment 3

[[e-mail] (:bhearsum)]

I don't know if RelEng has access to do this, or if we manage other calendar stuff on FTP. John, do you know?

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

These end up owned by tbirdbld on disk, so cleanup should live under the tbirdbld cron job. These are in SVN and are modifiable by RelEng; there is a doc on how to get started at https://wiki.mozilla.org/ReleaseEngineering/How_To/Modify_scripts_on_stage. You want the files/cron/tbirdbld. 

jhopkins has managed this cron before (although maybe not in this location), or I can take a look at this once we get to Q3.

Comment 5

[Stefan Sitter [:ssitter]]

Q3 has already finished, did you took a look as promised?

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I hope Q3 isn't finished just yet, but no I haven't had a look. Where can the version of gdata-provider files be found in the source repo ? The script looks up version in the hg repo.

Comment 7

[Stefan Sitter [:ssitter]]

The version of gdata-provider is automatically calculated from the Lightning version using the script calendar/providers/gdata/makeversion.py

Comment 8

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Note also I am pushing towards making the Lightning version also be auto-calculated, bug 940402

Comment 9

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

After some IRC discussion: John will be removing the old files once manually, afterwards I will fix bug 940402, then you can set things up to use Lightning's new makeversion.py to figure out the version number to remove. The same goes for gdata.

Comment 10

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

bug 940402 has landed, can we make this happen now? What needs to be done and who can do it?

Comment 11

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Nick, what else is needed to make this work? The scripts to calculate the version numbers for the gdata provider and lightning is available now:

http://mxr.mozilla.org/comm-central/source/calendar/lightning/build/makeversion.py
http://mxr.mozilla.org/comm-central/source/calendar/providers/gdata/makeversion.py

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: trim-nightlies.py (thunderbird)

Sorry this has sat this has sat for so long, but it isn't a high priority bug compared to others on my plate. I'll to try provide enough information so that you can determine the way forward.

tbirdbld has a number of crons, here's one example:
cd /pub/mozilla.org/thunderbird/nightly/latest-comm-central && /usr/bin/python ~/trim-nightlies.py https://hg.mozilla.org/comm-central/raw-file/default/mail/config/version.txt | sh

I've attached the python script trim-nightlies.py here. You can see it's expecting files to be named thunderbird-*, and to be able to get the version from a file checked into the tree. Piping to shell is ... very odd.

The could be modified somehow, or we could just use a sledgehammer like
  find .../lightning/nightly/latest-* -mtime +3

Comment 13

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: trim-nightlies.py (lightning)

Do you want this to be a new script, or do you want it integrated with the existing trim-nightlies.py? This version acts as a new script. I've taken over the weird bash piping from the Thunderbird script, but I am happy to remove it if you like.

Alternatively, I can combine the two scripts and use python to remove the files instead.

Let me know what you think.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

They look sufficiently divergent that I'm happy to go with separate scripts. I would point out that if the the versioning ever changes in the build system, then it is very likely that the script will get out of date. If you're OK with that I can get this into puppet.

Comment 15

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

I'm fine with that, I don't think the versioning for Lightning will change any time soon. The gdata provider might actually jump a few versions in the next few months, but then we can still push a change to this script.

Do you want this in a repository? I have [1] for the update.php script on calendar.mozilla.org, but I am not sure if the script to update that file on the webserver (see /data/genericrhel6/src/calendar.mozilla.org/update on genericrhel6) copies the whole repository or just the file. 

Aside from that, go ahead and put this on puppet for central and aurora.


[1] http://hg.mozilla.org/users/mozilla_kewis.ch/calendar-buildconfig/

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

This has to go into a private svn repository for puppet, so I'll take care of it.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [svn] Lightning cleanup

* this adds the two scripts on this bug. We currently use ~tbirdbld/trim-nightlies.py and I'm just committing it as-is at /usr/local/bin/trim-thunderbird-nightlies.py; the other is Fallen's mod of that for lightning
* tweak the thunderbird jobs to combine the existing ones for each branch, and extend to include latest*-l10n/*/xpi
* add the matching jobs for lightning, slightly different because it has platform subdirectories in the latest-*-l10n dirs, rather than flat plus platform dirs for xpis

Comment 18

[Justin Wood (:Callek)]

Comment on attachment 8466867 [details] [diff] [review]
[svn] Lightning cleanup

Review of attachment 8466867 [details] [diff] [review]:
-----------------------------------------------------------------

stamp

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8466867 [details] [diff] [review]
[svn] Lightning cleanup

Committed revision 91428

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Fallen, I did some spot checks on this and it appears to have landed cleanly. It should run for the first time in a few minutes, could you check that it's working as you expected ?

In retrospect every hour is probably over the top, once or a few times a day should be fine. There's also a related issue with the esr branches, which don't pick up new versions landing on default (for thunderbird at least); they keep building with an old version.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [svn] Use absolute paths

I forgot about cron's PATH not including /usr/local/bin, so we got some cron mail about it not being able to find the trim-* scripts. Fixed with this and landed as revision 91429.

Puppet should have that deployed before the next set of jobs.

Comment 22

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Seems to be working as expected, thanks a bunch!

Comment 23

[Stefan Sitter [:ssitter]]

What about hourly builds as mentioned in comment 1? For example <https://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-central-linux64/> seems to contain a copy of every build since November 2012. Similar for all the other builders there.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Lets have another bug for that please.