Closed Bug 410806 Opened 14 years ago Closed 12 years ago

move nightly updates to make_incremental_updates.py

Categories

(Release Engineering :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 511967

People

(Reporter: rhelmer, Assigned: coop)

References

Details

Attachments

(3 files, 2 obsolete files)

If we add a couple features to make_incremental_updates, it would be easy to start using it for nightly updates:

* callable as a library as well as a standalone app
* collects metadata needed for AUS as it goes (buildid, sha1sum, size, etc).

Patch coming up.
Assignee: nobody → rhelmer
This patch modifies make_incremental_patches.py in the following ways:

* config file handling is in main(), so we can pass a list directly to create_partial_patches() if using this as a library
* add ability to extract buildid from unpacked MAR (get_buildid())
* add ability to parse filenames (decode_filename()) 
* return a dict full of useful metadata from create_partial_patches(). This gives us everything we need to create AUS update.xml files.
Attachment #295380 - Flags: review?(benjamin)
 (In reply to comment #1)
> * return a dict full of useful metadata from create_partial_patches(). This
> gives us everything we need to create AUS update.xml files.

Actually returns a list of dicts (one dict per update generated).
Status: NEW → ASSIGNED
Here is a script which uses the changes to make_incremental_updates.py
Given a directory full of MARs and a URL, it creates partial MARs and AUS nightly-format snippet txt files.

It works as-is, but with some more work it could be called from a makefile target e.g.:

make nightly-updates PREVIOUS_URL='ftp://ftp.m.o/.../'

It automatically determines the OS, locale, version numbers etc. based on the contents of the FTP dir and the contents of the local dir. It can be run on one central host (as we do now) or on the individual builders (which I think we should move to), and supports locales (our current nightly script only supports en-US).
(In reply to comment #3)
> Created an attachment (id=295428) [details]
> create partials and AUS snippets for nightly builds
> 
> Here is a script which uses the changes to make_incremental_updates.py
> Given a directory full of MARs and a URL, it creates partial MARs and AUS
> nightly-format snippet txt files.
> 
> It works as-is, but with some more work it could be called from a makefile
> target e.g.:
> 
> make nightly-updates PREVIOUS_URL='ftp://ftp.m.o/.../'
> 
> It automatically determines the OS, locale, version numbers etc. based on the
> contents of the FTP dir and the contents of the local dir. It can be run on one
> central host (as we do now) or on the individual builders (which I think we
> should move to), and supports locales (our current nightly script only supports
> en-US).
> 

Minor nit - we are using os.path.getsize in other places and you use os.stat - would be great to just use the same thing everywhere.

Whiteboard: waiting for review
Priority: -- → P3
Comment on attachment 295380 [details] [diff] [review]
collect metadata during partial patch creation

>Index: make_incremental_updates.py

>+def decode_filename(filename):
>+    """ break filename into component parts
>+        for example: firefox-3.0b3pre.en-US.linux-i686.complete.mar
>+    """

Please add more to this docstring, specifying that it returns a dict and the keys that it sets
Attachment #295380 - Flags: review?(benjamin) → review+
This addresses schrep's comment about getsize() vs. os.stat() and bsmedberg's review comment about the decode_filename() docstring.

Speaking of which, I think we should make decode_filename() more flexible, but not sure how to do this (should be able to handle different filename formats, like release versus nightly). Perhaps make the regex that gets passed to re.search() optionally configurable?

Checking in what I have for now:

Checking in make_incremental_updates.py;
/cvsroot/mozilla/tools/update-packaging/make_incremental_updates.py,v  <--  make_incremental_updates.py
new revision: 1.3; previous revision: 1.2
done
Attachment #295380 - Attachment is obsolete: true
Whiteboard: waiting for review
Now with cmd line args, 4 spaces, more assertion+error handling, docstrings, etc.

The URL can only be file or FTP, because it tries to automatically determine the previous version. This feature needs more testing; long-term I'd like to be able to query a central store and not play this guessing game, but I think this would be ok for nightlies, for now.
Attachment #295428 - Attachment is obsolete: true
We'd like to use this from patcher for 1.8 branch, and it won't use the build ID yet anyway, so just print a warning if there's no application.ini or no build ID.
Attachment #296491 - Flags: review?(benjamin)
Whiteboard: waiting for review
Attachment #296491 - Flags: review?(benjamin) → review+
Comment on attachment 296491 [details] [diff] [review]
[checked in] not finding build ID shouldn't be fatal

Checking in make_incremental_updates.py;
/cvsroot/mozilla/tools/update-packaging/make_incremental_updates.py,v  <--  make_incremental_updates.py
new revision: 1.4; previous revision: 1.3
done
Attachment #296491 - Attachment description: not finding build ID shouldn't be fatal → [checked in] not finding build ID shouldn't be fatal
Whiteboard: waiting for review
Whiteboard: testing
Whiteboard: testing → testing in dev
I think this is a reasonable way to do this, but I have no time to work on it anytime soon.
Assignee: rhelmer → nobody
Status: ASSIGNED → NEW
Whiteboard: testing in dev
Component: Release Engineering → Release Engineering: Future
QA Contact: build → release
Ok so make_incremental_updates.py could be used as-is for nightly updates. If we could get all nightly MARs in one place, then the caching in make_incremental_updates.py will make the process very fast, but it can do one-offs to. The nightly update scripts would need to be modified to support this, not sure if they are in public repo yet?

attachment 296490 [details] [diff] [review] is an attempt to create an alternative to the nightly standalone update generator (which is different than the one used for releases, patcher). It might've been a mistake for me to tackle that in this bug too, perhaps it should be split out. It's not strictly necessary to fix this bug, but the old scripts are a hassle.
Assignee: nobody → ccooper
Component: Release Engineering: Future → Release Engineering
I'm working on this right now as part of bug 511967, but I'm not actually using  
make_incremental_updates.py. That seems like overkill when generating only a single partial update for a nightly on a build slave. For now I'm just calling the unpack shell scripts directly.

I think rhelmer's comments in bug 444050 are valid though. If we can break the monolithic python script into it's component scripts, I can just as easily call the helper scripts instead.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 511967
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.