move nightly updates to make_incremental_updates.py

RESOLVED DUPLICATE of bug 511967

Status

Release Engineering
General
P3
normal
RESOLVED DUPLICATE of bug 511967
10 years ago
4 years ago

People

(Reporter: rhelmer, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 2 obsolete attachments)

(Reporter)

Description

10 years ago
If we add a couple features to make_incremental_updates, it would be easy to start using it for nightly updates:

* callable as a library as well as a standalone app
* collects metadata needed for AUS as it goes (buildid, sha1sum, size, etc).

Patch coming up.
(Reporter)

Updated

10 years ago
Assignee: nobody → rhelmer
(Reporter)

Comment 1

10 years ago
Created attachment 295380 [details] [diff] [review]
collect metadata during partial patch creation

This patch modifies make_incremental_patches.py in the following ways:

* config file handling is in main(), so we can pass a list directly to create_partial_patches() if using this as a library
* add ability to extract buildid from unpacked MAR (get_buildid())
* add ability to parse filenames (decode_filename()) 
* return a dict full of useful metadata from create_partial_patches(). This gives us everything we need to create AUS update.xml files.
Attachment #295380 - Flags: review?(benjamin)
(Reporter)

Comment 2

10 years ago
 (In reply to comment #1)
> * return a dict full of useful metadata from create_partial_patches(). This
> gives us everything we need to create AUS update.xml files.

Actually returns a list of dicts (one dict per update generated).
Status: NEW → ASSIGNED
(Reporter)

Comment 3

10 years ago
Created attachment 295428 [details] [diff] [review]
create partials and AUS snippets for nightly builds

Here is a script which uses the changes to make_incremental_updates.py
Given a directory full of MARs and a URL, it creates partial MARs and AUS nightly-format snippet txt files.

It works as-is, but with some more work it could be called from a makefile target e.g.:

make nightly-updates PREVIOUS_URL='ftp://ftp.m.o/.../'

It automatically determines the OS, locale, version numbers etc. based on the contents of the FTP dir and the contents of the local dir. It can be run on one central host (as we do now) or on the individual builders (which I think we should move to), and supports locales (our current nightly script only supports en-US).

Comment 4

10 years ago
(In reply to comment #3)
> Created an attachment (id=295428) [details]
> create partials and AUS snippets for nightly builds
> 
> Here is a script which uses the changes to make_incremental_updates.py
> Given a directory full of MARs and a URL, it creates partial MARs and AUS
> nightly-format snippet txt files.
> 
> It works as-is, but with some more work it could be called from a makefile
> target e.g.:
> 
> make nightly-updates PREVIOUS_URL='ftp://ftp.m.o/.../'
> 
> It automatically determines the OS, locale, version numbers etc. based on the
> contents of the FTP dir and the contents of the local dir. It can be run on one
> central host (as we do now) or on the individual builders (which I think we
> should move to), and supports locales (our current nightly script only supports
> en-US).
> 

Minor nit - we are using os.path.getsize in other places and you use os.stat - would be great to just use the same thing everywhere.

(Reporter)

Updated

10 years ago
Whiteboard: waiting for review
(Reporter)

Updated

10 years ago
Priority: -- → P3
Comment on attachment 295380 [details] [diff] [review]
collect metadata during partial patch creation

>Index: make_incremental_updates.py

>+def decode_filename(filename):
>+    """ break filename into component parts
>+        for example: firefox-3.0b3pre.en-US.linux-i686.complete.mar
>+    """

Please add more to this docstring, specifying that it returns a dict and the keys that it sets
Attachment #295380 - Flags: review?(benjamin) → review+
(Reporter)

Comment 6

10 years ago
Created attachment 296188 [details] [diff] [review]
[checked in] collect metadata during partial patch creation [as landed]

This addresses schrep's comment about getsize() vs. os.stat() and bsmedberg's review comment about the decode_filename() docstring.

Speaking of which, I think we should make decode_filename() more flexible, but not sure how to do this (should be able to handle different filename formats, like release versus nightly). Perhaps make the regex that gets passed to re.search() optionally configurable?

Checking in what I have for now:

Checking in make_incremental_updates.py;
/cvsroot/mozilla/tools/update-packaging/make_incremental_updates.py,v  <--  make_incremental_updates.py
new revision: 1.3; previous revision: 1.2
done
Attachment #295380 - Attachment is obsolete: true
(Reporter)

Updated

10 years ago
Whiteboard: waiting for review
(Reporter)

Comment 7

10 years ago
Created attachment 296490 [details] [diff] [review]
create partials and AUS snippets for nightly builds

Now with cmd line args, 4 spaces, more assertion+error handling, docstrings, etc.

The URL can only be file or FTP, because it tries to automatically determine the previous version. This feature needs more testing; long-term I'd like to be able to query a central store and not play this guessing game, but I think this would be ok for nightlies, for now.
Attachment #295428 - Attachment is obsolete: true
(Reporter)

Comment 8

10 years ago
Created attachment 296491 [details] [diff] [review]
[checked in] not finding build ID shouldn't be fatal

We'd like to use this from patcher for 1.8 branch, and it won't use the build ID yet anyway, so just print a warning if there's no application.ini or no build ID.
Attachment #296491 - Flags: review?(benjamin)
(Reporter)

Updated

10 years ago
Whiteboard: waiting for review
Attachment #296491 - Flags: review?(benjamin) → review+
(Reporter)

Comment 9

10 years ago
Comment on attachment 296491 [details] [diff] [review]
[checked in] not finding build ID shouldn't be fatal

Checking in make_incremental_updates.py;
/cvsroot/mozilla/tools/update-packaging/make_incremental_updates.py,v  <--  make_incremental_updates.py
new revision: 1.4; previous revision: 1.3
done
Attachment #296491 - Attachment description: not finding build ID shouldn't be fatal → [checked in] not finding build ID shouldn't be fatal
(Reporter)

Updated

10 years ago
Whiteboard: waiting for review
(Reporter)

Updated

10 years ago
Whiteboard: testing
(Reporter)

Updated

9 years ago
Whiteboard: testing → testing in dev
(Reporter)

Comment 10

9 years ago
I think this is a reasonable way to do this, but I have no time to work on it anytime soon.
Assignee: rhelmer → nobody
Status: ASSIGNED → NEW
Whiteboard: testing in dev
Component: Release Engineering → Release Engineering: Future
QA Contact: build → release
(Reporter)

Comment 11

9 years ago
Ok so make_incremental_updates.py could be used as-is for nightly updates. If we could get all nightly MARs in one place, then the caching in make_incremental_updates.py will make the process very fast, but it can do one-offs to. The nightly update scripts would need to be modified to support this, not sure if they are in public repo yet?

attachment 296490 [details] [diff] [review] is an attempt to create an alternative to the nightly standalone update generator (which is different than the one used for releases, patcher). It might've been a mistake for me to tackle that in this bug too, perhaps it should be split out. It's not strictly necessary to fix this bug, but the old scripts are a hassle.
(Assignee)

Updated

8 years ago
Assignee: nobody → ccooper
Component: Release Engineering: Future → Release Engineering
(Assignee)

Comment 12

8 years ago
I'm working on this right now as part of bug 511967, but I'm not actually using  
make_incremental_updates.py. That seems like overkill when generating only a single partial update for a nightly on a build slave. For now I'm just calling the unpack shell scripts directly.

I think rhelmer's comments in bug 444050 are valid though. If we can break the monolithic python script into it's component scripts, I can just as easily call the helper scripts instead.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 511967
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.