721257 - figure out sdk-version-calculation approach that plays well with Flightdeck deployment

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: General, defect, P1)

Description

[Brian Warner [:warner :bwarner]]

We discovered this morning that the "get version from git metadata" code
included in addon-sdk-1.4 (bug 704519) conflicts with the deployment scheme
used by Flightdeck.

That code, used by "cfx" when it puts a .sdkVersion property into the
generated XPI, expects to be run in one of two environments:

* a git checkout (generally used by SDK developers and power-users), for
  which the code executes "git describe --tags --always --dirty" to come up
  with a fine-grained version string
* an unpacked release tarball (most users), for which the code parses some
  $Id$-style variables expanded by the "git archive" command which created
  the tarball

The code can also tolerate running in other environments (specifically an hg
checkout used by the releng machinery that still pulls from the output of the
git-to-hg bridge). However in those environments, .sdkVersion="unknown". In
addition, in addon-sdk-1.4 (which lacks the bug 718115 fix), a
scary-but-not-actually-fatal git error message is leaked to stderr.

It turns out the Flightdeck code is using the SDK in a way that lands in the
third case. Flightdeck, which lives in git, takes advantage of the fact that
the SDK is also managed in git, by using "git submodules" to check out
specific (multiple) versions of the SDK tree into subdirectories. This
reduces the size of the Flightdeck repository, and makes it possible to have
reduced-size working trees (by not checking out all of the supported SDK
versions), compared to committing a full copy of each supported SDK into
Flightdeck. Note that getting the SDK from a submodule is mostly equivalent
to getting it with a regular git checkout: the .git directory is still
present, and git-describe works as usual.

The Flightdeck deployment process then copies the Flightdeck source tree,
*omitting the .git directories*, to the public-facing servers. Now the
_version.py code finds itself without expanded $Id$-style variables, without
a .git directory, and probably without /usr/bin/git itself. It falls back to
saying .sdkVersion=unknown. (and without the bug 718115 fix, the stderr noise
triggers oops-cfx-has-failed code in Flightdeck, which then fails to serve
the generated XPI: this code has since been fixed).

NEEDS:

* Flightdeck-generated XPIs need to have correct .sdkVersion values,
  otherwise repacking becomes very hard
* Flightdeck deployment process can't depend upon having git on the frontend
  servers, and shouldn't depend upon copying .git metadata to those servers

WANTS:

* Flightdeck deployment process should remain simple
* SDK should avoid committing a version string into the code

OPTIONS:

* A: Flightdeck modifies their build/deployment process to look more like
     regular SDK users: unpack a released source tarball, maybe commit the
     results into the Flightdeck tree
* B: SDK changes the version-string management code, maybe hard-coding a
     version during each release, maybe getting the version from the name of
     the SDK's parent directory
* C: SDK adds an override-my-version-string flag, Flightdeck changes to pass
     this flag in (to versions which accept it).

This ticket was created to implement C, being the best short-term option
available (they want to release 1.4 support tomorrow). However, now I'm
thinking that option B with the from-the-parent-directory approach would be
better: cleaner and acceptable in the longer term.

Comment 1

[Sean McArthur [:seanmonstar]]

The third case (not solution) where someone could checkout the SDK and yet not be in a git repository points out that the SDK doesn't know it's own version without git's help.

I realize checking the version into source control can be an extra step, but it's also the common solution for most projects. It can simply be a high level version string, such as `version = "1.4"`, but still use the git-describe method if available. This way, if an add-on is created from a version of the SDK that isn't a part of git, it will still have a ballpark version that we can check for debugging purposes.

Specifically solving it just for FlightDeck could leave it so uses of the SDK from the hg bridge (or any other location not thought off) would mean that "unknown" versions could still show up.

Does the minor cost of updating the version string outweigh potentially getting add-ons submitted with .sdkVersion: "unknown"?

Comment 2

[Brian Warner [:warner :bwarner]]

Attachment: get version from name of parent directory, if necessary

I'm still checking to see if this works in the more general case (my upstream https://github.com/warner/python-versioneer project), but I think this might do the trick for Jetpack. If there's no .git directory, but the SDK is living in a subdirectory named "addon-sdk-NNN", then use NNN as the version string. That ought to survive the Flightdeck deployment process.

Comment 3

[Brian Warner [:warner :bwarner]]

Attachment: get version from name of parent directory, if necessary

updated patch: the previous version would have failed on windows (os.sep differences)

I've manually tested this in the four different environments: git checkout, unpacked tarball, .git-less checkout in a random directory name (which covers the hg-bridge case), and .git-less checkout in an "addon-sdk-1.4.1" directory (which covers the flightdeck case).

I think it's good to go. This patch won't require any changes to our build process. It does depend upon flightdeck continuing to use the same submodule directory names as they have been, but that seems pretty reasonable.

Comment 4

[Brian Warner [:warner :bwarner]]

oh, and this is also in https://github.com/warner/addon-sdk/tree/721257-versions in case that's an easier way to look at it.

Comment 5

[David Mason [:dcm :dmason]]

Comment on attachment 591714 [details] [diff] [review]
get version from name of parent directory, if necessary

Looks good

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/addon-sdk

https://github.com/mozilla/addon-sdk/commit/64f1d0a02b29027ae8c321815aa03bc125bee404
Bug 721257: get version from parentdir when .git is unavailable. r=dcm

This updates the embedded _version.py to upstream versioneer-0.6, which adds
code that can get the version number from the name of the parent directory.
For example, if the SDK finds itself living in a directory named
"addon-sdk-1.4.1", then (if it doesn't find variables expanded by
git-archive, and doesn't find a usable .git directory for git-describe) it
will strip the "addon-sdk-" prefix and describe itself as "1.4.1".

This should address the needs of Flightdeck (aka Addon Builder), which uses
git submodules to obtain a copy of the SDK, but then strips the .git
directories when deploying to the frontend web servers. This leaves neither
expanded variables nor .git metadata, but *does* use the addon-sdk-N.N/
directory name convention.

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

This was cherry-picked into the release branch as https://github.com/mozilla/addon-sdk/commit/4e64952bf877101570238e03d1c99392a4675543 and merged from there to stabilization in https://github.com/mozilla/addon-sdk/commit/ded70e01ec58260ffe574b25a00b9058aa5ae942 for whatever that's worth.

Comment 8

[Brian Warner [:warner :bwarner]]

All done: Flightdeck is deploying a new build that uses 1.4.1 (with this fix) now.