838733 - generate_diff.py should have a convenience mode for staging

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Jeff Hammel]

It is now relatively easy to mirror mozbase changes to mozilla-central
from github.  However, if you are developing mozbase-based
infrastructure on m-c, as we are for
https://wiki.mozilla.org/Auto-tools/Projects/Mozbase/Automation-Refactor
, we need a way to add the HEAD (prerelease) of certain packages to
e.g. a patch queue for development and testing.  This will avoid
multiple unnecessary releases and generally make life hummy-chummy.

Comment 1

[Jeff Hammel]

Looking at the code, I think I am inclined to add a --develop flag that will enable this for all non-specified dependencies.  That is, for

  ./generate_diff.py mozprocess

but not

  ./generate_diff.py mozrunner=5.14

(I could go with --staging or --stage if there are strong opinions about the flag name.)

The current behaviour if you do not specify a tag, generate_diff.py will read it from setup.py. Note that this is not what is desired for this bug, since this will require that the tags exist.  Since the tags are created at package release time, and we want to test with pre-released packages here (the whole point of this bug), that dog won't hunt.

The alternative that I can think of is using a special marker, e.g. `dev`, to point to dev versions:

  ./generate_diff.py mozrunner=dev

(Could also be `tip`, `HEAD` etc.)

Thinking it over, while the latter way appears more straight-forward, I prefer the former way for a few reasons:

* When you want to do this, you generally want to do this across the board.  `generate_diff.py mozrunner=dev mozprocess=dev mozprofile=dev` is just silly, and error-prone.

* While I may or may not add any along with this bugs, a --develop flag could also enable other conveniences for staging.  generate_diff.py, along with other write-in-pen scripts, is difficult to debug because it tries to ensure sanity before writing in pen, since you're sure as hell not going to ensure sanity otherwise. (To be fair, it writes in pen and reverts, but you need to make sure you are getting back to the right state.  Longer explanation available on request but please lets use email or irc; doesn't belong in this bug.) --develop could be used to make this somewhat easier.  You might also imagine other cute features for staging as would make whoever's life easier.

* As hinted at earlier, doing `mozrunner=HEAD` or other revision could be useful for staging (and, as per policy *STAGING ONLY!*), I view this as other functionality, not something we'll want in general.  Of course, I give HEAD as an example, but obviously not HEAD since that's the same as unspecified. Sigh.  Communicating this badly.  Hope you get idea. -- caveman

Comment 2

[Jeff Hammel]

Addendum: but meant to add, if anyone has any strong opinions...now's the time to speak up ;)

Comment 3

[Jeff Hammel]

P.P.S. Also, there is the niceity that if you have a command line for --develop and you bump versions, you should be able to use the same command line, sans `--develop`, post-version bumping assuming you don't screw up anything.  This makes me think that we have workflow, and where there is definable workflow, there is space to make things easier.  That whole "automation" thing the kids are talking about these days...

Comment 4

[Jeff Hammel]

Attachment: patch

Could do more here, but I *think* this suffices.

:ahal, feel free to pass on the review if you're busy and/or you think e.g. Will is more appropriate.

Comment 5

[Andrew Halberstadt [:ahal]]

Comment on attachment 724038 [details] [diff] [review]
patch

Review of attachment 724038 [details] [diff] [review]:
-----------------------------------------------------------------

Makes sense to me

Comment 6

[Jeff Hammel]

Thanks for the quick review.

Going to land and close the bug as this shouldn't hurt anything, but something may not be right.  See https://bugzilla.mozilla.org/show_bug.cgi?id=838374#c15 . Rather, something isn't right, but slowly marching forward...

Comment 7

[Jeff Hammel]

pushed: https://hg.mozilla.org/mozilla-central/rev/6c309875420c

Will close but reserving the right to reopen later

Comment 8

[Jeff Hammel]

Attachment: follow up: a bug fix and some more niceness

There is one serious bug fixed here:  git HEAD does not mean what i thought it meant.  Instead, AIUI, we want master.  

There's also the minor bug fix wrt forgetting store_true on --develop, a --no-check mode for easier debugging, and a TODO item for the future.

Comment 9

[Andrew Halberstadt [:ahal]]

Comment on attachment 724515 [details] [diff] [review]
follow up: a bug fix and some more niceness

Review of attachment 724515 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with one optional suggestion

::: testing/mozbase/generate_diff.py
@@ +360,5 @@
>  
>          # generate the diff and write to output file
> +        command = ['hg', 'addremove']
> +        # TODO: don't add untracked files via `hg addremove --exclude...`
> +        call(command, cwd=hg_root)

So does this mean that --no-check doesn't really work yet? If that's the case maybe we should just add it in when it does.

Comment 10

[Jeff Hammel]

--no-check works currently, for what I intended it for: not checking if any changes have been made to m-c or if any untracked files are there.

The TODO, while vaguely related, is another matter.  I'll file a follow-up about it.

Comment 11

[Jeff Hammel]

(In reply to Andrew Halberstadt [:ahal] from comment #9)
> Comment on attachment 724515 [details] [diff] [review]
> follow up: a bug fix and some more niceness
> 
> Review of attachment 724515 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r+ with one optional suggestion
> 
> ::: testing/mozbase/generate_diff.py
> @@ +360,5 @@
> >  
> >          # generate the diff and write to output file
> > +        command = ['hg', 'addremove']
> > +        # TODO: don't add untracked files via `hg addremove --exclude...`
> > +        call(command, cwd=hg_root)
> 
> So does this mean that --no-check doesn't really work yet? If that's the
> case maybe we should just add it in when it does.

pushed: https://hg.mozilla.org/mozilla-central/rev/8e68f4d73ec4

Comment 12

[Jeff Hammel]

Added minimal documentation at https://wiki.mozilla.org/Auto-tools/Projects/Mozbase#Using_generate_diff.py

Comment 13

[Jonathan Griffin (:jgriffin)]

(In reply to Jeff Hammel [:jhammel] from comment #12)
> Added minimal documentation at
> https://wiki.mozilla.org/Auto-tools/Projects/Mozbase#Using_generate_diff.py

Nice, thanks Jeff.