855893 - moving away from puppetagain to tree-generated tests.zip for mozharness testing

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Jeff Hammel]

Currently, mozbase as utilized by mozharness is done via uploading
python packages (via filing a bug, etc, as minimally documented at
https://wiki.mozilla.org/Auto-tools/Projects/Mozbase#Uploading_to_Puppetagain
) to i.e
https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Python .
As an outcome of the discussion from
https://wiki.mozilla.org/Auto-tools/Projects/Mozbase/Automation-Refactor/Meetings/2013-03-26
agreement was reached that it would be better to use the mozbase
modules from the associated tree generated to tests.zip.

The reasons for the decision as best I can represent is to avoid the
additional bug filing and upload step for the package as well as for
simplistic concurrency with the tree.

A few aspects of this I will detail.

Regarding https://bugzilla.mozilla.org/show_bug.cgi?id=855041#c0 :

> Aki, how hard would it be to set something up that mirrors the
> mozbase packages from pypi to puppetagain anytime they get updated?
> I guess it would be hard to make sure that nothing automatically
> starts using the new packages.

As I understood it, and to be fair I have had a very hard time
successfully communicating of late, that apropos mozbase that the
current ad hoc mirroring policy was what was desired.  Am I mistaken?
I thought we had decided against mirroring on version bump, even in
the interim.
The uploading to puppetagain will be unnecessary if this bug is
resolved and therefore such a structure would be unneeded.
If we do want to upload to puppetagain on release, versionbump.py
could be modified to do so.  E.g. if the current ([A*Team]file a bug +
[releng]upload) is the desired workflow, the script (or similar) could
be modified to automatically file the releng bug and (e.g.) either put
the packages at an appropriate location on (e.g.) people.m.o or
reference the pypi URL (the latter being my preference).  In any case,
if this bug is resolved, none of that is necessary as such.

This also (re)raises the issue of "should we move the mozbase
canonical source to mozilla-central (and kill the github repo)?".
While out of scope for this bug, it may be worth revisiting that
discussion.

Comment 1

[Jonathan Griffin (:jgriffin)]

The root issue I see is that currently, if a new version of a package is uploaded to puppetagain, tests that are using puppetagain packages will immediately use the new version, and this may cause test failures in TBPL that aren't attributable to any commit, which makes sheriffing effectively impossible.

I think moving to the in-tree mozbase is only one potential solution; another is to use strict versioning in mozbase packages and other tools (like Marionette) that use mozbase.  We'd have to bump setup.py of said tools in order to use new packages, which would leave a commit trail for sheriffs to follow.

Comment 2

[Jeff Hammel]

Yet another solution is to have requirements.txt files in-tree.  Regardless, the solution likely will affect at least ATeam and RelEng as stakeholders.

Comment 3

[Jeff Hammel]

Also, wrt moving to m-c, there is the guideline asserted in https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#11.29_Easy_for_a_dev_to_run_locally :

> Ideally part of mozilla-central (legacy exception being Talos).

Comment 4

[Jeff Hammel]

Regardless of the exact shape of solution, I believe requiring strict versions (in mozbase, for now) will eliminate this class of problems, or at least make quiet failures louder.  I realize that this goes against our decision made in October for versioning, but I did not anticipate this then.  While it is not "correct", neither is what we do; the latter is more commonly used and while easier to maintain for projects without this concern, but IMHO our need here >> convention.

See also https://plus.google.com/+IanBicking/posts/M8Wpsfsyykc

:ahal, :jgriffin, :wlach, etc, should we switch to a strict version model (and all the annoyance that goes with it) for making this class of failures easier to diagnose and/or should we discuss in another bug?

Comment 5

[Jeff Hammel]

While I have no interest in particularly defending my POV, I would tend towards a solution with requirements files in-tree and mozbase packages mirrored to m-c and puppetagain .

Comment 6

[William Lachance (:wlach)]

I think it'd be simplest to just use the in-tree version of mozbase for all tests, etc. corresponding to that version. In fact as I understand it this is what we *were* doing until recently. I guess it's only with mozharness that we decided to do something different, for what reason(s) I don't know.

All this stuff about requirements and versioning and puppetagain makes my brain hurt. If we can avoid having to worry about that stuff (at least when it comes to m-c) I think that'd be good.

Comment 7

[Jonathan Griffin (:jgriffin)]

The reason we had to move away from in-tree mozbase was because we needed to land some Marionette changes to fix some test harness problems, and these changes were dependent on some mozbase changes.  We couldn't land the mozbase changes on mozilla-b2g18 due to very tight landing restrictions at the time, so we shifted to using puppetagain.

Comment 8

[Jeff Hammel]

(In reply to William Lachance (:wlach) from comment #6)
> I think it'd be simplest to just use the in-tree version of mozbase for all
> tests, etc. corresponding to that version. In fact as I understand it this
> is what we *were* doing until recently. I guess it's only with mozharness
> that we decided to do something different, for what reason(s) I don't know.
> 
> All this stuff about requirements and versioning and puppetagain makes my
> brain hurt. If we can avoid having to worry about that stuff (at least when
> it comes to m-c) I think that'd be good.

I count this as a +1 for move mozbase canonical to m-c.  I'll file the bug when I kill my inbox

Comment 9

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #8)
> (In reply to William Lachance (:wlach) from comment #6)
> > I think it'd be simplest to just use the in-tree version of mozbase for all
> > tests, etc. corresponding to that version. In fact as I understand it this
> > is what we *were* doing until recently. I guess it's only with mozharness
> > that we decided to do something different, for what reason(s) I don't know.
> > 
> > All this stuff about requirements and versioning and puppetagain makes my
> > brain hurt. If we can avoid having to worry about that stuff (at least when
> > it comes to m-c) I think that'd be good.
> 
> I count this as a +1 for move mozbase canonical to m-c.  I'll file the bug
> when I kill my inbox

Nope, once again I can't understand anything.  I have no idea what is desired here

Comment 10

[Armen [:armenzg]]

Having strict version control in mozharness won't play well (with our current model - more info in my "another solution" at the end) as we can have branches like b2g18 wanting to use different versions of mozbase than m-c yet won't be able to make the distinction.

Moving the version control to m-c as jhammel mentions would be a good solution and I support this approach.
Other features:
* the ability to push to try and test new versions of python packages.
* the ability to ride the trains

Another solution (mentioned on https://bugzilla.mozilla.org/show_bug.cgi?id=791924#c8), is that through m-c we could control which revision of mozharness is used. This would allow for different branches to use different version of mozharness' scripts and configurations (which could include strict python version control).

Comment 11

[Andrew Halberstadt [:ahal]]

Order of preference:
1) use in-tree mozbase (like we used to)
2) use requirements.txt
3) use strict versioning (I'm not a fan of this approach)

Some questions that we haven't touched:
If doing 1), should we move the repo from github to m-c?
If doing 2), should we remove the mirror from m-c?

Fwiw, these were all questions I was planning on bringing up during next Tuesdays mozbase meeting. I'll send out a reminder email to a wider audience as I think we have lots of important things to talk about.

Comment 12

[Jonathan Griffin (:jgriffin)]

Another concern is making the experience of running the tests locally the same (at least in terms of Python packages versions) as being run in buildbot.  Strict versioning would accomplish this; the other solutions may or may not, depending on how they were implemented.

Good topics for discussion at the next mozbase meeting.

Comment 13

[Jeff Hammel]

(In reply to Andrew Halberstadt [:ahal] from comment #11)
> Order of preference:
> 1) use in-tree mozbase (like we used to)
> 2) use requirements.txt
> 3) use strict versioning (I'm not a fan of this approach)
> 
> Some questions that we haven't touched:
> If doing 1), should we move the repo from github to m-c?
> If doing 2), should we remove the mirror from m-c?
> 
> Fwiw, these were all questions I was planning on bringing up during next
> Tuesdays mozbase meeting. I'll send out a reminder email to a wider audience
> as I think we have lots of important things to talk about.

Sounds like a plan.  Discussion on this bug (or really in whatever forum works) prior to the meeting is also welcome, so if you know anyone that should be CCed or otherwise made aware of the issue, please do so!  While obviously post-meeting we're not going to immediately and irreversibly act, but a lot of time is lost (in my experience at Mozilla) making plans assuming that all the stakeholders are there and/or aware of the issue, and then re-making plans often haphazardly when the assumption is mistaken. It'd be nice if we could get closer to people->plan->action .

Regarding strict versioning, while I don't think it will be an enormous gain, I do think it would be of help.  I note that by itself it is *not* a solution to the problem, so I view it as less of a 3rd option as an independent option regardless of whether we go with 1. or 2. I *think* pegging strictly will help us see what has gone wrong and see it more quickly; that said, its not a blanket solution by itself.  if e.g. mozcrash == 0.5 is require and said version is not in-tree then we'll try to hit pypi.  (Note that this is not particularly different from what we have today, e.g. if we required mozcrash >= 0.5).

I agree that strictly pegging is less semantic and, sans deployment concerns, I would not consider it for a set of softwares like mozbase.  But if it does help diagnose production problems, I would lean towards doing it.  The overhead for mozbase maintenance is minimal.  At worst, you bump a few versions that you don't have to.  I basically agree with what Ian is saying in the link given in comment 4: neither >= nor == nor anything you can do in python versioning current or planned really suffices. If you're going to be wrong anyway.... >= is more flexible, for sure. But if == aids us...so be it.

I'm not necessarily pro-strict-versioning.  IFF it helps us (where 'us' equals, ultimately, Mozilla) in some term, then I'm for it.  If it makes no difference, then I mostly don't care and would very weakly side with less strict (>=). I don't in particular want to do anything more complicated than those two, if avoidable.  Again, per Ian's post, while setuptools supports complicated sets of version comparisons, you can't actually have something whereby you can meaningfully inform, on release of a new package, that the old package does not work with it.

My feelings briefly on:
> If doing 1), should we move the repo from github to m-c?
> If doing 2), should we remove the mirror from m-c?
Without going into too much discussion, if we're not getting any value from independent development outside of a staging area (IMHO, whatever advantages/disadvantages github in particular as a development platform have are irrelevant to this matter), then I see no reason not to move to m-c. My vision for mozbase and, larger, mozilla python, and largest, mozilla code in general, is an assemblage of library code and consumers of said library code (e.g. the `mozrunner` console script consumes the https://pypi.python.org/pypi/mozrunner library code, regardless that they're in the same package) and that there is no central or monolith; just ecology </soapbox>. But it feels like I'm mostly the one aiming towards that and I don't really want to fight that fight.  If we want to serve our right-now needs right now, lets do it.  The one thing I strongly want to avoid is a halfway solution, where we are paying for modularity (e.g. here in the cost of complexity of the mirroring, versionbumping, etc scripts) while getting none of its benefits.  And yes, sadly, this is the very short version ;)  I'll mostly try not to talk about this much Tuesday.

As for "2), should we remove the mirror from m-c?", I'm not sure exactly what you mean here; if you mean upload to puppet-again (which is technically not from m-c, at least usually if not always), then I think that depends on RelEng and infrastructure needs and concerns.  Is this what you meant, Andrew?  Could you elaborate?

Comment 14

[Andrew Halberstadt [:ahal]]

The plan is to remove the dependency from puppetagain on m-c/inbound, but leave it on b2g18. This will require some way of using different mozharness configs for different branches (or a way for the mozharness script to determine which branch it is running against).

Comment 15

[Andrew Halberstadt [:ahal]]

I'm thinking of also removing http://mxr.mozilla.org/mozilla-central/source/b2g/test/b2g-unittest-requirements.txt from the tree as it will no longer be used by anything. Though if I do remove it, then specifying the "use_puppetagain" flag (or whatever it will be called) in the mozharness configs will cause bustage.

Thoughts?

Comment 16

[Jonathan Griffin (:jgriffin)]

(In reply to Andrew Halberstadt [:ahal] from comment #15)
> I'm thinking of also removing
> http://mxr.mozilla.org/mozilla-central/source/b2g/test/b2g-unittest-
> requirements.txt from the tree as it will no longer be used by anything.
> Though if I do remove it, then specifying the "use_puppetagain" flag (or
> whatever it will be called) in the mozharness configs will cause bustage.
> 
> Thoughts?

Yes, I'd like to remove this in m-c; otherwise it's just another potential source of confusion.

Comment 17

[Andrew Halberstadt [:ahal]]

So my patch is green on Ash, but it seems to also make TBPL mark failures as green (i.e the debug reftests are still failing, yet they appear green). Investigating..

Comment 18

[Andrew Halberstadt [:ahal]]

Attachment: Patch 1.0 - use in-tree mozbase

Comment 19

[Andrew Halberstadt [:ahal]]

Attachment: Patch 1.0 (gecko) - keep using puppetagain on b2g18

This patch needs to land on b2g18 and b2g18v1.0.1 before the mozharness patch can land.

Comment 20

[Aki Sasaki (not active)]

Comment on attachment 758591 [details] [diff] [review]
Patch 1.0 - use in-tree mozbase

That works.
The tree config could actually specify this, which would allow for differing module lists to ride the trains, but this will do for now.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/f6b54fd6d733
https://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/fe588e629cbc

Comment 22

[Andrew Halberstadt [:ahal]]

Thanks Ryan! We don't want this on anything other than b2g18 branches though.

Comment 23

[Andrew Halberstadt [:ahal]]

(In reply to Aki Sasaki [:aki] from comment #20)
> Comment on attachment 758591 [details] [diff] [review]
> Patch 1.0 - use in-tree mozbase
> 
> That works.
> The tree config could actually specify this, which would allow for differing
> module lists to ride the trains, but this will do for now.

Yeah, that's true. Might be good to just make it read from testing/mozbase/packages.txt or something. But I'll leave that for another time.

https://hg.mozilla.org/build/mozharness/rev/381e5915bfd2

Comment 24

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/f6b54fd6d733

Comment 25

[Justin Wood (:Callek)]

Comment on attachment 758591 [details] [diff] [review]
Patch 1.0 - use in-tree mozbase

Review of attachment 758591 [details] [diff] [review]:
-----------------------------------------------------------------

Our jenkins reported 56 newly added pep8 violations with this patch