764592 - Versioning for the talos package for mozharness

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Jeff Hammel]

Currently, we use a talos.zip that is pointed to in-tree via
talos.json:

http://hg.mozilla.org/mozilla-central/file/47be099e523c/testing/talos/talos.json

This has the advantage that a particular mozilla-central changeset
will use a particular version of talos that *should* be compatible
with it.

We would like to get rid of talos.zip for several reasons, mostly that
it is hard to maintain (see bug 764588 for more detail).  Mozharness
supports installing from python packages via pip, which we would like
to use.

However, we do not want to lose the per-changeset correctness of which
talos to use.  This would be a step backwards and make life difficult
in general (bisection, results validation, etc).

So mozilla-central/testing/talos should contain some record of which
version of talos to use.

My proposal:

- Every time we want m-c to use a new talos, we bump the talos version
  (it is currently 0.0).  We release this version to the internal
  "pypi" (see bug 701506).

- we then update some stub in m-c (presumedly a mozharness .json file
  or perhaps a pip requirements file) to point to this particular
  version of talos to use.
  * conceivably, we can store other data that we want reflected on a
    per-changeset basis in testing/talos too
  * (these will presumedly point to locations in the build network.
    This is too bad as it will only allow those with build network
    access to reproduce any issues.  However, this is the same as with
    talos.json currently, not a step backwards).

I think that's about it.  The details need to be worked out but I
think the proposal works in theory.  It could be applied to other
harnesses going forward as well.

I wasn't sure whether to put this in the Talos component or here.  If
it should be moved, please feel free.

Comment 1

[Aki Sasaki (not active)]

Getting it out of triage, but we'll need this for bug 713055.

Comment 2

[Aki Sasaki (not active)]

Action items:
move this to wiki (where?) and close bug.

Comment 3

[Jeff Hammel]

I vote for https://wiki.mozilla.org/Buildbot/Talos or https://wiki.mozilla.org/Buildbot/Talos/Versioning if we want it on its own page.  Any votes or objections, :aki, :jmaher?  I'll do the move once we pick a place. And for good measure our next release to m-c we can give talos a real version

Comment 4

[Aki Sasaki (not active)]

Those pages work for me.

Something I noticed while running talos_script.py:

If we explicitly set talos_url, we actually don't have to version as strictly.
(I uploaded a talos-249452fec498.tar.gz to build.m.o/talos/findlinks and that works just fine as long as the config file sets the talos_url there)

We currently have http://hg.mozilla.org/build/talos/archive/tip.tar.gz as the default talos_url which means we always specify talos_url, so this versioning may only come into play if we set talos_url to None in a config file or command line option and set the pypi_url.

Comment 5

[Jeff Hammel]

So...we haven't been doing this. :) This is a bit of a chicken-and-egg problem: updating the version currently is pointless (and will probably result in complaints of "one more thing to update when we land talos.zip") and, theoretically, involves an upload of all dependencies to the releng pypi, which AIUI is done by filing a bug and putting the attachments god-knows where.  Though, being honest, its been such a long time since I've touched the talos+mozharness project that I forget on what we've decided and where/if it is documented.

Conversely, we need to do this when talos+mozharness goes into production.  Otherwise things will be awful.

I don't want to creep scope here, but this is related to the problem (sorta, and for now, anyway) that it is difficult to deploy talos.  Maybe its worth diagnosing this problem and making it easier?

Or I'd be happy to hear other suggestions for solving the problem: "I want to deploy talos tip.  What do I do to make this happen?" (I doubt anyone is surprised that my solution is "write a damn script that will do all the things")

Comment 6

[Aki Sasaki (not active)]

I think comment 4 may make this less problematic in production.
We may be able to wontfix this or implement it later.

I think comment 2 is still accurate, since this is a process change bug instead of a one-time fix and close bug.

Comment 7

[Aki Sasaki (not active)]

I'm not relying on version numbers; I'm relying on the talos.json talos_url pointing to a specific tarball.

WONTFIXing and removing the bug dependency since a) I'm not blocked on it and b) there doesn't seem to be movement here.  Feel free to reopen or file new if we want this versioning for other reasons.

Comment 8

[Jeff Hammel]

Sounds like a plan.  I do think we'll want/need to start versioning talos at some point, but we can punt for now

Comment 9

[Jeff Hammel]

(As usual for such things I'm content to wait until this becomes a problem instead of trying to impose process before things break.)