764588 - get rid of talos.zip

Status: RESOLVED DUPLICATE of bug 1048134

(Testing :: Talos, defect)

Description

[Jeff Hammel]

create_talos_zip.py and associated architecture is rather hard to
maintain:

1. we currently point to the HEAD version of most dependencies.  While
  we've had good luck with this so far, this means that any API
  changes in dependencies could kill us.  While the versions should be
  specified in setup.py (and they are), changing the dependencies here
  and in create_talos_zip.py requires a lot of (error-prone) manual
  work

2. we specify things on a per-file basis vs a package basis.  This
  means that if a new file is added that is a needed import (or
  otherwise), the resulting talos.zip will fail :( Likewise for
  file-deletion, etc.  This has kicked us in the past.

3. We will have multi-level dependencies.  Currently, all of our
dependencies are single level: that is talos depends on ${PACKAGE}. We
want to get talos on (e.g.) mozrunner, which depends on (e.g.) mozinfo
amongst other packages.  This means we need to unpack mozrunner in the
talos directory, mozinfo in the talos directory, and mozinfo in the
mozrunner directory.  Of course, add (e.g.) talos using mozprofile,
etc, and the picture gets even more complicated.

Example:

talos/
├── mozinfo
├── mozprofile
└── mozrunner
    ├── mozinfo
    └── mozprofile
        └── mozinfo

If you didn't understand that, good :) It is a bad way of doing things
in general.

4. while its no, say, `apt`, python (pip/easy_install/etc) has a
reliable, or at least tameable, way of resolving dependencies.  We
should avoid rolling our own, especially since we have at least three
of the folks on the python packaging authority at our company.

5. If we need binary dependencies, such as pywin32, we cannot
currently handle these either.

We need to have talos running in production on mozharness before this
can really be addressed, but noting for that day.

Comment 1

[Burak Yiğit Kaya [:BYK]]

Why don't we simply do python setup.py bdist --format=zip ?

Comment 2

[Jeff Hammel]

Most of what create_talos_zip.py does is bundle the dependencies in a directory structure (see comment 0).  setup.py bdist will not do this.  talos in production is currently unpackaged (that is, the dependencies are not installed and setup.py is not run). Hopefully with mozharness we can move away from this.

Comment 3

[Burak Yiğit Kaya [:BYK]]

@Jeff - Yeah, I realized that after *a lot* of digging through setup.py options. Such a mess. So what's the final goal of this bug then can you clarify? Like "being able to install Talos via pip" or what?

Comment 4

[Jeff Hammel]

(In reply to Burak Yiğit Kaya [:BYK] from comment #3)
> @Jeff - Yeah, I realized that after *a lot* of digging through setup.py
> options. Such a mess. So what's the final goal of this bug then can you
> clarify? Like "being able to install Talos via pip" or what?

in theory one should be able to install talos via pip:

pip install http://hg.mozilla.org/build/talos/archive/tip.tar.gz

works for me.  If it does not work in a particular situation, it is a bug that should be filed and fixed. It is not yet on pypi, though I imagine as we start versioning talos we will release it there too.

The purpose of this bug is mostly a tracking bug.  Once we no longer have to support talos.zip in production, the fix is pretty easy: hg rm create_talos_zip.py . But we obviously can't do that until the way of running in production is updated.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

update on this bug:
* desktop talos uses the repository, not talos.zip, thanks to mozharness.
* Android pandas will be using the repository, not talos.zip within the next few weeks, thanks to mozharness.
* Android tegras will continue to require a tests.zip, probably for the next 12 months.
** tegras will be decommissioned in Q2/Q3 2014, they will be replaced by Android 2.3 on Panda boards or something equivalent.  At this point, we will have all talos being pulled from the repository based on a given revision.

Can we close this bug as our path to remove talos.zip is converting to mozharness.

Comment 6

[Julien Pagès (:parkouss)]

Closing this as a dupe of bug 1048134. Feel free to reopen I'm wrong here.