Status

Testing
Talos
RESOLVED DUPLICATE of bug 1048134
5 years ago
2 years ago

People

(Reporter: Jeff Hammel, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: talos-android)

(Reporter)

Description

5 years ago
create_talos_zip.py and associated architecture is rather hard to
maintain:

1. we currently point to the HEAD version of most dependencies.  While
  we've had good luck with this so far, this means that any API
  changes in dependencies could kill us.  While the versions should be
  specified in setup.py (and they are), changing the dependencies here
  and in create_talos_zip.py requires a lot of (error-prone) manual
  work

2. we specify things on a per-file basis vs a package basis.  This
  means that if a new file is added that is a needed import (or
  otherwise), the resulting talos.zip will fail :( Likewise for
  file-deletion, etc.  This has kicked us in the past.

3. We will have multi-level dependencies.  Currently, all of our
dependencies are single level: that is talos depends on ${PACKAGE}. We
want to get talos on (e.g.) mozrunner, which depends on (e.g.) mozinfo
amongst other packages.  This means we need to unpack mozrunner in the
talos directory, mozinfo in the talos directory, and mozinfo in the
mozrunner directory.  Of course, add (e.g.) talos using mozprofile,
etc, and the picture gets even more complicated.

Example:

talos/
├── mozinfo
├── mozprofile
└── mozrunner
    ├── mozinfo
    └── mozprofile
        └── mozinfo

If you didn't understand that, good :) It is a bad way of doing things
in general.

4. while its no, say, `apt`, python (pip/easy_install/etc) has a
reliable, or at least tameable, way of resolving dependencies.  We
should avoid rolling our own, especially since we have at least three
of the folks on the python packaging authority at our company.

5. If we need binary dependencies, such as pywin32, we cannot
currently handle these either.

We need to have talos running in production on mozharness before this
can really be addressed, but noting for that day.
(Reporter)

Updated

5 years ago
Depends on: 713055
Why don't we simply do python setup.py bdist --format=zip ?
(Reporter)

Comment 2

5 years ago
Most of what create_talos_zip.py does is bundle the dependencies in a directory structure (see comment 0).  setup.py bdist will not do this.  talos in production is currently unpackaged (that is, the dependencies are not installed and setup.py is not run). Hopefully with mozharness we can move away from this.
@Jeff - Yeah, I realized that after *a lot* of digging through setup.py options. Such a mess. So what's the final goal of this bug then can you clarify? Like "being able to install Talos via pip" or what?
(Reporter)

Comment 4

5 years ago
(In reply to Burak Yiğit Kaya [:BYK] from comment #3)
> @Jeff - Yeah, I realized that after *a lot* of digging through setup.py
> options. Such a mess. So what's the final goal of this bug then can you
> clarify? Like "being able to install Talos via pip" or what?

in theory one should be able to install talos via pip:

pip install http://hg.mozilla.org/build/talos/archive/tip.tar.gz

works for me.  If it does not work in a particular situation, it is a bug that should be filed and fixed. It is not yet on pypi, though I imagine as we start versioning talos we will release it there too.

The purpose of this bug is mostly a tracking bug.  Once we no longer have to support talos.zip in production, the fix is pretty easy: hg rm create_talos_zip.py . But we obviously can't do that until the way of running in production is updated.
update on this bug:
* desktop talos uses the repository, not talos.zip, thanks to mozharness.
* Android pandas will be using the repository, not talos.zip within the next few weeks, thanks to mozharness.
* Android tegras will continue to require a tests.zip, probably for the next 12 months.
** tegras will be decommissioned in Q2/Q3 2014, they will be replaced by Android 2.3 on Panda boards or something equivalent.  At this point, we will have all talos being pulled from the repository based on a given revision.

Can we close this bug as our path to remove talos.zip is converting to mozharness.
Whiteboard: talos-android
Closing this as a dupe of bug 1048134. Feel free to reopen I'm wrong here.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1048134
You need to log in before you can comment on or make changes to this bug.