1071794 - create_talos_zip should pull dependencies from pypi, not github

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[William Lachance (:wlach)]

The github repo for mozbase is abandoned. For the deps we want to include in talos.zip, let's fetch them from pypi.

Comment 1

[Armen [:armenzg]]

I could be misunderstanding. Please correct me if needed!
I think we should spend time deprecating the need to create a talos zip for mobile.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

ideally yes, we just need the android 4.0 mozharness script to be re-written

Comment 3

[William Lachance (:wlach)]

Attachment: Update talos.zip creation process

This patch actually does a few things:

1. What this bug says it does.
2. Use latest versions of all mozbase packages (instead of ancient github versions)
3. Remove simplejson copy (don't need it anymore)
4. Create the .zip file from a temporary directory. This saves us the trouble of needing to clean up after ourselves (just blow away the temporary directory at the end and we're done).

Comment 4

[William Lachance (:wlach)]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #1)
> I could be misunderstanding. Please correct me if needed!
> I think we should spend time deprecating the need to create a talos zip for
> mobile.

Yes, absolutely we should. This is just a stopgap.

Comment 5

[William Lachance (:wlach)]

Try run for .zip generated with new script: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=9df20b75ad4a

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

before I review too much, these versions are pegged at certain versions of modules for a reason.  Maybe those reasons are not needed anymore.  Have you tested this creates a .zip file that we can use on try server to run tests on android?

Comment 7

[William Lachance (:wlach)]

(In reply to Joel Maher (:jmaher) from comment #6)
> before I review too much, these versions are pegged at certain versions of
> modules for a reason.  Maybe those reasons are not needed anymore.  Have you
> tested this creates a .zip file that we can use on try server to run tests
> on android?

Of course. :) That's what's being run through try right now.

Comment 8

[William Lachance (:wlach)]

Attachment: Same patch with versions pegged to current

Actually, it's probably better not to mess with the version numbers here. Let's try to just keep them as they are for now, we can always upgrade later quite easily.

This patch keeps the version in sync with setup.py and what we were using before. The only exception is mozcrash, because 0.9 was a .zip (and thus doesn't work with the fetching function), but I don't think that should cause any issues.

Comment 9

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) from comment #5)
> Try run for .zip generated with new script:
> https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=9df20b75ad4a

Try run looks good; I think we can commit this if it looks ok to :jmaher.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8494010 [details] [diff] [review]
Same patch with versions pegged to current

Review of attachment 8494010 [details] [diff] [review]:
-----------------------------------------------------------------

this looks like a great addition!

::: create_talos_zip.py
@@ +80,4 @@
>                     for f in httplib2_files]
>  
>  # all dependencies
> +manifest = datazilla_client + yaml + oauth2 + httplib2_oauth2

didn't you find we don't need yaml anymore?

Comment 11

[William Lachance (:wlach)]

(In reply to Joel Maher (:jmaher) from comment #10)
> Comment on attachment 8494010 [details] [diff] [review]
> ::: create_talos_zip.py
> @@ +80,4 @@
> >                     for f in httplib2_files]
> >  
> >  # all dependencies
> > +manifest = datazilla_client + yaml + oauth2 + httplib2_oauth2
> 
> didn't you find we don't need yaml anymore?

There's still stuff referencing yaml in the mozharness scripts, so it's probably not safe to remove yet. Thinking about it a bit more, I think changing that is pretty low priority. Others things would have vastly more bang/buck.

Comment 12

[William Lachance (:wlach)]

Pushed: https://hg.mozilla.org/build/talos/rev/c104a3296ee3

I also needed to update the version of mozlog used, because mozcrash now depends on structured logging. A retriggered try run of rck2 looked ok and jmaher gave approval on irc. :)