894980 - Return the try server the ability to test custom talos

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

Before, when the a-team wanted to test a new talos version, they would upload a new talos.zip to their people account, make a talos.json change to point to it and push it to try.

We have now lost that ability because we checkout the repo rather than download a talos.zip.
https://tbpl.mozilla.org/?tree=Try&jobname=talos&rev=1af89a37b3a5
https://tbpl.mozilla.org/php/getParsedLog.php?id=25372036&tree=Try&full=1#error0

I assume that talos.zip-url will be deprecated.
Why do we have a talos.tar.gz to download for? Is it so we can grab python packages?

As a workflow I think it would be ideal if we could grab new talos versions through a revision and a user repo.
Open to suggestions.

     "talos.zip": {
         "url": "http://build.mozilla.org/talos/zips/talos.a11542b55a70.zip",
         "path": ""
     },
     "global": {
         "talos_url": "http://puppetagain.pub.build.mozilla.org/data/python/packages/talos-7d4ae29b7d1f.tar.gz",
        "talos_revision": "7d4ae29b7d1f"
     },

Comment 1

[Aki Sasaki (not active)]

talos.zip-url will be deprecated.
global-talos_url was used before you made your change.

Maybe we can specify the talos_repo in addition to talos_revision?  Then people could create a user hg repo and push their changes to it, and point at that talos_repo and talos_revision.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

a custom talos user repo + revision would be just fine.

Comment 3

[Jason Yeo[:jyeo]]

Attachment: Use talos repo from talos json

I created another function to query_talos_revision for completeness sake. I hope that's okay.

Comment 4

[Jason Yeo[:jyeo]]

Attachment: Talos json with repo string

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 777289 [details] [diff] [review]
Talos json with repo string

We should keep talos.zip until we're fully rolled out; buildbot talos uses it.

So once mozharness talos is rolled out to all mozilla-central level branches, we can land a "remove talos.zip from talos.json" patch to mozilla-inbound, but we can't roll that out to aurora, beta, release, etc. until those branches are also using mozharness talos.

r=me for the talos_url removal and talos_repo addition.

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 777275 [details] [diff] [review]
Use talos repo from talos json

I would guess that self.has_cloned_talos isn't useful anymore, since that's the default (and only?) workflow.  I'm ok leaving that alone for now, though, unless you want to do cleanup.

This looks good, thanks!

Comment 7

[Jason Yeo[:jyeo]]

Attachment: s/talos_url/talos_repo/ pointing to hg.m.o/build/talos

Comment 8

[Armen [:armenzg]]

Comment on attachment 777361 [details] [diff] [review]
s/talos_url/talos_repo/ pointing to hg.m.o/build/talos

Carrying forward the r+ from obsolete patch.
I will land the patch once m-i is open.

I will merge mozharness tomorrow morning.

Comment 9

[Armen [:armenzg]]

Merged to production.

Comment 10

[Armen [:armenzg]]

Testing on try one last time to be sure:
https://tbpl.mozilla.org/?tree=Try&rev=b0b2e2e2cff8

-        "talos_url": "http://puppetagain.pub.build.mozilla.org/data/python/pack
-        "talos_revision": "7d4ae29b7d1f"
+        "talos_repo": " ssh://hg.mozilla.org/users/armenzg_mozilla.com/talos",
+        "talos_revision": "c41ec0e321c6"

Comment 11

[Armen [:armenzg]]

Trying now with http:// instead of ssh://
https://tbpl.mozilla.org/?tree=Try&rev=7e4be7297875

Comment 12

[Jason Yeo[:jyeo]]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #10)
> ssh://hg.mozilla.org/users/armenzg_mozilla.com/talos",

I'm not sure if ssh works here.

I pushed another try commit here: https://tbpl.mozilla.org/?tree=Try&rev=6b5220dc4064

-        "talos_url": "http://puppetagain.pub.build.mozilla.org/data/python/packages/talos-7d4ae29b7d1f.tar.gz",
-        "talos_revision": "7d4ae29b7d1f"
+        "talos_repo": "http://hg.mozilla.org/users/armenzg_mozilla.com/talos",
+        "talos_revision": "c41ec0e321c6"

Comment 13

[Armen [:armenzg]]

(In reply to Jason Yeo [:jyeo] from comment #12)
> (In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment
> #10)
> > ssh://hg.mozilla.org/users/armenzg_mozilla.com/talos",
> 
> I'm not sure if ssh works here.
> 
> I pushed another try commit here:
> https://tbpl.mozilla.org/?tree=Try&rev=6b5220dc4064
> 
> -        "talos_url":
> "http://puppetagain.pub.build.mozilla.org/data/python/packages/talos-
> 7d4ae29b7d1f.tar.gz",
> -        "talos_revision": "7d4ae29b7d1f"
> +        "talos_repo":
> "http://hg.mozilla.org/users/armenzg_mozilla.com/talos",
> +        "talos_revision": "c41ec0e321c6"

Comment 11 is a push without ssh:// (that is what I was meaning).

You can probably cancel your push from comment 12 if you believe the push from comment 11 is sufficient (please cancel if so to save resources).

Comment 14

[Aki Sasaki (not active)]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #10)
> Testing on try one last time to be sure:
> https://tbpl.mozilla.org/?tree=Try&rev=b0b2e2e2cff8
> 
> -        "talos_url":
> "http://puppetagain.pub.build.mozilla.org/data/python/pack
> -        "talos_revision": "7d4ae29b7d1f"
> +        "talos_repo": "
> ssh://hg.mozilla.org/users/armenzg_mozilla.com/talos",

Noting that you have a space before the ssh:// here, so it's probably not a valid test.

Comment 15

[Armen [:armenzg]]

jmaher, I believe you have a syntax error on your talos repo
http://hg.mozilla.org/users/jmaher_mozilla.com/tpain/rev/443f419ec164

09:44:04     INFO -    File "C:\slave\test\build\venv\lib\site-packages\talos\run_tests.py", line 289
09:44:04 CRITICAL -      except talosError, e:
09:44:04     INFO -           ^
09:44:04     INFO -  IndentationError: expected an indented block

Comment 16

[Jason Yeo[:jyeo]]

(In reply to Aki Sasaki [:aki] from comment #14)
> you have a space before the ssh:// here, so it's probably not a
> valid test.

I tried removing the space and executing it on the slave. Here's what I got:

[[e-mail] ~]$ hg --config ui.merge=internal:merge clone "ssh://hg.mozilla.org/users/armenzg_mozilla.com/talos" /builds/slave/talos-slave/test/build/talos_repo
remote: Permission denied (publickey).
abort: no suitable response from remote hg!
[[e-mail] ~]$

Let's just stick to read-only http addresses for the repo.

Comment 17

[Armen [:armenzg]]

(In reply to Jason Yeo [:jyeo] from comment #12)
> I pushed another try commit here:
> https://tbpl.mozilla.org/?tree=Try&rev=6b5220dc4064

10:00:56     INFO - Cloning http://hg.mozilla.org/users/armenzg_mozilla.com/talos to /builds/slave/talos-slave/test/build/talos_repo.

We're good to go:
https://hg.mozilla.org/integration/mozilla-inbound/rev/bb0b437e60c7

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/bb0b437e60c7

Comment 19

[Armen [:armenzg]]

Wee!

Comment 20

[Armen [:armenzg]]

We have the ability back.
We will have to fix bug 895721 before we deploy bug 713055 across the board.