721822 - move talos_from_code.py to the source tree

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

This should have been like this since the beginning.

I don't want to be supporting multiple code paths as we change the functionality of it.

Comment 1

[Armen [:armenzg]]

Attachment: download talos_from_code from source repo

Comment 2

[Armen [:armenzg]]

Attachment: talos_from_code.py

As found in revision 67ef2e00e8c7.

https://hg.mozilla.org/build/tools/raw-file/67ef2e00e8c7/scripts/talos/talos_from_code.py

Comment 3

[Armen [:armenzg]]

Attachment: remove talos_from_code.py from the tools repo

Comment 4

[Armen [:armenzg]]

Once this lands the work on bug 719567 becomes trivial.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 592205 [details] [diff] [review]
remove talos_from_code.py from the tools repo

Review of attachment 592205 [details] [diff] [review]:
-----------------------------------------------------------------

are there other files that reference this?  If not, r+

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 592202 [details] [diff] [review]
download talos_from_code from source repo

Review of attachment 592202 [details] [diff] [review]:
-----------------------------------------------------------------

to make this clear we will have:
<m-c>/testing/talos/talos_from_code.py
<m-c>/testing/talos/talos.json

my one question below and then r=me.

::: process/factory.py
@@ +7589,5 @@
>  
>          if self.customTalos is None and not self.remoteTests:
>              if self.talos_from_source_code:
>                  self.addStep(DownloadFile(
> +                    url=WithProperties("%(repo_path)s/raw-file/%(revision)s/testing/talos/talos_from_code.py"),

what if this doesn't exist?

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 592204 [details]
talos_from_code.py

>diff --git a/testing/talos/talos_from_code.py b/testing/talos/talos_from_code.py
>+def main():
>+
>+    # 3) download the necessary files
>+    # NOTE: there are two different path ways depending on the contents of the talos.json
>+    if get_value(jsonFilename, "downloadables") != None:
>+        downloadables = get_value(jsonFilename, "downloadables")
>+        print "INFO: talos.json URL: %s" % options.talos_json_url
>+        try:
>+            for key in downloadables:
>+                downloadUrl = get_value(jsonFilename, key)

I thought talos.json would look like:
{ "downloadables": {"talos_zip": "http://talos.zip", "pageloader_xpi": "http://pageloader.xpi" } }

From reading this it would be more like:
{ "downloadables": ["talos_zip", "pageloader_xpi"],
  "talos_zip": "http://talos.zip",
  "pageloader_xpi": "http://pageloader.xpi"
}

>+def download_file(url, saveAs=None):
>+    '''
>+    It downloads a file from the URL indicated and can be saved locally with
>+    a different name if needed.
>+    '''
>+    req = urllib2.Request(url)
>+    filename = get_filename_from_url(url)
>+    f = urllib2.urlopen(req)
we should put error handling around this?  

>+    local_file = open(saveAs if saveAs else filename, 'wb')
>+    local_file.write(f.read())
>+    local_file.close()
>+    return filename

return filename or local_file's name?



Do we need specific key values?  I don't think so since we are just saving the filename of the file we downloaded.  Also for pageloader, is it always downloaded to the same spot?  Not sure how that works.

A lot of questions I have here, lets address those before a r+.

Comment 8

[Armen [:armenzg]]

(In reply to Joel Maher (:jmaher) from comment #5)
> Comment on attachment 592205 [details] [diff] [review]
> remove talos_from_code.py from the tools repo
> 
> Review of attachment 592205 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> are there other files that reference this?  If not, r+

Nope. See http://mxr.mozilla.org/build/search?string=talos_from_code (to download the file and then to run it).

(In reply to Joel Maher (:jmaher) from comment #6)
> ::: process/factory.py
> @@ +7589,5 @@
> >  
> >          if self.customTalos is None and not self.remoteTests:
> >              if self.talos_from_source_code:
> >                  self.addStep(DownloadFile(
> > +                    url=WithProperties("%(repo_path)s/raw-file/%(revision)s/testing/talos/talos_from_code.py"),
> 
> what if this doesn't exist?

We have this "if self.talos_from_source_code:" in [1]

I have to backport to mozilla-aurora and mozilla-beta.

[1] http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l7553

Comment 9

[Armen [:armenzg]]

Attachment: add talos_from_code.py

My bad. I attached the wrong patch.

This is a no-op change. The same as doing:
cd testing/talos && wget https://hg.mozilla.org/build/tools/raw-file/67ef2e00e8c7/scripts/talos/talos_from_code.py

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 592213 [details] [diff] [review]
add talos_from_code.py

Review of attachment 592213 [details] [diff] [review]:
-----------------------------------------------------------------

this is everything but the pageloader.xpi bit, correct.

Comment 11

[Armen [:armenzg]]

Comment on attachment 592213 [details] [diff] [review]
add talos_from_code.py

http://hg.mozilla.org/mozilla-central/rev/8a59519e137e

Comment 12

[Armen [:armenzg]]

Attachment: change "Exception as e" to "Exception, e"

Reviewed on IRC. Landed on m-c as DONTBUILD and NPOTB:
http://hg.mozilla.org/mozilla-central/rev/b6221faacd5f

Comment 13

[Armen [:armenzg]]

Attachment: [m-c/talos_from_code.py] pageloader.xpi support

I can go back to the "downloadables" model if you prefer to.

Comment 14

[Armen [:armenzg]]

Attachment: [mozilla-tests/config.py] no-op change for merge days

Change the code to allow being taken into consideration on merge days.

Comment 15

[Armen [:armenzg]]

Comment on attachment 593489 [details] [diff] [review]
[m-c/talos_from_code.py] pageloader.xpi support

I can go back to the "downloadables" model if you prefer to.

Comment 16

[Armen [:armenzg]]

Comment on attachment 593489 [details] [diff] [review]
[m-c/talos_from_code.py] pageloader.xpi support

Attached to the wrong bug.

Comment 17

[Armen [:armenzg]]

Comment on attachment 592202 [details] [diff] [review]
download talos_from_code from source repo

Landed on "default":
http://hg.mozilla.org/build/buildbotcustom/rev/a38537fa29d3

talos_from_code.py landed on m-c last week and it should be merged in today to mozilla-aurora.
I expect the project_branches to have already pulled from mozilla-central by now.

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 593490 [details] [diff] [review]
[mozilla-tests/config.py] no-op change for merge days

Review of attachment 593490 [details] [diff] [review]:
-----------------------------------------------------------------

love the comment block!

Comment 19

[Armen [:armenzg]]

Comment on attachment 593490 [details] [diff] [review]
[mozilla-tests/config.py] no-op change for merge days

buildbot-configs: a0c69b4685c2

Comment 20

[Armen [:armenzg]]

attachment 592202 [details] [diff] [review] has to land on production.
attachment 592205 [details] [diff] [review] has to land once the previous patch lands and sticks.

Comment 21

[Armen [:armenzg]]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #20)
> attachment 592202 [details] [diff] [review] has to land on production.

This is live in production since 8:20 AM PDT.
We're now using talos_from_code.py from in-source repo.

Comment 22

[Armen [:armenzg]]

Comment on attachment 592202 [details] [diff] [review]
download talos_from_code from source repo

I had to back this out since not all branches have the new talos.json format and would cause them to burn.
http://hg.mozilla.org/build/buildbotcustom/rev/9f654f282cef

I landed on tools a fix to support both talos.json formats for now:
http://hg.mozilla.org/build/tools/rev/d4eb54621cce

We will switch over in a week or two when every branch has pulled from mozilla-central.

Comment 23

[Armen [:armenzg]]

Comment on attachment 592202 [details] [diff] [review]
download talos_from_code from source repo

http://hg.mozilla.org/build/buildbotcustom/rev/dbbcb17309a7

Comment 24

[Armen [:armenzg]]

This has been live for the last 22 hours.

http://hg.mozilla.org/build/buildbotcustom/rev/51316d5bdbb8

Comment 25

[Armen [:armenzg]]

Comment on attachment 592205 [details] [diff] [review]
remove talos_from_code.py from the tools repo

http://hg.mozilla.org/build/tools/rev/bfc1615253df