986209 - Make it possible to run gaia try jobs *without* doing a build

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

The gaia team would like to be able to trigger gaia try jobs against "the latest B2G desktop builds available", in order to reduce turnaround time.

In this mode, gaia try would behave similar to how Travis does...it just finds the most recent B2G desktop build that's available on ftp.mozilla.org, and runs tests against it.

We could add logic to the mozharness scripts that handle these test jobs to pull down the most recent build from m-c to run against, but we'd need a way to schedule the test jobs in the absence of a build job.  In buildbot lingo, I think this means we'd need to trigger a sendchange either with a null build url, or with the 'latest' build url.

Potentially, this behavior could be gated on something in the commit message, similar to how try syntax is handled.

:catlee, can you estimate how hard this might be?

Comment 1

[Jonathan Griffin (:jgriffin)]

Talked with aki about this in the b2g-cross-functional meeting.  It seemed like a viable approach might be to:

- make a 'dummy' no-op build that just serves to kick off tests
- add a 'latest' link for B2G desktop builds that test jobs could use to locate the latest build

According to gwagner, this is medium-high priority, since it's the last blocker to moving off of Travis. I've volunteered myself and ahal to help if there's anything we can do.

Comment 2

[Aki Sasaki (not active)]

Attachment: (needs testing) post_upload

Comment 3

[Aki Sasaki (not active)]

Comment on attachment 8399743 [details] [diff] [review]
(needs testing) post_upload

This patch works, but created a mozilla-central-linux32_gecko/1396323260/latest softlink that points to mozilla-central-linux32_gecko/1396323260/en-US, rather than a mozilla-central-linux32_gecko/latest softlink that points to mozilla-central-linux32_gecko/1396323260 .

Comment 4

[Aki Sasaki (not active)]

Attachment: post_upload2

This patch:

* adds a --release-to-latest-tinderbox-builds option, although I debated making this set behavior
* adds an avoid_race_condition option to rel_symlink()
* removes shadow central stuff

Comment 5

[Aki Sasaki (not active)]

Attachment: post_upload_custom

Call post_upload.py with --release-to-latest-tinderbox-builds.
This turns it on for all MBF-based builds (including b2g desktop)... not sure how much we should worry about earlier builds finishing later.

Comment 6

[Chris AtLee [:catlee]]

Isn't this use case exactly what the taskcluster work is aiming to do?

Comment 7

[Andrew Halberstadt [:ahal]]

Taskcluster is still a month or two out technically (and depends on sheriffs using treeherder which may be even longer). James Lal said if doing this wasn't too difficult it would be good to land it in the meantime, otherwise if it was a big complicated change, we could just wait for taskcluster.

Comment 8

[Jonathan Griffin (:jgriffin)]

Using TaskCluster to do this instead has a lot of dependencies...all the B2G-related tests (including emulator ones) have to be stood up in TaskCluster, and we have to run them side-by-side for a while to verify we're getting consistent results and a similar number of intermittents.

TaskCluster currently only runs the tests we don't run in TBPL...namely, gaia-build tests and linter tests.

Treeherder has to be deployed, and the sheriffs need to go through some iterations on it to iron out the rough edges, and then sheriffing has to be transitioned to it.

As ahal says, this is likely to be a couple of months of work, realistically, and it's hoped that we can use gaia try in the interim, if that's less work.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8400255 [details] [diff] [review]
post_upload2

>diff --git a/stage/post_upload.py b/stage/post_upload.py
>+def rel_symlink(_to, _from, avoid_race_condition=False):
...
>+        if os.path.exists(_from):
>+            if os.path.isdir(_from):
>+                shutil.rmtree(_from)
>+            else:
>+                os.unlink(_from)

There's theoretically an issue if two builds try to update the latest symlink at the same time, eg unlink() fails because a very slightly earlier job has just done that. We'd should only hit it if I/O operations are slow though. Locking or a retry could be used to avoid that.

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 8400255 [details] [diff] [review]
post_upload2

Pretty sure this is no-op until the buildbotcustom piece is live.

http://hg.mozilla.org/build/tools/rev/c616fa4b85e0

Checked into productdelivery:
Sending        files/bin/post_upload.py
Transmitting file data .
Committed revision 85578.

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 8400256 [details] [diff] [review]
post_upload_custom

https://hg.mozilla.org/build/buildbotcustom/rev/115390977e9f

Comment 12

[Aki Sasaki (not active)]

This was delayed a bit while we debated whether to try getting this in Task Cluster...

Next steps:
* Figure out how best to trigger jobs from the commit hook.  Leaning towards a sendchange but self-serve may also be a solution.
** jhford is blocked on me for this.
* We need a way to see these results.  https://bugzilla.mozilla.org/show_bug.cgi?id=914632#c2 says we need an hg repo, which might be overkill since we won't be creating builds here, but would possibly lead to a gaia-inbound (though I don't know if the patches slated for gaia-try are going to be riskier or more experimental than what would belong in gaia-inbound).
** hg repo
** configs for that repo in buildbot-configs/mozilla and mozilla-tests
** tbpl support for that repo

I think that's it, but it's certainly possible I'm missing something.
Also juggling Flame builds as I haven't found another person to own one of these bugs yet.

Comment 13

[Armen [:armenzg]]

Live in production.

Comment 14

[[e-mail] (:bhearsum)]

I turned the post_upload.py into a no-op because of bustage:
09:55 < nagios-releng> Fri 06:55:50 PDT [4446] buildbot-master111.srv.releng.scl3.mozilla.com:Command Queue is CRITICAL: 5 dead items (http://m.mozilla.org/Command+Queue)
09:55 < nagios-releng> Fri 06:55:50 PDT [4447] buildbot-master112.srv.releng.scl3.mozilla.com:Command Queue is CRITICAL: 5 dead items (http://m.mozilla.org/Command+Queue)
09:55 < nagios-releng> Fri 06:55:50 PDT [4448] buildbot-master51.srv.releng.use1.mozilla.com:Command Queue is CRITICAL: 1 dead item (http://m.mozilla.org/Command+Queue)
09:55 < nagios-releng> Fri 06:55:59 PDT [4449] buildbot-master53.srv.releng.usw2.mozilla.com:Command Queue is CRITICAL: 2 dead items (http://m.mozilla.org/Command+Queue)
09:58 < catlee> hm
09:59 -!- bholley [[e-mail]] has joined #buildduty
09:59 < catlee> OSError: [Errno 40] Too many levels of symbolic links: '/home/ftp/pub/firefox/tinderbox-builds/mozilla-inbound-linux/1396616987/tmpurkgMy'
09:59 < catlee> is that from aki's stuff?


https://hg.mozilla.org/build/tools/rev/f64669f52207

Comment 15

[Chris AtLee [:catlee]]

We were getting symlinks that were pointing at themselves. e.g.
lrwxrwxrwx 1 ffxbld firefox 10 Apr  4 07:29 ./b2g-inbound-linux/1396613985 -> 1396613985

Comment 16

[Aki Sasaki (not active)]

Ugh, sorry.  I did test.  I'm not sure how that happened, but I'll try to figure it out.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Stale latest links have been removed in firefox, mobile, and b2g tinderbox-builds/* dirs.

Comment 18

[Aki Sasaki (not active)]

Attachment: best effort softlink

I tested with:

* existing softlink (new softlink works)
* existing file (softlink works)
* existing non-writable file (softlink works if directory is writable)
* existing directory (softlink is created inside the directory, which is wrong, but probably harmless)
* existing non-writable directory (softlink fails with an STDERR message)

I'm thinking a best effort softlink is our best option here.  Thoughts?

Comment 19

[Aki Sasaki (not active)]

Attachment: gaia-try-tbpl

Comment 20

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8403041 [details] [diff] [review]
gaia-try-tbpl

Review of attachment 8403041 [details] [diff] [review]:
-----------------------------------------------------------------

Let's put it under Try for consistency.

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 8402993 [details] [diff] [review]
best effort softlink

Leaning towards latest nightlies, which already have a latest dir.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

If we go back to latest for tinderbox-builds, then we should take into account builds finishing out of sequence compared to pushes. I think we'd expect latest to mean latest code, instead of latest build.

Comment 23

[Aki Sasaki (not active)]

That would be different than our latest assumption for both nightlies and releases, where it means latest build, not necessarily latest code.

Comment 24

[Jonathan Griffin (:jgriffin)]

Since these try jobs will be running against somewhat arbitrary gecko, I'm not sure the difference matters here.  Latest build is probably fine.

Comment 25

[Aki Sasaki (not active)]

Attachment: gaia-try-configs wip

Comment 26

[Aki Sasaki (not active)]

Attachment: gaia-try-configs

I'm triggering tests!
Now working on getting them to not be perma-red wastes of infrastructure time...

Comment 27

[Aki Sasaki (not active)]

Attachment: gaia-try-custom

Comment 28

[Aki Sasaki (not active)]

Attachment: gaia-try-mh

This isn't completely blowing up.
Gaia integration isn't finding the mozbase requirements txt.  I hope we're looking at the test zip, and not looking at the actual tree, because gaia-try isn't going to have an actual tree.

Comment 29

[Jonathan Griffin (:jgriffin)]

Gaia-try uses test.zip; it doesn't look for an actual tree.

Comment 30

[Aki Sasaki (not active)]

Attachment: gaia-try-mh

This patch:

* adds a config file for gaia-try
* allows for specifying a different location for gaia.json (optional; we could also have the file exist in a b2g/config/ dir in gaia-try, but I figured having the file at the root level would be clearer to the casual observer)
* fixes the |git remote -v| directory
* allows for setting installer_url and test_url via commandline instead of requiring they live in buildprops.json.

I tested an earlier version of this patch on ash and it didn't look like it was causing issues.  I'll still want a good cycle on Cypress on default after landing, though.

I got further with gaia-integration by allowing for setting self.test_url even though gaia-integration has require_test_zip=False for some reason [?].  Now it gets past the mozbase requirements file and is dying on npm commands on a build slave, which is probably a good sign.

Comment 31

[Aki Sasaki (not active)]

Comment on attachment 8403041 [details] [diff] [review]
gaia-try-tbpl

https://hg.mozilla.org/webtools/tbpl/rev/d999ce1616ba

Comment 32

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8408451 [details] [diff] [review]
gaia-try-mh

Review of attachment 8408451 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm!

Comment 33

[Aki Sasaki (not active)]

Comment on attachment 8408451 [details] [diff] [review]
gaia-try-mh

https://hg.mozilla.org/build/mozharness/rev/68dc075549a2

Comment 34

[Aki Sasaki (not active)]

Comment on attachment 8400256 [details] [diff] [review]
post_upload_custom

https://hg.mozilla.org/build/buildbotcustom/rev/643fb06fdae4

Comment 35

[Jordan Lund (:jlund)]

mozharness patch is in production: http://hg.mozilla.org/build/mozharness/rev/c9615cf1645a :)

Comment 36

[Aki Sasaki (not active)]

Comment on attachment 8408002 [details] [diff] [review]
gaia-try-configs

https://hg.mozilla.org/build/buildbot-configs/rev/05b13219f810

Comment 37

[Aki Sasaki (not active)]

Comment on attachment 8408003 [details] [diff] [review]
gaia-try-custom

https://hg.mozilla.org/build/buildbotcustom/rev/be9e4dfbc276

Comment 38

[Ed Morley [:emorley]]

TBPL part in production :-)

Comment 39

[Jordan Lund (:jlund)]

buildbot-config patch is in production: http://hg.mozilla.org/build/buildbot-configs/rev/46353eebf660
buildbotcustom patch is in production: http://hg.mozilla.org/build/buildbotcustom/rev/22ee9ed82c3b

:)

Comment 40

[Aki Sasaki (not active)]

Attachment: fix_gaia-try_uploads

Comment 41

[Aki Sasaki (not active)]

I see a green test run!
http://buildbot-master04.srv.releng.usw2.mozilla.com:8201/builders/b2g_ubuntu32_vm%20gaia-try%20opt%20test%20reftest/builds/0
Hopefully this latest patch will fix the log upload so it appears on tbpl properly.

Comment 42

[Aki Sasaki (not active)]

Attachment: self-serve

And I think this adds it to self-serve?

Comment 43

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8414923 [details] [diff] [review]
fix_gaia-try_uploads

This patch seems more related to our traditional way of uploading builds, or pulling packages, which may not apply to gaia-try.

For postrun failures, I think you need to duplicate the block starting at
 http://hg.mozilla.org/build/puppet/file/5f842bcfcacd/modules/buildmaster/templates/postrun-default.cfg.erb#l20
for gaia-try. I tried that on bm106 and got this was uploaded
 http://ftp.mozilla.org/pub/mozilla.org/b2g/try-builds/[e-mail]-3d288cf7101e/gaia-try-macosx64_gecko/gaia-try_mountainlion-b2gdt_test-gaia-ui-test-bm106-tests1-macosx-build0.txt.gz

Comment 44

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [puppet] Add gaia-try ssh config

Comment 45

[Aki Sasaki (not active)]

Comment on attachment 8415412 [details] [diff] [review]
[puppet] Add gaia-try ssh config

We could do that, or maybe

(bb08)deathduck:/src/gaia-try/buildbotcustom/bin [10:28:22] (default)
711$ hg diff
diff --git a/bin/postrun.py b/bin/postrun.py
--- a/bin/postrun.py
+++ b/bin/postrun.py
@@ -63,17 +63,17 @@ class PostRunner(object):
                        self.config['statusdb.master_name']]
         if "nightly" in builder.name:
             upload_args.append("--nightly")
         if builder.name.startswith("release-"):
             upload_args.append("--release")
             upload_args.append(
                 "%s/%s" % (info.get('version'), info.get('build_number')))

-        if branch and 'try' in branch:
+        if branch and branch.startswith('try'):
             upload_args.append("--try")
         elif branch == 'shadow-central':
             upload_args.append("--shadow")

         if 'l10n' in builder.name:
             upload_args.append("--l10n")

         if product:

?

Comment 46

[Aki Sasaki (not active)]

Comment on attachment 8414940 [details] [diff] [review]
self-serve

jlund irc review

Comment 47

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8415412 [details] [diff] [review]
[puppet] Add gaia-try ssh config

Landed as is, 
  https://hg.mozilla.org/build/puppet/rev/cd7fdf51bc1b

because we're already getting --try but also the wrong keys, eg

/builds/buildbot/tests1-macosx/bin/python /builds/buildbot/tests1-macosx/lib/python2.7/site-packages/buildbotcustom/bin/log_uploader.py -r 2 -t 10 --master-name bm106-tests1-macosx --try --product b2g --platform macosx64_gecko --branch gaia-try --user ffxbld -i /home/cltbld/.ssh/ffxbld_dsa stage.mozilla.org /builds/buildbot/tests1-macosx/master/gaia-try_mountainlion-b2gdt_test-gaia-ui-test 1

and that is teh fail.

Comment 48

[Aki Sasaki (not active)]

Comment on attachment 8414940 [details] [diff] [review]
self-serve

http://hg.mozilla.org/build/tools/rev/58165b25abd4

Comment 49

[Aki Sasaki (not active)]

https://tbpl.mozilla.org/?tree=Gaia-Try is now showing logs/jobs properly.
Self serve has is_done status: https://secure.pub.build.mozilla.org/buildapi/self-serve/gaia-try/rev/697bc885c5f5139b213e4b72be05e8fe7de0ea0c/is_done

Thanks Nick and Catlee!

I think we're done here.

Comment 50

[Aki Sasaki (not active)]

We don't upload crashreporter symbols to latest-nightly/ so debug tests (and any crashes in opt tests most likely) are burning.

I'm probably going to force Nick to review my post_upload.py patch.

Comment 51

[Aki Sasaki (not active)]

Attachment: post_upload4

Back to best-effort softlink!

Comment 52

[Aki Sasaki (not active)]

Comment on attachment 8416016 [details] [diff] [review]
post_upload4

https://hg.mozilla.org/build/tools/rev/1fb7364360a1
(bb08)deathduck:/src/clean/svn/productdelivery/files/bin [12:21:15]
758$ svn ci
Sending        post_upload.py
Transmitting file data .
Committed revision 86794.

Comment 53

[Aki Sasaki (not active)]

Comment on attachment 8400256 [details] [diff] [review]
post_upload_custom

http://hg.mozilla.org/build/buildbotcustom/rev/a1c93d45703b

Comment 54

[Aki Sasaki (not active)]

Attachment: use_tinderbox-builds-latest

I'm going to give this a whirl in staging.

Comment 55

[Aki Sasaki (not active)]

Comment on attachment 8416071 [details] [diff] [review]
use_tinderbox-builds-latest

Crap, this doesn't account for debug.

Comment 56

[Aki Sasaki (not active)]

Attachment: (needs testing) buildbot-configs opt_extra_args, debug_extra_args

Comment 57

[Aki Sasaki (not active)]

Attachment: (needs testing) buildbotcustom allow for opt_extra_args, debug_extra_args

Comment 58

[Aki Sasaki (not active)]

Attachment: (configs) opt_extra_args, debug_extra_args

This is going to be fragile around merge day; ideally I fix bug 979554 before next merge day and we upload the gecko/crashreporter/tests files without version numbers.

I tested as far as downloading/extracting the installer, test zip, and crashreporter symbols, which works.

Comment 59

[Aki Sasaki (not active)]

Attachment: (custom) deal with opt_extra_args / debug_extra_args

Comment 60

[Aki Sasaki (not active)]

Attachment: (configs) opt_extra_args, debug_extra_args

Fix the linux32 links

Comment 61

[Aki Sasaki (not active)]

Attachment: wah

Comment 62

[Aki Sasaki (not active)]

Comment on attachment 8416648 [details] [diff] [review]
wah

https://hg.mozilla.org/build/buildbot-configs/rev/bf8352926462

Comment 63

[Aki Sasaki (not active)]

Comment on attachment 8416619 [details] [diff] [review]
(custom) deal with opt_extra_args / debug_extra_args

https://hg.mozilla.org/build/buildbotcustom/rev/0122c7db5121

Comment 64

[Aki Sasaki (not active)]

Attachment: no_coallescing

afaict, coallesce_jobs does nothing.
enable_merging specifies whether coallescing happens.

Comment 65

[Aki Sasaki (not active)]

Comment on attachment 8417563 [details] [diff] [review]
no_coallescing

https://hg.mozilla.org/build/buildbot-configs/rev/202f1d9100d0

Comment 66

[Chris Cooper [:coop] (he/him)]

Merged and deployed to production.

Comment 67

[Aki Sasaki (not active)]

I don't see coalescing anymore.
The symbol downloads should work ondemand for opt tests; the debug tests were turned off.  Resolving fixed again.