1120990 - Prototype retrigger-based bisection tool

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Armen [:armenzg]]

Prototype of a command-line tool that can be used by sheriffs and others to automate retrigger-based bisection. This could be used to help bisect new intermittent oranges, and to backfill jobs that have been skipped due to coalescing. Integration with Treeherder or other service will be done later.

The proposed named at the moment is TriggerCI.

Comment 1

[Armen [:armenzg]]

I will be using the bisection tool to work on the prototype now that I have most pieces in place:
https://github.com/armenzg/mozilla_ci_tools/tree/bisection

Comment 2

[Armen [:armenzg]]

This etherpad keeps track of what scope this prototype will have:
https://etherpad.mozilla.org/bisection-prototype-tool

I can get some help with getting ideas on how to track jobs triggered.
The current approach is trigger and forget.

Comment 3

[Armen [:armenzg]]

Currently you can trigger an X number of jobs for a buildername for a range of 1 changesets (I just have not had time to test more than 1).

In the bisection branch we can run this.
We tell it to trigger 5 'Ubuntu VM 12.04 x64 try opt test mochitest-1', since there are 4 pending jobs of that same builder, it will only trigger 1 more.

NOTE: In this run I'm using --dry-run

For some reason, I believe that "cedar" is being excluded from being added to the buildjson data (bug 1131742).
Card added to Trello for a hackery to work around it.
For now, I will use "ash" instead.

armenzg@armenzg-thinkpad:~/repos/ci_tools$ python scripts/mozbisect.py -b "Ubuntu VM 12.04 x64 try opt test mochitest-1" --start-rev 98e9b20f9d4d --end-rev 98e9b20f9d4d --repo-name try --num 5 --debug --dry-run
02/10/2015 03:40:06 INFO:	 Setting DEBUG level
02/10/2015 03:40:06 DEBUG:	 Determine repository associated to try
02/10/2015 03:40:06 DEBUG:	 Loading /home/armenzg/repos/ci_tools/repositories.txt
02/10/2015 03:40:06 DEBUG:	 About to fetch https://hg.mozilla.org/try/json-pushes?fromchange=98e9b20f9d4d&tochange=98e9b20f9d4d&version=2
02/10/2015 03:40:06 INFO:	 Starting new HTTPS connection (1): hg.mozilla.org
02/10/2015 03:40:07 DEBUG:	 "GET /try/json-pushes?fromchange=98e9b20f9d4d&tochange=98e9b20f9d4d&version=2 HTTP/1.1" 200 55
02/10/2015 03:40:07 DEBUG:	 We want to have 5 jobs of Ubuntu VM 12.04 x64 try opt test mochitest-1 on revision 98e9b20f9d4d
02/10/2015 03:40:07 DEBUG:	 About to fetch https://secure.pub.build.mozilla.org/buildapi/self-serve/try/rev/98e9b20f9d4d?format=json
02/10/2015 03:40:07 INFO:	 Starting new HTTPS connection (1): secure.pub.build.mozilla.org
02/10/2015 03:40:08 DEBUG:	 "GET /buildapi/self-serve/try/rev/98e9b20f9d4d?format=json HTTP/1.1" 200 652
02/10/2015 03:40:08 DEBUG:	 Find jobs matching 'Ubuntu VM 12.04 x64 try opt test mochitest-1'
02/10/2015 03:40:08 DEBUG:	 We have matched 4 jobs.
02/10/2015 03:40:08 DEBUG:	 We found 4 pending jobs, 0 running jobs and 0 successful_jobs.
02/10/2015 03:40:08 DEBUG:	 We have found 4 jobs matching 'Ubuntu VM 12.04 x64 try opt test mochitest-1' on 98e9b20f9d4d. We need to trigger more.
02/10/2015 03:40:08 DEBUG:	 We want to trigger 'Ubuntu VM 12.04 x64 try opt test mochitest-1' on revision '98e9b20f9d4d' a total of 1 times.
02/10/2015 03:40:08 DEBUG:	 Buildername Ubuntu VM 12.04 x64 try opt test mochitest-1 is valid.
02/10/2015 03:40:08 DEBUG:	 The build job that triggers Ubuntu VM 12.04 x64 try opt test mochitest-1 is Linux x86-64 try build
02/10/2015 03:40:08 DEBUG:	 Buildername Linux x86-64 try build is valid.
02/10/2015 03:40:08 DEBUG:	 About to fetch https://secure.pub.build.mozilla.org/buildapi/self-serve/try/rev/98e9b20f9d4d?format=json
02/10/2015 03:40:08 INFO:	 Starting new HTTPS connection (1): secure.pub.build.mozilla.org
02/10/2015 03:40:09 DEBUG:	 "GET /buildapi/self-serve/try/rev/98e9b20f9d4d?format=json HTTP/1.1" 200 652
02/10/2015 03:40:09 DEBUG:	 Find jobs matching 'Linux x86-64 try build'
02/10/2015 03:40:09 DEBUG:	 We have matched 1 jobs.
02/10/2015 03:40:09 DEBUG:	 List of matching jobs:
02/10/2015 03:40:09 DEBUG:	 {u'build_id': 62599025, u'status': 0, u'branch': u'try', u'buildername': u'Linux x86-64 try build', u'claimed_by_name': u'buildbot-master75.srv.releng.use1.mozilla.com:/builds/buildbot/try1/master', u'buildnumber': 6530, u'starttime': 1423510870, u'requests': [{u'complete_at': 1423512711, u'complete': 1, u'buildername': u'Linux x86-64 try build', u'claimed_at': 1423512332, u'priority': 0, u'submittime': 1423510827, u'reason': u'scheduler', u'branch': u'try', u'request_id': 61477630, u'revision': u'98e9b20f9d4d5e8b7fee9464324484d2adc78863'}], u'endtime': 1423512711, u'revision': u'98e9b20f9d4d5e8b7fee9464324484d2adc78863'}
02/10/2015 03:40:09 DEBUG:	 We found a successful job. We don't search anymore.
02/10/2015 03:40:09 DEBUG:	 There is a job that has completed successfully.
02/10/2015 03:40:09 DEBUG:	 {u'build_id': 62599025, u'status': 0, u'branch': u'try', u'buildername': u'Linux x86-64 try build', u'claimed_by_name': u'buildbot-master75.srv.releng.use1.mozilla.com:/builds/buildbot/try1/master', u'buildnumber': 6530, u'starttime': 1423510870, u'requests': [{u'complete_at': 1423512711, u'complete': 1, u'buildername': u'Linux x86-64 try build', u'claimed_at': 1423512332, u'priority': 0, u'submittime': 1423510827, u'reason': u'scheduler', u'branch': u'try', u'request_id': 61477630, u'revision': u'98e9b20f9d4d5e8b7fee9464324484d2adc78863'}], u'endtime': 1423512711, u'revision': u'98e9b20f9d4d5e8b7fee9464324484d2adc78863'}
02/10/2015 03:40:09 DEBUG:	 Job identified with claimed_at value: 1423512332 run on 2015-02-09
02/10/2015 03:40:12 DEBUG:	 We are going to look for 61477630 through the jobs run on 2015-02-09.
02/10/2015 03:40:12 DEBUG:	 Found {u'slave_id': 7346, u'master_id': 136, u'builder_id': 245542, u'request_ids': [61477630], u'id': 58610396, u'buildnumber': 6530, u'reason': u'scheduler', u'requesttime': 1423510827, u'result': 0, u'starttime': 1423510869, u'endtime': 1423512710, u'properties': {u'basedir': u'/builds/slave/try-l64-0000000000000000000000', u'toolsdir': u'/builds/slave/try-l64-0000000000000000000000/scripts', u'appName': u'Firefox', u'got_revision': u'98e9b20f9d4d', u'jsshellUrl': u'http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/jsshell-linux-x86_64.zip', u'buildnumber': 6530, u'slavename': u'try-linux64-spot-074', u'buildername': u'Linux x86-64 try build', u'appVersion': u'38.0a1', u'comments': u'try: build nothing', u'script_repo_checkout': u'/builds/slave/try-l64-0000000000000000000000/scripts', u'platform': u'linux64', u'master': u'http://buildbot-master75.srv.releng.use1.mozilla.com:8101/', u'testresults': u"[('num_ctors', 'num_ctors', 87, '87')]", u'branch': u'try', u'stage_platform': u'linux64', u'revision': u'98e9b20f9d4d5e8b7fee9464324484d2adc78863', u'product': u'firefox', u'request_times': {u'61477630': 1423510827}, u'repository': u'', u'log_url': u'http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/try-linux64-bm75-try1-build6530.txt.gz', u'buildid': u'20150209114122', u'request_ids': [61477630], u'scheduler': u'try-firefox', u'sourcestamp': u'98e9b20f9d4d', u'num_ctors': u'87', u'commit_titles': [u'try: build nothing'], u'symbolsUrl': u'http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.crashreporter-symbols.zip', u'packageUrl': u'http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.tar.bz2', u'script_repo_revision': u'2f5ae1f6ccaa', u'script_repo_url': u'https://hg.mozilla.org/users/armenzg_mozilla.com/mozharness', u'project': u'', u'testsUrl': u'http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.tests.zip', u'uploadFiles': u"[u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/firefox-38.0a1.en-US.linux-x86_64.tar.bz2', u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/linux-x86_64/xpi/firefox-38.0a1.en-US.langpack.xpi', u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/firefox-38.0a1.en-US.linux-x86_64.tests.zip', u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/firefox-38.0a1.en-US.linux-x86_64.crashreporter-symbols.zip', u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/firefox-38.0a1.en-US.linux-x86_64.txt', u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/firefox-38.0a1.en-US.linux-x86_64.json', u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/firefox-38.0a1.en-US.linux-x86_64.mozinfo.json', u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/jsshell-linux-x86_64.zip', u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/sccache.log.gz', u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/host/bin/mar', u'/builds/slave/try-l64-0000000000000000000000/build/src/obj-firefox/dist/host/bin/mbsdiff']", u'builduid': u'c80bfb9367814e4f963dceb49b8998b1', u'repo_path': u'try'}}
02/10/2015 03:40:13 DEBUG:	 We want to find the files needed to trigger Linux x86-64 try build
02/10/2015 03:40:13 DEBUG:	 We are going to test if we can reach http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.tar.bz2
02/10/2015 03:40:13 INFO:	 Starting new HTTP connection (1): ftp.mozilla.org
02/10/2015 03:40:13 DEBUG:	 "HEAD /pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.tar.bz2 HTTP/1.1" 200 0
02/10/2015 03:40:13 DEBUG:	 We are going to test if we can reach http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.tests.zip
02/10/2015 03:40:13 INFO:	 Starting new HTTP connection (1): ftp.mozilla.org
02/10/2015 03:40:13 DEBUG:	 "HEAD /pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.tests.zip HTTP/1.1" 200 0
02/10/2015 03:40:13 INFO:	 We were going to post to this url: https://secure.pub.build.mozilla.org/buildapi/self-serve/try/builders/Ubuntu VM 12.04 x64 try opt test mochitest-1/98e9b20f9d4d
02/10/2015 03:40:13 INFO:	 With this payload: {'files': '["http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.tar.bz2", "http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.tests.zip"]', 'properties': '{"branch": "try", "revision": "98e9b20f9d4d"}'}
02/10/2015 03:40:13 INFO:	 With these files: [u'http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.tar.bz2', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-98e9b20f9d4d/try-linux64/firefox-38.0a1.en-US.linux-x86_64.tests.zip']

Comment 4

[Armen [:armenzg]]

We are happy with the current prototype:
https://mozilla-ci-tools.readthedocs.org/en/latest/scripts.html#trigger-py