Last Comment Bug 989131 - Write Gaia commit-hook that can trigger Gaia tryserver runs
: Write Gaia commit-hook that can trigger Gaia tryserver runs
Status: RESOLVED FIXED
:
Product: Firefox OS
Classification: Client Software
Component: General (show other bugs)
: unspecified
: All All
-- normal (vote)
: ---
Assigned To: John Ford [:jhford] CET/CEST Berlin Time
:
:
Mentors:
Depends on: 993774 1007437 1009695
Blocks: 914632
  Show dependency treegraph
 
Reported: 2014-03-27 16:23 PDT by Jonathan Griffin (:jgriffin)
Modified: 2014-05-13 13:08 PDT (History)
9 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description User image Jonathan Griffin (:jgriffin) 2014-03-27 16:23:13 PDT
We want to be able to trigger Gaia tryserver runs based on Gaia pull requests, so that we can stop using Travis.

To do this, we'll need to have a github commit hook that can fire off a gaia tryserver run, and then report status back.

Aki or catlee can probably estimate how difficult this is on the buildbot side.

Test jobs will need to be modified to pull in pull request refs.  Test jobs currently pull from our hg mirror of gaia.  Hal, does our hg mirror contain pull request refs?  If not, we may need to modify our test jobs to use the git.mozilla.org mirror instead.
Comment 1 User image Dylan Oliver [:doliver] 2014-03-27 16:39:39 PDT
Adding jhford to work on the hook.
Comment 2 User image John Ford [:jhford] CET/CEST Berlin Time 2014-03-28 10:49:11 PDT
Github doesn't have traditional github commit hook support, but it does send out notifications to repositories on each push event.  Writing a receiver for that is trivial.  I'm not familiar with how the gaia try runs will be triggered, so some information on that would be great.

Reporting status back to the PR is slightly more challenging.  There are APIs for pull request status, but iirc, those are undocumented APIs.  Travis is open source, so we should be able to figure out how they use them from looking there.

Last time I looked at TBPL, we still didn't have a solid way to answer "Are all runs done for this job".  Has that changed?

For status reporting, we can only use the single pass/fail thing if we know that *all* jobs are done.

I'll start working on this on Monday if there are no objections.
Comment 3 User image Hal Wine [:hwine] (use NI) 2014-03-28 11:04:00 PDT
(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #2)
> Last time I looked at TBPL, we still didn't have a solid way to answer "Are
> all runs done for this job".  Has that changed?

TBPL doesn't, but buildapi does -- see the "is_done" request in https://wiki.mozilla.org/ReleaseEngineering/Applications/BuildAPI#Job_Requests
Comment 4 User image Gregor Wagner [:gwagner] 2014-04-01 15:58:26 PDT
(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #2)
> Github doesn't have traditional github commit hook support, but it does send
> out notifications to repositories on each push event.  Writing a receiver
> for that is trivial.  I'm not familiar with how the gaia try runs will be
> triggered, so some information on that would be great.
> 
> Reporting status back to the PR is slightly more challenging.  There are
> APIs for pull request status, but iirc, those are undocumented APIs.  Travis
> is open source, so we should be able to figure out how they use them from
> looking there.
> 

Lets start with a simple approach here. For a first version is should be enough to have a link to the try run in the PR. Reporting the actual test results back to github can be added later and we shouldn't block on this.
Comment 5 User image John Ford [:jhford] CET/CEST Berlin Time 2014-04-01 17:33:39 PDT
(In reply to Gregor Wagner [:gwagner] from comment #4) 
> Lets start with a simple approach here. For a first version is should be
> enough to have a link to the try run in the PR. Reporting the actual test
> results back to github can be added later and we shouldn't block on this.

Yes.  Incremental is how I'll do this.  I have the push event receiver code running on my personal server right now, but it mostly works.  Once I have the triggering working, I'll work on the reporting.

(In reply to Hal Wine [:hwine] (use needinfo) from comment #3)
> TBPL doesn't, but buildapi does -- see the "is_done" request in
> https://wiki.mozilla.org/ReleaseEngineering/Applications/
> BuildAPI#Job_Requests

huzzah, thanks!

Aki, can you provide details of what's needed and how to trigger a gaia try run?
Comment 6 User image Aki Sasaki [:aki] 2014-04-04 10:25:09 PDT
Sorry, I don't have this clear yet.  On my list.
Comment 7 User image Aki Sasaki [:aki] 2014-04-04 13:35:07 PDT
John: where is the hook going to run from?  What's it capable of?

I'm currently guessing it's running on github or external to our build network, and it can generate a POST request.  I think the information needed will be the pre-built gecko package url + test url, the gaia repo url + gaia revision.  I think if you're external we'll need to use buildapi in some fashion.
Comment 8 User image Aki Sasaki [:aki] 2014-04-04 14:35:13 PDT
Alternately an hg commit to a gecko tree; would your hook be capable of that?
Comment 9 User image Aki Sasaki [:aki] 2014-04-07 16:44:04 PDT
(In reply to Aki Sasaki [:aki] from comment #8)
> Alternately an hg commit to a gecko tree; would your hook be capable of that?

Looks like this is the direction we'll go.
We'll be creating a json file in http://hg.mozilla.org/integration/gaia-try to trigger the tests.
I don't yet know the contents of the json file, but I imagine that'll be the user (email?), gaia repo url, gaia repo revision would be in here.
Comment 10 User image John Ford [:jhford] CET/CEST Berlin Time 2014-04-08 14:50:09 PDT
(In reply to Aki Sasaki [:aki] from comment #9)
> Looks like this is the direction we'll go.
> We'll be creating a json file in http://hg.mozilla.org/integration/gaia-try
> to trigger the tests.
> I don't yet know the contents of the json file, but I imagine that'll be the
> user (email?), gaia repo url, gaia repo revision would be in here.

Is that repo going to be a full blown gecko tree?  I thought that the gecko that'd be used for gaia-try is just the latest B2G desktop build.

If it needs to be a full blown gecko tree, that slightly bumps the time it'll take to submit each job, but is certainly possible.  Adding that information should.

I will only have information from Github, so unless we have a service that maps Github credentials to LDAP, I'll just forge the author using their Github information.
Comment 11 User image Aki Sasaki [:aki] 2014-04-08 14:58:41 PDT
No, it'll be an empty repo except that json file.
We'll also need to get a latest url from somewhere.  Maybe we can put that in the json, and the commit hook process can bump the repo/revision info but leave the installer url alone.
Comment 12 User image John Ford [:jhford] CET/CEST Berlin Time 2014-04-08 17:03:43 PDT
(In reply to Aki Sasaki [:aki] from comment #11)
> No, it'll be an empty repo except that json file.
> We'll also need to get a latest url from somewhere.  Maybe we can put that
> in the json, and the commit hook process can bump the repo/revision info but
> leave the installer url alone.

phew.  this sounds like a workable solution to me.  Please let me know when it's ready for testing.
Comment 13 User image Aki Sasaki [:aki] 2014-04-17 11:56:22 PDT
John:
If you're waiting for what the file will look like, I'm pretty sure it's going to look like this: https://hg.mozilla.org/users/asasaki_mozilla.com/hg-git/file/432ac6cedc75/gaia.json

It'll be a 'gaia.json' file in https://hg.mozilla.org/integration/gaia-try

I suggest looking for that file and reading it into a dict, and falling back to {} if it doesn't, then populating the git_revision and remote, then writing to the file, committing, pushing without -f, and hg stripping/updating/re-committing/repushing in a retry loop to avoid losing push races.
http://hg.mozilla.org/build/mozharness/file/fb0bae844d4a/scripts/bump_gaia_json.py#l189 and http://hg.mozilla.org/build/mozharness/file/fb0bae844d4a/scripts/b2g_bumper.py#l245 for examples.

This won't be fully testable with triggered tests until bug 986209 is reviewed/landed/reconfiged, but hopefully this unblocks you a bit further.
Comment 14 User image John Ford [:jhford] CET/CEST Berlin Time 2014-04-17 17:59:44 PDT
Aki, thanks for the update!  I'll start landing to that repository as soon as everything is in place.

As we discussed in person just now, I would like to inform developers of when their job is done as well as where they can check status.  I think the easiest way for status is to comment on the PR when triggering the job with a link to the TBPL page for the commit and a message that the jobs were triggered.  As neither of us are sure whether the Buildapi contains information on the overall outcome of the job, you mentioned creating a downstream job that is done when all the tests are done.  If that downstream job is able to hit a www-accessible api endpoint, I can have my hook comment on the PR when hit.

The most simple thing I can think of is to do something like:

curl -Li http://www.example.com:7040/tests-finished?commit=abcd123[&outcome=pass|fail]

Does that still work for you?
Comment 15 User image Aki Sasaki [:aki] 2014-04-22 09:52:47 PDT
(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #14)
> Aki, thanks for the update!  I'll start landing to that repository as soon
> as everything is in place.
> 
> As we discussed in person just now, I would like to inform developers of
> when their job is done as well as where they can check status.  I think the
> easiest way for status is to comment on the PR when triggering the job with
> a link to the TBPL page for the commit and a message that the jobs were
> triggered.  As neither of us are sure whether the Buildapi contains
> information on the overall outcome of the job, you mentioned creating a
> downstream job that is done when all the tests are done.  If that downstream
> job is able to hit a www-accessible api endpoint, I can have my hook comment
> on the PR when hit.
> 
> The most simple thing I can think of is to do something like:
> 
> curl -Li
> http://www.example.com:7040/tests-finished?commit=abcd123[&outcome=pass|fail]
> 
> Does that still work for you?

Evidently the AggregatingScheduler solution I was thinking about is really only good for one set of builders (e.g. releases) and doesn't work well when multiple instances of each builder is running... we should most likely use buildapi's is_done.  If that has the info you need, will that be sufficient for you?
Comment 16 User image John Ford [:jhford] CET/CEST Berlin Time 2014-04-24 11:57:29 PDT
(In reply to Aki Sasaki [:aki] from comment #15)
> Evidently the AggregatingScheduler solution I was thinking about is really
> only good for one set of builders (e.g. releases) and doesn't work well when
> multiple instances of each builder is running... we should most likely use
> buildapi's is_done.  If that has the info you need, will that be sufficient
> for you?

is_done should have the info I need.  Can you confirm that the gaia_try runs will work with is_done?
Comment 17 User image Aki Sasaki [:aki] 2014-04-24 12:04:02 PDT
I think the easiest way to know is to test.
https://tbpl.mozilla.org/?tree=Gaia-Try is there now, and in theory we'll launch the tests as soon as we start populating gaia.json in http://hg.mozilla.org/integration/gaia-try .
Do you want me to create a dummy, or do you want to start pushing pull request entries there?
Comment 18 User image John Ford [:jhford] CET/CEST Berlin Time 2014-04-29 14:39:34 PDT
Aki, can you add the tree to https://treestatus.mozilla.org/gaia-try please?

remote: ************************** ERROR ****************************
remote: Error accessing https://treestatus.mozilla.org/gaia-try?format=json :
remote: HTTP Error 404: NOT FOUND
remote: Unable to check if the tree is open - treating as if CLOSED.
remote: To push regardless, include "CLOSED TREE" in your push comment.
remote: *************************************************************
Comment 19 User image Aki Sasaki [:aki] 2014-04-29 14:44:31 PDT
I don't have perms.  :hwine and sheriffs do.
Comment 20 User image Hal Wine [:hwine] (use NI) 2014-04-29 16:25:27 PDT
created - in case any correction needed, faster service in #releng where the sheriffs hang out.
Comment 21 User image Aki Sasaki [:aki] 2014-04-29 18:04:14 PDT
https://bugzilla.mozilla.org/show_bug.cgi?id=986209#c41
Comment 22 User image John Ford [:jhford] CET/CEST Berlin Time 2014-05-06 13:55:46 PDT
This is in production.  I'm going to mark this as fixed, further issues should be filed as new bugs.

Thanks everyone!

Note You need to log in before you can comment on or make changes to this bug.