Write Gaia commit-hook that can trigger Gaia tryserver runs

RESOLVED FIXED

Status

Firefox OS
General
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: jgriffin, Assigned: jhford)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
We want to be able to trigger Gaia tryserver runs based on Gaia pull requests, so that we can stop using Travis.

To do this, we'll need to have a github commit hook that can fire off a gaia tryserver run, and then report status back.

Aki or catlee can probably estimate how difficult this is on the buildbot side.

Test jobs will need to be modified to pull in pull request refs.  Test jobs currently pull from our hg mirror of gaia.  Hal, does our hg mirror contain pull request refs?  If not, we may need to modify our test jobs to use the git.mozilla.org mirror instead.
(Reporter)

Updated

3 years ago
Blocks: 914632
Adding jhford to work on the hook.
Github doesn't have traditional github commit hook support, but it does send out notifications to repositories on each push event.  Writing a receiver for that is trivial.  I'm not familiar with how the gaia try runs will be triggered, so some information on that would be great.

Reporting status back to the PR is slightly more challenging.  There are APIs for pull request status, but iirc, those are undocumented APIs.  Travis is open source, so we should be able to figure out how they use them from looking there.

Last time I looked at TBPL, we still didn't have a solid way to answer "Are all runs done for this job".  Has that changed?

For status reporting, we can only use the single pass/fail thing if we know that *all* jobs are done.

I'll start working on this on Monday if there are no objections.
Assignee: nobody → jhford
(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #2)
> Last time I looked at TBPL, we still didn't have a solid way to answer "Are
> all runs done for this job".  Has that changed?

TBPL doesn't, but buildapi does -- see the "is_done" request in https://wiki.mozilla.org/ReleaseEngineering/Applications/BuildAPI#Job_Requests
(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #2)
> Github doesn't have traditional github commit hook support, but it does send
> out notifications to repositories on each push event.  Writing a receiver
> for that is trivial.  I'm not familiar with how the gaia try runs will be
> triggered, so some information on that would be great.
> 
> Reporting status back to the PR is slightly more challenging.  There are
> APIs for pull request status, but iirc, those are undocumented APIs.  Travis
> is open source, so we should be able to figure out how they use them from
> looking there.
> 

Lets start with a simple approach here. For a first version is should be enough to have a link to the try run in the PR. Reporting the actual test results back to github can be added later and we shouldn't block on this.
(In reply to Gregor Wagner [:gwagner] from comment #4) 
> Lets start with a simple approach here. For a first version is should be
> enough to have a link to the try run in the PR. Reporting the actual test
> results back to github can be added later and we shouldn't block on this.

Yes.  Incremental is how I'll do this.  I have the push event receiver code running on my personal server right now, but it mostly works.  Once I have the triggering working, I'll work on the reporting.

(In reply to Hal Wine [:hwine] (use needinfo) from comment #3)
> TBPL doesn't, but buildapi does -- see the "is_done" request in
> https://wiki.mozilla.org/ReleaseEngineering/Applications/
> BuildAPI#Job_Requests

huzzah, thanks!

Aki, can you provide details of what's needed and how to trigger a gaia try run?
Flags: needinfo?(aki)

Comment 6

3 years ago
Sorry, I don't have this clear yet.  On my list.

Comment 7

3 years ago
John: where is the hook going to run from?  What's it capable of?

I'm currently guessing it's running on github or external to our build network, and it can generate a POST request.  I think the information needed will be the pre-built gecko package url + test url, the gaia repo url + gaia revision.  I think if you're external we'll need to use buildapi in some fashion.
Flags: needinfo?(jhford)

Comment 8

3 years ago
Alternately an hg commit to a gecko tree; would your hook be capable of that?

Comment 9

3 years ago
(In reply to Aki Sasaki [:aki] from comment #8)
> Alternately an hg commit to a gecko tree; would your hook be capable of that?

Looks like this is the direction we'll go.
We'll be creating a json file in http://hg.mozilla.org/integration/gaia-try to trigger the tests.
I don't yet know the contents of the json file, but I imagine that'll be the user (email?), gaia repo url, gaia repo revision would be in here.
(In reply to Aki Sasaki [:aki] from comment #9)
> Looks like this is the direction we'll go.
> We'll be creating a json file in http://hg.mozilla.org/integration/gaia-try
> to trigger the tests.
> I don't yet know the contents of the json file, but I imagine that'll be the
> user (email?), gaia repo url, gaia repo revision would be in here.

Is that repo going to be a full blown gecko tree?  I thought that the gecko that'd be used for gaia-try is just the latest B2G desktop build.

If it needs to be a full blown gecko tree, that slightly bumps the time it'll take to submit each job, but is certainly possible.  Adding that information should.

I will only have information from Github, so unless we have a service that maps Github credentials to LDAP, I'll just forge the author using their Github information.
Flags: needinfo?(jhford)

Comment 11

3 years ago
No, it'll be an empty repo except that json file.
We'll also need to get a latest url from somewhere.  Maybe we can put that in the json, and the commit hook process can bump the repo/revision info but leave the installer url alone.
(In reply to Aki Sasaki [:aki] from comment #11)
> No, it'll be an empty repo except that json file.
> We'll also need to get a latest url from somewhere.  Maybe we can put that
> in the json, and the commit hook process can bump the repo/revision info but
> leave the installer url alone.

phew.  this sounds like a workable solution to me.  Please let me know when it's ready for testing.
Depends on: 993774

Comment 13

3 years ago
John:
If you're waiting for what the file will look like, I'm pretty sure it's going to look like this: https://hg.mozilla.org/users/asasaki_mozilla.com/hg-git/file/432ac6cedc75/gaia.json

It'll be a 'gaia.json' file in https://hg.mozilla.org/integration/gaia-try

I suggest looking for that file and reading it into a dict, and falling back to {} if it doesn't, then populating the git_revision and remote, then writing to the file, committing, pushing without -f, and hg stripping/updating/re-committing/repushing in a retry loop to avoid losing push races.
http://hg.mozilla.org/build/mozharness/file/fb0bae844d4a/scripts/bump_gaia_json.py#l189 and http://hg.mozilla.org/build/mozharness/file/fb0bae844d4a/scripts/b2g_bumper.py#l245 for examples.

This won't be fully testable with triggered tests until bug 986209 is reviewed/landed/reconfiged, but hopefully this unblocks you a bit further.
Flags: needinfo?(aki)
Aki, thanks for the update!  I'll start landing to that repository as soon as everything is in place.

As we discussed in person just now, I would like to inform developers of when their job is done as well as where they can check status.  I think the easiest way for status is to comment on the PR when triggering the job with a link to the TBPL page for the commit and a message that the jobs were triggered.  As neither of us are sure whether the Buildapi contains information on the overall outcome of the job, you mentioned creating a downstream job that is done when all the tests are done.  If that downstream job is able to hit a www-accessible api endpoint, I can have my hook comment on the PR when hit.

The most simple thing I can think of is to do something like:

curl -Li http://www.example.com:7040/tests-finished?commit=abcd123[&outcome=pass|fail]

Does that still work for you?
Flags: needinfo?(aki)

Comment 15

3 years ago
(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #14)
> Aki, thanks for the update!  I'll start landing to that repository as soon
> as everything is in place.
> 
> As we discussed in person just now, I would like to inform developers of
> when their job is done as well as where they can check status.  I think the
> easiest way for status is to comment on the PR when triggering the job with
> a link to the TBPL page for the commit and a message that the jobs were
> triggered.  As neither of us are sure whether the Buildapi contains
> information on the overall outcome of the job, you mentioned creating a
> downstream job that is done when all the tests are done.  If that downstream
> job is able to hit a www-accessible api endpoint, I can have my hook comment
> on the PR when hit.
> 
> The most simple thing I can think of is to do something like:
> 
> curl -Li
> http://www.example.com:7040/tests-finished?commit=abcd123[&outcome=pass|fail]
> 
> Does that still work for you?

Evidently the AggregatingScheduler solution I was thinking about is really only good for one set of builders (e.g. releases) and doesn't work well when multiple instances of each builder is running... we should most likely use buildapi's is_done.  If that has the info you need, will that be sufficient for you?
Flags: needinfo?(aki)
Flags: needinfo?(jhford)
(In reply to Aki Sasaki [:aki] from comment #15)
> Evidently the AggregatingScheduler solution I was thinking about is really
> only good for one set of builders (e.g. releases) and doesn't work well when
> multiple instances of each builder is running... we should most likely use
> buildapi's is_done.  If that has the info you need, will that be sufficient
> for you?

is_done should have the info I need.  Can you confirm that the gaia_try runs will work with is_done?
Flags: needinfo?(jhford)

Comment 17

3 years ago
I think the easiest way to know is to test.
https://tbpl.mozilla.org/?tree=Gaia-Try is there now, and in theory we'll launch the tests as soon as we start populating gaia.json in http://hg.mozilla.org/integration/gaia-try .
Do you want me to create a dummy, or do you want to start pushing pull request entries there?
Aki, can you add the tree to https://treestatus.mozilla.org/gaia-try please?

remote: ************************** ERROR ****************************
remote: Error accessing https://treestatus.mozilla.org/gaia-try?format=json :
remote: HTTP Error 404: NOT FOUND
remote: Unable to check if the tree is open - treating as if CLOSED.
remote: To push regardless, include "CLOSED TREE" in your push comment.
remote: *************************************************************

Comment 19

3 years ago
I don't have perms.  :hwine and sheriffs do.
Flags: needinfo?(hwine)
created - in case any correction needed, faster service in #releng where the sheriffs hang out.
Flags: needinfo?(hwine)

Comment 21

3 years ago
https://bugzilla.mozilla.org/show_bug.cgi?id=986209#c41
This is in production.  I'm going to mark this as fixed, further issues should be filed as new bugs.

Thanks everyone!
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Depends on: 1007435
Depends on: 1007437
No longer depends on: 1007435
Depends on: 1009695
You need to log in before you can comment on or make changes to this bug.