522139 - Test Server for Mobile Framework must detect completed builds from build system

Status: RESOLVED INVALID

(Testing :: General, defect)

Description

[cmtalbert]

The test server must be able to detect completed builds from the build system.

The best long term strategy here is to create an interface on the server that can capture buildbot "sendchange" notifications.  This way, the entire mobile testing system can be seamlessly integrated into the current build system.  This is probably also the most difficult mechanism to use.

A simpler strategy would be to detect when a new build arrives on the tinderbox-builds FTP site and pull the builds and tests from there.  This is essentially the way the buildbot system works that drives Talos, so it is not entirely without precedent. This way is easier since we do not have to adhere to any of the rules that interfacing with buildbot would place upon us.

A decision has yet to be made.

Comment 1

[Mikeal Rogers [:mikeal] (mrogers@)]

I wrote code that checks a given set of target products and build branches for new builds.

It's working and posting the newBuild information to testbot.

This is a simple solution that will scale up easily in the future since a custom sendchange in buildbot could just POST to the same interface the poller/scraper code is using now.

I don't know what else needs to be done before we can mark this "fixed" since the code is now working, detects new builds and triggers a newBuild operation in testbot which in turn creates a set of job objects for each set of tests that need to be run for that build.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

how does the server get notified of a new build?  Does it get an event from buildbot, or does it poll the build directories?

Another bug we need to figure out is what is the set of jobs required once a build is pushed and how do we define this for each and every build.

Comment 3

[Bob Moss :bmoss]

I think Mikeal has the first one covered. On the second question I think we need to start simple and build out. So maybe the first iteration is just run them "all" like we do today. In the future we need to have a way to be smart about which tests are run for automated builds based upon several criteria (such as; What platform is it targeted at?, What code areas were touched?, etc...) There is also the piece where someone wants to run specific tests/suites against either an existing build (think try server like here) or just against a specific build.

Comment 4

[Mikeal Rogers [:mikeal] (mrogers@)]

(In reply to comment #2)
> how does the server get notified of a new build?  Does it get an event from
> buildbot, or does it poll the build directories?

It polls the HTTP interface for ftp.mozilla.org and scrapes the HTML for the pages it cares about. It's surprisingly simple and reliable using pyquery.

> Another bug we need to figure out is what is the set of jobs required once a
> build is pushed and how do we define this for each and every build.

So, what I've done is make this part incredibly simple and module.

Anyone who wants to run a testbot writes up a "manager" object. In the manager object they define how they want to handle various events.

I wrote a MozillaManager and implemented the "new_build" method. Whenever a new build comes in this method gets called and passed the build information, it's just is to return all the job objects that need to be created for that build.

http://github.com/mikeal/testbot/blob/master/testbot/__init__.py#L45

Right now i have it creating three new test jobs for each new build. This is just for testing purposes but it's very simple to customize this to create different kinds of jobs for different types of builds and to change how the jobs look since it's all just JSON there isn't any restrictions in place on the formatting.

The manager will also end up being responsible for handling the get_job logic for the clients and we can customize that as well. Right now, since I'm sticking the entire build document in each job document writing a view for jobs by platform is trivial and spooling out the jobs to that client platform will also be trivial.

A try server like button is actually trivial here since there is little friction to creating jobs and creating build documents, it's just a matter of writing up the HTML.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Actually we need more logic than what we are already doing.  Right now for Firefox, the logic is [talos, mochitest, everythingelse].  We don't have a clear definition of each one, but that can be determined.  But for Fennec it is [talos, mochitest1, mochitest2, mochitest3, mochitest4, mochitest-chrome, reftest, crashtest, xpcshell].

So right there we need to have logic to do those two separate things.  I think for Windows Mobile this will even change some more and add more complexity.

Comment 6

[Mikeal Rogers [:mikeal] (mrogers@)]

Surely the logic is going to grow, for testing purposes right now we can stick with something simple but it's designed to grow.

All the information we have about the build is passed to this function so it can easily create different sets of jobs for different kinds of builds. I think at this point we can move on and add that logic later once we're out of the early development phase :)

Comment 7

[Aki Sasaki (not active)]

(In reply to comment #5)
> Actually we need more logic than what we are already doing.  Right now for
> Firefox, the logic is [talos, mochitest, everythingelse].  We don't have a
> clear definition of each one, but that can be determined.  But for Fennec it is
> [talos, mochitest1, mochitest2, mochitest3, mochitest4, mochitest-chrome,
> reftest, crashtest, xpcshell].
> 
> So right there we need to have logic to do those two separate things.  I think
> for Windows Mobile this will even change some more and add more complexity.

We're constantly doing further splitting on both desktop and Maemo, so easy configurability would definitely be a plus.

Re ftp polling, this works, to a degree, but a) tends to be laggier than a sendchange and b) can run into mid-upload issues (trying to download a tarball that's still being uploaded).  And c) it's harder to re-trigger tests on a previous tarball without renaming it or touching it or whatever.  We don't necessarily have to use the exact sendchange api to trigger builds: we can hit a url or whatever via a shell command, but having another way to specify a file would be good.

Comment 8

[Mikeal Rogers [:mikeal] (mrogers@)]

> Re ftp polling, this works, to a degree, but a) tends to be laggier than a
> sendchange and

Most definitely, this is just the simplest fastest way to move on with the development.

In the future we can a custom sendchange that does and HTTP POST to the same interface and benefit from more realtime notifications without any further modifications to testbot.

> b) can run into mid-upload issues (trying to download a tarball
> that's still being uploaded).  And

This will be an issue eventually but ATM the pool of devices will lag behind the latest checkin a bit so we probably won't see it until down the line when we start adding more devices to the pool. Once we start seeing it, we can go ahead and replace with a custom sendchange.

> c) it's harder to re-trigger tests on a
> previous tarball without renaming it or touching it or whatever.  We don't
> necessarily have to use the exact sendchange api to trigger builds: we can hit
> a url or whatever via a shell command, but having another way to specify a file
> would be good.

So, testbot doesn't directly correlate a build with a set of jobs like this. A new build triggers the creation of a series of jobs but it's simple to go in and create a bunch of new jobs for an existing build document.

We most definitely will want to hook in to sendchange and send the HTTP POST from there instead of this polling script. But changes that need to go in to buildbot infrastructure have a delay in process and rollout time so my goal was to have something that we can use now and move on with further development (there is no shortage of code to write) until we have the time to add the sendchange support. Right now there are so many other things we need to worry about writing in the short term that I didn't want to block on anything.

Comment 9

[Aki Sasaki (not active)]

Makes perfect sense :)

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

is this bug still valid? Seems like something we could close

Comment 11

[cmtalbert]

Good lord, ancient. --> INVALID

We have long since solved this many other ways.