Closed Bug 669930 Opened 13 years ago Closed 13 years ago

Publish Build Faster metrics

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: catlee, Assigned: catlee)

Details

(Whiteboard: [buildfaster:p1]) 

We need some simple metrics for how long builds / tests / waits are taking.

http://arewebuildingfastyet.com ?
Chris,
We are pulling data out of orange factor right now on specific slow tests in mochitest-plain.  I have :samliu and :harth working on building a dashboard for it.  So if you let :samliu know how to grab the data out of your database, he can include the specific buildbot setup/teardown times and include those in the dashboard.

So let him know how to get to your data and some thoughts on how you want it displayed and he can get started.
Would there be anyway of getting this information along with the buildslave?  I would love to be able to identify a Tegra that is starting to act-up.
(In reply to comment #1)
> Chris,
> We are pulling data out of orange factor right now on specific slow tests in
> mochitest-plain.  I have :samliu and :harth working on building a dashboard
> for it.  So if you let :samliu know how to grab the data out of your
> database, he can include the specific buildbot setup/teardown times and
> include those in the dashboard.
> 
> So let him know how to get to your data and some thoughts on how you want it
> displayed and he can get started.

We store per-step information in our statusdb, I have a snapshot here:
http://people.mozilla.org/~catlee/statusdb-2011-02-01.sql.bz2

It can be a bit challenging to get at exactly the data you want though :)
Hi Chris,

A few questions:
1. How often are these snapshots made / is it automated / where do they end up?
2. I was thinking about how to extract the data, do you happen to have ideas on how I can do that efficiently? Last I was informed I would be getting everything in json but I don't think this is the case?

Thanks for your help!
(In reply to comment #4)
> Hi Chris,
> 
> A few questions:
> 1. How often are these snapshots made / is it automated / where do they end
> up?

They're made manually right now. Depending on what data you need, I'm sure we can get you r/o access to the mysql database.

> 2. I was thinking about how to extract the data, do you happen to have ideas
> on how I can do that efficiently? Last I was informed I would be getting
> everything in json but I don't think this is the case?

The detail on how long individual steps take aren't available from our build json. Maybe they're in the ES database? They are in this statusdb snapshot.
Thanks Chris,

Hm so if I want to get data out of this snapshot do you have any recommendations on how that can be accomplished? My thought was importing into a local mysql and extracting from there, but that seems inefficient and I was wondering if there might be a better way?
(In reply to comment #6)
> Thanks Chris,
> 
> Hm so if I want to get data out of this snapshot do you have any
> recommendations on how that can be accomplished? My thought was importing
> into a local mysql and extracting from there, but that seems inefficient and
> I was wondering if there might be a better way?

It really depends on what you're looking for...

Maybe if I exported data for setup/teardown, and wait times that would be helpful?
FTR
I copy/pasted all the info from this report:
https://build.mozilla.org/buildapi/reports/testruns
Nevermind, I loaded the database locally and will handle getting data. Thanks! :)
Would a report like this help?

submitted_at,revision,os,jobtype,uid,results,wait_time,start_time,finish_time,elapsed
2011-07-07T03:08:40,b2622d5c857ab7e1cda6340e51227e1ab9efbb59,android,opt build,63e86b391675467091b52e02419b1973,0,0:00:00,2011-07-07T03:08:40,2011-07-07T04:49:27,1:40:47
2011-07-07T03:08:40,b2622d5c857ab7e1cda6340e51227e1ab9efbb59,linux32,opt build,63e86b391675467091b52e02419b1973,0,0:00:01,2011-07-07T03:08:41,2011-07-07T04:40:36,1:31:55
2011-07-07T03:08:40,b2622d5c857ab7e1cda6340e51227e1ab9efbb59,linux64,opt build,63e86b391675467091b52e02419b1973,0,0:00:01,2011-07-07T03:08:41,2011-07-07T04:46:44,1:38:03
2011-07-07T03:08:40,b2622d5c857ab7e1cda6340e51227e1ab9efbb59,osx10.6,opt build,63e86b391675467091b52e02419b1973,0,0:00:04,2011-07-07T03:08:44,2011-07-07T06:19:10,3:10:26
2011-07-07T03:08:40,b2622d5c857ab7e1cda6340e51227e1ab9efbb59,win32,opt build,63e86b391675467091b52e02419b1973,0,0:00:10,2011-07-07T03:08:50,2011-07-07T06:25:58,3:17:08
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,linux32,debug build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:02,2011-07-07T03:08:57,2011-07-07T03:55:18,0:46:21
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,osx10.5,debug build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:02,2011-07-07T03:08:57,2011-07-07T04:06:44,0:57:47
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,linux32,opt build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:03,2011-07-07T03:08:58,2011-07-07T04:23:12,1:14:14
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,linux32,opt build,8adc5b8486f141139e74ef5fc3dc98f3,1,0:00:03,2011-07-07T03:08:58,2011-07-07T04:16:25,1:07:27
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,linux64,opt build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:03,2011-07-07T03:08:58,2011-07-07T07:27:16,4:18:18
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,android,opt build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:03,2011-07-07T03:08:58,2011-07-07T04:30:09,1:21:11
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,linux64,debug build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:03,2011-07-07T03:08:58,2011-07-07T05:06:16,1:57:18
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,osx10.6,opt build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:03,2011-07-07T03:08:58,2011-07-07T06:02:14,2:53:16
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,win32,opt build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:03,2011-07-07T03:08:58,2011-07-07T06:06:13,2:57:15
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,android,opt build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:03,2011-07-07T03:08:58,2011-07-07T04:38:41,1:29:43
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,win32,debug build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:03,2011-07-07T03:08:58,2011-07-07T04:43:02,1:34:04
2011-07-07T03:08:55,3f9e716164ef3c1e26285ce1915d23475bbfb04c,osx10.6,debug build,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:04,2011-07-07T03:08:59,2011-07-07T04:49:40,1:40:41
2011-07-07T03:40:07,3f9e716164ef3c1e26285ce1915d23475bbfb04c,linux32,debug test,8adc5b8486f141139e74ef5fc3dc98f3,0,0:00:16,2011-07-07T03:40:23,2011-07-07T04:08:28,0:28:05
Oh yeah actually that would help, I was wondering in that statusdb snapshot, is it intentional that the request tables are all empty? I've been asked to find these metrics and haven't had much success:

    Bw - time waiting for builder (avg over all OSes)
    Ba - time active building (avg over all OSes)
    Tw - time waiting for test machine (avg over all OSes)
    S1 - setup time
    Ta - time running tests
    S2 - time teardown & reboot

Also, when this dashboard goes into production is it intended to query the statusdb directly?

The only thing I've managed to do with regards to this is getting elapsed times for builds in the steps table.
Yes :) So is there a web resource I can request from that will give me this reliably?

Thanks
Sam
Let me know how the data looks, and then I can automate a daily dump of it.
This will be uploaded to http://build.mozilla.org/builds/buildfaster.csv.gz every day. The data will include the past 60 days of data.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.