492406 - Figure out how to get SeaMonkey data on graphs server

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Robert Kaiser]

In bug 492221, we're doing a short-term fix to just disable SeaMonkey posting data to the graphs server, but alice and me agreed on IRC that we want to look into actually getting data and graphs up on the new server after the transition.

We need to configure the server to the the machines from the cb network post to it, and do eventual configuration on the SeaMonkey buildbots and the server to report the graphs accurately.

Currently, it's Codesighs (Linux, Mac) and the Trace-Malloc (Linux) numbers that we are generating now, basically as a side effect of using NightlyBuildFactory.

Comment 1

[[e-mail] (:bhearsum)]

KaiRo, should this be assigned to you?

Comment 2

[Robert Kaiser]

Ben, it probably ought to be assigned to alice and dependent on finishing the whole graphs server transition. I can't do anything on this before the server side isn't taken care of.

Comment 3

[[e-mail] (:bhearsum)]

I'm going to put this into future for now. Once the graph server side is taken care off I guess we can come back to this.

After that, I think the biggest thing here is poking a hole in the firewall - I could be wrong though.

Comment 5

[Robert Kaiser]

Alice, do we have a chance to get this done now?

Comment 6

[Aki Sasaki (not active)]

I don't think there's a firewall involved -- graphs.m.o is on the internet is it not?

There is, however, an ACL whitelist in apache for the cgi used to submit data.  Server Ops is able to update this whitelist... they'll need the list of IPs that need access.

Comment 7

[Robert Kaiser]

Does it need to be specific IPs (would be all SeaMonkey build slaves, I guess) or can access for the whole community network be granted?

Comment 8

[Aki Sasaki (not active)]

Assuming that's a subnet, I bet that's possible. IT would know better.

Comment 9

[alice nodelman [:alice] [:anode]]

To get this going we're going to need:

- figure out how to securely allow access by these machines to the graph server reporting system
- list of the names of all the tests are going to be reporting
- list of all the machines names that will be reporting (so that they can be added to the graph server db)

We could start by attempting to get things reporting to graphs-stage, I have access there and can do the necessary db updates.

Look at http://hg.mozilla.org/graphs/file/bbb11e47c50d/sql/data.sql for how the machines/tests/os/etc are to be added to the graph server db so that I can be provided with working sql insert statements.

Comment 10

[Robert Kaiser]

I'm trying my best to figure out the SQL, let's see if I get it right, as documentation is not overwhelming in this file...

insert into branches values (NULL,"SeaMonkey");
insert into branches values (NULL,"SeaMonkey2.0");

insert into machines values (NULL,6,0,NULL,"Linux_comm-1.9.1",1,unix_timestamp());
insert into machines values (NULL,6,0,NULL,"Linux_comm-1.9.1_leak_test",1,unix_timestamp());
insert into machines values (NULL,6,0,NULL,"Linux_comm-central-trunk",1,unix_timestamp());
insert into machines values (NULL,6,0,NULL,"Linux_comm-central-trunk_leak_test_leak_test",1,unix_timestamp());
insert into machines values (NULL,7,0,NULL,"OS_X_10.5_comm-1.9.1",1,unix_timestamp());
insert into machines values (NULL,7,0,NULL,"OS_X_10.5_comm-1.9.1_leak_test",1,unix_timestamp());
insert into machines values (NULL,7,0,NULL,"OS_X_10.5_comm-central-trunk",1,unix_timestamp());
insert into machines values (NULL,7,0,NULL,"OS_X_10.5_comm-central-trunk_leak_test",1,unix_timestamp());
insert into machines values (NULL,8,0,NULL,"WINNT_5.2_comm-1.9.1_leak_test",1,unix_timestamp());
insert into machines values (NULL,8,0,NULL,"WINNT_5.2_comm-central-trunk_leak_test",1,unix_timestamp());


It's just codesighs and leak tests, i.e. everything the build machines run. We don't have talos, but I think it would be nice to have graphs of the other things to be able to track and compare them.

Comment 11

[alice nodelman [:alice] [:anode]]

I've applied the changes to the graphs-stage db.  So, now you just have to attempt to send data there and then we can shake out the issues.

If all goes well there then we can move things over to production graphs.

Comment 12

[Robert Kaiser]

When I set the graph server to graphs-stage.mozilla.org, I now get the following:
"Warning: error when trying to post codesighs, trying again in 5 seconds"
(repeated up to at least 320 seconds, same with the leak test stuff).

Might the community subnet not have access to push data there?

Comment 13

[alice nodelman [:alice] [:anode]]

You may need an IT bug to ensure that seamonkey builders have permission to post to graph server.  

Is there any more information in the logs?

Comment 14

[Robert Kaiser]

Filed bug 544131 - unfortunately those "error when trying to post xxx" warnings are all we get.

Comment 15

[Robert Kaiser]

Yay, bug 544131 helped, it looks like a first data set has been posted to graphs-stage!

Comment 16

[Robert Kaiser]

I'm still seeing one error, but that is really on the graph server side:

results not added, response: 
No machine_name called 'Linux_comm-central-trunk_leak_test' can be found
  File "/var/www/html/graphs2/server/pyfomatic/collect.py", line 255, in handleRequest
    metadata = MetaDataFromTalos(databaseCursor, databaseModule, inputStream)
  File "/var/www/html/graphs2/server/pyfomatic/collect.py", line 59, in __init__
    self.doDatabaseThings(databaseCursor)
  File "/var/www/html/graphs2/server/pyfomatic/collect.py", line 88, in doDatabaseThings
    raise DatabaseException("No machine_name called '%s' can be found" % self.machine_name)

Comment 17

[Robert Kaiser]

(In reply to comment #10)
> insert into machines values
> (NULL,6,0,NULL,"Linux_comm-central-trunk_leak_test_leak_test",1,unix_timestamp());

Oops. that should be:
insert into machines values (NULL,6,0,NULL,"Linux_comm-central-trunk_leak_test",1,unix_timestamp());

Comment 18

[Robert Kaiser]

OK, with that change, looks like all machines that currently run can post to graphs-stage, all trees are green. :)

Let's leave them running this way over the weekend, so that we get a bit of data up there and can see if things look right. If so, I'll re-post the SQL for the production server.

Comment 19

[Robert Kaiser]

OK, things look good on graphs-stage, let' go for pushing this to the real graphs server!

Here's the correct SQL statements:

insert into branches values (NULL,"SeaMonkey");
insert into branches values (NULL,"SeaMonkey2.0");

insert into machines values
(NULL,6,0,NULL,"Linux_comm-1.9.1",1,unix_timestamp());
insert into machines values
(NULL,6,0,NULL,"Linux_comm-1.9.1_leak_test",1,unix_timestamp());
insert into machines values
(NULL,6,0,NULL,"Linux_comm-central-trunk",1,unix_timestamp());
insert into machines values
(NULL,6,0,NULL,"Linux_comm-central-trunk_leak_test",1,unix_timestamp());
insert into machines values
(NULL,7,0,NULL,"OS_X_10.5_comm-1.9.1",1,unix_timestamp());
insert into machines values
(NULL,7,0,NULL,"OS_X_10.5_comm-1.9.1_leak_test",1,unix_timestamp());
insert into machines values
(NULL,7,0,NULL,"OS_X_10.5_comm-central-trunk",1,unix_timestamp());
insert into machines values
(NULL,7,0,NULL,"OS_X_10.5_comm-central-trunk_leak_test",1,unix_timestamp());
insert into machines values
(NULL,8,0,NULL,"WINNT_5.2_comm-1.9.1_leak_test",1,unix_timestamp());
insert into machines values
(NULL,8,0,NULL,"WINNT_5.2_comm-central-trunk_leak_test",1,unix_timestamp());

Comment 20

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 21

[Robert Kaiser]

Switched SeaMonkey configuration to graphs.m.o server in http://hg.mozilla.org/build/buildbot-configs/rev/c75a480422d9 - this should be fixed now.

Thanks to Alice for your help here!