587812 - "invalid data, not a float" in tp4 log caused graph server post failure

Status: RESOLVED DUPLICATE of bug 595958

(Testing :: Talos, defect)

Description

[[e-mail] (:bhearsum)]

In a recent tp4 run (http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1281992509.1281993573.19525.gz) there was an error right at the end of the test:
NOISE: __start_cc_report
NOISE: _x_x_mozilla_cycle_collect,41793
NOISE: __end_cc_report
NOISE: __startTimestamp1281993381726__endTimestamp
NOISE: 
NOISE: __startSecondTimestamp1281993382504__endSecondTimestamp
Invalid data, not a float
Completed test tp4: 
		Stopped Mon, 16 Aug 2010 14:16:34


Which later seems to have caused an error when posting to the Graph Server:
FAIL: Failed to send data 5 times... quitting
RETURN:send failed, graph server says:
RETURN:to determine average from 'test_run_values' for  5073835 - local variable 'values' referenced before assignment
RETURN:  File "/data/www/graphs.mozilla.org/server/pyfomatic/collect.py", line 257, in handleRequest
RETURN:    average = valuesReader(databaseCursor, databaseModule, inputStream, metadata)
RETURN:  File "/data/www/graphs.mozilla.org/server/pyfomatic/collect.py", line 215, in valuesReader
RETURN:    raise DatabaseException("to determine average from 'test_run_values' for  %s - %s" % (metadata.test_run_id, str(x)))
RETURN:
RETURN:


...which turned the build orange.

Comment 1

[[e-mail] (:bhearsum)]

Alice, have you ever seen anything like this before?

Comment 2

[[e-mail] (:bhearsum)]

Actually, I think this probably a bug in Talos itself, should go in the right component.

Comment 3

[alice nodelman [:alice] [:anode]]

That looks more like a graph server failure...

Comment 4

[[e-mail] (:bhearsum)]

Looks to me like bad data was sent to the graph server....

These "Invalid data, not a float" are scattered throughout the log. (Also, haven't seen any graph server changes other than data.sql updates in awhile.)

Comment 5

[Chris AtLee [:catlee]]

I think the problem comes for NaNs:

NOISE: _x_x_mozilla_page_load,327.25252525252523,NaN,NaN

Comment 6

[alice nodelman [:alice] [:anode]]

Those NaNs are built into the pageloader report.js:

    report += "_x_x_mozilla_page_load,"+avgmed+",NaN,NaN\n";  // max and min are just 0, ignored

Been like that since the beginning.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

hit this today on the 4.0b5 release Linux64 tp4:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Release/1283266348.1283267327.18375.gz

Comment 8

[alice nodelman [:alice] [:anode]]

Patch for this in bug 595958.