458937 - moz2 buildbots + talos unable to POST to the graph server

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

filing as a blocker because this is holding up 3.1b1 release work and keeping the tree closed.

Comment 1

[[e-mail] (:bhearsum)]

FAIL: graph server unreachable
FAIL: timed out
FAIL: error in post data

Comment 2

[Dave Miller [:justdave]]

lots of timeout errors in the log, restarted apache, so far looks good.  No idea what got it bogged down.

Comment 3

[Mark Smith [:xb95]]

Long running queries, this one was going for nearly 10 minutes.  When this is happening, INSERT statements are blocked.  I've never seen this before and believe it will be partially or mostly ameliorated by the switch to InnoDB tonight.

SELECT             dataset_values.dataset_id, dataset_values.time, dataset_values.value,             dataset_branchinfo.branchid, dataset_extra_data.data         FROM dataset_values         LEFT JOIN             dataset_branchinfo ON                 dataset_values.dataset_id = dataset_branchinfo.dataset_id AND                 dataset_values.time = dataset_branchinfo.time         LEFT JOIN             dataset_extra_data ON                 dataset_values.dataset_id = dataset_extra_data.dataset_id AND                 dataset_values.time = dataset_extra_data.time         WHERE             dataset_values.dataset_id IN (53216,53252,53234,53274)                       ORDER BY 
            dataset_values.dataset_id, dataset_values.time