Open Bug 1672825 Opened 4 years ago Updated 4 years ago

Identify results with mulitple data points per push

Categories

(Testing :: Performance, task, P3)

task

Tracking

(Not tracked)

People

(Reporter: davehunt, Unassigned)

Details

With bug 1672740 and bug 1672822 we've stumbled on results that are mixed in Perfherder when running in multiple variations. This should be something we can identify by looking for results that consistently report more than one data point per commit. Let's use this bug to track identifying and filing bugs for each occurrence, and to discuss ways to prevent or get early notification of future occurrences.

The following query helped to identify pushes with more than one datapoint. Some of these are due to retriggers, and a lot of them appear to be related to bug 1672740. Perhaps we should rerun this query once that bug is resolved.

SELECT
  count(*),
  push_timestamp,
  datum.signature_id,
  signature.framework_id,
  signature.suite,
  signature.test,
  datum.push_id,
  push.revision,
  job_type.symbol
FROM performance_datum as datum
INNER JOIN performance_signature AS signature ON signature.id = datum.signature_id
INNER JOIN push ON push.id = datum.push_id
INNER JOIN job ON job.id = datum.job_id
INNER JOIN job_type ON job_type.id = job.job_type_id
WHERE datum.repository_id = 77
  AND framework_id NOT IN (2, 14)  -- exclude build_metrics and vcs
  AND push_timestamp > DATE_SUB(CURDATE(), INTERVAL 1 DAY)
GROUP BY signature_id, push_id
HAVING count(*) > 1
ORDER BY count(*) DESC

This improved query shows all signatures that have more than one job symbol, and I believe confirms that the open bugs account for all issues:

SELECT
  datum.signature_id,
  signature.framework_id,
  signature.suite,
  signature.test,
  push.revision,
  COUNT(DISTINCT(CONCAT(job_group.symbol, '(', job_type.symbol, ')'))) AS symbol_count,
  job_group.name,
  job_group.symbol AS job_group_symbol,
  GROUP_CONCAT(DISTINCT(job_group.symbol)) AS job_group_symbol_concat,
  job_type.symbol AS job_type_symbol,
  taskcluster_metadata.task_id AS task_id
FROM performance_datum AS datum
INNER JOIN performance_signature AS signature ON signature.id = datum.signature_id
INNER JOIN push ON push.id = datum.push_id
INNER JOIN job ON job.id = datum.job_id
INNER JOIN job_type ON job_type.id = job.job_type_id
INNER JOIN job_group ON job_group.id = job.job_group_id
INNER JOIN taskcluster_metadata ON taskcluster_metadata.job_id = job.id
WHERE datum.repository_id = 77
  AND framework_id NOT IN (2, 14)  -- exclude build_metrics and vcs
  AND push_timestamp > DATE_SUB(CURDATE(), INTERVAL 1 DAY)
GROUP BY datum.signature_id
HAVING COUNT(DISTINCT(CONCAT(job_group.symbol, '(', job_type.symbol, ')'))) > 1
ORDER BY symbol_count DESC
You need to log in before you can comment on or make changes to this bug.