If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

The fetch_allthethings task takes 10 mins and makes 140,000 MySQL requests

NEW
Unassigned

Status

Tree Management
Treeherder: Data Ingestion
P1
normal
9 months ago
9 months ago

People

(Reporter: emorley, Unassigned)

Tracking

(Blocks: 1 bug)

Details

(Reporter)

Description

9 months ago
For example:
https://rpm.newrelic.com/accounts/677903/applications/14179733/transactions?tw%5Bend%5D=1484251707&tw%5Bstart%5D=1484165307#id=5b224f746865725472616e73616374696f6e2f43656c6572792f66657463682d616c6c7468657468696e6773222c22225d

A single run of the task takes on average 540 seconds and makes 141,300 MySQL requests.

MySQL breakdown:

MySQL commit:    15700
MySQL runnable_job select:    15700
MySQL build_platform select:    15700
MySQL machine_platform select:    15700
MySQL option_collection select:    15700
MySQL job_type select:    15700
MySQL runnable_job update:    15700
MySQL job_group select:    15700
MySQL option select:    15700
(Reporter)

Comment 1

9 months ago
So the problem is that currently update_runnable_jobs_table() loops through allthethings.json
and for each entry:
* does a get_or_create() on each of:
  - BuildPlatform
  - MachinePlatform
  - JobGroup
  - JobType
  - OptionCollection
* Then using those IDs created the RunnableJob object

Instead it should do some aggregation on the reference data.
You need to log in before you can comment on or make changes to this bug.