The fetch-runnablejobs task takes 10 mins and makes 140,000 MySQL requests

RESOLVED WONTFIX

Status

P1
normal
RESOLVED WONTFIX
2 years ago
a year ago

People

(Reporter: emorley, Unassigned)

Tracking

Details

(Reporter)

Description

2 years ago
For example:
https://rpm.newrelic.com/accounts/677903/applications/14179733/transactions?tw%5Bend%5D=1484251707&tw%5Bstart%5D=1484165307#id=5b224f746865725472616e73616374696f6e2f43656c6572792f66657463682d616c6c7468657468696e6773222c22225d

A single run of the task takes on average 540 seconds and makes 141,300 MySQL requests.

MySQL breakdown:

MySQL commit:    15700
MySQL runnable_job select:    15700
MySQL build_platform select:    15700
MySQL machine_platform select:    15700
MySQL option_collection select:    15700
MySQL job_type select:    15700
MySQL runnable_job update:    15700
MySQL job_group select:    15700
MySQL option select:    15700
(Reporter)

Comment 1

2 years ago
So the problem is that currently update_runnable_jobs_table() loops through allthethings.json
and for each entry:
* does a get_or_create() on each of:
  - BuildPlatform
  - MachinePlatform
  - JobGroup
  - JobType
  - OptionCollection
* Then using those IDs created the RunnableJob object

Instead it should do some aggregation on the reference data.
(Reporter)

Comment 2

a year ago
This task now only takes 5 mins and makes a third of the DB requests - presumably since we run fewer jobs on buildbot (the taskcluster jobs have their data fetched another way). As such let's just wontfix this given buildbot going away.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → WONTFIX
Summary: The fetch_allthethings task takes 10 mins and makes 140,000 MySQL requests → The fetch-runnablejobs task takes 10 mins and makes 140,000 MySQL requests
You need to log in before you can comment on or make changes to this bug.