1524521 - Provisioning local perf data is hard

Status: RESOLVED FIXED

(Tree Management :: Perfherder, enhancement, P3)

Description

[Ionuț Goldan [:igoldan]]

Doing local backend development that implies database schema updates can easily become difficult to test. ATM, the docs don't mention how developers should ingest performance data.

Without this performance data, the local UI's is basically empty and useless. You don't have any alert widgets or graph data points to interact with.

I noticed we have some Perfherder specific workers, according to the existing Procfile, but haven't yet looked over their definitions. Even if this is the main mechanism for creating local data, it's not too pleasant: you would have to wait for at least 2 days to gather enough data.

:emorley if you're aware of a faster method for provisioning local perf data or there's something I'm missing about this, do share. This kind of local development will happen almost daily in H2.

Next steps I see here, after confirming my assumptions:

• fix the old script that directly and quickly imported existing perf data based on project & signatures (bug 1429809)
• provide Perfherder docs for developers, alongside Treeherder's so it'll be easier for contributors to follow.

Comment 1

[Ed Morley [:emorley]]

Hi :-)

Getting data into Treeherder in general is slightly harder now that buildbot is EOL, since one cannot use the builds-4hr trick to ingest historic jobs, and instead the local environment has to listen to Pulse as new jobs complete (see also bug 1312054).

That said, for the Perfherder use-case, even builds-4hr wouldn't have been enough, since I imagine (/it sounds like) you need several days of data at a minimum.

Realistically I think this use-case is only ever going to work by having an actual copy of ~production data, rather than trying to come up with a way to download a few days of jobs from the prod DB.

There are currently a few ways to do this:

1. Push code to prototype, which has all new job data, but no recent sheriff provided data (unless it's been recently reset back to the latest prod DB snapshot; which is something that can be requested)
2. Pointing the local dev environment at the treeherder-prod-ro replica, which is always up to date, but is only useful when testing read only changes
3. Requesting that a new temporary RDS instance be spun up based on the latest prod DB snapshot (taken daily), and then pointing the local dev environment at that. This is like #2 except can be used for writing too
4. (For UI-only changes) Using yarn start, which points the local dev UI at the production API

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4808

Comment 3

[Sarah Clements [:sclements]]

Ionut should this bug be closed since your pr was merged?

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Sarah Clements [:sclements] from comment #3)

Ionut should this bug be closed since your pr was merged?

I know the PR has a small bug: it's not able to download data which has been touched by a user, because it cannot access that kind of data.
To close this bug, I just need to link it against a detailed description of this problem.

Comment 5

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/5301

Comment 6

[Ionuț Goldan [:igoldan]]

The fix got merged to master.

Comment 7

[Ionuț Goldan [:igoldan]]

This got deployed to production.

Comment 8

[Armen [:armenzg]]

Is this code in the docs?

Comment 9

[Ionuț Goldan [:igoldan]]

(In reply to Armen [:armenzg] from comment #8)

Is this code in the docs?

I'm afraid no, it isn't mentioned in the docs.