1461482 - Verify VCS Perfherder data for Windows tasks is being ingested

Status: RESOLVED FIXED

(Testing Graveyard :: ActiveData, defect, P1)

Description

[Gregory Szorc [:gps]]

I'm able to run ActiveData queries against the VCS Perfherder metrics being recorded by `hg robustcheckout`.

However, I'm selecting the raw data and am not seeing any data from Windows workers. There's a chance I'm doing it wrong. But some of the aggregate data seems to reflect no data for Windows.

https://taskcluster-artifacts.net/Yl5tnpaBQTGjduaj76r2PQ/0/public/logs/live_backing.log is an example of a Windows task whose data is not ingested. Search for "PERFHERDER_DATA". The fact it is at the beginning of a line may confuse the log parser?

Comment 1

[Gregory Szorc [:gps]]

FWIW, I have other patches in flight that attempt to make Firefox CI tasks use run-task. At that point, logs will be prefixed with the typical time prefix and this may cause ActiveData to start recognizing the PERFHERDER_DATA lines.

So if this is a non-trivial amount of work, ping me before you invest much time and I may be able to save you the work!

Comment 2

[Kyle Lahnakoski [:ekyle]]

Sorry, I never even saw this bug. There is a dup somewhere, which I also did nothing on. needinfo myself so I remember to act on this.

Comment 3

[Kyle Lahnakoski [:ekyle]]

p1, just to verify before closing

Comment 4

[Kyle Lahnakoski [:ekyle]]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=build-win64%2Fpgo

Comment 5

[Kyle Lahnakoski [:ekyle]]

Turns out it was two problems. The multiline json (with taskcluster timestamps)

[taskcluster 2019-03-28T17:35:29.768Z]

and the PERFHERDER record has no timestamp

Here is the file

https://taskcluster-artifacts.net/Qu-_x508RWys08fcgMtVgQ/0/public/logs/live_backing.log

Here is he task record

{
    "from":"task",
    "select":"_id",
    "where":{"eq":{"task.id":"Qu-_x508RWys08fcgMtVgQ"}}
}

Here are the (new) perf records

{
    "from":"perf",
    "select":["_id","run.key","result.framework.name"],
    "where":{"and":[{"eq":{"task.id":"Qu-_x508RWys08fcgMtVgQ"}}]},
    "limit":1000
}

Processing is proceeding atarting a few hours back.