1558055 - Store recording date of test page recordings in perfherder data

Status: NEW

(Testing :: Raptor, enhancement, P3)

Description

[Corey Dow-Hygelund [:ccd]]

Raptor replays recordings of web pages for testing using Mitmproxy. This enables comparison of builds across time, due to the reference pages being consistent. However, these are updated regularly, to maintain consistency with the current version of the pages.

These updates can result in very different behavior of the pages during testing. Therefore, knowledge of these snapshot updates is key in appropriately analyzing and modeling the data.

This request is to

1. Determine if the data is currently coming out of TaskCluster. If not, it needs to be added.
2. Add a field to the existing Perfherder DB schema(s) representing the date of snapshot update.
3. Populate the fields for subsequent snapshot updates.

Comment 1

[Vicky Chin [:vchin]]

Rob, what information is currently in the Raptor DB schema(s)?

Other information that would also be useful:
Firefox Version/Build
Device used to capture
OS level
Tracking Protection On/Off
List of Firefox Prefs (might have to be a pared down list)
Mitmproxy prefs

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Looking at the mitmproxy recording for amazon.com in tp6-1 I see a timestamp that resolves to 18 July 2018 18:46:06.313. The manifest was updated on 24 Jul 2018 via bug 1454053. Rather than maintaining the dates separately, could we use these timestamps to indicate when these recordings were last updated? We could potentially include this as extra data in the Perfherder results, or output the dates to the console.

What does "determine if the data is currently coming out of TaskCluster" mean here? What DB schema are we referring to? I'm not aware of a Raptor DB, but perhaps this means the data published to Perfherder? We could create an additional file to be included in the recording archive with additional metadata. Currently I believe the only metadata we have is the filename, which we are planning to use to include platform/device/app details used during the recordings.

Comment 3

[Robert Wood [:rwood]]

(In reply to Vicky Chin [:vchin] from comment #1)

Rob, what information is currently in the Raptor DB schema(s)?

Other information that would also be useful:
Firefox Version/Build
Device used to capture
OS level
Tracking Protection On/Off
List of Firefox Prefs (might have to be a pared down list)
Mitmproxy prefs

Hi Vicky, all data submitted by Raptor is in the PERFHERDER_DATA json and ingested by perfherder and stored in perfherder DB; as for the exact schemas of those I don't know but :ekyle would be able to help us there - there will be job info i.e. versions, devices, OS level etc that would come from the taskcluster job itself and not Raptor.

As for prefs set maybe we could create a taskcluster artifact somehow that would be a list of browser prefs set during the test job run. Otherwise the prefs are found in https://searchfox.org/mozilla-central/source/testing/profiles (common, perf, and raptor) plus additional prefs set in Raptor to turn on the proxy.

:ekyle is working on the ability to annotate perf data; I would expect that would give us the ability to add annotations whenver the mitmproxy recordings have been updated. :ekyle, should we add an extra data field in all Raptor tp6 test results that includes the date of the mitmproxy recording? Or should we wait and use annotations when that is available, what do you think? Thanks!

Comment 4

[Corey Dow-Hygelund [:ccd]]

:davehunt

Publishing the data to Perfherder is desired. :ekyle was unsure if these dates were available in the properties coming out of taskcluster. The methods being proposed would fulfill the need.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Corey Dow-Hygelund [:ccd] from comment #4)

Publishing the data to Perfherder is desired. :ekyle was unsure if these dates were available in the properties coming out of taskcluster. The methods being proposed would fulfill the need.

There's no way (that I know of) for Perfherder to handle this supplementary data. I suspect a better long term approach may be to publish this kind of data to ekyle's proposed annotation service, which Perfherder and other tools could use to annotate the visualisations when the recordings are updated. I'll let ekyle respond to the open needinfo for his thoughts on what we could do in the short term, but I'll also raise a bug for extracting the timestamp of the recording and at least adding it to the log.

Comment 6

[Kyle Lahnakoski [:ekyle]]

https://github.com/klahnakoski/annotations/issues/2