1279294 - new tp5o_scroll subtests names ending in .cssom* are not showing up in perfherder_data

Status: RESOLVED WORKSFORME

(Testing :: Talos, defect)

Description

[Avi Halachmi (:avih)]

+++ This bug was initially created as a clone of Bug #1279292 +++

This try push add new subtests to tscrollx and tp5o_scroll (instead of N subtests it will now have 3N subtests):

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b1047ee23ca1&selectedJob=22226513


perfherder subtests view for tp5o_scroll Linux 64 e10s opt (both old/new are the same revision of the try push):

https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=try&originalRevision=b1047ee23ca1&newProject=try&newRevision=b1047ee23ca1&originalSignature=c050d478f9344390dd49397560f7a07ea5397cae&newSignature=c050d478f9344390dd49397560f7a07ea5397cae&framework=1


Expected:
Each web page name should have 3 results (name, name.CSSOM, name.CSSOM.checkerboard).

Actual:
Each web page name shows only one result (<original-name>) - <original-name>.CSSOM and <original-name>.CSSOM.checkerboard are missing for all pages.


Sample tp5o_scroll test log:

http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-b1047ee23ca111be6a38fbcee47e08a43dff98cf/try-linux64/try_ubuntu64_hw_test-g1-e10s-bm103-tests1-linux-build1091.txt.gz


--------
tscrollx on the same try push has a different issue where one of every two new subtests are showing up. Joel thinks these are different issues - bug 1279292 is for the tscrollx issue.

Comment 1

[William Lachance (:wlach)]

From the log it looks like the subtests have exactly the same name. Search for the PERFHERDER_DATA and run it through a linter.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

in this case it is an issue in pageloader where we split the url on '/' and keep the first part dropping the page name (which is where we append .cssom and .cssom.checkerboard)

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

the magic is here:
https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/results.py#266

Comment 4

[Avi Halachmi (:avih)]

(In reply to William Lachance (:wlach) from comment #1)
> From the log it looks like the subtests have exactly the same name.

Not to me.

I've only modified the test, and now the test itself successfully reports different names. The following part of the log is after all the test runs have completed, but before the data is submitted further:

> 01:00:18     INFO -  PROCESS | 19603 | ------- Summary: start -------
> 01:00:18     INFO -  PROCESS | 19603 | Number of tests: 151
> 01:00:18     INFO -  PROCESS | 19603 |
> 01:00:18     INFO -  PROCESS | 19603 | [#0] 163.com/www.163.com/index.html  Cycles:12  Average:3.93  Median:3.91  stddev:0.15 (3.8%)  stddev-sans-first:0.07
> 01:00:18     INFO -  PROCESS | 19603 | Values: 4.4  3.8  3.9  4.0  3.8  3.9  3.8  4.0  3.9  3.8  3.9  3.9
> 01:00:18     INFO -  PROCESS | 19603 |
> 01:00:18     INFO -  PROCESS | 19603 | [#1] 163.com/www.163.com/index.html.CSSOM  Cycles:12  Average:3.45  Median:3.45  stddev:0.03 (0.9%)  stddev-sans-first:0.03
> 01:00:18     INFO -  PROCESS | 19603 | Values: 3.4  3.4  3.5  3.4  3.4  3.5  3.4  3.5  3.4  3.4  3.5  3.5
> 01:00:18     INFO -  PROCESS | 19603 |
> 01:00:18     INFO -  PROCESS | 19603 | [#2] 163.com/www.163.com/index.html.CSSOM.checkerboard  Cycles:12  Average:0.00  Median:0.00  stddev:0.00 (NaN%)  stddev-sans-first:0.00
> 01:00:18     INFO -  PROCESS | 19603 | Values: 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
> 01:00:18     INFO -  PROCESS | 19603 |
> 01:00:18     INFO -  PROCESS | 19603 | [#3] 56.com/www.56.com/index.html  Cycles:12  Average:7.36  Median:7.41  stddev:0.12 (1.6%)  stddev-sans-first:0.12
> 01:00:18     INFO -  PROCESS | 19603 | Values: 7.4  7.5  7.4  7.4  7.5  7.3  7.5  7.2  7.4  7.2  7.4  7.2
> ...


> Searchfor the PERFHERDER_DATA and run it through a linter.

Not really sure what this means, where should one search, and whom you're addressing.

If this is a problem with my modification of the test itself, I'd be happy to fix it, but as far as I can tell the new tests report new names correctly, but further along the line something is not behaving as expected.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

:rwood recently did some work to fix display names from pageloader for perf-reftest and quantum_pageload, lets ensure this is fixed as well.

Comment 6

[Robert Wood [:rwood]]

This has already been fixed, i.e. some sample output in production from today for tp5o_scroll:

06:30:47     INFO -  PID 3573 | |8;bbc.co.uk/www.bbc.co.uk/news/index.html;4.338599999999999;4.360700000000001;4.332499999999999;4.225600000000001;4.497649999999998;4.380299999999998;4.431900000000001;4.38425;4.12875;4.2772000000000014;4.31925;4.2573
06:30:47     INFO -  PID 3573 | |9;CSSOM.bbc.co.uk/www.bbc.co.uk/news/index.html;4.642787685645884;4.499395505317207;4.576210715206973;4.674289342092008;4.549008458852768;4.576909078657627;4.303875324325027;4.537331312894821;4.5968142446875575;4.422456133251007;4.532421394013891;4.449248156008812

And appears correctly in perfherder subtests also:

https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=mozilla-central&newProject=mozilla-inbound&newRevision=9bc216bce3c6cb524c3937a66fa923e1a8b0894e&originalSignature=0913f3d989fce87f5ef592453b86412101a1739b&newSignature=0913f3d989fce87f5ef592453b86412101a1739b&framework=1&selectedTimeRange=172800

And perhderder-data.json it looks good:

http://mozilla-releng-blobs.s3.amazonaws.com/blobs/Mozilla-Inbound-Non-PGO/sha512/7d85b1e2a68f696b34668365f6b269238218efa01fccd9f907ba3c5f9a5cfabdd7435ac5b0592937cf50a149ea14a0544a75f6dcf90cf55e30d2f53e6c716d36

However the *.checkerboard results are still not showing up; that's Bug 1279292