1416889 - [MDC1] Validate os and gfx driver for ubuntu16 on moonshots+xen.

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[:dhouse]

rwood/joel found that talos test results are inconsistent (see bug 
1416501). We need to be certain we have the graphics driver set up and exposes its settings to the graphics settings from the ui.

Comment 1

[:dhouse]

I've set up the the vm #265 on moon-chassis-6 with the same ubuntu16 image as on the 30 vm's QA is testing (266-270 on moon-chassis-6 and 276-300 on moon-chassis-7).

Comment 2

[:dhouse]

I'm testing:
1. HARDWARE? if we get consistent results on the iX hardware running Ubuntu 16.04.3, matching moonshot hardware (Testing an lts upgrade first, then I'll try a kickstart re-image). - talos-linux64-ix-085 (loaner for rwood. hijacked by permission)
2. OS_REL? if we can run Ubuntu 12.04, matching current iX hardware, on the moonshot hardware (network or other drivers may be an issue), and if so then:
2a. if we get the same inconsistent results. - t-linux64-xe-265 (loaner for dhouse)
3. GFX_DRV? if we can run the talos tests
3a. without passthrough gpu?
3b. with a new intel driver? (https://bugzilla.mozilla.org/show_bug.cgi?id=1359190#c11)

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

these are all great ideas- once we have a bit more data we will hopefully have an understanding of the root cause.  I am thinking it is the VM, but it could be drivers (or drivers+VM)

Comment 4

[:dhouse]

Test results so far:
1. HARDWARE? upgrade to 16.04.3 went without problems (12 -> 14 -> 16), but the worker failed on various python deps. I manually fixed a couple, but it appeared that the series of failures would continue and require more effort. So I stopped, and kicked off a new imaging at 16.04.3 from pxeboot/kickstart on the iX
2. OS_REL? re-imaging my moonshot loaner at 12.04 went okay, only a couple of puppet tweaks were needed because of no systemd and turning off iptables. but the worker hasn't picked up any jobs, and there is no /builds/slave/test for me to manually kick off the example test with
3. GFX_DRV?
3a. without passthrough gpu? I tried switching from a gpu worker to headless, but the same as with #2 I don't have the test setup /builds/slave/test
3b. with a new intel driver? I did not try yet.

Comment 5

[:dhouse]

Joel or Robert, how can I get the test setup? With a new machine (t-linux64-xe-265 and t-linux64-xe-275, and probably also talos-linux64-ix-085 when I'm finished imaging it), I have /builds/slave/talos-data but not /builds/slave/test. How do I set this up correctly?

I want to be able to:
```

cd /builds/slave/test

/tools/buildbot/bin/python scripts/scripts/talos_script.py --suite perf-reftest-singletons-e10s --add-option --webServer,localhost --branch-name Mozilla-Inbound-Non-PGO --cfg talos/linux_config.py --download-symbols ondemand --use-talos-json --blob-upload-branch Mozilla-Inbound-Non-PGO

```

Comment 6

[Robert Wood [:rwood]]

(In reply to Dave House [:dhouse] from comment #5)
> Joel or Robert, how can I get the test setup? With a new machine
> (t-linux64-xe-265 and t-linux64-xe-275, and probably also
> talos-linux64-ix-085 when I'm finished imaging it), I have
> /builds/slave/talos-data but not /builds/slave/test. How do I set this up
> correctly?
> 
> I want to be able to:
> ```
> 
> cd /builds/slave/test
> 
> /tools/buildbot/bin/python scripts/scripts/talos_script.py --suite
> perf-reftest-singletons-e10s --add-option --webServer,localhost
> --branch-name Mozilla-Inbound-Non-PGO --cfg talos/linux_config.py
> --download-symbols ondemand --use-talos-json --blob-upload-branch
> Mozilla-Inbound-Non-PGO
> 
> ```

On the new taskcluster XE hardware there's a 'workspace' folder, so I am able to just run this for example:

cd /home/cltbld

/home/cltbld/workspace/mozharness/scripts/talos_script.py --config-file /home/cltbld/workspace/mozharness/configs/talos/linux_config.py --no-read-buildbot-config --installer-url=https://queue.taskcluster.net/v1/task/XZx6HBLBSx2UXJ9abym-ag/artifacts/public/build/target.tar.bz2 --test-packages-url=https://queue.taskcluster.net/v1/task/XZx6HBLBSx2UXJ9abym-ag/artifacts/public/build/target.test_packages.json --suite=perf-reftest-singletons-e10s --add-option --webServer,localhost --use-talos-json --branch-name try --download-symbols=ondemand

On the existing buildbot IX hardware the command above is correct in /builds/slave/test. When I get the loaners those folders already exist - so I don't know how they are setup in the first place, sorry. :(

Comment 7

[Kendall Libby [:fubar] (he/him)]

Establishing base lines is useful, but I'd like to have a better understanding of exactly what the noisiest tests are doing under the hood, so that we can try and focus our troubleshooting, rather than making assumptions. For example, in 1416501#c8 there's some good data, but what is happening in the "bloom-basic.html" test? 

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=144bb1e5b3a6&newProject=try&newRevision=b76509fb7649a0089e3cf35e7e78c8c02c9c1fe1&framework=1

Let's focus on the most egregious outliers: "tp5o_webext XRes opt e10s", "glvideo Mean tick time across 100 ticks: opt e10s", "tp5o XRes opt e10s stylo_disabled"

There's also a bunch of tweaks that RelEng made on various OS in the past that we should look at and determine if they're applicable to Ubuntu 16.04: https://wiki.mozilla.org/ReferencePlatforms

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I see these commands needed to setup:
mkdir -p /home/cltbld/workspace
mkdir -p /home/cltbld/workspace/build/upload/logs
mkdir -p /home/cltbld/artifacts/public
mkdir -p /home/cltbld/workspace/build/blobber_upload_dir
curl --fail -o mozharness.zip --retry 10 -L https://queue.taskcluster.net/v1/task/XZx6HBLBSx2UXJ9abym-ag/artifacts/public/build/mozharness.zip
MOZHARNESS_PATH=/home/cltbld/workspace/mozharness
export MOZ_SOURCE_REPO=https://hg.mozilla.org/try
export MOZ_SOURCE_CHANGESET=b76509fb7649a0089e3cf35e7e78c8c02c9c1fe1
mkdir -p /home/cltbld/bin
cd /home/cltbld/workspace
python2.7 /home/cltbld/workspace/mozharness/scripts/talos_script.py  --config-file /home/cltbld/workspace/mozharness/configs/talos/linux_config.py --no-read-buildbot-config --installer-url=https://queue.taskcluster.net/v1/task/XZx6HBLBSx2UXJ9abym-ag/artifacts/public/build/target.tar.bz2 --test-packages-url=https://queue.taskcluster.net/v1/task/XZx6HBLBSx2UXJ9abym-ag/artifacts/public/build/target.test_packages.json --suite=chromez-e10s --add-option --webServer,localhost --add-option --webServer,localhost --use-talos-json --branch-name try --download-symbols=ondemand 


that should get you setup, there might be more to do afterwards.  as a note, this will use the revision of a try push I did:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&revision=b76509fb7649&selectedJob=142752812

that is from November 7th and has about 5 days left before expiring.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

for xres, that might be a value we stop recording in general- that is data from xrestop and it too noisy already on the IX machines.

:rwood, can you help sort out what bloom-basic is doing?

Comment 10

[Robert Wood [:rwood]]

bloom_basic is a part of perf-reftest-singletons (and perf-reftest); :bholley is the author so knows the details however this is from the wiki [1]:

"Bloom_basic ensures that we correctly fast-reject ancestor selectors that clearly don't match any ancestor elements. If the bloom filter isn't working correctly, the 'test' page will take orders of magnitude longer. The 'ref' page will take somewhat longer, but not nearly as much."

[1] https://wiki.mozilla.org/Buildbot/Talos/Tests#bloom_basic_singleton

Comment 11

[Kendall Libby [:fubar] (he/him)]

(In reply to Robert Wood [:rwood] from comment #10)
>
> [1] https://wiki.mozilla.org/Buildbot/Talos/Tests

Aaaah, there's the magic. Thank you.

Joel, on the tp5 tests you said in email that they're using http://localhost/. What's serving that data?

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

the test harness that runs the test launches a python webserver:
https://searchfox.org/mozilla-central/source/testing/talos/talos/run_tests.py#80

this is using mozhttpd:
https://searchfox.org/mozilla-central/source/testing/mozbase/mozhttpd/mozhttpd/mozhttpd.py

historically we had a static apache webserver on the IX machines, this made a local development environment much different, so we switched a few years ago to a python dynamic webserver.

The webserver does fileIO to read the source from the tree and then serves it to firefox via http://localhost:XXXX- so there is fileIO there depending on how Firefox caches data, etc.

We could also be having contention between the python webserver/harness (the harness does realtime log parsing as well) and firefox- the contention would be on CPU if I had to guess.

the CPU usage on the XE vs IX seems to be similar when looking at top, maybe we need to look at more samples, etc.

Comment 13

[Kendall Libby [:fubar] (he/him)]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #12)
> 
> The webserver does fileIO to read the source from the tree and then serves
> it to firefox via http://localhost:XXXX- so there is fileIO there depending
> on how Firefox caches data, etc.

Offhand, I'm leaning towards i/o and caching. The spiky timing results in 1416501#c8 feels like a cache warming up.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

yeah, lets see if removing the VM gives us more stable numbers- that would give us places to look.  I assume this is OS level caching, vs firefox caching- we are not hitting memory limits from what I can tell, so we are not paging out data from firefox- although I admit I am not an expert!

Comment 15

[:dhouse]

Here's where I"m at with the base testing I had started with (TODO: next to dig into the details as recommended :)).
talos-linux64-ix-085 - Ubuntu 16.04.3 to match moonshot hardware
t-linux64-xe-265 - Ubuntu 12.04 to match ix hardware
t-linux64-xe-271 - Ubuntu 16.04.3 with Xen (vCPU 6, highest priority)
t-linux64-xe-272 - Ubuntu 16.04.3 with Xen (vCPU 6, 3 sockets 2 cores each, highest priority)
t-linux64-xe-273 - Ubuntu 16.04.3 with Xen (gpu turned off for host, passthrough on)
t-linux64-xe-275 - Ubuntu 16.04.3 without the passthrough GPU attached

Comment 16

[Robert Wood [:rwood]]

(In reply to Dave House [:dhouse] from comment #15)
> Here's where I"m at with the base testing I had started with (TODO: next to
> dig into the details as recommended :)).
> talos-linux64-ix-085 - Ubuntu 16.04.3 to match moonshot hardware
> t-linux64-xe-265 - Ubuntu 12.04 to match ix hardware
> t-linux64-xe-271 - Ubuntu 16.04.3 with Xen (vCPU 6, highest priority)
> t-linux64-xe-272 - Ubuntu 16.04.3 with Xen (vCPU 6, 3 sockets 2 cores each,
> highest priority)
> t-linux64-xe-273 - Ubuntu 16.04.3 with Xen (gpu turned off for host,
> passthrough on)
> t-linux64-xe-275 - Ubuntu 16.04.3 without the passthrough GPU attached

Awesome! If you want any help connecting to these instances to get talos numbers just let me know.

Comment 17

[:dhouse]

With the gpu disabled, I get similar noisy results. So I think that rules out the gfx driver:
```t-linux64-xe-275 with gpu disabled
          "name": "bloom-basic.html",
          "replicates": [
            61.625,
            106.85000000000002,
            132.98000000000002,
            189.81500000000005,
            211.10500000000002,
            49.389999999999986,
            54.44999999999999,
            218.21999999999997,
            46.93000000000001,
            43.275000000000034,
            152.515,
            194.76,
            50.650000000000034,
            151.815,
            154.51500000000004
```

Running Ubuntu16 on ix hardware, we get low-noise. So that partly rules out the OS change:
```talos-linux64-ix-085
      "name": "perf_reftest_singletons",
      "subtests": [
        {
          "alertThreshold": 5.0,
          "lowerIsBetter": true,
          "name": "bloom-basic.html",
          "replicates": [
            90.595,
            88.64999999999999,
            85.83,
            99.86999999999999,
            86.545,
            100.23,
            91.985,
            91.9,
            86.57999999999998,
            87.36999999999999,
            87.69500000000002,
            91.85,
```

Comment 18

[:dhouse]

I see some larger variance on the ix in Ubuntu16, but not as significant as on the moonshot:
```talos-linux64-ix-085
      "name": "perf_reftest_singletons",
      "subtests": [
        {
          "alertThreshold": 5.0,
          "lowerIsBetter": true,
          "name": "bloom-basic.html",
          "replicates": [
            88.33999999999997,
            105.33500000000001,
            104.825,
            88.24500000000002,
            87.14499999999998,
            87.19000000000001,
            100.52999999999999,
            99.205,
            85.85499999999999,
            88.305,
            87.57999999999998,
            86.55499999999999,
            85.985,
            85.56,
            90.69000000000001
          ],
          "unit": "ms",
          "value": 87.38499999999999
```

Comment 19

[Kendall Libby [:fubar] (he/him)]

Can we gather more system metrics while tests are running, to try and narrow down the scope?  eg iostat, vmstat, etc.  The `top` output in bug 1416501 seems to indicate that cpu and memory are both fine, but it's really light on real data (esp I/O).

Comment 20

[:dhouse]

I'm looking at oprofile, and other recommended host-level performance monitoring for xen. I narrowed my testing down on Friday to four tests from the perf_reftest_singletons suite (bloom-basic, coalesce-1, tiny-traversal-singleton, nth-index-2). I've run many tests and only collected the talos metrics for those, or while watching the performance monitors on xencenter. 

There is not a significant load on the system during these tests. But the scheduling or i/o priority may be causing the noise despite not fighting for resources.

I've tested with stepping through cpu count, core topology, cpu scheduling weight, forcing dom0 to a higher priority, and with a xen-recommended tcp limit increase. I found only one variable that changes the variance/noise on the test results: running with a single cpu decreases the noise significantly. But once I run with two cpus we get the same noise. I did not see any other change in the noise with any change to the system tuning.

Comment 21

[:dhouse]

I'm seeing the noise even when I put in a tpdelay of multiple seconds between each test page load.

I also see the noise when I have pinned the host, dom0, to 2 cpus and the vm to the other 6 cpus.

I am able to reproduce the basic cpu performance variability with a bash loop in each vm I've tested on, and I get consistent (non-noisy) results from the host running the same test.

Comment 22

[Joel Maher ( :jmaher ) (UTC -8)]

are there different settings that Xen can use?  This is sounding like this is less of a driver issue or resource issue and more of a Xen VM layer issue.

Comment 23

[:dhouse]

The rtds scheduler gives lower noise on the tests that I have been reviewing (over multiple full runs of perf-reftest-singletons):
I tried out the credit2 and rtds schedulers to see if they give more consistent results. The experimental single-core arin653 hung on vm-start. Credit2 did not show any difference from credit. The rtds scheduler gave different results, and with tuning appears to give lower variance, often around 2% for the tests I had been focused on. I am seeing typically 30-50% noise under the credit schedulers (regardless of the weight settings and pinning/separating Dom0 from the vm).
```
[root@xenserver272 ~]# xl cpupool-list 
Name               CPUs   Sched     Active   Domain count
Pool-0               4      rtds       y          1
host                 4      rtds       y          1

xl sched-rtds -d LinuxVM -v all -p 2000 -b 1800
                          VCPUs-params (MRW): weight: 65535
                             VCPUs-max ( RW): 4
                      VCPUs-at-startup ( RW): 4

PID 7849 | ------- Summary: start -------
PID 7849 | Number of tests: 16
PID 7849 | 
PID 7849 | [#0] bloom-basic.html  Cycles:15  Average:325.73  Median:317.72  stddev:19.27 (6.1%)  stddev-sans-first:11.42
PID 7849 | Values: 382.9  320.9  316.7  313.7  352.7  316.1  321.5  313.1  314.7  313.8  317.9  317.4  326.3  340.7  317.7
PID 7849 | 
PID 7849 | [#1] bloom-basic-2.html  Cycles:15  Average:318.00  Median:316.45  stddev:5.21 (1.6%)  stddev-sans-first:5.30
PID 7849 | Values: 314.2  316.5  315.0  316.5  317.0  317.1  313.8  326.9  330.3  314.8  314.6  325.2  319.4  313.6  315.2
PID 7849 | 
PID 7849 | [#2] style-sharing.html  Cycles:15  Average:34.25  Median:33.28  stddev:3.00 (9.0%)  stddev-sans-first:3.09
PID 7849 | Values: 32.9  32.9  34.3  34.0  32.7  33.3  37.0  31.1  31.8  42.5  33.1  38.8  33.9  32.0  33.6
PID 7849 | 
PID 7849 | [#3] style-sharing-style-attr.html  Cycles:15  Average:34.39  Median:34.13  stddev:1.95 (5.7%)  stddev-sans-first:2.01
PID 7849 | Values: 35.2  36.4  35.8  33.6  31.2  32.2  37.0  32.7  36.4  35.5  37.2  33.3  32.5  32.7  34.1
PID 7849 | 
PID 7849 | [#4] display-none-1.html  Cycles:15  Average:0.35  Median:0.21  stddev:0.18 (84.0%)  stddev-sans-first:0.18
PID 7849 | Values: 0.4  0.2  0.5  0.6  0.6  0.6  0.4  0.2  0.4  0.2  0.2  0.2  0.2  0.2  0.2
PID 7849 | 
PID 7849 | [#5] only-children-1.html  Cycles:15  Average:0.39  Median:0.36  stddev:0.15 (40.8%)  stddev-sans-first:0.15
PID 7849 | Values: 0.4  0.3  0.3  0.3  0.6  0.4  0.2  0.4  0.3  0.4  0.4  0.7  0.2  0.6  0.3
PID 7849 | 
PID 7849 | [#6] slow-selector-1.html  Cycles:15  Average:0.28  Median:0.35  stddev:0.14 (41.1%)  stddev-sans-first:0.15
PID 7849 | Values: 0.3  0.4  0.4  0.2  0.2  0.2  0.1  0.2  0.4  0.3  0.3  0.6  0.4  0.2  0.2
PID 7849 | 
PID 7849 | [#7] slow-selector-2.html  Cycles:15  Average:0.28  Median:0.32  stddev:0.13 (40.5%)  stddev-sans-first:0.13
PID 7849 | Values: 0.2  0.4  0.2  0.4  0.2  0.4  0.3  0.3  0.3  0.6  0.2  0.2  0.1  0.3  0.1
PID 7849 | 
PID 7849 | [#8] style-attr-1.html  Cycles:15  Average:31.59  Median:31.49  stddev:0.81 (2.6%)  stddev-sans-first:0.84
PID 7849 | Values: 31.5  31.1  31.6  31.0  32.5  32.6  31.5  30.6  33.2  30.9  30.9  31.2  32.2  30.6  32.4
PID 7849 | 
PID 7849 | [#9] coalesce-1.html  Cycles:15  Average:1162.37  Median:1161.60  stddev:15.24 (1.3%)  stddev-sans-first:12.87
PID 7849 | Values: 1130.3  1162.6  1153.5  1199.8  1153.0  1164.3  1161.0  1169.5  1153.0  1161.6  1170.6  1152.9  1175.3  1154.6  1173.5
PID 7849 | 
PID 7849 | [#10] coalesce-2.html  Cycles:15  Average:977.08  Median:979.97  stddev:12.82 (1.3%)  stddev-sans-first:12.88
PID 7849 | Values: 988.6  955.0  985.2  963.7  991.2  980.0  994.0  974.6  986.2  981.8  970.2  971.5  957.1  991.7  965.5
PID 7849 | 
PID 7849 | [#11] parent-basic-singleton.html  Cycles:15  Average:1183.83  Median:1184.48  stddev:7.82 (0.7%)  stddev-sans-first:8.07
PID 7849 | Values: 1186.8  1183.7  1177.7  1179.9  1186.9  1184.5  1189.9  1192.5  1194.1  1169.3  1192.0  1177.4  1193.0  1175.0  1174.6
PID 7849 | 
PID 7849 | [#12] tiny-traversal-singleton.html  Cycles:15  Average:8168.33  Median:8142.33  stddev:113.13 (1.4%)  stddev-sans-first:117.16
PID 7849 | Values: 8142.3  8100.9  8232.4  8025.9  8350.6  8086.1  8154.0  8302.0  8321.7  8104.0  8213.2  8064.2  8321.3  8030.7  8075.5
PID 7849 | 
PID 7849 | [#13] nth-index-1.html  Cycles:15  Average:30.93  Median:30.97  stddev:0.47 (1.5%)  stddev-sans-first:0.49
PID 7849 | Values: 31.0  30.4  31.0  31.9  31.6  31.0  30.2  30.5  31.4  31.2  30.6  30.8  31.2  30.8  30.4
PID 7849 | 
PID 7849 | [#14] nth-index-2.html  Cycles:15  Average:27.44  Median:26.60  stddev:1.97 (7.4%)  stddev-sans-first:1.72
PID 7849 | Values: 31.3  25.8  27.0  26.0  26.7  26.5  28.7  26.3  32.3  26.6  26.6  26.0  26.2  28.6  27.1
PID 7849 | 
PID 7849 | [#15] bidi-resolution-1.html  Cycles:15  Average:2122.54  Median:2119.01  stddev:41.27 (1.9%)  stddev-sans-first:42.36
PID 7849 | Values: 2144.4  2124.7  2143.9  2147.3  2152.9  2119.0  2108.6  2084.8  2089.8  2067.2  2139.6  2091.3  2105.1  2084.7  2234.8
PID 7849 | -------- Summary: end --------

Comment 24

[Joel Maher ( :jmaher ) (UTC -8)]

if rtds gives us less noise, could we try that out at scale and measure the noise/stability over a few hundred jobs?

Comment 25

[:dhouse]

Joel, that sounds good to me. Can I make the change at any time? I'll change all of the QA moonshot pool to use the rtds scheduler. I will be particularly interested in seeing if this only reduces the noise on part of the tests.

I don't know the optimal rtds scheduler settings (time period and hardware budget (combined cpu/memory/io as a single resource)) but I think that will vary depending on the test. From the cpu-bound testing I've done, and running a subset from perf-reftest-singleton, I've seeing the lowest variance with 80-90% budget at low periods, 1-2 ms. But I have only run a scattering of tests. 

https://wiki.xenproject.org/wiki/Xen_Project_Schedulers

Comment 26

[Joel Maher ( :jmaher ) (UTC -8)]

I would go ahead an make the changes on all the existing machines- it is just :rwood and I using them via try and we are not using them so far today- happy to get tests running and look for patterns or more clarifying data.

Comment 27

[Robert Wood [:rwood]]

(In reply to Dave House [:dhouse] from comment #23)
...
> 
> PID 7849 | ------- Summary: start -------
> PID 7849 | Number of tests: 16
> PID 7849 | 
> PID 7849 | [#0] bloom-basic.html  Cycles:15  Average:325.73  Median:317.72 
> stddev:19.27 (6.1%)  stddev-sans-first:11.42
> PID 7849 | Values: 382.9  320.9  316.7  313.7  352.7  316.1  321.5  313.1 
> 314.7  313.8  317.9  317.4  326.3  340.7  317.7

Interesting - less noise but alot slower.. i.e. compare with bloom-basic results in comment 17

Comment 28

[:dhouse]

Attachment: test_xen_set.sh

Attaching some of my test scripts. Used like `./test_xen_set.sh 271` to run the series against xenserver271/t-linux64-xe-271. Expects the talos tests to be set up already.

Comment 29

[:dhouse]

Attachment: setup_talos_tests.sh

Script to configure a linux vm for testing. Pulls and patches the talos suite (to only run a subset of 5. Used like `for I in {270..273}; do ssh root@t-linux64-xe-${I}.test.releng.mdc1.mozilla.com 'bash -s' < setup_tests.sh; done`

Comment 30

[:dhouse]

Attachment: cpu_tests.tar.gz

Attaching some of my earlier test results with the standard scheduler (credit) and a single cpu pool shared with dom0 and other settings default; changing vcpu count, cores-per-socket, and scheduling weight on the vm.

Plotdata pulls from json files to review the standard deviation:
```
for I in {271..273}; do grep -o '[0-9]\+\] [^ ]\+.html.*stddev:[0-9\.]\+ ([0-9\.%]\+)' ~/logs/t-linux64-xe-${I}_*json|sed 's/.*_cp\([0-9]\)th\([0-9]\)co\([0-9]\)_\([0-9]\+\)_.*\([0-9]\+\)] \([^ ]\+\).html.*Average:\([0-9\.]\+\) .*stddev:\([^ ]\+\) (\([^)]\+\))$/\5 \6 \1 \2 \3 \4 \7 \8 \9/' > plotdata_${I}.dat; done
[...]
gnuplot> fn(n) = (n>128) ? ((n>256) ? 3 : 2 ) : 1; set linestyle 2 lt 1 lw 3 ; plot "plotdata_270.dat" using ($1 * 100 + ($3+$4+$5) * fn($6) * 2):7:8 with yerror linestyle 2;

```

This is the data showing that the stddev only decreased at 1 cpu and did not appear to have a relationship to the core count or having more cpus.

Comment 31

[:dhouse]

Attachment: 271_plot.png

Comment 32

[:dhouse]

Attachment: variance.sh

The basic script I've been using to test cpu performance variance.

Comment 33

[:dhouse]

I met with Jake and we are composing a test set for scheduler parameters to apply to the QA pool.

Comment 34

[:dhouse]

Attachment: wip cpupool_testing.sh

started work for creating the cpupools. Better to save it in the configuration. The `xl sched-rtds` setting can be done on the fly as it applies immediately to a running vm (in a pool).

One bug found is that the `xl sched-rtds` always displays the default values for period and quota (possibly tracked in https://lists.gt.net/xen/devel/425613). Changing the values changes the results. So I think this is just a bug in the printing of the parameter state.

Comment 35

[:dhouse]

Attachment: rt_test.sh

a script for testing cpu pool
to be used by rtfulltest.sh

Comment 36

[:dhouse]

Attachment: rtfulltest.sh

script to test over combinations of cpu,core,and rt(period, budget)

Comment 37

[:dhouse]

Attachment: rtfulltest.sh

Comment 38

[:dhouse]

Joel, I am finishing setting up the moonshot QA pool (t-linux64-xe- 266-270 and 276-300) with a range of Xen scheduler parameters. So i'm hoping that you or Robert can kick off a number of tests that cover the pool and then determine if there are better results in the mix.

--Just a heads-up that I was slow in getting the pool set up yesterday, but they are nearly ready.

Comment 39

[:dhouse]

Joel, they're ready:
I've updated the QA pool to have the following scheduling settings:
```
host, period, %budget, dom0cpus, vmcpus, cores, pool (0=shared), 2nd pool cpus
276, 10000, 40, 4, 4, 4, vm, 4
277, 10000, 40, 4, 4, 1, vm, 4
278, 10000, 40, 8, 8, 8, 0, 0
279, 10000, 40, 8, 8, 1, 0, 0
280, 10000, 40, 8, 4, 4, 0, 0
281, 10000, 40, 8, 4, 1, 0, 0
282, 10000, 40, 1, 7, 7, vm, 7
283, 10000, 40, 1, 7, 1, vm, 7
284, 10000, 40, 1, 7, 1, 0, 0
285, 10000, 40, 8, 8, 1, 0, 0
286, 5000, 40, 8, 8, 1, 0, 0
287, 2500, 40, 8, 8, 1, 0, 0
288, 1000, 40, 8, 8, 1, 0, 0
289, 10000, 40, 8, 8, 4, 0, 0
290, 5000, 40, 8, 8, 4, 0, 0
291, 2500, 40, 8, 8, 4, 0, 0
292, 1000, 40, 8, 8, 4, 0, 0
293, 10000, 40, 8, 8, 8, 0, 0
294, 5000, 40, 8, 8, 8, 0, 0
295, 2500, 40, 8, 8, 8, 0, 0
296, 1000, 40, 8, 8, 8, 0, 0
297, 1000, 20, 8, 8, 8, 0, 0
298, 2500, 20, 8, 8, 8, 0, 0
299, 5000, 20, 8, 8, 8, 0, 0
300, 10000, 20, 8, 8, 8, 0, 0
```
These vary the time period for the scheduler and the cpu/cores given to the vm. I have kept hosts 266-270 with defaults as a control group.

Most are in a shared pool with "dom0" (domain 0 is the Xen host). Most are also with 8 cpus for dom0. A few are set with separate cpu pools to test if separating dom0 gives more stability to the vm.

Comment 40

[:dhouse]

The "control group" settings are:
266, 10000, 40, 4, 6, 3, 0(credit not rtds), 0
267, 10000, 40, 4, 6, 3, 0(credit not rtds), 0
268, 10000, 40, 4, 6, 3, 0(credit not rtds), 0
269, 10000, 40, 4, 6, 3, 0(credit not rtds), 0
270, 10000, 40, 4, 6, 3, 0(rtds), 0

Comment 41

[:dhouse]

Attachment: qa_pool_setup.tgz

Comment 42

[Robert Wood [:rwood]]

(In reply to Dave House [:dhouse] from comment #39)
> Joel, they're ready:
> I've updated the QA pool to have the following scheduling settings:
...

Awesome! I'll start pushes to try

Comment 43

[Joel Maher ( :jmaher ) (UTC -8)]

for reference, original data:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=144bb1e5b3a6&newProject=try&newRevision=b76509fb7649a0089e3cf35e7e78c8c02c9c1fe1&framework=1&filter=singl

here is my try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b5ed3a2d5561744890295593d16e431bd3c8bfa7&filter-searchStr=other

I focused on perf-reftest-singleton and tp6, also did a bunch of 'other' retriggers, but those are crashing frequently in the a11y test.

here is the compare view comparing against itself:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=b5ed3a2d5561744890295593d16e431bd3c8bfa7&newProject=try&newRevision=b5ed3a2d5561744890295593d16e431bd3c8bfa7&framework=1&filter=singlet

the IX machines have 30.6 +- 0.7%
the control for singleton is 26 +- 3.0%

the other values fall into ~110 or ~127 (note, much slower than 26- including runtimes and timeouts).  As for the noise, we see the majority in 0.4 -> 1.0; a few outliers higher than that:
10000, 40, 8, 1, 0, 0
5000, 40, 8, 4, 0, 0

the configs with < 0.5% noise:
10000, 40, 1, 7, vm, 7
2500, 40, 8, 0, 0
5000, 40, 8, 0, 0 (0.39%)


lets look at tp6, we have 4 pages- amazon, google, facebook, youtube.

amazon:
ix: 727.77 ± 5.66% 
control: 1,258.59 ± 8.46%

configs post between 2900-3400 (again much slower), here are the noisiest configs:
5000, 40, 8, 0, 0       -  2,916.08 ± 3.85%
10000, 40, 8, 0, 0      -  2,963.75 ± 4.24% 
10000, 40, 1, 7, vm, 7  -  2,949.43 ± 4.20%
10000, 40, 1, 7, 0, 0   -  2,903.00 ± 3.90% 
1000, 40, 8, 4, 0, 0    -  2,949.00 ± 3.67%

and here are the lowest noise configs:
10000, 40, 8, 4, 1, 0, 0  -  3,243.25 ± 0.82%
2500, 40, 8, 1, 0, 0      -  2,990.44 ± 1.88%
10000, 40, 4, 1, vm, 4    -  3,418.17 ± 1.43%


facebook.com has been noisy on the ix machines, let see what we have here:
ix: 329.67 ± 6.41%
control: 462.50 ± 9.68%


noisiest configs:
10000, 40, 1, 7, vm, 7   -  2,474.36 ± 4.60% 
10000, 40, 8, 1, 0, 0    -  2,523.82 ± 4.25%
5000, 20, 8, 0, 0        -  2,456.33 ± 4.64%

quietest configs:
10000, 40, 8, 4, 1, 0, 0 -  2,825.25 ± 1.41% 
10000, 40, 8, 4, 0, 0    -  2,801.13 ± 1.50% 
10000, 40, 4, 1, vm, 4   -  2,836.17 ± 1.05%
10000, 40, 1, 7, 0, 0    -  2,552.61 ± 1.45%


these are good configs for tp6, but don't necessarily post good numbers for perf-reftest-singletons:
10000, 40, 8, 4, 1, 0, 0
10000, 40, 4, 1, vm, 4

More eyes would help for sure.

Comment 44

[:dhouse]

Thank you Joel for the data! Can we run the set of tests again to make sure the results are consistent?

Comment 45

[Joel Maher ( :jmaher ) (UTC -8)]

:dhouse, I have retriggered the jobs 300+ times, the results are pretty consistent, I can add a few hundred more retriggers if that would help.

Comment 46

[:dhouse]

Joel and Robert, I've updated the pool 266-270,276-299 to have a new set of Xen scheduler periods and quotas (https://docs.google.com/spreadsheets/d/1aZzt_rkvjFtb6FaHprNllpXkgl53g27_Ldbg5ozijVE/edit#gid=518031591). Please run some tests on these and we'll see if the larger period and quota gives better performance while still being less variant.

Comment 47

[Joel Maher ( :jmaher ) (UTC -8)]

data is in - 

try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f91b363a7f53c79d5d9ce2b76cb64c54a330a38d

compare view:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=f91b363a7f53c79d5d9ce2b76cb64c54a330a38d&newProject=try&newRevision=f91b363a7f53c79d5d9ce2b76cb64c54a330a38d&framework=1&filter=singleton


perf-reftest-singletons:
IX: 30.76 ± 0.68%
MDC-#1: 26.37 ± 2.96% (first experiment Nov 7th)
Control: 22.60 ± 1.67%
Unknown: 23.63 ± 3.83% (probably original Nov 7th machines)

276-281:
* values are lower 22-28, but noise is high: 2.6-5.5

282:
* value is 33 (ok), and noise is 7.3+ (insane!)

283-290:
* values are higher 40-60, noise is still high: 2-5

291-299:
* values are higher: 55-80, noise is low: <1 (except 293/294- noise is 3-4)



tp6 amazon:
IX: 727.77 ± 5.66%
MDC-#1: 1,226.28 ± 8.56% (first experiment Nov 7th)
Control: 1,130.61 ± 13.14%
Unknown: 1,264.68 ± 10.05% (probably original Nov 7th machines)

all values are between 1078-2600 and noise is between 1.44-23

Here is the machine that is the best for this test:
* t-linux64-xe-277: 1,079.15 ± 3.21% 

And the worse offenders:
* t-linux64-xe-289: 1,971.80 ± 12.12%
* t-linux64-xe-290: 2,013.75 ± 23.77%
* t-linux64-xe-296: 2,609.42 ± 17.59%


tp6 facebook:
IX: 329.67 ± 6.41%
MDC-#1: 462.50 ± 9.68% (first experiment Nov 7th)
Control: 439.12 ± 7.58%
Unknown: 469.97 ± 8.68% (probably original Nov 7th machines)

all values are between 432-1563 and noise is between 1-14

Here are the machines with the best data:
* t-linux64-xe-277: 432.50 ± 4.12%
* t-linux64-xe-278: 446.62 ± 4.40%

And the worse offenders:
* t-linux64-xe-287: 1,399.63 ± 14.28%
* t-linux64-xe-284: 844.63 ± 13.55% 


So t-linux64-xe-277 is our best bet- with probably a few contenders nearby.  I still want to see what bare metal gets us- one machine is all I ask :)

Comment 48

[:dhouse]

I'm setting up the pool with new Xen scheduler settings: lower schedule periods with high quotas.
This can complete the set to show us if high quotas give us higher performance in shorter periods also (or if the shorter periods slow things down too much).

Re: persistent ;) request for a bare metal test: I'm also going to do a poc on the side to run the tests in a docker container on the xenserver host (no network, so I'll be pushing the image and test data to it). This can give us a bare-metal test to verify we can get consistent results on the moonshot hardware.

Comment 49

[:dhouse]

Joel, the pool is set with the Dec1_lowPhighQ (https://docs.google.com/spreadsheets/d/1aZzt_rkvjFtb6FaHprNllpXkgl53g27_Ldbg5ozijVE/edit#gid=389821689) xen scheduler settings now and ready for another test run.

Comment 50

[Joel Maher ( :jmaher ) (UTC -8)]

I have been waiting fora build the last few hours:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9fe175e5fd9d28fdb7b210569d432300dfd31b45

so we don't confuse things, I will wait and retrigger- then update this bug over the weekend when I get some time to see the results and look over them.

Comment 51

[Joel Maher ( :jmaher ) (UTC -8)]

as a note, the singleton results are in:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=9fe175e5fd9d28fdb7b210569d432300dfd31b45&newProject=try&newRevision=9fe175e5fd9d28fdb7b210569d432300dfd31b45&framework=1&filter=singl

from earlier- perf-reftest-singleton:
IX: 30.76 ± 0.68%
MDC-#1: 26.37 ± 2.96% (first experiment Nov 7th)

some of the more reasonable machines:
t-linux64-xe-278: 54.66 ± 0.83%
t-linux64-xe-279: 59.83 ± 0.40% 
t-linux64-xe-284: 55.39 ± 0.62%
t-linux64-xe-290: 60.43 ± 0.50% 

these 4 machines have <2x runtime of IX and reasonable noise levels.  Still waiting on tp6 results.

Comment 52

[Joel Maher ( :jmaher ) (UTC -8)]

looking at tp6 amazon:
IX: 727.77 ± 5.66%
MDC-#1: 1,226.28 ± 8.56% (first experiment Nov 7th)

here are the best machines:
t-linux64-xe-281: 2,150.75 ± 0.35%
t-linux64-xe-280: 1,865.85 ± 5.35%
t-linux64-xe-286: 2,045.00 ± 3.79% 
t-linux64-xe-290: 2,007.36 ± 8.05%
t-linux64-xe-291: 1,857.88 ± 6.89% 

none of these results are <2x the IX machine and the noise is pretty high.


lets look at facebook:
IX: 329.67 ± 6.41%
MDC-#1: 462.50 ± 9.68% (first experiment Nov 7th)

here are the best machines:
t-linux64-xe-278: 1,173.08 ± 3.08%

that is 3x+ longer than the IX machine which one could argue is not acceptable

So far I don't see much here- possibly the data you see with the these results will help tweak results going forward.

Comment 53

[:dhouse]

So this looks like the lower period with high quota gave good results for perf-reftests-singleton but not as good as the longer periods (from Nov30) for tp6.

I'll set up the pool with period+quota ranging around the best (host#277) values from the Nov30 high Period tests ("Dec2_midP_highQ(277)" sheet from the shared spreadsheet, https://docs.google.com/spreadsheets/d/1aZzt_rkvjFtb6FaHprNllpXkgl53g27_Ldbg5ozijVE/edit#gid=971154523).

Comment 54

[:dhouse]

Joel, the pool is now set with xen scheduler period/quotas ranging around the settings from #277 on Thursday (period 500000, quota 100%). So from 200000 to 700000 ~90-100%.

Please kick off a new set of tests when you get a chance. I'm hoping this gives us a balance between the good results for perf-reftest-singleton's good results yesterday at a low period, high quota, and the tp6 good results from Thursday with higher periods.

Comment 55

[Joel Maher ( :jmaher ) (UTC -8)]

are these all the same config, or do I need to split them out by machine name still?

Comment 56

[:dhouse]

The hosts are different configs again (period and quotas listed on the spreadsheet here: https://docs.google.com/spreadsheets/d/1aZzt_rkvjFtb6FaHprNllpXkgl53g27_Ldbg5ozijVE/edit#gid=971154523) - 700000 down to 200000 with each at 100,98,96,92,88 % quota (no 88% for 200000).

Comment 57

[Joel Maher ( :jmaher ) (UTC -8)]

the data is here now (I did trigger all tests, so please read carefully if you do use data as we have stylo-disabled, etc.):
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=9fe175e5fd9d28fdb7b210569d432300dfd31b45&newProject=try&newRevision=9fe175e5fd9d28fdb7b210569d432300dfd31b45&framework=1&filter=singl

from earlier- perf-reftest-singleton:
IX: 30.76 ± 0.68%
MDC-#1: 26.37 ± 2.96% (first experiment Nov 7th)

all of this data is showing really good numbers and a mix of noise, here are the best machines:
t-linux64-xe-268: 22.31 ± 0.80% 
t-linux64-xe-277: 22.44 ± 1.18%
t-linux64-xe-287: 23.38 ± 0.90%
t-linux64-xe-290: 23.67 ± 1.04%
t-linux64-xe-294: 23.34 ± 1.09%

and the worse:
t-linux64-xe-267: 22.85 ± 3.20%
t-linux64-xe-274: 24.04 ± 3.93% 
t-linux64-xe-281: 23.71 ± 3.28%
t-linux64-xe-296: 30.64 ± 3.95%
t-linux64-xe-297: 25.56 ± 3.86%
t-linux64-xe-298: 25.12 ± 3.89%
t-linux64-xe-299: 27.28 ± 4.53% 



looking at tp6 amazon:
IX: 727.77 ± 5.66%
MDC-#1: 1,226.28 ± 8.56% (first experiment Nov 7th)

all of the data is higher than IX, but many hit the original experiment and lower noise, here are the best machines:
t-linux64-xe-267: 1,110.56 ± 3.96%
t-linux64-xe-269: 1,083.39 ± 4.78%
t-linux64-xe-278: 1,098.50 ± 2.01%
t-linux64-xe-280: 1,124.95 ± 4.85% 
t-linux64-xe-286: 1,116.39 ± 4.52%

and worse:
t-linux64-xe-266: 1,121.25 ± 10.28%
t-linux64-xe-281: 1,126.44 ± 10.45%
t-linux64-xe-288: 1,169.13 ± 11.18% 
t-linux64-xe-294: 1,201.83 ± 11.71% 
t-linux64-xe-295: 1,356.21 ± 21.40%
t-linux64-xe-296: 1,282.42 ± 15.80% 
t-linux64-xe-297: 1,280.83 ± 18.87%
t-linux64-xe-299: 1,367.38 ± 11.07% 


lets look at facebook:
IX: 329.67 ± 6.41%
MDC-#1: 462.50 ± 9.68% (first experiment Nov 7th)

and the best machines:
t-linux64-xe-266: 450.50 ± 5.31%
t-linux64-xe-269: 441.22 ± 3.45% 
t-linux64-xe-287: 436.85 ± 3.18% 
t-linux64-xe-289: 447.83 ± 4.79%

and worse:
t-linux64-xe-267: 440.11 ± 9.37%
t-linux64-xe-286: 444.17 ± 9.25% 
t-linux64-xe-293: 436.31 ± 9.49%
t-linux64-xe-295: 486.57 ± 14.78%
t-linux64-xe-298: 488.63 ± 9.51% 

I am not sure if this gives us enough data to make a full decision.  It does seem like machines 290+ are not worthwhile configs, that still leaves us with 24 other configs to choose from- possibly we need to look at a few other scenarios at a larger scale (i.e. all the tests).

It looks as though we are getting a bit closer- maybe a few more cycles and a few controlled experiments on our top 5 or 6 configs.

Comment 58

[Joel Maher ( :jmaher ) (UTC -8)]

we have all the machines running the same config as we have for -277 from the last big run.  I ran all the tests and this is what I see:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=067ce16f1775&newProject=try&newRevision=a87f167cbe275b3f999293d034a728f94e04ba2e&framework=1&filter=noise
 - As a note, I also did a fresh push for the IX machines so we can capture any test/number change

the noise is still 50%+ worse overall.

perf-reftest-singletons is showing same data, faster runtimes and lower noise, same results for tp6 amazon and tp6 facebook.

Looking at this from a point of view of which tests have >3% noise and that noise is 2x+ the noise of IX machines:
* a11y
* cpstartup
* displaylist_mutate
* glvideo
* kraken
* sessionrestore
* sessionrestore_many_windows
* sessionrestore_no_auto_restore
* tabpaint
* tart
* tcanvasmark
* xres (xrestop measurement is 7x higher on the new machines!)
* tp6 google
* tp6 youtube
* tpaint
* ts_paint
* tsvgr_opacity
* tsvgx

Of that 18, 5 are specifically startup related- glvideo is interesting- the numbers are 3x higher and the noise is 10x more.  This is a test for webgl video rendering (I am not sure exactly what it measures- I have asked the original author to document it better).  

Right now I am not excited about this config- a lot of tests are much slower and much noisier.  Should we try an experiment on various configs with glvideo and a startup test like cpstartup?

Comment 59

[:dhouse]

Let's do as you suggested: experiment with various configs with glvideo and cpstartup. Could you include tp6 for comparison?

What do you think for the configs? I'm thinking we can test ranging from 500ms to 800ms at 96% and 98% and add off the ends down to 4ms and up to 2s. I want to check those extremes to see if there are unexpected results for glvideo or cpstartup in those ranges where we already tested tp6/perf-reftest-singleton (https://docs.google.com/spreadsheets/d/1aZzt_rkvjFtb6FaHprNllpXkgl53g27_Ldbg5ozijVE/edit#gid=1630623628).

Comment 60

[Joel Maher ( :jmaher ) (UTC -8)]

on the latest round we are testing for glvideo and cpstartup.

CPSTARTUP:
IX: 262.35 ± 1.08% 

best machines:
t-linux64-xe-269: 355.25 ± 1.98%
t-linux64-xe-290: 358.00 ± 2.22% 

worse machines:
t-linux64-xe-296: 481.75 ± 14.26% 
t-linux64-xe-299: 614.83 ± 12.96%



GLVIDEO:
6.60 ± 0.50%

best machines:
t-linux64-xe-266: 29.23 ± 1.52%
t-linux64-xe-267: 29.09 ± 2.28% 
t-linux64-xe-281: 29.12 ± 2.25% 
t-linux64-xe-284: 29.35 ± 2.09% 
t-linux64-xe-286: 29.12 ± 2.26% 
t-linux64-xe-295: 29.62 ± 2.13% 

worse machines:
t-linux64-xe-296: 33.04 ± 9.39%
t-linux64-xe-297: 34.95 ± 9.16% 
t-linux64-xe-298: 39.35 ± 8.27% 


another test, display list mutate:
IX: 5,145.10 ± 1.64%

best:
t-linux64-xe-274: 3,662.88 ± 2.95%
t-linux64-xe-283: 4,146.78 ± 3.58%

worse:

t-linux64-xe-287: 4,148.28 ± 10.21% 
t-linux64-xe-292: 4,518.14 ± 11.00%
t-linux64-xe-293: 4,456.31 ± 10.75% 
t-linux64-xe-296: 4,976.43 ± 11.42%
t-linux64-xe-297: 5,882.09 ± 10.78% 
t-linux64-xe-298: 6,864.70 ± 11.81% 


I think this config set we need to drop 290-299, that is showing pretty crazy results- I recall those machines as bad last week as well.

This still doesn't give us a place to start- maybe 266 or 269 as a config to test all- I could gather more test data on these machines.  Possibly we need to tweak the OS installation instead of just the scheduler.

Do we have xen client software on the machines?  Can we turn off firewalls?  Is the graphics adapter proxies or shared with the host?

Comment 61

[Joel Maher ( :jmaher ) (UTC -8)]

here is the data for the latest round:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=b87969248e6ea43f417c5e65615dce012cc85e1c&newProject=try&newRevision=b87969248e6ea43f417c5e65615dce012cc85e1c&framework=1

Comment 62

[:dhouse]

Joel, how about we try all the tests at the setting for #266 in the last round (2seconds 96% quota).

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #60)
> I think this config set we need to drop 290-299, that is showing pretty
> crazy results- I recall those machines as bad last week as well.

I put low periods on those machines again. So I think that is why they're significantly worse again.

> This still doesn't give us a place to start- maybe 266 or 269 as a config to
> test all- I could gather more test data on these machines.  Possibly we need
> to tweak the OS installation instead of just the scheduler.

We can tweak the OS installation in addition to the scheduler, but I don't think the OS is causing the noise or performance difference from ix (since I ran the same setup as iX on a moonshot xen vm and saw the same noise).

> Do we have xen client software on the machines?  Can we turn off firewalls? 
> Is the graphics adapter proxies or shared with the host?

We have the xen client utils installed on these and I tested with the latest version of that yesterday.

I'll test with disabling the host-based firewall; We tested with turning it off weeks ago, and I don't remember any difference. But I'll test it to get some data to verify.

The graphics card is set as passthrough only and not shared with the host. However, I don't know if it has shared memory or dedicated everything for the gpu.

Comment 63

[Robert Wood [:rwood]]

Attachment: XE-300-about-support.txt

You've probably been over this already alot... but attaching about:support from the XE-300 loaner in case there's something that can help.

There's a lot of "WebGL Driver Extensions" listed. As well as GPU and more graphics info:

GPU #1
Active: Yes
Description: Intel Open Source Technology Center -- Mesa DRI Intel(R) Iris Pro Graphics P580 (Skylake GT4e)
Vendor ID: Intel Open Source Technology Center
Device ID: Mesa DRI Intel(R) Iris Pro Graphics P580 (Skylake GT4e)
Driver Version: 3.0 Mesa 12.0.6

HW_COMPOSITING:
blocked by default: Acceleration blocked by platform
OPENGL_COMPOSITING:
unavailable by default: Hardware compositing is disabled
WEBRENDER:
opt-in by default: WebRender is an opt-in feature
unavailable by runtime: Hardware compositing is disabled
OMTP:
disabled by default: Disabled by default

Comment 64

[Joel Maher ( :jmaher ) (UTC -8)]

running all the tests on the recent configs, I see this comparison:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=067ce16f1775&newProject=try&newRevision=ded00c40c84c5b57c8d4b550ccdee8d7db1319cd&framework=1

I am still waiting for many results, but the results that have come in are not looking any better.

Comment 65

[:dhouse]

I walked through the issue with a Citrix Xenserver team member this afternoon. We reviewed the stats while running the perf-reftest-singleton suite. He checked the disk usage, and free memory and suggested we try with more memory allocated to dom0. We tested with 4gb for dom0, and no change in variance was visible (still around 37% from my bash tests and 113% bloom-basic at 8vcpus, 8gb vm memory). We applied the patches to #300 to verify if that changed the state. We also tested with 8gb memory for dom0 and that appeared to reduce the variance (3.7% on perf-reftest-singleton) but at 10x slower performance. I am confirming all the patches are applied and retesting with high dom0 memory.

Comment 66

[Joel Maher ( :jmaher ) (UTC -8)]

this sounds promising :dhouse.  I have data from the recent config (matching 266 from before):
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=067ce16f1775&newProject=try&newRevision=23288b31915627b51a64c482d106621055b06182&framework=1

I noticed the overall times are not too bad, but the noise is bad, I did some work locally to remove the 'outliers' and the noise went from 75 down to 52 (which is the noise we see on AWS).  Our IX machines have noise of 35 (if I remove 'outliers' it is 31.5).

outliers are defined as tests which are >2 stddev away from the median- I remove them and recalculate the data.

I am still baffled as to why the xrestop is showing almost 10x more in the VM than the IX machines- that might be an area to focus on.

Comment 67

[:dhouse]

Jake got a bare-metal machine set up: t-linux64-xe-264.

Joel, please add #264 to your pool and kick off tests to make sure we get the expected good low-noise results from there. I'll be interested in seeing if xrestop and glvideo tests are as similarly slow as under xen. I think they could be slow because of the graphics card being slower, and this could confirm that.

I ran some of the talos tests on there for initial comparison:
```
PID 4031 | [#0] bloom-basic.html  Cycles:20  Average:40.04  Median:39.53  stddev:2.00 (5.1%)  stddev-sans-first:2.05
PID 4554 | [#0] bloom-basic.html  Cycles:20  Average:40.63  Median:39.62  stddev:2.46 (6.2%)  stddev-sans-first:2.53
PID 5068 | [#0] bloom-basic.html  Cycles:20  Average:39.22  Median:39.19  stddev:0.51 (1.3%)  stddev-sans-first:0.52
PID 8206 | [#0] bloom-basic.html  Cycles:20  Average:40.36  Median:39.73  stddev:2.32 (5.8%)  stddev-sans-first:2.38
PID 4031 | [#1] coalesce-1.html  Cycles:20  Average:209.40  Median:208.98  stddev:3.37 (1.6%)  stddev-sans-first:3.40
PID 4554 | [#1] coalesce-1.html  Cycles:20  Average:207.73  Median:208.25  stddev:2.18 (1.0%)  stddev-sans-first:1.98
PID 5068 | [#1] coalesce-1.html  Cycles:20  Average:211.62  Median:211.67  stddev:2.33 (1.1%)  stddev-sans-first:2.24
PID 8206 | [#1] coalesce-1.html  Cycles:20  Average:203.81  Median:204.24  stddev:1.76 (0.9%)  stddev-sans-first:1.79
PID 4031 | [#2] tiny-traversal-singleton.html  Cycles:20  Average:1462.17  Median:1463.88  stddev:10.75 (0.7%)  stddev-sans-first:10.57
PID 4554 | [#2] tiny-traversal-singleton.html  Cycles:20  Average:1452.54  Median:1456.65  stddev:13.97 (1.0%)  stddev-sans-first:14.32
PID 5068 | [#2] tiny-traversal-singleton.html  Cycles:20  Average:1426.15  Median:1426.17  stddev:15.33 (1.1%)  stddev-sans-first:14.01
PID 8206 | [#2] tiny-traversal-singleton.html  Cycles:20  Average:1425.20  Median:1425.61  stddev:15.56 (1.1%)  stddev-sans-first:13.05
PID 4031 | [#3] nth-index-2.html  Cycles:20  Average:4.94  Median:4.88  stddev:0.21 (4.3%)  stddev-sans-first:0.22
PID 4554 | [#3] nth-index-2.html  Cycles:20  Average:4.95  Median:4.89  stddev:0.23 (4.6%)  stddev-sans-first:0.23
PID 5068 | [#3] nth-index-2.html  Cycles:20  Average:4.83  Median:4.81  stddev:0.13 (2.7%)  stddev-sans-first:0.13
PID 8206 | [#3] nth-index-2.html  Cycles:20  Average:4.86  Median:4.82  stddev:0.17 (3.4%)  stddev-sans-first:0.17
```

Same set of tests on #300 (matches latest config for Xen scheduler: 600ms period on rtds with 96% quota, 8vcpus):
```
PID 5266 | [#0] bloom-basic.html  Cycles:20  Average:122.06  Median:179.63  stddev:53.61 (29.8%)  stddev-sans-first:53.47
PID 5767 | [#0] bloom-basic.html  Cycles:20  Average:112.83  Median:205.58  stddev:63.25 (30.8%)  stddev-sans-first:58.72
PID 6271 | [#0] bloom-basic.html  Cycles:20  Average:114.93  Median:196.62  stddev:62.68 (31.9%)  stddev-sans-first:64.03
PID 6795 | [#0] bloom-basic.html  Cycles:20  Average:86.75  Median:40.26  stddev:57.12 (141.9%)  stddev-sans-first:58.34
PID 5266 | [#1] coalesce-1.html  Cycles:20  Average:283.02  Median:260.35  stddev:88.89 (34.1%)  stddev-sans-first:90.93
PID 5767 | [#1] coalesce-1.html  Cycles:20  Average:284.31  Median:259.01  stddev:65.76 (25.4%)  stddev-sans-first:64.84
PID 6271 | [#1] coalesce-1.html  Cycles:20  Average:260.49  Median:245.26  stddev:51.42 (21.0%)  stddev-sans-first:52.37
PID 6795 | [#1] coalesce-1.html  Cycles:20  Average:253.68  Median:243.66  stddev:49.77 (20.4%)  stddev-sans-first:50.70
PID 5266 | [#2] tiny-traversal-singleton.html  Cycles:20  Average:1767.61  Median:1785.88  stddev:116.91 (6.5%)  stddev-sans-first:117.59
PID 5767 | [#2] tiny-traversal-singleton.html  Cycles:20  Average:1740.74  Median:1715.70  stddev:110.22 (6.4%)  stddev-sans-first:111.87
PID 6271 | [#2] tiny-traversal-singleton.html  Cycles:20  Average:1792.51  Median:1848.89  stddev:165.99 (9.0%)  stddev-sans-first:167.58
PID 6795 | [#2] tiny-traversal-singleton.html  Cycles:20  Average:1734.12  Median:1754.63  stddev:126.46 (7.2%)  stddev-sans-first:123.69
PID 5266 | [#3] nth-index-2.html  Cycles:20  Average:4.82  Median:4.68  stddev:0.70 (14.9%)  stddev-sans-first:0.71
PID 5767 | [#3] nth-index-2.html  Cycles:20  Average:4.81  Median:4.36  stddev:1.05 (24.2%)  stddev-sans-first:1.08
PID 6271 | [#3] nth-index-2.html  Cycles:20  Average:4.56  Median:4.41  stddev:0.35 (8.0%)  stddev-sans-first:0.32
PID 6795 | [#3] nth-index-2.html  Cycles:20  Average:4.75  Median:4.44  stddev:0.62 (13.9%)  stddev-sans-first:0.63
```

Comment 68

[Joel Maher ( :jmaher ) (UTC -8)]

my try run is still going, I don't see t-linux64-xe-264 as an option:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=5ac9ec6be7c7668278ee495ea4367bd06de0fb0e&newProject=try&newRevision=5ac9ec6be7c7668278ee495ea4367bd06de0fb0e&framework=1&filter=linux64-xe

is it possible this doesn't have a taskcluster worker up and running?  Maybe we can log in and run a few tests manually?  Or we could turn off the other machines and run some tests via try on this.

Comment 69

[:dhouse]

Hi Joel, it was because I rebooted last night and it ended up going into the installer again. I'm bringing it up now and I'll verify that the taskcluster worker is running.

Comment 70

[Robert Wood [:rwood]]

Something interesting I spotted in the /var/logs/syslog on loaner 300, no idea if it's worthwhile but just in case:

Dec  8 13:33:41 t-linux64-xe-300.test.releng.mdc1.mozilla.com su[3138]: 08/12/2017 13:33:41 Wireframing: -wireframe mode is in effect for window moves.
Dec  8 13:33:41 t-linux64-xe-300.test.releng.mdc1.mozilla.com su[3138]: 08/12/2017 13:33:41   If this yields undesired behavior (poor response, painting
Dec  8 13:33:41 t-linux64-xe-300.test.releng.mdc1.mozilla.com su[3138]: 08/12/2017 13:33:41   errors, etc) it may be disabled:
Dec  8 13:33:41 t-linux64-xe-300.test.releng.mdc1.mozilla.com su[3138]: 08/12/2017 13:33:41    - use '-nowf' to disable wireframing completely.
Dec  8 13:33:41 t-linux64-xe-300.test.releng.mdc1.mozilla.com su[3138]: 08/12/2017 13:33:41    - use '-nowcr' to disable the Copy Rectangle after the
Dec  8 13:33:41 t-linux64-xe-300.test.releng.mdc1.mozilla.com su[3138]: 08/12/2017 13:33:41      moved window is released in the new position.
Dec  8 13:33:41 t-linux64-xe-300.test.releng.mdc1.mozilla.com su[3138]: 08/12/2017 13:33:41   Also see the -help entry for tuning parameters.

Comment 71

[Robert Wood [:rwood]]

Also these errors seem to appear often in the syslog, even while running talos tests, fyi in case they mean anything:

Dec  8 16:47:08 t-linux64-xe-300.test.releng.mdc1.mozilla.com com.canonical.Unity.Scope.LocalFiles[1398]: (unity-files-daemon:3660): unity-files-daemon-WARNING **: daemon.vala:491: Error performing global search 't': GDBus.Error:org.freedesktop.DBus.Error.UnknownMethod: No such interface 'org.gnome.zeitgeist.Index' on object at path /org/gnome/zeitgeist/index/activity
Dec  8 16:47:08 t-linux64-xe-300.test.releng.mdc1.mozilla.com kernel: [92527.778581] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#3channel#0 (csrow:3 channel:0 page:0x0 offset:0x0 grain:8)
Dec  8 16:47:08 t-linux64-xe-300.test.releng.mdc1.mozilla.com kernel: [92527.778596] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#3channel#1 (csrow:3 channel:1 page:0x0 offset:0x0 grain:8)
Dec  8 16:47:08 t-linux64-xe-300.test.releng.mdc1.mozilla.com com.canonical.Unity.Scope.LocalFiles[1398]: (unity-files-daemon:3660): unity-files-daemon-WARNING **: daemon.vala:491: Error performing global search 'te': GDBus.Error:org.freedesktop.DBus.Error.UnknownMethod: No such interface 'org.gnome.zeitgeist.Index' on object at path /org/gnome/zeitgeist/index/activity
Dec  8 16:47:09 t-linux64-xe-300.test.releng.mdc1.mozilla.com com.canonical.Unity.Scope.LocalFiles[1398]: (unity-files-daemon:3660): unity-files-daemon-WARNING **: daemon.vala:491: Error performing global search 'ter': GDBus.Error:org.freedesktop.DBus.Error.UnknownMethod: No such interface 'org.gnome.zeitgeist.Index' on object at path /org/gnome/zeitgeist/index/activity
Dec  8 16:47:09 t-linux64-xe-300.test.releng.mdc1.mozilla.com com.canonical.Unity.Scope.LocalFiles[1398]: (unity-files-daemon:3660): unity-files-daemon-WARNING **: daemon.vala:491: Error performing global search 'term': GDBus.Error:org.freedesktop.DBus.Error.UnknownMethod: No such interface 'org.gnome.zeitgeist.Index' on object at path /org/gnome/zeitgeist/index/activity

Comment 72

[Joel Maher ( :jmaher ) (UTC -8)]

:dhouse, can we close this bug out?

Comment 73

[:dhouse]

I'll close this one also. I am continuing to work with Citrix on trying to achieve less variance on a Xen vm, but we don't need this bug for that.