Closed Bug 1239301 Opened 5 years ago Closed 5 years ago

Green up accessability Linux64 debug tests under docker

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(firefox46 fixed)

RESOLVED FIXED
mozilla46
Tracking Status
firefox46 --- fixed

People

(Reporter: armenzg, Unassigned)

References

Details

Attachments

(1 file)

We want to run Linux64 debug test jobs under docker.
We're getting closer to it, however, we now have to figure test issues.
In this bug we would like to look at fixing mochitest-other.
There are a lot of a11y test failures [1] which we would need assitance on how to proceed to get them green.

There's a screenshot of the environment [2] but I can't spot anything out of order.

dbolter: would you be able to have a quick look at this bug and see who could help us figure out how to move forward? I will be at the office today if you would like to go over it together.

Here are the steps to reproduce locally:
# First time prep steps
sudo apt-get install v4l2loopback-dkms
# Every time after your host is rebooted
sudo modprobe v4l2loopback # This will create a device under /dev/video*

docker run -ti \
  -e GECKO_HEAD_REPOSITORY='https://hg.mozilla.org/try/' \
  -e GECKO_HEAD_REV='1f1ac2c7d13832b4434fd5040b2219dddaa212d4' \
  -e MOZHARNESS_CONFIG='mozharness/configs/unittests/linux_unittest.py mozharness/configs/remove_executables.py
' \
  -e MOZHARNESS_SCRIPT='mozharness/scripts/desktop_unittest.py' \
  -e MOZHARNESS_URL='https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/mozharness.zip' \
  -e MOZILLA_BUILD_URL='https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/target.tar.bz2' \
  -e NEED_PULSEAUDIO='true' \
  -e NEED_WINDOW_MANAGER='true' \
  -e START_VNC='true' \
  -e SKIP_MOZHARNESS_RUN='true' \
  --device /dev/video1:/dev/video1 # <-- Normally the last one under /dev/video*
  armenzg/desktop-test:0.5.5

# Now inside of docker
./bin/test.sh
# You can now connect to VNC on your host (xtightvncviewer 172.17.0.2)
cd workspace
sudo -E -u worker python2.7 /home/worker/workspace/mozharness/scripts/desktop_unittest.py --config-file mozharness/configs/unittests/linux_unittest.py --config-file mozharness/configs/remove_executables.py --no-read-buildbot-config --installer-url=https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/target.tar.bz2 --test-packages-url=https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/test_packages.json --download-symbols=ondemand --mochitest-suite=chrome,a11y --total-chunk=1 --this-chunk=1

Remember you can run mozharness with --no-run-tests and then do this:
source /home/worker/workspace/build/venv/bin/activate
sudo -E -u worker  /home/worker/workspace/build/venv/bin/python -u /home/worker/workspace/build/tests/mochitest/runtests.py --total-chunks 1 --this-chunk 1 --appname=/home/worker/workspace/build/application/firefox/firefox --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/target.crashreporter-symbols.zip --certificate-path=tests/certs --setpref=webgl.force-enabled=true --quiet --log-raw=/home/worker/workspace/build/blobber_upload_dir/chrome_raw.log --log-errorsummary=/home/worker/workspace/build/blobber_upload_dir/chrome_errorsummary.log --use-test-media-devices --screenshot-on-fail --chrome

Running with --no-run-tests once and then activating the venv is the preferred way as it won't have to start everything from scratch.

If you close your VNC connection, you will have to call ./bin/test.sh again (maybe just call `x11vnc &` alone).

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=1f1ac2c7d138&selectedJob=15368035
https://public-artifacts.taskcluster.net/IGF3dIgoQUyoM70EBdVQ2A/0/public/logs/live_backing.log
[2] https://public-artifacts.taskcluster.net/IGF3dIgoQUyoM70EBdVQ2A/0/public/test_info//mozilla-test-fail-screenshot_NDespk.png
referencing the log from [1] in comment 0, here are some of the first errors from the log file:
23:11:58     INFO -  502 INFO TEST-PASS | chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html | getTextAtOffset (line start): wrong end offset(got '2', expected: '2'), offset: 1, id:  'li2' ;
23:11:58     INFO -  503 INFO TEST-PASS | chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html | getTextAtOffset (line start): wrong text (got '• ', expected: '• '), offset: 2, id:  'li2' ;
23:11:58     INFO -  504 INFO TEST-PASS | chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html | getTextAtOffset (line start): wrong start offset(got '0', expected: '0'), offset: 2, id:  'li2' ;
23:11:58     INFO -  505 INFO TEST-PASS | chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html | getTextAtOffset (line start): wrong end offset(got '2', expected: '2'), offset: 2, id:  'li2' ;
23:11:58     INFO -  506 INFO TEST-UNEXPECTED-FAIL | chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html | getTextAtOffset (line start): wrong text (got '• a long and ', expected: '• a long '), offset: 0, id:  'li3' ;
23:11:58     INFO -  testTextHelper@chrome://mochitests/content/a11y/accessible/tests/mochitest/text.js:596:5
23:11:58     INFO -  testTextSuperHelper@chrome://mochitests/content/a11y/accessible/tests/mochitest/text.js:526:11
23:11:58     INFO -  testTextAtOffset@chrome://mochitests/content/a11y/accessible/tests/mochitest/text.js:141:3
23:11:58     INFO -  doTest@chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html:129:7
23:11:58     INFO -  setTimeout handler*SimpleTest_setTimeoutShim@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:622:12
23:11:58     INFO -  addA11yLoadEvent/waitForDocLoad/<@chrome://mochitests/content/a11y/accessible/tests/mochitest/common.js:134:9
23:11:58     INFO -  setTimeout handler*SimpleTest_setTimeoutShim@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:622:12
23:11:58     INFO -  waitForDocLoad@chrome://mochitests/content/a11y/accessible/tests/mochitest/common.js:124:5
23:11:58     INFO -  SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:735:59
23:11:58     INFO -  507 INFO TEST-PASS | chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html | getTextAtOffset (line start): wrong start offset(got '0', expected: '0'), offset: 0, id:  'li3' ;
23:11:58     INFO -  Not taking screenshot here: see the one that was previously logged
23:11:58     INFO -  508 INFO TEST-UNEXPECTED-FAIL | chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html | getTextAtOffset (line start): wrong end offset(got '13', expected: '9'), offset: 0, id:  'li3' ;
23:11:58     INFO -  testTextHelper@chrome://mochitests/content/a11y/accessible/tests/mochitest/text.js:604:5
23:11:58     INFO -  testTextSuperHelper@chrome://mochitests/content/a11y/accessible/tests/mochitest/text.js:526:11
23:11:58     INFO -  testTextAtOffset@chrome://mochitests/content/a11y/accessible/tests/mochitest/text.js:141:3
23:11:58     INFO -  doTest@chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html:129:7
23:11:58     INFO -  setTimeout handler*SimpleTest_setTimeoutShim@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:622:12
23:11:58     INFO -  addA11yLoadEvent/waitForDocLoad/<@chrome://mochitests/content/a11y/accessible/tests/mochitest/common.js:134:9
23:11:58     INFO -  setTimeout handler*SimpleTest_setTimeoutShim@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:622:12
23:11:58     INFO -  waitForDocLoad@chrome://mochitests/content/a11y/accessible/tests/mochitest/common.js:124:5
23:11:58     INFO -  SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:735:59
23:11:58     INFO -  Not taking screenshot here: see the one that was previously logged
23:11:58     INFO -  509 INFO TEST-UNEXPECTED-FAIL | chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html | getTextAtOffset (line start): wrong text (got '• a long and ', expected: '• a long '), offset: 1, id:  'li3' ;
23:11:58     INFO -  testTextHelper@chrome://mochitests/content/a11y/accessible/tests/mochitest/text.js:596:5
23:11:58     INFO -  testTextSuperHelper@chrome://mochitests/content/a11y/accessible/tests/mochitest/text.js:526:11
23:11:58     INFO -  testTextAtOffset@chrome://mochitests/content/a11y/accessible/tests/mochitest/text.js:141:3
23:11:58     INFO -  doTest@chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html:129:7
23:11:58     INFO -  setTimeout handler*SimpleTest_setTimeoutShim@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:622:12
23:11:58     INFO -  addA11yLoadEvent/waitForDocLoad/<@chrome://mochitests/content/a11y/accessible/tests/mochitest/common.js:134:9
23:11:58     INFO -  setTimeout handler*SimpleTest_setTimeoutShim@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:622:12
23:11:58     INFO -  waitForDocLoad@chrome://mochitests/content/a11y/accessible/tests/mochitest/common.js:124:5
23:11:58     INFO -  SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:735:59
23:11:58     INFO -  510 INFO TEST-PASS | chrome://mochitests/content/a11y/accessible/tests/mochitest/text/test_lineboundary.html | getTextAtOffset (line start): wrong start offset(got '0', expected: '0'), offset: 1, id:  'li3' ;
23:11:58     INFO -  Not taking screenshot here: see the one that was previously logged



reading the test-pass messages at the top of the snippet indicates "TEST-PASS | ... wrong text ...", which looks identical the the TEST-UNEXPECTED-FAIL messages.

keep in mind that this is a docker container with linux64 created a bit by hand.  Maybe there is a package we need to install or a setting we need to tweak to make a11y tests work reliably.  Do ask for help, we would love to see these running green sooner than later!
Haven't looked deeply here and nothing immediately comes to mind. Trevor any ideas or hunches?
Flags: needinfo?(tbsaunde+mozbugs)
I only looked at the first failure, but are the browser windows the same size on the new and old test configs?  It seems like the issue might be the new ones are somewhat wider.
Flags: needinfo?(tbsaunde+mozbugs)
I can run the tests on the releng machines versus inside of docker and see if there are any window differences IIUC.
I can reproduce the issue outside of docker on my local host (Ubutun 14.04 LTS).
I think docker and my local host are reporting the issues correctly rather the releng production machines.
I think the issue lays on the releng machines by having something funky with the fonts being used (the fonts look more coarse than on docker or my local machine).
Look at the font rendering on the left (releng host) and the fonts on the right (docker image)
http://people.mozilla.org/~armenzg/sattap/f129ac5a.png

The screen resolutions are the same in both cases.

Steps to run on docker [1]
Steps to run on localhost [2]
Steps to run on releng loaner [3]

You can load the second screenshot of [4] and [5] side by side to see how the rendering is different.

dbolter, tbsaunde: where do we go from here?
I'm afraid that trying to fix the releng loaner will go very badly as it could affect every suite on every tree.


[1] 
Use steps on comment 0 to get started and then run this to run the test and keep it open (--keep-open true):
/home/worker/workspace/build/venv/bin/python -u /home/worker/workspace/build/tests/mochitest/runtests.py --appname=/home/worker/workspace/build/application/firefox/firefox --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/target.crashreporter-symbols.zip --certificate-path=tests/certs --setpref=webgl.force-enabled=true --quiet --log-raw=/home/worker/workspace/build/blobber_upload_dir/chrome_raw.log --log-errorsummary=/home/worker/workspace/build/blobber_upload_dir/chrome_errorsummary.log --use-test-media-devices --screenshot-on-fail --a11y --keep-open true /home/worker/workspace/build/tests/mochitest/a11y/accessible/tests/mochitest/text/test_lineboundary.html



[2]
Running locally on my *host* (not docker):
curl --fail -o mozharness.zip --retry 10 -L https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/mozharness.zip
unzip mozharness.zip
# Some modifications on the Mozharness arguments to run on my local host
cd mozharness
# Apply this patch to fix a regression -- https://hg.mozilla.org/integration/mozilla-inbound/rev/349d3d8a41d9
python scripts/desktop_unittest.py --config-file configs/unittests/linux_unittest.py --no-read-buildbot-config --installer-url=https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/target.tar.bz2 --test-packages-url=https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/test_packages.json --download-symbols=ondemand --mochitest-suite=a11y --no-run-tests --cfg developer_config.py
source build/venv/bin/activate
cd build
python -u `pwd`/tests/mochitest/runtests.py --appname=`pwd`/application/firefox/firefox --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/target.crashreporter-symbols.zip --certificate-path=tests/certs --setpref=webgl.force-enabled=true --quiet --log-raw=`pwd`/blobber_upload_dir/chrome_raw.log --log-errorsummary=`pwd`/blobber_upload_dir/chrome_errorsummary.log --use-test-media-devices --screenshot-on-fail --a11y --keep-open true `pwd`/tests/mochitest/a11y/accessible/tests/mochitest/text/test_lineboundary.html

[3]
Running it on a releng machine:
curl --fail -o mozharness.zip --retry 10 -L https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/mozharness.zip
unzip mozharness.zip
# Some modifications on the Mozharness arguments to run on my local host
cd mozharness
# I don't use developer_config.py
python scripts/desktop_unittest.py --config-file configs/unittests/linux_unittest.py --no-read-buildbot-config --installer-url=https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/target.tar.bz2 --test-packages-url=https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/test_packages.json --download-symbols=ondemand --mochitest-suite=a11y --no-run-tests
source build/venv/bin/activate
cd build
python -u `pwd`/tests/mochitest/runtests.py --appname=`pwd`/application/firefox/firefox --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=https://queue.taskcluster.net/v1/task/JarpTXRFT3CJe3YakW3nZw/artifacts/public/build/target.crashreporter-symbols.zip --certificate-path=tests/certs --setpref=webgl.force-enabled=true --quiet --log-raw=`pwd`/blobber_upload_dir/chrome_raw.log --log-errorsummary=`pwd`/blobber_upload_dir/chrome_errorsummary.log --use-test-media-devices --screenshot-on-fail --a11y --keep-open true `pwd`/tests/mochitest/a11y/accessible/tests/mochitest/text/test_lineboundary.html

[4]
Screenshots of the opened window on docker and my local host:
http://people.mozilla.org/~armenzg/sattap/4dacfe74.png
http://people.mozilla.org/~armenzg/sattap/2a5254f8.png

[5]
Screenshots from releng host
http://people.mozilla.org/~armenzg/sattap/55cca797.png
http://people.mozilla.org/~armenzg/sattap/3a86e837.png
More clearly seen in this screenshot:
http://people.mozilla.org/~armenzg/sattap/63243994.png

Notice the lines breaking in different positions.

If we could skip the tests on the releng hosts we could fix the tests for docker (or local hosts).

Here's also a link to the test file and the various people whom have touched it:
https://hg.mozilla.org/mozilla-central/annotate/878033e3ca98726f11dd3a30b0768961d221d22a/accessible/tests/mochitest/text/test_lineboundary.html
interesting find!  this might justify what Trevor said about window size.  Our iframes are set at the same size for the testing area- maybe this is a font rendering issue or some display difference.
It is not a window size issue afaict. I could be missing something.
:jfkthame, can you give us some of your expert guidance on the font difference here:
http://people.mozilla.org/~armenzg/sattap/63243994.png

if we just need to force a specific font/fallback/option to match these up it would be nice.
Flags: needinfo?(jfkthame)
The screenshots there show that the two systems have different font rendering options in effect: the right-hand image has subpixel antialiasing, whereas the left has no smoothing/antialiasing at all. In general, you probably want to fix this so as to be testing a more consistent configuration. This is probably a system-level font preference; there may be a GUI settings tool to adjust it (on my Ubuntu running XFCE, there's Settings / Appearance / Fonts which allows me to turn antialiasing on/off), or else it can be configured through fontconfig's setup (:karlt could probably give you the correct incantations).

But that's not the real issue here; the important difference isn't how the fonts are rendered, but that the testcase isn't line-breaking as intended.

The reason for this is that the testcase is using inappropriate CSS units at this point:

    <li id="li8" style="width:10ex; font-family:monospace; font-size:10pt;">a long and winding road that lead me to your door</li>

The testcase clearly expects the <li> here to get a specific set of line-breaks, and it sets a monospaced font to try and ensure that, avoiding font-specific differences in character widths in proportional fonts. The problem is the use of "ex" as the unit of width. The size of "ex" does not bear any specific relationship to the width of the characters in the font; see http://www.w3.org/TR/css3-values/#ex. So here, we end up with a line width that is dependent on the x-height of the monospaced font, and this may well vary between platforms / default fonts.

Fortunately, there's a simple fix: use the "ch" unit, which is defined in terms of character width (specifically, the width of "0", but as we're using a monospaced font...) and will give a line width that more reliably fits a known number of characters, resulting in predictable line breaks. It looks like replacing "10ex" with "8ch" here should give the desired result.
Flags: needinfo?(jfkthame)
I have verified that this patch (applied by manually hacking the test on my docker instance) passes where before the patch it would fail for me!

So far the 'oth' jobs look green in the try push, just waiting on windows.

:jfkthame, thanks so much for not only dropping hints about what to do, but actually explaining a lot of this and providing a patch!
I've modified ~/.fonts.conf to match what the releng production hosts are using.
Making that change makes the font look the same and tests pass.
We should land both the test fix plus matching our image to the releng hosts.
I'm hoping that my next push disabling anti-aliasing will show many more suites go green (hopefully reftests).

Once we don't run the tests on releng hosts anymore we can go back and enable anti-aliasing to match the standard set up that our users use.

If the plan does not work for you, please let me know. I will file a bug for post-transition to undo this change.
Flags: needinfo?(armenzg)
all the 'oth' jobs are green now!
As a reference, here's the patch to disable anti-aliasing:
https://hg.mozilla.org/try/rev/3e922419e51f
Removing NI as I've filed the future bug to undo the disabling of the anti-aliasing if people believe this is important to put resources into it.
Flags: needinfo?(armenzg)
> But that's not the real issue here; the important difference isn't how the
> fonts are rendered, but that the testcase isn't line-breaking as intended.
> 
> The reason for this is that the testcase is using inappropriate CSS units at
> this point:
> 
>     <li id="li8" style="width:10ex; font-family:monospace;
> font-size:10pt;">a long and winding road that lead me to your door</li>
> 
> The testcase clearly expects the <li> here to get a specific set of
> line-breaks, and it sets a monospaced font to try and ensure that, avoiding

yeah

> font-specific differences in character widths in proportional fonts. The
> problem is the use of "ex" as the unit of width. The size of "ex" does not
> bear any specific relationship to the width of the characters in the font;
> see http://www.w3.org/TR/css3-values/#ex. So here, we end up with a line
> width that is dependent on the x-height of the monospaced font, and this may
> well vary between platforms / default fonts.
> 
> Fortunately, there's a simple fix: use the "ch" unit, which is defined in
> terms of character width (specifically, the width of "0", but as we're using
> a monospaced font...) and will give a line width that more reliably fits a
> known number of characters, resulting in predictable line breaks. It looks
> like replacing "10ex" with "8ch" here should give the desired result.

You know much more aboutthat than me so I'll just trust you on this being the right way to get the linewrapping to work.
Attachment #8708264 - Flags: review?(tbsaunde+mozbugs) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/d0c07c27b85b3a5103364cf4c7230ee32c452ef1
Bug 1239301 - Use 'ch' units rather than 'ex' units when specifying a width that is intended to fit a specific number of characters. r=tbsaunde
https://hg.mozilla.org/mozilla-central/rev/d0c07c27b85b
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla46
You need to log in before you can comment on or make changes to this bug.