1281241 - run some if not all linux unittests tests on m3.medium instances, consider upgrading llvmpipe/mesa as well

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

let the good times roll!

We have a problem where many tests are experiencing failures- it looks like this is an issue with the hardware or the software in the container running the tests.

To solve this we will try:
* running unittests on try with m3.medium instances (instead of m1.medium) so we have more than 1 core available to the docker container and theoretically the tests
* upgrading mesa/llvmpipe in the docker image

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

first try push is up here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8b8b07888730

ideally before we go live, I would like to see us run 50 instances of any job and see no more than 3 jobs reported as orange, ideally less than 3.  lets see if this stuff runs as expected, we can retrigger jobs to see trends.

Comment 2

[Armen [:armenzg]]

I'm trying to understand the context.

Did we disable a bunch of tests to make things work on m1.medium?
Where are you experiencing failures?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

m3.medium is >1 core and it had a larger percentage of test failures (iirc 5% more failures) than m1.medium.  I cannot find the spreadsheet right now.  Jeff has a use case, possibly he can outline some more specific failures he is seeing.

Comment 4

[Armen [:armenzg]]

Would we want to narrow this down to a number of suites? (the ones Jeff cares about)

Comment 5

[Jeff Muizelaar [:jrmuizel]]

(In reply to Joel Maher (:jmaher) from comment #3)
> m3.medium is >1 core and it had a larger percentage of test failures (iirc
> 5% more failures) than m1.medium.  I cannot find the spreadsheet right now. 
> Jeff has a use case, possibly he can outline some more specific failures he
> is seeing.

m3.medium is still 1 core. However, m3.medium is faster and cheaper than m1.medium. We wanted the extra performance and newer cpu instructions to reduce the likelihood of some timeouts during test runs when using llvmpipe.

Comment 6

[Jeff Muizelaar [:jrmuizel]]

These jobs all failed with:
 [taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"Yw8NRCthSbK5HC_6tJTIMA"} has failed. This may indicate an error with the registry, image name, or an authentication error. Try pulling the image locally to ensure image exists. ENOSPC, write

Comment 7

[Greg Arndt [:garndt]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #6)
> These jobs all failed with:
>  [taskcluster:error] Pulling docker image
> {"path":"public/image.tar","type":"task-image","taskId":
> "Yw8NRCthSbK5HC_6tJTIMA"} has failed. This may indicate an error with the
> registry, image name, or an authentication error. Try pulling the image
> locally to ensure image exists. ENOSPC, write

Unfortunately sometimes this happens where docker dies when importing an image.  Usually retriggering the task solves the issue.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

odd, I am not able to figure out how to get this going.  Using the taskcluster/desktop-test image (the latest two tags from hub.docker.com), I see failures where we use fail to download test-linux.sh as it is in a hardcoded path and that has since moved:
curl --fail -o ./test-linux.sh --retry 10 https://hg.mozilla.org/try//raw-file/a1c987825a8ffd17026792296f58583cc95011cc/testing/taskcluster/scripts/tester/test-linux.sh

^ testing/taskcluster -> taskcluster/ci/legacy

So it appears we need the magic generated image.tar instead.  We have dozens of failed jobs with the same error message, I don't think this is an intermittent failure.

:garndt, I could use your help to solve this.  you can see the changes I made to run this on try;
https://hg.mozilla.org/try/rev/8b8b0788873027aa5f1116fbd4e8403d7678445a

possibly there is something else on the host os or tc-worker that we need to setup properly before running jobs on there?

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Joel, the location of the script is coded here:
https://dxr.mozilla.org/mozilla-central/source/testing/docker/desktop-test/bin/test.sh#32

What we would need is a new bug with a fix for the new location of the script and get this pushed to inbound. The desktop-test task will automatically be triggered.

Also do you run your tests from in-tree or external? If in-tree you shouldn't have to create the docker image but could reference its last task id.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

I had tried to reference a static image on hub.docker.com/taskcluster/desktop-test, but those images are out of date by months and do not include recent changes.  I don't know the process for updating those, and it is a ~6 hour cycle for me to update docker images if I were to test on my own.

the dynamically generated docker images (image.tar) uses the correct location for test-linux.sh- possibly I just need some education on how to use the image.tar from in-tree auto generated images.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I will walk through with Joel on IRC. Lets see if we can get this managed to work before garndt comes online.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It's mystic. Something is clearly not working and when I look at the live log I would say it's a malformed download of the docker image. To test this I was using the same task id and triggered a test for our firefox-ui-functional tests via my external script in mozmill-ci. Interestingly this task doesn't seem to fail in extracting the docker archive:

https://tools.taskcluster.net/task-inspector/#NjzHovwMSwSnw66fzxClKA/0

Also I wonder why it takes that long until tasks for medium worker are getting started. Even if no other test is running and the queue is empty you have to wait up to 18min before the task gets started. Maybe its the special hardware specs which are different to desktop test workers? The workers for our firefox-ui-tests take about 4-5 min to spin up.

https://tools.taskcluster.net/task-inspector/#G1VdyT1cSwqIzIH4ZBZNUA/0

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

ok, doing a fresh push, seemed to resolve this:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c42f80fbc92b38ea45f8d601494375b4943d460f

now to do another push with all the mochitests:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a3e72eee9c87

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

ok, we hit a roadblock here, the failure to extract the image is because we run out of disk space, we have 4GB total on m3.medium- and get this, to get more disk we need to go with 2 vCPU.

Jeff, do you have a preference on m3.large vs c3.large?  c3.large has half the RAM, but a slightly faster processor.  Possibly we could restrict docker to use only 1 cpu, although that would not be ideal.  Keep in mind that some of our tests (like web-platform-tests) already run on larger instances, I think c3.xlarge, so we could try that out without getting any custom AMI images setup.

Comment 15

[Jeff Muizelaar [:jrmuizel]]

2 vCPU is even better for us. My preference would be c4.large, c3.large, m3.large.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

running on m3.large:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b8b1440a46186b5e5cf6e0ce87247bc5cbe5ca57&filter-searchStr=tc

Jeff, If there are certain jobs/tests that you want to look at, please retrigger away.

This weekend if all is well so far, I will ensure I have enough clients and do 50 retriggers on each job, that will give us enough data to ensure we are not introducing noisier jobs/tests.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

it appears that retriggers are not working on taskcluster jobs right now, so I cannot determine if these failed tests are perma fail or not.  I assume this will get fixed today.

Jeff, please give me some direction here to know what issues to look for.

Comment 18

[Greg Arndt [:garndt]]

Joel, I left a comment on the other bug about retriggers, but it seems that you can retrigger them from the staging site so that you're unblocked.

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

as a note, that is:
https://treeherder.allizom.org/#/jobs?repo=try&revision=b8b1440a46186b5e5cf6e0ce87247bc5cbe5ca57&filter-searchStr=tc

Comment 20

[Jeff Muizelaar [:jrmuizel]]

The jobs results don't look too bad. I'm going to have Andrew investigate to see if we can't make things more green (especially bc2)

Comment 21

[Andrew Comminos [:acomminos]]

Most of these seem not too terrible. bc3 on e10s appears to be permafailing due to perf-related changes.

For the intermittents, many of the crashes appear to be happening within native cairo;

https://public-artifacts.taskcluster.net/emzV5D06SIe6iXYZpp4Ukw/0/public/logs/live_backing.log
https://public-artifacts.taskcluster.net/YmFWEXy_SL6fomYoTJzRnA/0/public/logs/live_backing.log
https://public-artifacts.taskcluster.net/Igf_QOQVTza4aNuKg92akg/0/public/logs/live_backing.log

Unfortunately, we don't have frame pointers, but the calls are almost certainly being made from within GTK. Going to see if I can reproduce on an old build of gtk+cairo.

Comment 22

[Joel Maher ( :jmaher ) (UTC -8)]

I would say we have a list of tests to fix, here they are by job type:
linux64 opt:
* mochitest-3: dom/events/test/test_bug659071.html, dom/filesystem/tests/test_basic.html
* a11y: all kinds of badness
* bc4: toolkit/modules/tests/browser/browser_FinderHighlighter.js
* bc7: oddness in browser/base/content/test/general/*
* c3: layout/xul/test/test_windowminmaxsize.xul 
* mochitest-e10s-3: dom/html/test/test_fullscreen-api.html 
* mochitest-e10s-5: dom/push/test/test_serviceworker_lifetime.html
* mochitest-e10s-8: gfx/layers/apz/test/mochitest/test_group_touchevents.html
* mochitest-e10s-10: toolkit/components/extensions/test/mochitest/test_ext_notifications.html
* bc-e10s-1: toolkit/modules/tests/browser/browser_FinderHighlighter.js
* bc-e10s-3: toolkit/components/perfmonitoring/tests/browser/browser_compartments.js
* bc-e10s-5: browser/components/privatebrowsing/test/browser/*
* bc-e10s-6: browser/components/sessionstore/test/*
* bc-e10s-7: oddness in browser/base/content/test/general/*

linux64 debug:
* mochitest-10: toolkit/components/prompts/test/test_subresources_prompts.html
* c3: toolkit/content/tests/chrome/test_popup_anchoratrect.xul (this fails frequently on m1.medium)
* mochitest-e10s-3: dom/html/test/test_fullscreen-api.html
* mochitest-e10s-7: leak in /tests/dom/workers/test/serviceworkers

keep in mind that this could be partially related to the base revision I pushed with, the tree and tests change all the time- overall I think most of these are valid and unique.  While they might exist as *known* failures, there is a good chance they are happening much more frequently.

luckily for these mochitests there are <15 issues to sort out, a few look hard though.

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

just checking in here, are there any updates?

Comment 24

[(not currently active) Ted Mielczarek]

(In reply to Andrew Comminos [:acomminos] from comment #21)
> Most of these seem not too terrible. bc3 on e10s appears to be permafailing
> due to perf-related changes.
> 
> For the intermittents, many of the crashes appear to be happening within
> native cairo;

We should be able to get debug symbols for these libraries. Someone just needs to grab a loaner test instance and follow these steps:
https://bugzilla.mozilla.org/show_bug.cgi?id=528231#c30

(If someone else does that work, you can give me the symbols.zip and I'll upload it.)

Comment 25

[Andrew Comminos [:acomminos]]

(In reply to Joel Maher (:jmaher: pto- back july 7th) from comment #23)
> just checking in here, are there any updates?

I believe we're going to go ahead with this- I'll be starting on fixing the failures in the near future, once my backlog is clear.

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #24)
> (In reply to Andrew Comminos [:acomminos] from comment #21)
> > Most of these seem not too terrible. bc3 on e10s appears to be permafailing
> > due to perf-related changes.
> > 
> > For the intermittents, many of the crashes appear to be happening within
> > native cairo;
> 
> We should be able to get debug symbols for these libraries. Someone just
> needs to grab a loaner test instance and follow these steps:
> https://bugzilla.mozilla.org/show_bug.cgi?id=528231#c30
> 
> (If someone else does that work, you can give me the symbols.zip and I'll
> upload it.)

I've asked for a loaner to symbolicate, thanks!

Comment 26

[Andrew Comminos [:acomminos]]

Bug 1285561 appears to have fixed up the a11y crashes; not sure if other tests were affected.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1777b364e978

Comment 27

[Joel Maher ( :jmaher ) (UTC -8)]

there was a nice list of tests in comment 22, do you think most of those are fixed?  should we be pushing and collecting a large volume of data to see where we stand?

Comment 28

[Andrew Comminos [:acomminos]]

Not quite yet I don't think- my recent push (https://treeherder.mozilla.org/#/jobs?repo=try&revision=1777b364e978) suggests that many of the intermittents documented in comment 22 are still valid, save for a11y (and some other nonspecific oranges). I'm going to continue investigating these, particularly high-volume failures such as dom/html/test/test_fullscreen-api.html.

Comment 29

[Joel Maher ( :jmaher ) (UTC -8)]

oh great, maybe in a short while this will be ready to go live :)

Comment 30

[Andrew Comminos [:acomminos]]

Most of the high volume intermittents (above 30%) should be fixed now;

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b8d2a3e1e038&filter-searchStr=tc-m

There are still some more frequently occurring failures- these seem to be quite hard to track down.

Since our primary purpose of using dual-core instances is to run llvmpipe faster for GL composition, jrmuizel and I were discussing potentially running a tier-2 "tc-gl-M-(e10s)" set of tests. An additional benefit of this would be ensuring that we still test the basic composition path on Linux. These tests would run on the m3.large instances with layers.acceleration.force-enabled set to true.

Since failure rate is saner now, what do you think Joel?

Comment 31

[Joel Maher ( :jmaher ) (UTC -8)]

we could switch just the 'gl' jobs to m3.medium for now, and eventually move other jobs.  The e10s tests look more problematic, and add debug and it gets even messier.

in fact, everything but browser-chrome (bc*), mochitest-media (mda), and mochitest-plain could be ported over given the results of the try push from comment 30.  Maybe doing that, seeing how it sorts out and reassessing a week later to get a list of tests to clean up for the remaining jobs would be a good route to go.  Ideally we can get to 100% on the m3.mediums.

to fix this we would need to modify:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/desktop-test/tests.yml

and for each test we care about add:
instance-size: large

for example, webgl:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/desktop-test/tests.yml#291

Comment 32

[Andrew Comminos [:acomminos]]

Attachment: Bug 1281241 - Use large desktop-test instances by default on TaskCluster.

Review commit: https://reviewboard.mozilla.org/r/72376/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/72376/

Comment 33

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8782149 [details]
Bug 1281241 - Use large desktop-test instances by default on TaskCluster.

https://reviewboard.mozilla.org/r/72376/#review69984

<p>to confirm, this will run all tests on the new instance type except for mochitest-plain and mochitest-browser-chrome?  This means, marionette, cppunit, jittests, mochitest-other, mochitest-devtools, reftest, crashtest, web-platform-tests, etc. will all run on the new instance type.  I do like the usage of 'legacy'.</p>
<p>Lastly, do we have a current list of bugs or tests which are holding us back from making mochitest-plain and browser-chrome from running on desktop-test-large ?</p>

Comment 34

[Andrew Comminos [:acomminos]]

Yup, everything except for plain and browser-chrome mochitest suites.

I'm currently working on making this bug depend on the remaining intermittents.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dd82326944a4
Use large desktop-test instances by default on TaskCluster. r=jmaher

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f5ed7f38160e
followup - Use legacy taskcluster instances for XPCShell and ASAN devtools. r=philor

Comment 37

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/dd82326944a4
https://hg.mozilla.org/mozilla-central/rev/f5ed7f38160e

Comment 38

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I assume this change also affected our firefox-ui-tests which use desktop-test with the Ubuntu 16.04 docker images? I ask because since this patch landed ALL of our intermittent test failures are gone! No single one re-appeared! This is freaking cool!

Would we have to make a further change to allow our qa-3-linux-fx-tests workers to use the same?

Comment 39

[Joel Maher ( :jmaher ) (UTC -8)]

I am not sure what the qa-3-linux-fx-tests workers are, but this change should be easy to test out on try- look at the changesets and try it out on the try server.

that is exciting that changing the worker type to a multi core machine solved a lot of the intermittents!

Comment 40

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Joel Maher ( :jmaher) from comment #39)
> I am not sure what the qa-3-linux-fx-tests workers are, but this change
> should be easy to test out on try- look at the changesets and try it out on
> the try server.

Those are workers we use for the firefox-ui-tests as triggered by mozmill-ci. Basically they use the desktop-test docker image too. From recent fxfn jobs I can see worker types of desktop-test-large:

https://queue.taskcluster.net/v1/task/Bvdxx6KZQ625cBEdIfDmXA

So it means that some TC admins would have to update our workers? I don't see a possibility to do so via a task definition.

> that is exciting that changing the worker type to a multi core machine
> solved a lot of the intermittents!

Not only a lot, but all for us! Since Friday we haven't had a single intermittent failure for fx-ui-tests anymore!

Comment 41

[Joel Maher ( :jmaher ) (UTC -8)]

not clear what information is needed from me

Comment 42

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sorry, I actually wanted to ni? dustin. Dustin, can you please have a look at comment 40? If it warrants a new bug I can file one. Thanks.

Comment 43

[Dustin J. Mitchell [:dustin] (he/him)]

There are two options, really -- we can update the `qa-3-linux-fx-tests` workerType, or just use the desktop-test-large workerType.  The latter makes more sense, as there's not much reason to segregate these tasks into their own workerType.

In fact, that appears to be the case aleady -- I don't see any indication of a different workerType here:

https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/desktop-test/tests.yml#59

I also don't see "linux-fx-tests" anywhere in dxr.  I think we set up that workerType earlier, when you were scheduling this out-of-tree, and now that it's in-tree you're already completely migrated.  So as far as the firefox-ui-tests, you've already won :)

Comment 44

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It's hard to keep this conversation about fx ui tests in two different bugs (see also bug 1296547). In short we setup a different worker type initially by your request, so we also can get our own queue. If we start running tests for Nightly builds for all locales, I don't want to blow up the desktop-test-large queue that heavily. Shall we continue here or on bug 1296547?

Comment 45

[Dustin J. Mitchell [:dustin] (he/him)]

Here is good.

If we're not talking about 10,000 tasks, then there's no real worry about blowing up that queue -- it regularly has over 2000 instances running.  I think it's better to use one workerType than to run the same tasks on different workerTypes between nightly and on-commit.

Comment 46

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Dustin, I created the following Github issue to start this conversion:
https://github.com/mozilla/mozmill-ci/issues/812. Please have a look. Thanks.

Comment 47

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Just to note here, the desktop-test-large worker type has drastically improved the download of the docker image compared to the old desktop-test worker. Maybe its not for all machines but in the following case we had only about 3min compared to ~20min before!

https://tools.taskcluster.net/task-inspector/#WW_mn3EqTZKyRGfZHdRV1g/0

Comment 48

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: move some tests from m1.medium to m3.large (default instance type)

this will leave two remaining tests (out of 5 originally):
* browser-chrome
* asan devtools

here is a try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=79642bc50a9a8628564638c853c71147c39d53df

browser-chrome could move if we address bug 1395539 and bug 1384879.  Possibly there are other hurdles for browser-chrome tests with more retriggers, etc.

I haven't tested devtools on asan yet.

Comment 49

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d1df54aa8b21
move mochitest-plain, screenshots, and xpcshell off m1.medium. r=gbrown

Comment 50

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/d1df54aa8b21

Comment 51

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.