963885 - Disable mozdevice tests

Status: RESOLVED INVALID

(Testing :: Mozbase, defect)

Description

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=33508666&tree=Mozilla-Inbound is... hellishly awful.

devicemanager hits failures; that's the nature of its job, do stuff on totally unreliable hardware. Because we expect it to fail, and we want to rerun the job on a less awful slave, buildbot parses logs for "Remote Device Error" and "devicemanager.DMError", and when they are found sets the job status to RETRY so that the job will be rerun.

In the case of that log, we spent a bit of money to create a Linux PGO build, uploaded it, triggered every flavor of unittest and talos test on it, and then ran make check on it. The mozdevice tests hit an intermittent failure, which included "Remote Device Error" in its output, so rather than turning the job orange and highlighting the failure message so the intermittent could be filed, starred, and fixed, we spent a bit of money to create *another* Linux PGO build, uploaded it, triggered a whole new set of every flavor of unittest and talos test on it... every single intermittent devicemanager test failure costs us a second build, a second mochitest-1, a second mochitest-n, a second reftest, a second tsvg, etc.

We very much need to STOP THAT, and only after having stopped it figure out what we can do instead to allow the mozdevice tests to run in make check.

Comment 1

[Phil Ringnalda (:philor)]

Disabled with the same sledgehammer that bug 885784 used, in https://hg.mozilla.org/mozilla-central/rev/9e06d42c2a6a

Comment 2

[Phil Ringnalda (:philor)]

Bug 885784 says it disabled them in Gecko 24, have to look at whether they also got reenabled in 24 and are thus retry-prone on esr24, since bug 886540 fails to mention when it hit m-c.

Comment 3

[Phil Ringnalda (:philor)]

Oh, m-c isn't the canonical repo, so someone will have to upstream that, won't they?

Comment 4

[William Lachance (:wlach)]

Didn't we talk about making the mozbase unit tests their own job at some point?

Comment 5

[(not currently active) Ted Mielczarek]

I think it was mentioned, but I don't recall having any serious discussion about it.

Comment 6

[Andrew Halberstadt [:ahal]]

Yeah, I think we agreed it would be nice.. though I'm not sure who's going to find time to actually get it done.

Couldn't we also just modify buildbot/mozharness to only retry test jobs and leave builds orange?

Comment 7

[Andrew Halberstadt [:ahal]]

Also, I'm pretty sure the DM tests just use a mock agent, so if there are intermittents, it probably isn't due to flaky hardware, but something needs fixing in the tests themselves.

Comment 8

[William Lachance (:wlach)]

(In reply to Andrew Halberstadt [:ahal] from comment #7)
> Also, I'm pretty sure the DM tests just use a mock agent, so if there are
> intermittents, it probably isn't due to flaky hardware, but something needs
> fixing in the tests themselves.

Hard to say for sure, but I suspect at this point it's just that the timeouts in the sut tests are too short. The socket timeout is only 5 seconds for the sut tests, which probably isn't long enough given all the crazy things that can happen on production VMs.

Comment 9

[Dan Minor [:dminor]]

I can take a look at running mozbase unit tests from the test package. Filed Bug 966441 to track this.

Comment 10

[Phil Ringnalda (:philor)]

(In reply to Phil Ringnalda (:philor) from comment #3)
> Oh, m-c isn't the canonical repo, so someone will have to upstream that,
> won't they?

Or, instead, someone could just clobber my disabling when they merge from github for the last time, and cause us to go back to running these tests, and go back to retriggering an entire build when a single test intermittently fails, and go back to running two sets of every single test suite when a single test intermittently fails.

Comment 11

[William Lachance (:wlach)]

(In reply to Phil Ringnalda (:philor) from comment #10)
> (In reply to Phil Ringnalda (:philor) from comment #3)
> > Oh, m-c isn't the canonical repo, so someone will have to upstream that,
> > won't they?
> 
> Or, instead, someone could just clobber my disabling when they merge from
> github for the last time, and cause us to go back to running these tests,
> and go back to retriggering an entire build when a single test
> intermittently fails, and go back to running two sets of every single test
> suite when a single test intermittently fails.

Heh, m-c is now the canonical repo for mozbase, so we can fix this directly without worrying about this.

I think it's best just to leave this be until bug 966441 is fixed though.

Comment 12

[Phil Ringnalda (:philor)]

So how many times per day is it currently causing builds to be retried, and thus causing pointless entire second runs of tests?

Comment 13

[William Lachance (:wlach)]

So :ahal moved the mozbase tests out of tree (yay!). However, I think we may just want to leave these disabled as they don't add a lot of value (we're not using devicemanagerSUT for anything anymore AFAIK).

I think philor's disabling might have gotten reverted. Should we add it back here?

http://searchfox.org/mozilla-central/source/testing/mozbase/moz.build#10

Comment 14

[Andrew Halberstadt [:ahal]]

I think philor's disabling either never landed or got reverted way earlier on. Here is where the test-manifest.ini got replaced with mozbuild:
http://searchfox.org/mozilla-central/diff/6adcf5b4567ec44beac2051a93d1ceaef83ee2dc/testing/mozbase/test-manifest.ini

Note that change happened in bug 1317970 which landed 3 months ago, and even before that point, the mozdevice tests were enabled. So I think it's likely safe to RESOLVE INVALID this.

Comment 15

[William Lachance (:wlach)]

Ok, probably best to focus on just removing this code altogether (bug 1340584).