1095712 - [meta] Re-enable various disabled, currently flaky marionette tests

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::Calendar, defect)

Description

[Gareth Aye [:gaye] (back from PTO)]

We have a number of marionette tests in calendar which were turned off after being plagued with intermittent failure. This bug will track failure investigation and re-enabling work.

Comment 1

[Gareth Aye [:gaye] (back from PTO)]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25950

This turns on a few of the flaky calendar test cases to run as a separate job on TC to help us investigate failures.

Comment 2

[James Lal [:lightsofapollo]]

Comment on attachment 8519144 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25950

Kevin wanted to make sure your okay with this first.

Comment 3

[Kevin Grandon :kgrandon]

Before I review.. what is the benefit of keeping two manifests currently? Is there a difference in environments? Should tests behave the same way in both?

Comment 4

[James Lal [:lightsofapollo]]

Auto reruns are the big difference the question here is should we allow somewhat flakey tests if the auto-reruns are good enough to catch them.

Comment 5

[Kevin Grandon :kgrandon]

Comment on attachment 8519144 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25950

Oh right, that makes more sense, we're re-enabling the calendar tests, just on TC.

I'd like to see a *huge* run, like ~20+ times to see what kind of intermittent rate we're looking at first.

Also it looks like some tests are currently perma-failing, so those would need to be addressed first: https://treeherder.allizom.org/ui/#/jobs?repo=gaia&revision=f711886d925712b351a4bbb5428759ed3e2db7d3

Comment 6

[Gareth Aye [:gaye] (back from PTO)]

Ah good catch I'll fix that permared now.

Comment 7

[James Lal [:lightsofapollo]]

Comment on attachment 8519144 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25950

I think kgrandon makes a good point... code wise for this PR's its an r+ so feel free to land this once the tests are greened up.

Comment 9

[GitHub Autolander Bot]

Attachment: [gaia] gaye:1095712-reenable-calendar-marionette-tests > mozilla-b2g:master

Comment 10

[Gareth Aye [:gaye] (back from PTO)]

Comment on attachment 8611352 [details] [review]
[gaia] gaye:1095712-reenable-calendar-marionette-tests > mozilla-b2g:master

r=self this should close bug 1060447, bug 1059159, and bug 1007519

Comment 11

[Gareth Aye [:gaye] (back from PTO)]

Er just bug 1059159 and bug 1007519

Comment 12

[GitHub Autolander Bot]

Pull request has landed in master: https://github.com/mozilla-b2g/gaia/commit/a7069b52dd8430712885f4f45ea6e3465461ba69

Comment 13

[Kevin Grandon :kgrandon]

After this landed we started seeing a large amount of test failures on b2g-inbound, mainly on Gij1. They look like the following: https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/Wx-Wjs7yRiqf17oF5ytoUQ/1/public/logs/live_backing.log

This could often cause the entire test run to fail, see: https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=4195d8d8fdf0

I guess retries aren't working fully here?

Backed out for now: https://github.com/mozilla-b2g/gaia/commit/1870f546a92b99c6e1aa2e98e56d9b28eeee61d3

Comment 14

[Kevin Grandon :kgrandon]

Does marionette-js-runner@1.1.1 affect this at all? Seems like that landed after this patch did.

Comment 15

[Gareth Aye [:gaye] (back from PTO)]

Yeah I meant to upgrade marionette-js-runner before landing this. Things are working for me locally and on try. Is there any way I can test on b2g-inbound before landing again?

Comment 16

[Gareth Aye [:gaye] (back from PTO)]

Actually I'm fairly certain this is the issue I fixed in marionette-js-runner@1.1.1. The reason the retry isn't being kicked off is because the jsmarionette process isn't dying. I can add an extra layer of protection there (in the retry routine).

Comment 17

[GitHub Autolander Bot]

Attachment: [gaia] gaye:bug-1095712-retry-script-should-timeout > mozilla-b2g:master

Comment 18

[Gareth Aye [:gaye] (back from PTO)]

Comment on attachment 8612240 [details] [review]
[gaia] gaye:bug-1095712-retry-script-should-timeout > mozilla-b2g:master

r=self

Comment 19

[Kevin Grandon :kgrandon]

Alright, let's try it and monitor b2g-inbound then. If it does fail then we can look at running on b2g-inbound using the old gaia revision + try run thing, but that's quite painful. Thanks!

Comment 20

[Gareth Aye [:gaye] (back from PTO)]

Wow you're awake!

Comment 21

[Gareth Aye [:gaye] (back from PTO)]

Okay https://github.com/mozilla-b2g/gaia/commit/1e4cf425efbab4ea1e7993595f7654a26313407a retry now kicks in if the jsmarionette process is alive but not doing anything.

Comment 22

[GitHub Autolander Bot]

Attachment: [gaia] gaye:bug-1095712-reenable-calendar-marionette-tests > mozilla-b2g:master

Comment 23

[Gareth Aye [:gaye] (back from PTO)]

Comment on attachment 8612245 [details] [review]
[gaia] gaye:bug-1095712-reenable-calendar-marionette-tests > mozilla-b2g:master

r=self

Comment 24

[GitHub Autolander Bot]

http://docs.taskcluster.net/tools/task-graph-inspector/#o3uI7vvkRLmTqARjVwpr6Q

The pull request failed to pass integration tests. It could not be landed, please try again.

Comment 25

[Gareth Aye [:gaye] (back from PTO)]

So the reason that this failed was because of a permared introduced in bug 1168061 (see https://github.com/mozilla-b2g/gaia/commit/9dfedd7d35da00fa9b08dcffc3ab4f47da54e4f0). Will give this another shot.

Comment 26

[GitHub Autolander Bot]

http://docs.taskcluster.net/tools/task-graph-inspector/#NJAvO9x5SqKt67BOr0lgow

The pull request failed to pass integration tests. It could not be landed, please try again.

Comment 27

[Kevin Grandon :kgrandon]

I think this is an autolander bug (where you need to rebase). Seems to work fine, so let's land manually: https://github.com/mozilla-b2g/gaia/commit/8c9b6f0c3c281ba6e5fad7ecf5515006b0f2005f

Comment 28

[Kevin Grandon :kgrandon]

Sorry, I forgot you wanted to leave this open.

Comment 29

[Gareth Aye [:gaye] (back from PTO)]

Cool thanks I figured that's what was going on and rebased

Comment 30

[Gareth Aye [:gaye] (back from PTO)]

Backed out the change that triggers retry when the test process hangs for >=30s https://github.com/mozilla-b2g/gaia/commit/6b567638ce9a2ad716908a1e3a4eaad86b34bef9. It's possible we just need to bump that number since only a couple test cases seemed upset

Comment 31

[GitHub Autolander Bot]

Attachment: [gaia] gaye:bug-1095712-retry-script-should-timeout-redux > mozilla-b2g:master

Comment 32

[Gareth Aye [:gaye] (back from PTO)]

Comment on attachment 8612302 [details] [review]
[gaia] gaye:bug-1095712-retry-script-should-timeout-redux > mozilla-b2g:master

Second try and bumping the timeout from 30s to 120s. I'm going to wait for a while before asking autolander to stage it though to make sure inbound is looking healthy.

Comment 33

[GitHub Autolander Bot]

Attachment: [gaia] gaye:bug-1095712-jsmarionette-hotfix > mozilla-b2g:master

Comment 34

[Gareth Aye [:gaye] (back from PTO)]

Comment on attachment 8612472 [details] [review]
[gaia] gaye:bug-1095712-jsmarionette-hotfix > mozilla-b2g:master

r=self

Comment 35

[Gareth Aye [:gaye] (back from PTO)]

Landed hotfix to marionette-js-runner for a small js issue in the code that handles python crashes  https://github.com/mozilla-b2g/gaia/commit/326c7623a88597e1dfa92bda84bb8e3d27b572d8

Comment 36

[BMO Automation]

Firefox OS is not being worked on