[meta] Re-enable various disabled, currently flaky marionette tests

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
4 years ago
8 months ago

People

(Reporter: gaye, Assigned: gaye)

Tracking

({leave-open})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(6 attachments)

(Assignee)

Description

4 years ago
We have a number of marionette tests in calendar which were turned off after being plagued with intermittent failure. This bug will track failure investigation and re-enabling work.
(Assignee)

Updated

4 years ago
(Assignee)

Updated

4 years ago
Summary: Re-enable various disabled, currently flaky marionette tests → [meta] Re-enable various disabled, currently flaky marionette tests
(Assignee)

Comment 1

4 years ago
Created attachment 8519144 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25950

This turns on a few of the flaky calendar test cases to run as a separate job on TC to help us investigate failures.
Attachment #8519144 - Flags: review?(jlal)
Comment on attachment 8519144 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25950

Kevin wanted to make sure your okay with this first.
Attachment #8519144 - Flags: ui-review?(kgrandon)
Before I review.. what is the benefit of keeping two manifests currently? Is there a difference in environments? Should tests behave the same way in both?
Flags: needinfo?(gaye)
Auto reruns are the big difference the question here is should we allow somewhat flakey tests if the auto-reruns are good enough to catch them.
Flags: needinfo?(gaye)
Comment on attachment 8519144 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25950

Oh right, that makes more sense, we're re-enabling the calendar tests, just on TC.

I'd like to see a *huge* run, like ~20+ times to see what kind of intermittent rate we're looking at first.

Also it looks like some tests are currently perma-failing, so those would need to be addressed first: https://treeherder.allizom.org/ui/#/jobs?repo=gaia&revision=f711886d925712b351a4bbb5428759ed3e2db7d3
Attachment #8519144 - Flags: ui-review?(kgrandon)
(Assignee)

Updated

4 years ago
Blocks: 1027729
(Assignee)

Comment 6

4 years ago
Ah good catch I'll fix that permared now.
Comment on attachment 8519144 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25950

I think kgrandon makes a good point... code wise for this PR's its an r+ so feel free to land this once the tests are greened up.
Attachment #8519144 - Flags: review?(jlal)
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1164849
Created attachment 8611352 [details] [review]
[gaia] gaye:1095712-reenable-calendar-marionette-tests > mozilla-b2g:master
(Assignee)

Updated

3 years ago
Assignee: nobody → gaye
(Assignee)

Comment 10

3 years ago
Comment on attachment 8611352 [details] [review]
[gaia] gaye:1095712-reenable-calendar-marionette-tests > mozilla-b2g:master

r=self this should close bug 1060447, bug 1059159, and bug 1007519
Attachment #8611352 - Flags: review+
(Assignee)

Updated

3 years ago
Keywords: checkin-needed, leave-open

Updated

3 years ago
Keywords: checkin-needed

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Assignee)

Updated

3 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
After this landed we started seeing a large amount of test failures on b2g-inbound, mainly on Gij1. They look like the following: https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/Wx-Wjs7yRiqf17oF5ytoUQ/1/public/logs/live_backing.log

This could often cause the entire test run to fail, see: https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=4195d8d8fdf0

I guess retries aren't working fully here?

Backed out for now: https://github.com/mozilla-b2g/gaia/commit/1870f546a92b99c6e1aa2e98e56d9b28eeee61d3
Flags: needinfo?(gaye)
Does marionette-js-runner@1.1.1 affect this at all? Seems like that landed after this patch did.
(Assignee)

Comment 15

3 years ago
Yeah I meant to upgrade marionette-js-runner before landing this. Things are working for me locally and on try. Is there any way I can test on b2g-inbound before landing again?
Flags: needinfo?(gaye)
(Assignee)

Comment 16

3 years ago
Actually I'm fairly certain this is the issue I fixed in marionette-js-runner@1.1.1. The reason the retry isn't being kicked off is because the jsmarionette process isn't dying. I can add an extra layer of protection there (in the retry routine).
Created attachment 8612240 [details] [review]
[gaia] gaye:bug-1095712-retry-script-should-timeout > mozilla-b2g:master
(Assignee)

Comment 18

3 years ago
Comment on attachment 8612240 [details] [review]
[gaia] gaye:bug-1095712-retry-script-should-timeout > mozilla-b2g:master

r=self
Attachment #8612240 - Flags: review+
Alright, let's try it and monitor b2g-inbound then. If it does fail then we can look at running on b2g-inbound using the old gaia revision + try run thing, but that's quite painful. Thanks!
(Assignee)

Comment 20

3 years ago
Wow you're awake!
(Assignee)

Comment 21

3 years ago
Okay https://github.com/mozilla-b2g/gaia/commit/1e4cf425efbab4ea1e7993595f7654a26313407a retry now kicks in if the jsmarionette process is alive but not doing anything.
Created attachment 8612245 [details] [review]
[gaia] gaye:bug-1095712-reenable-calendar-marionette-tests > mozilla-b2g:master
(Assignee)

Comment 23

3 years ago
Comment on attachment 8612245 [details] [review]
[gaia] gaye:bug-1095712-reenable-calendar-marionette-tests > mozilla-b2g:master

r=self
Attachment #8612245 - Flags: review+
(Assignee)

Updated

3 years ago
Keywords: checkin-needed

Updated

3 years ago
Keywords: checkin-needed
http://docs.taskcluster.net/tools/task-graph-inspector/#o3uI7vvkRLmTqARjVwpr6Q

The pull request failed to pass integration tests. It could not be landed, please try again.
(Assignee)

Comment 25

3 years ago
So the reason that this failed was because of a permared introduced in bug 1168061 (see https://github.com/mozilla-b2g/gaia/commit/9dfedd7d35da00fa9b08dcffc3ab4f47da54e4f0). Will give this another shot.
(Assignee)

Updated

3 years ago
Keywords: checkin-needed

Updated

3 years ago
Keywords: checkin-needed
http://docs.taskcluster.net/tools/task-graph-inspector/#NJAvO9x5SqKt67BOr0lgow

The pull request failed to pass integration tests. It could not be landed, please try again.
I think this is an autolander bug (where you need to rebase). Seems to work fine, so let's land manually: https://github.com/mozilla-b2g/gaia/commit/8c9b6f0c3c281ba6e5fad7ecf5515006b0f2005f
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
Sorry, I forgot you wanted to leave this open.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 29

3 years ago
Cool thanks I figured that's what was going on and rebased
(Assignee)

Comment 30

3 years ago
Backed out the change that triggers retry when the test process hangs for >=30s https://github.com/mozilla-b2g/gaia/commit/6b567638ce9a2ad716908a1e3a4eaad86b34bef9. It's possible we just need to bump that number since only a couple test cases seemed upset
Created attachment 8612302 [details] [review]
[gaia] gaye:bug-1095712-retry-script-should-timeout-redux > mozilla-b2g:master
(Assignee)

Comment 32

3 years ago
Comment on attachment 8612302 [details] [review]
[gaia] gaye:bug-1095712-retry-script-should-timeout-redux > mozilla-b2g:master

Second try and bumping the timeout from 30s to 120s. I'm going to wait for a while before asking autolander to stage it though to make sure inbound is looking healthy.
Attachment #8612302 - Flags: review+
Created attachment 8612472 [details] [review]
[gaia] gaye:bug-1095712-jsmarionette-hotfix > mozilla-b2g:master
(Assignee)

Comment 34

3 years ago
Comment on attachment 8612472 [details] [review]
[gaia] gaye:bug-1095712-jsmarionette-hotfix > mozilla-b2g:master

r=self
Attachment #8612472 - Flags: review+
(Assignee)

Comment 35

3 years ago
Landed hotfix to marionette-js-runner for a small js issue in the code that handles python crashes  https://github.com/mozilla-b2g/gaia/commit/326c7623a88597e1dfa92bda84bb8e3d27b572d8

Comment 36

8 months ago
Firefox OS is not being worked on
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago8 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.