1222215 - [gatt] Stop using so many retries for Gij suite

Status: RESOLVED FIXED

(Testing Graveyard :: JSMarionette, defect)

Description

[Aus Lacroix [:aus]]

This keeps on coming back to haunt us. I certainly have been a vocal advocate for disabling this and have tried before only to have to back it out to prevent the suite from being hidden.

Let's try this again, with more feeling.

Comment 1

[GitHub Autolander Bot]

Attachment: [gaia] nullaus:bug1222215 > mozilla-b2g:master

Comment 2

[Gregor Wagner [:gwagner]]

Much appreciated Aus!
Lets use pine if the orange rate is too high for master.

Comment 3

[Michael Henretty [:mikehenrty][:mhenretty]]

<3

Comment 4

[Johnny Stenback (:jst)]

Good to see this happen, with a focus on quality I can't think of anything else that would have more bang for the buck than getting our test automation into check. Thanks guys!

Comment 5

[Aus Lacroix [:aus]]

(In reply to Gregor Wagner [:gwagner] from comment #2)
> Much appreciated Aus!
> Lets use pine if the orange rate is too high for master.

Sounds good.

(In reply to Johnny Stenback  (:jst, [e-mail]) from comment #4)
> Good to see this happen, with a focus on quality I can't think of anything
> else that would have more bang for the buck than getting our test automation
> into check. Thanks guys!

No problem! I'm happy to see momentum to get this to happen! :)

Comment 6

[Michael Henretty [:mikehenrty][:mhenretty]]

A lot of the failures [1] look totally fixable on our side, and I'm not seeing any of the dreaded "cannot call send of undefined" stuff. Can I help out with fixing these tests? We could file blocking bugs for every bad intermittent, then fix those and rebase until we get a nice greeny run. What do you think Aus? Or do you have other plans for how to proceed here?

1.) https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=80a5920fbf8c49400f457501cf80b81fd30468de

Comment 7

[Aus Lacroix [:aus]]

(In reply to Michael Henretty [:mhenretty] from comment #6)
> A lot of the failures [1] look totally fixable on our side, and I'm not
> seeing any of the dreaded "cannot call send of undefined" stuff. Can I help
> out with fixing these tests? We could file blocking bugs for every bad
> intermittent, then fix those and rebase until we get a nice greeny run. What
> do you think Aus? Or do you have other plans for how to proceed here?
> 
> 1.)
> https://treeherder.mozilla.org/#/
> jobs?repo=gaia&revision=80a5920fbf8c49400f457501cf80b81fd30468de

I was going to take a first pass and try to fix as many as possible. I may seek out individual test owners for help and obviously would accept patches ;)

Comment 8

[Michael Henretty [:mikehenrty][:mhenretty]]

New run with some fixes:

https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=f738dc459f86b50a4f331908c23f0a3df2366a0f

Comment 9

[Michael Henretty [:mikehenrty][:mhenretty]]

Aus, is there a code change that I could make in Gaia that would allow me to run only 1 Gij test? We used to be able to do this with Travis, and it was really nice for sanity testing these test fixes.

Comment 10

[Michael Henretty [:mikehenrty][:mhenretty]]

Nevermind Aus, thanks to help from David Flanagan I found you can do this:

https://github.com/mozilla-b2g/gaia/pull/33085/files#diff-a0a482120c4a4d9bc37bdcb393e42fffR23

Comment 11

[David Flanagan [:djf]]

I'm concerned about the bookmark_test.js and bookmark_order_test.js failures I was investigating in bug 1223070. With this patch applied, they seem to fail every single time. But without this patch, they seem to pass consistently, without requiring retries.

I wonder if gaia-marionette-retry is doing something to the test environment other than just retrying the tests. What if instead of switching to marionette-mocha to run the tests, we just modified this line in gaia-marionette-retry:

  runTests(filenames, args, 5 /* retry */).then(summarize);

To pass a retry value of 0 instead of 5. That would presumably reduce the number of retries without changing anything else...

I haven't tried this myself.

Comment 12

[Aus Lacroix [:aus]]

(In reply to David Flanagan [:djf] from comment #11)
> I'm concerned about the bookmark_test.js and bookmark_order_test.js failures
> I was investigating in bug 1223070. With this patch applied, they seem to
> fail every single time. But without this patch, they seem to pass
> consistently, without requiring retries.

No environment difference. But, there is a process between the parent mocha instance, and the child, which could certainly influence timing. :/

Comment 13

[Michael Henretty [:mikehenrty][:mhenretty]]

Ok, we've cleared all the blockers for this bug, and the latest try run [1] is looking green enough that I think this is ready to go. Let's land this sucka!

1.) https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=151f7ba90f63

Comment 14

[Michael Henretty [:mikehenrty][:mhenretty]]

master: https://github.com/mozilla-b2g/gaia/commit/9f665863ea9c3dd9585905ef002e8fa06713d820

Comment 15

[Michael Henretty [:mikehenrty][:mhenretty]]

Since landing this we have been able to identify and fix many of our biggest test harness issues. Bug 1236376, bug 1236378, and bug 1104285 most notably so far. But until we fix the dreaded 1175116, we cannot hope to meet the visibility requirements for Gij. So I'm going to back this out over the weekend since we already have all the inforamtion we need from it. Next week we will see about relanding it to work on bug 1175116.

1.) https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy

Comment 16

[Michael Henretty [:mikehenrty][:mhenretty]]

reverted in master: https://github.com/mozilla-b2g/gaia/commit/2f1677b552872e71d308c445cad6880097602b5b

Comment 17

[Michael Henretty [:mikehenrty][:mhenretty]]

Welp, backing this out re-broke the TV tests. For better or worse, we are stuck with this for now.
https://treeherder.mozilla.org/#/jobs?repo=gaia-master&revision=2f1677b55287

re-landed to master: https://github.com/mozilla-b2g/gaia/commit/d047650229cdb8aaa258a5819a077660b96ff17b