Closed Bug 1222215 Opened 4 years ago Closed 4 years ago

[gatt] Stop using so many retries for Gij suite

Categories

(Testing Graveyard :: JSMarionette, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aus, Assigned: aus)

References

Details

Attachments

(1 file)

This keeps on coming back to haunt us. I certainly have been a vocal advocate for disabling this and have tried before only to have to back it out to prevent the suite from being hidden.

Let's try this again, with more feeling.
Status: NEW → ASSIGNED
Much appreciated Aus!
Lets use pine if the orange rate is too high for master.
Good to see this happen, with a focus on quality I can't think of anything else that would have more bang for the buck than getting our test automation into check. Thanks guys!
(In reply to Gregor Wagner [:gwagner] from comment #2)
> Much appreciated Aus!
> Lets use pine if the orange rate is too high for master.

Sounds good.

(In reply to Johnny Stenback  (:jst, jst@mozilla.com) from comment #4)
> Good to see this happen, with a focus on quality I can't think of anything
> else that would have more bang for the buck than getting our test automation
> into check. Thanks guys!

No problem! I'm happy to see momentum to get this to happen! :)
A lot of the failures [1] look totally fixable on our side, and I'm not seeing any of the dreaded "cannot call send of undefined" stuff. Can I help out with fixing these tests? We could file blocking bugs for every bad intermittent, then fix those and rebase until we get a nice greeny run. What do you think Aus? Or do you have other plans for how to proceed here?

1.) https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=80a5920fbf8c49400f457501cf80b81fd30468de
Flags: needinfo?(aus)
(In reply to Michael Henretty [:mhenretty] from comment #6)
> A lot of the failures [1] look totally fixable on our side, and I'm not
> seeing any of the dreaded "cannot call send of undefined" stuff. Can I help
> out with fixing these tests? We could file blocking bugs for every bad
> intermittent, then fix those and rebase until we get a nice greeny run. What
> do you think Aus? Or do you have other plans for how to proceed here?
> 
> 1.)
> https://treeherder.mozilla.org/#/
> jobs?repo=gaia&revision=80a5920fbf8c49400f457501cf80b81fd30468de

I was going to take a first pass and try to fix as many as possible. I may seek out individual test owners for help and obviously would accept patches ;)
Flags: needinfo?(aus)
Depends on: 1223321
Depends on: 1223070
Depends on: 1205907
Aus, is there a code change that I could make in Gaia that would allow me to run only 1 Gij test? We used to be able to do this with Travis, and it was really nice for sanity testing these test fixes.
Flags: needinfo?(aus)
Nevermind Aus, thanks to help from David Flanagan I found you can do this:

https://github.com/mozilla-b2g/gaia/pull/33085/files#diff-a0a482120c4a4d9bc37bdcb393e42fffR23
Flags: needinfo?(aus)
I'm concerned about the bookmark_test.js and bookmark_order_test.js failures I was investigating in bug 1223070. With this patch applied, they seem to fail every single time. But without this patch, they seem to pass consistently, without requiring retries.

I wonder if gaia-marionette-retry is doing something to the test environment other than just retrying the tests. What if instead of switching to marionette-mocha to run the tests, we just modified this line in gaia-marionette-retry:

  runTests(filenames, args, 5 /* retry */).then(summarize);

To pass a retry value of 0 instead of 5. That would presumably reduce the number of retries without changing anything else...

I haven't tried this myself.
(In reply to David Flanagan [:djf] from comment #11)
> I'm concerned about the bookmark_test.js and bookmark_order_test.js failures
> I was investigating in bug 1223070. With this patch applied, they seem to
> fail every single time. But without this patch, they seem to pass
> consistently, without requiring retries.

No environment difference. But, there is a process between the parent mocha instance, and the child, which could certainly influence timing. :/
No longer depends on: 1192249
No longer depends on: 1222355
No longer depends on: 1180236
No longer depends on: 1106194
No longer depends on: 1191036
No longer depends on: 1163853
No longer depends on: 1189882
No longer depends on: 1221876
No longer depends on: 1223321
No longer depends on: 1235531
Ok, we've cleared all the blockers for this bug, and the latest try run [1] is looking green enough that I think this is ready to go. Let's land this sucka!

1.) https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=151f7ba90f63
master: https://github.com/mozilla-b2g/gaia/commit/9f665863ea9c3dd9585905ef002e8fa06713d820
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Depends on: 1236308
Since landing this we have been able to identify and fix many of our biggest test harness issues. Bug 1236376, bug 1236378, and bug 1104285 most notably so far. But until we fix the dreaded 1175116, we cannot hope to meet the visibility requirements for Gij. So I'm going to back this out over the weekend since we already have all the inforamtion we need from it. Next week we will see about relanding it to work on bug 1175116.

1.) https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy
reverted in master: https://github.com/mozilla-b2g/gaia/commit/2f1677b552872e71d308c445cad6880097602b5b
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Welp, backing this out re-broke the TV tests. For better or worse, we are stuck with this for now.
https://treeherder.mozilla.org/#/jobs?repo=gaia-master&revision=2f1677b55287

re-landed to master: https://github.com/mozilla-b2g/gaia/commit/d047650229cdb8aaa258a5819a077660b96ff17b
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.