Moving back to gaia rev 6b0a8108eac92ab3b5e73946c57f750fe8744356 changes the output to this: OK TEST-START test_sms_contact.py test_sms_contact (test_sms_contact.TestContacts) ... ERROR But there is no stack or "TEST-UNEXPECTED-FAIL". Perhaps this has always been happening and we didn't realize. Some of the commits I rolled back to test this were for bug 927404 which improves error reporting. Thoughts? I won't have any more time to look at this today unfortunately.
Reuben also indicates over in bug 914191 comment 22 that the underlying failure could be due to webidl changes.
So I fixed users of the Contacts API, but now I'm getting timeouts on tests that use the Contacts app: https://travis-ci.org/mozilla-b2g/gaia/jobs/12764456 TEST-START test_keyboard.py test_keyboard_basic (test_keyboard.TestKeyboard) ... ERROR ====================================================================== ERROR: None ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/marionette_client-0.6.0-py2.7.egg/marionette/marionette_test.py", line 132, in run testMethod() File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard.py", line 40, in test_keyboard_basic self.wait_for_condition(lambda m: len(contacts_app.contacts) == 1) File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 691, in wait_for_condition raise TimeoutException(message) TEST-UNEXPECTED-FAIL | test_keyboard.py test_keyboard.TestKeyboard.test_keyboard_basic | TimeoutException: Condition timed out
Yeah, I think it's just "multiple tests were broken, but reporting that a test was broken was also broken so nobody knew." But they sure were nice and green, while green was a lie, weren't they? Blocks reopening b2g-inbound, though, unless you want to declare the current state of the suite worthless, hide them everywhere, and go through a round of make-it-acceptable-to-unhide.
Severity: normal → blocker
Reuben, I backed out the keyboard changes causing that earlier today. Do you have that in your pull?
(In reply to Ben Kelly [:bkelly] from comment #5) > Reuben, I backed out the keyboard changes causing that earlier today. Do > you have that in your pull? I rebased my PR and got the same failures: https://travis-ci.org/mozilla-b2g/gaia/jobs/12769292
Oh, nevermind, the Contacts tests are timing out on master right now, so I'm just going to merge.
Re comment #0, Ben that test shouldn't run on Travis (it sends a real SMS). Were you running locally? Re comment #7, Reuben a timeout in gaia-ui is usually the same as a failure. The pattern of failures looks like something in the pull is not working and as such all the tests are broken. I'm not in a position to debug right now but I'd say something in the mock_contact.py is not right so when the test tries to setUp by pushing the mock into the contacts Db it's failing and the test has no data to use. Did you run them locally before submitting the pull and get a pass? test_clock_set_alarm_repeat failure is unrelated. Either that or a HTML Locator needs to be updated in the tests but that seems unlikely as the app itself wasn't change in the pull.
Meh, I'm not gaining anything by having b2g-inbound closed, it just batches up all the pushes that continue going to gaia anyway.
Severity: blocker → normal
(In reply to Zac C (:zac) from comment #8) > Re comment #7, Reuben a timeout in gaia-ui is usually the same as a failure. > The pattern of failures looks like something in the pull is not working and > as such all the tests are broken. I'm not in a position to debug right now > but I'd say something in the mock_contact.py is not right so when the test > tries to setUp by pushing the mock into the contacts Db it's failing and the > test has no data to use. The exact same failure was there before I merged my PR so I assumed it was an existing problem.
That's not true, you can see in the job history of Travis that the failure types after each of your commits were different as you progressively fixed the problems!
Yes, that was in my PR. Here are some builds from before I merged, showing the same problem: https://travis-ci.org/mozilla-b2g/gaia/jobs/12763843 https://travis-ci.org/mozilla-b2g/gaia/jobs/12759159 https://travis-ci.org/mozilla-b2g/gaia/jobs/12757154 https://travis-ci.org/mozilla-b2g/gaia/jobs/12756590
What I mean is the failures in those jobs are not the same as the failures we see in your merge pull request commit. Were you trying to fix those failures? Was there a Gecko or contacts API changed that caused the original failures? because these tests were fine when I left the office on Friday evening, they are definitely not some long-term persisent reds.
Created attachment 819549 [details] Gaia PR pointer
Assignee: nobody → reuben.bmo
Attachment #819549 - Flags: review?(bkelly)
Reuben, looking better! I'll fix the alarmclock tests this morning and we'll try and get Travis green.
Comment on attachment 819549 [details] Gaia PR pointer Reuben, please address the comments from Github: - photo is a sequence<blob> so the I think we need an array there - correct bug number in commit/PR subject With those changes r=me. We have more fixes that need to be made, but lets not block getting these in.
Attachment #819549 - Flags: review?(bkelly) → review+
note, the tree is closed till this issue get fixed
Due to the general lack of coordination on these fixes Rik filed bug #928915 to disable all of the failing tests.
Since I believe reuben had addressed the questions/issues from GH I went ahead and merged this: https://github.com/mozilla-b2g/gaia/commit/ef8730359a3eb8033c60ef93a01a757b9876f792 I will look at re-enabling tests now.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.