988118 - Gaia unit test failures on TBPL due to the camera merge

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Camera, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=36696317&tree=B2g-Inbound

b2g_ubuntu64_vm b2g-inbound opt test gaia-unit on 2014-03-25 17:32:08 PDT for push ba5dabcb650f
slave: tst-linux64-spot-966

17:44:30     INFO -  gaia-unit-tests TEST-UNEXPECTED-FAIL | camera/test/unit/lib/sounds_test.js | Sounds Sounds#createAudio() Should set the mozAudioChannel type | expected false to be truthy
17:44:30     INFO -  gaia-unit-tests INFO       | stack trace:
17:44:30     INFO -      Error: expected false to be truthy
17:44:30     INFO -          at chaiAssert (http://camera.gaiamobile.org:8080/common/test/helper.js:33:1)
17:44:30     INFO -          at get (http://camera.gaiamobile.org:8080/common/vendor/chai/chai.js:352:1)
17:44:30     INFO -          at ok (http://camera.gaiamobile.org:8080/common/vendor/chai/chai.js:1225:5)
17:44:30     INFO -          at (anonymous) (http://camera.gaiamobile.org:8080/test/unit/lib/sounds_test.js:176:7)
17:44:30     INFO -          at wrapper (http://camera.gaiamobile.org:8080/common/test/mocha_generators.js:62:13)
17:44:30     INFO -          at run (http://camera.gaiamobile.org:8080/common/vendor/mocha/mocha.js:3709:7)
17:44:30     INFO -          at runTest (http://camera.gaiamobile.org:8080/common/vendor/mocha/mocha.js:4081:5)
17:44:30     INFO -          at (anonymous) (http://camera.gaiamobile.org:8080/common/vendor/mocha/mocha.js:4127:7)
17:44:30     INFO -          at next (http://camera.gaiamobile.org:8080/common/vendor/mocha/mocha.js:4007:7)
17:44:30     INFO -          at (anonymous) (http://camera.gaiamobile.org:8080/common/vendor/mocha/mocha.js:4016:7)
17:44:30     INFO -          at next (http://camera.gaiamobile.org:8080/common/vendor/mocha/mocha.js:3964:16)
17:44:30     INFO -          at (anonymous) (http://camera.gaiamobile.org:8080/common/vendor/mocha/mocha.js:3984:5)
17:44:30     INFO -          at (anonymous) (http://camera.gaiamobile.org:8080/common/vendor/mocha/mocha.js:4932:28)
17:44:30     INFO -  gaia-unit-tests TEST-END   | camera/test/unit/lib/sounds_test.js | Sounds#createAudio()

Since nobody's around to look into these failures, b2g-inbound and Gaia are both closed until this is resolved.

Comment 1

[Jason Smith [:jsmith]]

Maybe bug 986276 is the cause? That seems to be the last patch that modified that unit test.

Comment 2

[Kevin Grandon :kgrandon]

Reverted: https://github.com/mozilla-b2g/gaia/commit/9b093b55fd98d8a86251397a196fd9591a35a88e

The problem is that this is work on a feature branch, and there is no bug tracking the work to merge the work into master. Because of this I'm leaving this bug as open ni? on Diego.

I might recommend landing in chunks of 3-4 commits over a week or two - just a suggestion, that's what they did on the recent bubble tea branch.

Comment 3

[Justin D'Arcangelo [:justindarc]]

Looks like we had disabled the `sounds_test.js` suite back in Bug 958200 for this exact same reason:

https://bugzilla.mozilla.org/show_bug.cgi?id=958200#c21

Comment 4

[Kevin Grandon :kgrandon]

If you do want to try re-landing this - definitely use gaia-try first, and ensure you have a green build before merging.

https://wiki.mozilla.org/ReleaseEngineering/TryServer#Using_a_custom_Gaia

I know it sucks that tbpl runs tests differently than travis. This is causing a huge problem recently and I'll follow up this week as to what we can do to fix this.

Comment 5

[John Hu [:johnhu][:johu][:醬糊小弟]]

(In reply to Justin D'Arcangelo [:justindarc] from comment #3)
> Looks like we had disabled the `sounds_test.js` suite back in Bug 958200 for
> this exact same reason:
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=958200#c21

It is true that the patch of bug 986276 re-enabled the `sounds_test.js`.

I don't know the environment of TBPL. But according to the code[1][2], it should work. It just checks *mozAudioChannelType*.

To fix it, we may just re-disable it.

[1] https://github.com/dmarcos/gaia/blob/711a2bd41d756287a8731b83abea06b9e554311e/apps/camera/test/unit/lib/sounds_test.js#L175-L176
[2] https://github.com/dmarcos/gaia/blob/711a2bd41d756287a8731b83abea06b9e554311e/apps/camera/js/lib/sounds.js#L151-L155

Comment 6

[Justin D'Arcangelo [:justindarc]]

Attachment: pull-request (master)

This PR reverts the revert and fixes the TBPL blacklist to skip the `sounds_test.js` suite.

kgrandon: Setting R? so you can look at testing this on gaia-try before merging. Thanks!

Comment 7

[Kevin Grandon :kgrandon]

This is currently running on try: https://tbpl.mozilla.org/?tree=Try&rev=4effd5fed078

Comment 8

[Diego Marcos [:dmarcos]]

Thanks Kevin for helping. Where do I have to look to see the results of the unit tests?

Comment 9

[Diego Marcos [:dmarcos]]

I'm really sorry for this. I think this indicates that something either on Travis (we have to match a TBPL environment) or TBPL (make easier to run stuff there) has to change. In the case of camera we feel we're fighting the environment to get stuff shipped. We made a big effort to add more than 200 hundred unit tests and we spent almost a day fighting with Travis to get green due to again differences between our local environments and travis: Travis env is case sensitive and one of the dependencies for one of the tests was capitalized and we were exceeding the node maxBuffer for one of the build tests (bug 988010). If we don't make shipping easier and more transparent two things will happen:

1. Some people will stop caring about doing any contribution of substance since it's so difficult to ship anything. A quasi heroic attitude is needed to get anything done and it's pretty exhausting to play hero all the time.
2. Some obstreperous people will find ways to game the system to follow the path of least resistance to do what they have to do. For example, I'm tempted to disable all unit tests on the repo and only run them on my local environment where I find them useful.

Comment 10

[Diego Marcos [:dmarcos]]

Attachment: Landable revert of the revert + patch to fix the unit test

Comment 11

[Diego Marcos [:dmarcos]]

Attachment: Landable revert of the revert + patch to fix the unit test

Comment 12

[Kevin Grandon :kgrandon]

Diego - I agree with the current pain points, and it is something we are looking to solve in the future by unifying the two environments with great integration into Github. The project is called Task Cluster, but it is still a ways away. In the meantime I think we can adjust our processes a bit to help ease the burden. Landing smaller chunks to master more frequently would probably help quite a bit for now. Relief should be coming in the next few months :)

Comment 13

[Kevin Grandon :kgrandon]

TBPL looks pretty happy, but travis failed. I am pretty sure it's due to an intermittent failure in another app so I restarted the job. Let's see what it comes back with.

Comment 14

[Kevin Grandon :kgrandon]

Attachment: Pull Request

Comment 15

[Kevin Grandon :kgrandon]

Looks like this landed, https://github.com/mozilla-b2g/gaia/commit/e328318d5e27a3253061fb1679ab9abc82e60fcc.

Let's keep a close eye on it.

Comment 16

[David Flanagan [:djf]]

Bulk edit for camera bugs.

If earlier comments do not show how this bug landed to master, it probably landed as part of https://github.com/mozilla-b2g/gaia/pull/17599 which merged the camera-new-features branch into master.

This bug was uplifted from master to v1.4 as part of https://github.com/mozilla-b2g/gaia/commit/a8190d08e61316a86bba572ba8d894d081a20530