1073048 - Running gaia-ui-test with "--type smoketest-bluetooth" runs only 1 test on my device

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Martijn Wargers (dead)]

I'm running the gaia-ui-tests on MacOSX and the bluetooth tests can't work there, because I can't install pybluez on MacOSX.
So I need to run the tests without the tests that use bluetooth.
In that cases I can add -bluetooth to type.

However, when I do that, e.g:
GAIATEST_SKIP_WARNING=1 gaiatest --address localhost:2828 --profile $HOME/B2G/gaia/profile --testvars /Users/mwargers/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/testvars.json --restart --type smoketest-bluetooth /Users/mwargers/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/manifest.ini

Then it's only running 1 test, namely test_settings_dsds_cell_data.py TestDSDSCellData.test_cell_data_for_two_sims
And it skips the other 127 smoketests.

It seems to me something is going wrong here.

http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/functional/settings/manifest.ini#3
In that manifest.ini file, bluetooth = false

Is it perhaps the case that when bluetooth is not set at all in the manifest that bluetooth = true by default or something?

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I think you need --type=b2g+smoketest-bluetooth

Comment 2

[Martijn Wargers (dead)]

(In reply to Dave Hunt (:davehunt) from comment #1)
> I think you need --type=b2g+smoketest-bluetooth

I did that, that's still running only that 1 test I mentioned.

Comment 3

[Martijn Wargers (dead)]

Ok, when linking to:
http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/manifest.ini
It goes well, that's because there's a bluetooth = false setting in there.

So I guess it's indeede the case that when something is not set, it is automatically set to true.

Comment 4

[Martijn Wargers (dead)]

I guess it goes wrong here somewhere: http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/runner/base.py#742

Comment 5

[Martijn Wargers (dead)]

Attachment: manifest.diff

Adding bluetooth = false to every manifest.ini would be a possibility, but I don't think it's really sensible. It doesn't really scale. Also, there are more of these variables that could be added.

This is a git diff, I don't know why /.idea/modules.xml has changes to it.
I'm not sure how to make a PR (=pull request) of this.

Anyway, I don't think this patch is sensible anyway.

Btw, I noticed this doesn't have smoketest set at all:
http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/unit/manifest.ini
That means running with --type=b2g-smoketest doesn't run anything in that direcotry at all.
GAIATEST_SKIP_WARNING=1 gaiatest --address localhost:2828 --profile $HOME/B2G/gaia/profile --testvars /Users/mwargers/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/testvars.json --restart --type=b2g-smoketest /Users/mwargers/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/tests/unit/manifest.ini
I don't think that makes sense.

Comment 6

[Martijn Wargers (dead)]

I guess just accepting that using "-" on type is not working always as expected, except when using it on the top level manifest.
Feel free to close this bug.

Comment 7

[Martijn Wargers (dead)]

This was very likely fixed by bug 1099041.