977618 - make test-integration builds profile every time

Status: RESOLVED INVALID

(Firefox OS Graveyard :: Gaia, defect)

Description

[Dale Harvey (:daleharvey)]

even when I only change the tests

Comment 1

[Dale Harvey (:daleharvey)]

Attachment: https://github.com/mozilla-b2g/gaia/pull/16718

Comment 2

[Julien Wajsberg [:julienw]]

Dale, tell me if I'm wrong here, but ideally you'd want "APP=xxx make test-integration" to not run the profile generation steps. I think it used to be like this and this regressed with the recent build changes, but maybe we could ask Yuren to fix this instead of adding yet another env variable.

What do you think?

Obviously the real long-term goal would be to not renegerate the profile if we don't change any code.

Comment 3

[Dale Harvey (:daleharvey)]

Having make test-integration build the whole profile is on purpose, I actually regressed it recently by having it only build the apps profile and it broke the tests which have undocumented interdependencies (mostly every test at least depends on the system app)

I will post what I think is the correct solution to the mailing list, I dont think fixing APP is what we should be doing, its all fundamentally messy, and in the meantime to having a proper fix it would be nice to have this noprofile option

Comment 4

[Dale Harvey (:daleharvey)]

https://bugzilla.mozilla.org/show_bug.cgi?id=978697 is what I think is the correct solution

Comment 5

[Julien Wajsberg [:julienw]]

Comment on attachment 8383124 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/16718

Hey Yuren,

I don't feel comfortable reviewing this, maybe you'll have good ideas to share.

Comment 6

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Dale, you can use below commands to run integration test without building profile everytime:

PROFILE_FOLDER=profile-test make (only run it in first time)
make test-integration-test

Comment 7

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 8383124 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/16718

I also add instructions on README.md

https://github.com/mozilla-b2g/gaia/blob/master/README.md#running-tests-without-building-profile

Comment 8

[Dale Harvey (:daleharvey)]

Awesome, as this was a hack, happy its not needed + documented

However I want to make a note, we have not trivial problem of having too many build entry points into the same thing, bin/test-integration / make test-integration / tests/travis/script, they are often incompatible, take different arguments and different syntax, its bad API design and its confusing developers. 

running the integration tests should require developers to clone gaia and |make test-integration|, if we have different options to run the tests they should be different options to make test-integration