827302 - TBPL should run Gaia tests in order to prevent regressions from the platform

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

There was many regressions with the B2G desktop build running Gaia in the past. Also in order to catch regressions as fast as possible it would be awesome to have tests running on TBPL. It's a terrible loss of time for Gaia developers when the build is broken the morning by some platform changes.

What can we do to have desktop builds running Gaia tests on TBPL?

Lukas do you have any suggestions?

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

Talked with jgriffin in #b2g and this is already running on the cedar branch and work tracked in bug 802317 so marking this as a dupe.

Comment 2

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

That does not seems to resolve this bug. This bug is about getting tests living on Gaia (unit, integration, performance?) running on TBPL.

The bug that has been resolved is about Gaia UI tests which are written by a different team and maintain in a separate repository.

All tests should run on TBPL, not only UI tests.  

Also ideally they should live on the mozilla-central/inbound page of TBPL because I doubt that platform developers will look at the gaia-master tree.

Asking tef? since this discussion is here for so long and it has not yet been resolved in a useful way for Gecko/Gaia developers.

The next step imho would be to run Gaia unit tests and Gaia integration tests (and not only Gaia UI tests). That would motivate Gaia devs to look at TBPL. Then moving those results to the main page for *landings* (Gecko and Gaia). That would let Gecko devs see if they broke us.

Comment 3

[Jason Smith [:jsmith]]

Umm...why is this nomed to block? Infrastructure work doesn't inherently block the release - worst case, we'll still go out the door without it finished. We should fix this though.

Comment 4

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

(In reply to Jason Smith [:jsmith] from comment #3)
> Umm...why is this nomed to block? Infrastructure work doesn't inherently
> block the release - worst case, we'll still go out the door without it
> finished. We should fix this though.

I have seen some infrastructure bug blocking the release though. And even if I agree on the idea this one is not really an infrastructure bug. This is about running the unit/integrations tests on every commit and have some visual results. I consider this a Gaia bug.

Comment 5

[Lukas Blakk [:lsblakk] use ?needinfo]

This is not a blocker for 1.0.1 it's definitely NPOTB, we are planning test on checkin (Gaia) for v1.2 and this may or may not be the bug to talk about it in - does RelEng not have a bug already?  If not, we should move this to their component and continue the discussion.

Comment 6

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

(In reply to [e-mail] from comment #5)
> This is not a blocker for 1.0.1 it's definitely NPOTB, 

The situation is impossible since months. I kept receiving emails/complains about our testing in Gaia. In the meantime I keep telling to people that testing won't happen until there is a proper CI otherwise nobody will look at the tests and people on one APP will broke the others and Gecko folks will break us easily. 

So I disagree about the tef- status.

Comment 7

[Jonathan Griffin (:jgriffin)]

Gaia unit tests are coming soon (as soon as this week, on cedar).  Gaia UI tests are a bit farther out...2 to 3 week, minimum.

Comment 8

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Vivien: I think we need to focus on getting testing up and running for m-c + gaia-master first, before we can worry about b2g18 + v1-train. I'd rather focus our efforts on the longer term and get solid testing for future releases, than spend too much effort on things that will "just" help for v1.1.

Comment 9

[Daniel Coloma:dcoloma]

This seems to be not my decision anymore, handing over decision to Lukas again

Comment 10

[Julien Wajsberg [:julienw]]

The gaia Unit Tests won't likely work on v1-train at all, this would take a huge workload to make them work.

However b2g18 + master should work, but I'm not sure this is so meaningful.

Comment 11

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

(In reply to Jonas Sicking (:sicking) from comment #8)
> Vivien: I think we need to focus on getting testing up and running for m-c +
> gaia-master first, before we can worry about b2g18 + v1-train. I'd rather
> focus our efforts on the longer term and get solid testing for future
> releases, than spend too much effort on things that will "just" help for
> v1.1.

I'm not trying to tef? block on this to have it running on b2g-18 + v1-train since there is actually no much tests written. But I want it to be running asap in order to make sure we have it ready for the next release.

tef? is just a way for me to raise the priority on this!

Comment 12

[Jason Smith [:jsmith]]

tef? isn't the appropriate flag then. If we need raise priority, I'd bring this up in the gaia/b2g weekly meeting, mailing lists, etc.

Comment 13

[Jonathan Griffin (:jgriffin)]

This is already running FYI on cedar:  https://tbpl.mozilla.org/?tree=Cedar&jobname=gaia

The tests seem quite unstable atm; as soon as they're green (either through fixing the tests or disabling them), we can roll them out more broadly.

Comment 14

[Jonathan Griffin (:jgriffin)]

related bugs:  833666, 868597, 866908, 866909

Comment 15

[Jonathan Griffin (:jgriffin)]

This general problem has been fixed; there are open bugs for specific issues with specific test harnesses.