787578 - B2G Updates: Get updater automated tests running for B2G

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect, P1)

Description

[Marshall Culpepper [:marshall_law]]

There are currently a large set of mochitests and xpcshell tests for the updater that we should get running in automation for B2G. See:

- http://dxr.mozilla.org/mozilla-central/toolkit/mozapps/update/test/
- http://dxr.mozilla.org/mozilla-central/toolkit/mozapps/update/test_svc/
- http://dxr.mozilla.org/mozilla-central/toolkit/mozapps/update/test_timermanager/

Comment 1

[Jonathan Griffin (:jgriffin)]

I can turn these on any time you tell me they're ready.

Comment 2

[Andrew Overholt [:overholt]]

Updates are very important and these large sets warrant a blocking+ IMO.

Comment 3

[Brian R. Bondy [:bbondy]]

I've worked on these tests a lot before and there will be some work here relating to signing, so I'll take this bug

Comment 4

[Brian R. Bondy [:bbondy]]

Unassigned based on dependencies

Comment 5

[Jonathan Griffin (:jgriffin)]

Removing mochitest-chrome as a dep, since the existing mochitest-chrome tests for the updater are Firefox-specific.

Comment 6

[Andrew Overholt [:overholt]]

Has there been any progress here?  Are we still blocked on getting the xpcshell tests going?

Comment 7

[Brian R. Bondy [:bbondy]]

I think this is blocked on the dependent bug which looks like it should land soon.

Comment 8

[Doug Turner (:dougt)]

Brian, what are the next steps here?

Comment 9

[Brian R. Bondy [:bbondy]]

Could you provide logs for the failing tests? Are other xpcshell-tests up and running now?

Comment 10

[Robert Strong (they/them - no direct email)]

(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> Removing mochitest-chrome as a dep, since the existing mochitest-chrome
> tests for the updater are Firefox-specific.
Note that the mochitest-chrome tests are actually toolkit specific and work with Firefox, Thunderbird, and SeaMonkey.

Comment 11

[Robert Strong (they/them - no direct email)]

dougt, can you help out getting the info requested in comment #9?

Comment 12

[Alex Keybl [:akeybl]]

Marking for C2, given this meets the criteria of known P1/P2 blocking-basecamp+ bugs at the end of C1.

Comment 13

[Brian R. Bondy [:bbondy]]

(In reply to Brian R. Bondy [:bbondy] from comment #9)
> Could you provide logs for the failing tests? Are other xpcshell-tests up
> and running now?

Could someone answer the second part of this?  Are other xpcshell-tests up and running now?

Comment 14

[Jonathan Griffin (:jgriffin)]

ahal is working on getting xpcshell tests for B2G running on TPBL, see bug 797153

Comment 15

[Brian R. Bondy [:bbondy]]

CC akeybl: See Comment 14
Then I suggest that should also be a C2 blocker. 
Unassigned myself again since I can't work on it.

Comment 16

[Andrew Halberstadt [:ahal]]

Bug 797153 was just a small piece, there's still a lot of work to be done. But if I drop everything I think I can get xpcshell tests running by C2. But there are still some unknowns to me (e.g bug 815641)

Comment 17

[Brian R. Bondy [:bbondy]]

I think this should be re-evaluated for consideration for C2 given the last couple comments.

Comment 18

[Andrew Halberstadt [:ahal]]

The biggest issue blocking me is bug 816086. If I can get that figured out by the end of the week I think there is a decent chance of at least getting xpcshell tests running on cedar by Dec 10th (main tracking bug for that is bug 809233).

Comment 19

[Lawrence Mandel [:lmandel] (use needinfo)]

Andrew, assigning to you as it looks like you're working on this.

Comment 20

[Brian R. Bondy [:bbondy]]

Feel free to file followups for specific app update xpcshell test failures to me.

Comment 21

[Andrew Halberstadt [:ahal]]

(In reply to Marshall Culpepper [:marshall_law] from comment #0)
> - http://dxr.mozilla.org/mozilla-central/toolkit/mozapps/update/test/
> - http://dxr.mozilla.org/mozilla-central/toolkit/mozapps/update/test_svc/

These two sets of test don't seem to be in the tests.zip. Filed bug 817651 to get them added.

Comment 22

[Andrew Halberstadt [:ahal]]

Update: xpcshell looks to be on track to ready to go on m-c by monday (+ time for releng to change and update the configs)

However that still won't change the fact that the update tests aren't being added to the tests.zip because MOZ_UPDATER isn't being defined in the build (bug 817651)

Comment 23

[Faramarz Rashed]

:ahal, there seems to be a bunch of dependency here.  what's the ETA here since this missed the C2 milestone?

Comment 24

[Lawrence Mandel [:lmandel] (use needinfo)]

Andrew has previously told me that he expected this work to be complete and deployed by mid week (roughly Wednesday). Andrew, is this still accurate?

Comment 25

[Andrew Halberstadt [:ahal]]

(In reply to Lawrence Mandel [:lmandel] from comment #24)
> Andrew has previously told me that he expected this work to be complete and
> deployed by mid week (roughly Wednesday). Andrew, is this still accurate?

Yep, they're running green on Cedar. Just need to land a manifest change on the core branches and wait for releng to enable the config (which is already reviewed).

Comment 26

[Andrew Halberstadt [:ahal]]

Just waiting for the latest patch in bug 807125 to land

Comment 27

[Andrew Halberstadt [:ahal]]

All has landed, they should go live on all branches either today or tomorrow.

Comment 28

[Andrew Halberstadt [:ahal]]

xpcshell tests are now unhidden on green on inbound (and m-c after next merge) and aurora. Unfortunately on b2g18 they're hitting bug 811830 for some reason. There is a separate bug I'm working on (bug 821012) which I believe will workaround the issue in bug 811830.

In the mean time there are several update tests that seem to fail which are outside of my domain knowledge to look into.

* toolkit/mozapps/update/test/unit/test_0030_general.js
* toolkit/mozapps/update/test/unit/test_0081_prompt_uiAlreadyOpen.js
* toolkit/mozapps/update/test/unit/test_bug595059.js
* all tests in toolkit/mozapps/update/test/unit/xpcshell_updater.ini

I haven't filed bugs for these yet, but if someone does please make it block bug 820380

Comment 29

[Andrew Halberstadt [:ahal]]

The problem on the mozilla-b2g18 branch isn't bug 811830 after all, it's just that the version of remotexpcshelltests.py is out of date. I filed bug 821333 to fix the problem.

Comment 30

[Andrew Halberstadt [:ahal]]

Filed bug 821343 and bug 821344 to look into the test failures in comment 28

Comment 31

[Andrew Halberstadt [:ahal]]

These are now running on all branches including b2g18 (though they are still hidden due to one of the tests timing out, will disable it soon).

This bug is more or less finished from my point of view. Lets use bug 821343 and bug 821344 to track progress on fixing remaining problems with the update tests.