1107678 - [meta] Enable tests on b2g emulator KK on Treeherder

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Emulator, defect)

Description

[StevenLee[:slee]]

This bug is to track the progress of tests on b2g kk emulator.

Comment 1

[Bobby Chien]

Hi James,

could we have update about testing environment preparation? Let us know if you back from holiday. Many thanks.

Comment 2

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Change the component because I think this isn't only for video/audio modules but in general we want to enable tests that already run on emulator-ics. Feel free to correct me!

Comment 3

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Attachment: [01] Bug 1107678: Add automated tests to emulator-kk on cedar

As emulator-kk should start up reliably now, we want to run tests on it. As a first step, we'd like to execute the tests on cedar to get an idea what's missing.

Comment 4

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Thank you. Landed this patch in

  https://hg.mozilla.org/build/buildbot-configs/rev/d8340f29efb0

Comment 5

[Thomas Zimmermann [:tzimmermann] [:tdz]]

The cedar tree is at

  https://tbpl.mozilla.org/?tree=Cedar

Comment 6

[Rail Aliiev [:rail]]

FTR, the changes are not in production yet. There should be an extra comment here when they are.

Comment 7

[Thomas Zimmermann [:tzimmermann] [:tdz]]

On irc, :ahal mentioned something about cedar re-configuration after the patch lands. Will that happen automatically or is there something to do from my side?

Comment 8

[Rail Aliiev [:rail]]

A buildduty person will do this. Usually it happens almost every day. The bug will be updated when it's in production.

Comment 9

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/d8340f29efb0

Comment 10

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Let's see what happens...

  https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=6bdbe725837d

Comment 11

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Josh,

can you try to find someone from RelEng to help with the integration of emulator-kk tests on Cedar and b2g-inbound? Thanks!

Comment 12

[Thomas Zimmermann [:tzimmermann] [:tdz]]

(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #10)
> Let's see what happens...
> 
>   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=6bdbe725837d

It looks like Marionette still doesn't work on emulator-kk. With bug 1124567 being fixed, it should now startup reliably. I guess that either the test scripts or Marionette need to be updated.

Comment 13

[Josh Cheng [:josh]]

Dear HsinYi,
Could you please help to find someone for checking this bug?
Thanks!

Comment 14

[Hsin-Yi Tsai (she/her) [:hsinyi]]

(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #12)
> (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #10)
> > Let's see what happens...
> > 
> >   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=6bdbe725837d
> 
> It looks like Marionette still doesn't work on emulator-kk. With bug 1124567
> being fixed, it should now startup reliably. I guess that either the test
> scripts or Marionette need to be updated.

From log on emulator-ics on treeherder, there's a message "MARIONETTE LOG: INFO: received mozbrowserloadend for app" after the message "MARIONETTE LOG: INFO: waiting for mozbrowserloadend." However, in [1] marionette was waiting for mozbrowserloadend but didn't receive it eventually. This seems the cause of the failure.

[1] http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-emulator-kk/1424269816/cedar_ubuntu64_vm-b2g-emulator-kk_test-marionette-webapi-bm123-tests1-linux64-build0.txt.gz

(In reply to Josh Cheng [:josh] from comment #13)
> Dear HsinYi,
> Could you please help to find someone for checking this bug?
> Thanks!

Hi Josh,
Looks Thomas is still actively working on this. Could you elaborate more on what additional support you need? Thank you thank you.

Comment 15

[Josh Cheng [:josh]]

Hi HsinYi,
I thought Thomas ask for our help in comment 11?
Thanks

Comment 16

[Hsin-Yi Tsai (she/her) [:hsinyi]]

(In reply to Josh Cheng [:josh] from comment #15)
> Hi HsinYi,
> I thought Thomas ask for our help in comment 11?
> Thanks

I see! Then for this request, I think we need to ask for some help from RelEng team.

I noted that emulator-kk tests on taskCluster had been enabled on try but not on b2g-inbound yet. I am not sure of the situation on b2g-inbound, i.e. what the current status and the plan for that branch. Do you happen to know that, Ryan? Also, is comment 11 something you could help? Thank you!

Comment 17

[James Lal [:lightsofapollo]]

I think for TC bits it is just a matter of copying the configs for emualtor-kk (then tests) as seen here https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/branches/try/job_flags.yml#130 to here https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/job_flags.yml

Comment 18

[Ryan VanderMeulen [:RyanVM]]

I'm not in RelEng, so no, I can't help. Maybe catlee can connect you with someone. And as for TaskCluster, I've had little to no involvement in that project, so I don't know what the status is with respect to enabling things on b2g-inbound. James would be your best bet for that.

Comment 19

[Thomas Zimmermann [:tzimmermann] [:tdz]]

> (In reply to Josh Cheng [:josh] from comment #13)
> > Dear HsinYi,
> > Could you please help to find someone for checking this bug?
> > Thanks!
> 
> Hi Josh,
> Looks Thomas is still actively working on this. Could you elaborate more on
> what additional support you need? Thank you thank you.

Well, I'm working on it on and off; whenever I have some time.

I was hoping for someone from releng or ateam to look at the error logs from cedar and suggest possible problems and solutions. The whole test infrastructure is mostly a black box to me.

Next I'll try what James suggested in comment 17 and see what happens.

Comment 20

[Thomas Zimmermann [:tzimmermann] [:tdz]]

(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #12)
> (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #10)
> > Let's see what happens...
> > 
> >   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=6bdbe725837d
> 
> It looks like Marionette still doesn't work on emulator-kk. With bug 1124567
> being fixed, it should now startup reliably. I guess that either the test
> scripts or Marionette need to be updated.

Looks like the CPP errors were transient. In recent pushes, these tests succeed.

Comment 21

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Attachment: Bug 1107678: Set Task Cluster job flags for emulator-kk on cedar

Copies TC try job flags for cedar branch, similar to what has been suggested in comment 17.

Comment 22

[Thomas Zimmermann [:tzimmermann] [:tdz]]

(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #21)
> Created attachment 8571929 [details] [diff] [review]
> Bug 1107678: Set Task Cluster job flags for emulator-kk on cedar
> 
> Copies TC try job flags for cedar branch, similar to what has been suggested
> in comment 17.

Let's see...

https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=e1c0048e8615

Comment 23

[James Lal [:lightsofapollo]]

Sorry my comment was directed at turning this on for b2g-inbound/mozilla-inbound/fx-team cedar has specific configs (see the branches folder for overrides per branch https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/branches/cedar/job_flags.yml

results would show up here for now https://treeherder.allizom.org/#/jobs?repo=cedar (note allizom not mozilla)

Comment 24

[Hsin-Yi Tsai (she/her) [:hsinyi]]

(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #22)
> (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #21)
> > Created attachment 8571929 [details] [diff] [review]
> > Bug 1107678: Set Task Cluster job flags for emulator-kk on cedar
> > 
> > Copies TC try job flags for cedar branch, similar to what has been suggested
> > in comment 17.
> 
> Let's see...
> 
> https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=e1c0048e8615

So, I checked marionette-webapi test log on cedar again. Basically Marionette-webapi startup correctly. I saw a bunch of marionette-webapi test passed (e.g. test_battery_status_not_charging.js, telephony tests, etc), though there still were several tests failed, such as test_dom_BluetoothManager_enabled.js, test_outgoing.js, test_massive_incoming_delete.js, etc. Thus, I'd suggest the next step is to ask for help from each module to investigate those failures first.

b.t.w., I saw emulator-kk auto tests on try Monday but the tests disappear now. Were they disabled?

Comment 25

[Hsin-Yi Tsai (she/her) [:hsinyi]]

(In reply to Hsin-Yi Tsai [:hsinyi] from comment #24)
> (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #22)
> > (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #21)
> > > Created attachment 8571929 [details] [diff] [review]
> > > Bug 1107678: Set Task Cluster job flags for emulator-kk on cedar
> > > 
> > > Copies TC try job flags for cedar branch, similar to what has been suggested
> > > in comment 17.
> > 
> > Let's see...
> > 
> > https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=e1c0048e8615
> 
> So, I checked marionette-webapi test log on cedar again. Basically
> Marionette-webapi startup correctly. I saw a bunch of marionette-webapi test
> passed (e.g. test_battery_status_not_charging.js, telephony tests, etc),
> though there still were several tests failed, such as
> test_dom_BluetoothManager_enabled.js, test_outgoing.js,
> test_massive_incoming_delete.js, etc. Thus, I'd suggest the next step is to
> ask for help from each module to investigate those failures first.
> 
> b.t.w., I saw emulator-kk auto tests on try Monday but the tests disappear
> now. Were they disabled?

And for marionette tests, it seems there have been busted issues since emulator-ics. Not really sure if marionette test bustage is kk-related or not.

Comment 26

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Hey,

Yes, maybe there's a problem with Mn on cedar. James told me yesterday that the new Task Cluster system can also run new tests on try. Let's see if this changes anything.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=38ca75fadf25

Comment 27

[Thomas Zimmermann [:tzimmermann] [:tdz]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=51f5b1ce471a

Comment 28

[Thomas Zimmermann [:tzimmermann] [:tdz]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8eb6e526b0a4

Comment 29

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Change the bug title as TPBL is deprecated. Treeherder is what we really care.

Comment 30

[Hsin-Yi Tsai (she/her) [:hsinyi]]

emulator-kk and kk tesets have been enabled by default on staging treeherder https://treeherder.allizom.org/#/jobs?repo=try

James' team will be working on make the kk status as good as ics. Then we need every team's assistance in porting kk tests and resolving test failure.

Comment 31

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Let's see what works

  https://treeherder.mozilla.org/#/jobs?repo=try&revision=c10d67943be4

Comment 32

[James Lal [:lightsofapollo]]

Here is a earlier try push with some changes that are about to land... https://treeherder.allizom.org/#/jobs?repo=try&revision=7b179bb38322

Comment 33

[Thomas Zimmermann [:tzimmermann] [:tdz]]

(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #31)
> Let's see what works
> 
>   https://treeherder.mozilla.org/#/jobs?repo=try&revision=c10d67943be4

This had an internal OS error. Let's try again

  https://treeherder.allizom.org/#/jobs?repo=try&revision=a6265865b27a

Comment 34

[Thomas Zimmermann [:tzimmermann] [:tdz]]

https://treeherder.allizom.org/#/jobs?repo=try&revision=ee7afc4b37c3

Comment 35

[Dylan Oliver [:doliver]]

Thomas or Hsin-Yi, it's been a while since this one was touched and we are trying to prioritize the open build/config requests. Can you provide an update on whether this is still valid and needed, and what help you think we need from RelEng to complete it?

Comment 36

[Hsin-Yi Tsai (she/her) [:hsinyi]]

(In reply to Dylan Oliver [:doliver] from comment #35)
> Thomas or Hsin-Yi, it's been a while since this one was touched and we are
> trying to prioritize the open build/config requests. Can you provide an
> update on whether this is still valid and needed, and what help you think we
> need from RelEng to complete it?

Hi Dylan,
Per previous discussion, emu-x86-kk (bug 1144970) is the focus, and the team isn't working on this. Once we have x86-kk enabled on TC then kk (arm-based) isn't necessary.  According to this, I'd say this bug (targeting on arm-based) is not needed.

Comment 37

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Please see Hsinyi's answer.