Closed Bug 1107678 Opened 7 years ago Closed 6 years ago

[meta] Enable tests on b2g emulator KK on Treeherder

Categories

(Firefox OS Graveyard :: Emulator, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(tracking-b2g:+)

RESOLVED WONTFIX
tracking-b2g +

People

(Reporter: slee, Unassigned)

References

Details

(Whiteboard: [b2g-build-support])

Attachments

(2 files)

This bug is to track the progress of tests on b2g kk emulator.
tracking-b2g: --- → +
Component: TaskCluster → Video/Audio
Product: Testing → Core
Hi James,

could we have update about testing environment preparation? Let us know if you back from holiday. Many thanks.
Flags: needinfo?(jlal)
Change the component because I think this isn't only for video/audio modules but in general we want to enable tests that already run on emulator-ics. Feel free to correct me!
Component: Video/Audio → Emulator
Product: Core → Firefox OS
Summary: [meta] Enable tests on b2g emulator KK → [meta] Enable tests on b2g emulator KK on TBPL
Depends on: 1108271
No longer blocks: emulator-l_taskcluster
Depends on: 1124567
No longer depends on: 1108271
Blocks: 1033935
As emulator-kk should start up reliably now, we want to run tests on it. As a first step, we'd like to execute the tests on cedar to get an idea what's missing.
Assignee: nobody → tzimmermann
Status: NEW → ASSIGNED
Attachment #8562853 - Flags: review?(rail)
Attachment #8562853 - Flags: review?(rail) → review+
FTR, the changes are not in production yet. There should be an extra comment here when they are.
On irc, :ahal mentioned something about cedar re-configuration after the patch lands. Will that happen automatically or is there something to do from my side?
Flags: needinfo?(rail)
A buildduty person will do this. Usually it happens almost every day. The bug will be updated when it's in production.
Flags: needinfo?(rail)
Josh,

can you try to find someone from RelEng to help with the integration of emulator-kk tests on Cedar and b2g-inbound? Thanks!
Flags: needinfo?(jocheng)
(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #10)
> Let's see what happens...
> 
>   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=6bdbe725837d

It looks like Marionette still doesn't work on emulator-kk. With bug 1124567 being fixed, it should now startup reliably. I guess that either the test scripts or Marionette need to be updated.
Dear HsinYi,
Could you please help to find someone for checking this bug?
Thanks!
Flags: needinfo?(jocheng) → needinfo?(htsai)
(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #12)
> (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #10)
> > Let's see what happens...
> > 
> >   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=6bdbe725837d
> 
> It looks like Marionette still doesn't work on emulator-kk. With bug 1124567
> being fixed, it should now startup reliably. I guess that either the test
> scripts or Marionette need to be updated.

From log on emulator-ics on treeherder, there's a message "MARIONETTE LOG: INFO: received mozbrowserloadend for app" after the message "MARIONETTE LOG: INFO: waiting for mozbrowserloadend." However, in [1] marionette was waiting for mozbrowserloadend but didn't receive it eventually. This seems the cause of the failure.

[1] http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-emulator-kk/1424269816/cedar_ubuntu64_vm-b2g-emulator-kk_test-marionette-webapi-bm123-tests1-linux64-build0.txt.gz

(In reply to Josh Cheng [:josh] from comment #13)
> Dear HsinYi,
> Could you please help to find someone for checking this bug?
> Thanks!

Hi Josh,
Looks Thomas is still actively working on this. Could you elaborate more on what additional support you need? Thank you thank you.
Flags: needinfo?(htsai) → needinfo?(jocheng)
Hi HsinYi,
I thought Thomas ask for our help in comment 11?
Thanks
Flags: needinfo?(htsai)
(In reply to Josh Cheng [:josh] from comment #15)
> Hi HsinYi,
> I thought Thomas ask for our help in comment 11?
> Thanks

I see! Then for this request, I think we need to ask for some help from RelEng team.

I noted that emulator-kk tests on taskCluster had been enabled on try but not on b2g-inbound yet. I am not sure of the situation on b2g-inbound, i.e. what the current status and the plan for that branch. Do you happen to know that, Ryan? Also, is comment 11 something you could help? Thank you!
Flags: needinfo?(ryanvm)
Flags: needinfo?(jocheng)
Flags: needinfo?(htsai)
I think for TC bits it is just a matter of copying the configs for emualtor-kk (then tests) as seen here https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/branches/try/job_flags.yml#130 to here https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/job_flags.yml
Flags: needinfo?(jlal)
I'm not in RelEng, so no, I can't help. Maybe catlee can connect you with someone. And as for TaskCluster, I've had little to no involvement in that project, so I don't know what the status is with respect to enabling things on b2g-inbound. James would be your best bet for that.
Flags: needinfo?(ryanvm) → needinfo?(catlee)
> (In reply to Josh Cheng [:josh] from comment #13)
> > Dear HsinYi,
> > Could you please help to find someone for checking this bug?
> > Thanks!
> 
> Hi Josh,
> Looks Thomas is still actively working on this. Could you elaborate more on
> what additional support you need? Thank you thank you.

Well, I'm working on it on and off; whenever I have some time.

I was hoping for someone from releng or ateam to look at the error logs from cedar and suggest possible problems and solutions. The whole test infrastructure is mostly a black box to me.

Next I'll try what James suggested in comment 17 and see what happens.
(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #12)
> (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #10)
> > Let's see what happens...
> > 
> >   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=6bdbe725837d
> 
> It looks like Marionette still doesn't work on emulator-kk. With bug 1124567
> being fixed, it should now startup reliably. I guess that either the test
> scripts or Marionette need to be updated.

Looks like the CPP errors were transient. In recent pushes, these tests succeed.
Copies TC try job flags for cedar branch, similar to what has been suggested in comment 17.
(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #21)
> Created attachment 8571929 [details] [diff] [review]
> Bug 1107678: Set Task Cluster job flags for emulator-kk on cedar
> 
> Copies TC try job flags for cedar branch, similar to what has been suggested
> in comment 17.

Let's see...

https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=e1c0048e8615
Sorry my comment was directed at turning this on for b2g-inbound/mozilla-inbound/fx-team cedar has specific configs (see the branches folder for overrides per branch https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/branches/cedar/job_flags.yml

results would show up here for now https://treeherder.allizom.org/#/jobs?repo=cedar (note allizom not mozilla)
(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #22)
> (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #21)
> > Created attachment 8571929 [details] [diff] [review]
> > Bug 1107678: Set Task Cluster job flags for emulator-kk on cedar
> > 
> > Copies TC try job flags for cedar branch, similar to what has been suggested
> > in comment 17.
> 
> Let's see...
> 
> https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=e1c0048e8615

So, I checked marionette-webapi test log on cedar again. Basically Marionette-webapi startup correctly. I saw a bunch of marionette-webapi test passed (e.g. test_battery_status_not_charging.js, telephony tests, etc), though there still were several tests failed, such as test_dom_BluetoothManager_enabled.js, test_outgoing.js, test_massive_incoming_delete.js, etc. Thus, I'd suggest the next step is to ask for help from each module to investigate those failures first.

b.t.w., I saw emulator-kk auto tests on try Monday but the tests disappear now. Were they disabled?
(In reply to Hsin-Yi Tsai [:hsinyi] from comment #24)
> (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #22)
> > (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #21)
> > > Created attachment 8571929 [details] [diff] [review]
> > > Bug 1107678: Set Task Cluster job flags for emulator-kk on cedar
> > > 
> > > Copies TC try job flags for cedar branch, similar to what has been suggested
> > > in comment 17.
> > 
> > Let's see...
> > 
> > https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=e1c0048e8615
> 
> So, I checked marionette-webapi test log on cedar again. Basically
> Marionette-webapi startup correctly. I saw a bunch of marionette-webapi test
> passed (e.g. test_battery_status_not_charging.js, telephony tests, etc),
> though there still were several tests failed, such as
> test_dom_BluetoothManager_enabled.js, test_outgoing.js,
> test_massive_incoming_delete.js, etc. Thus, I'd suggest the next step is to
> ask for help from each module to investigate those failures first.
> 
> b.t.w., I saw emulator-kk auto tests on try Monday but the tests disappear
> now. Were they disabled?

And for marionette tests, it seems there have been busted issues since emulator-ics. Not really sure if marionette test bustage is kk-related or not.
Hey,

Yes, maybe there's a problem with Mn on cedar. James told me yesterday that the new Task Cluster system can also run new tests on try. Let's see if this changes anything.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=38ca75fadf25
Flags: needinfo?(catlee)
Change the bug title as TPBL is deprecated. Treeherder is what we really care.
Summary: [meta] Enable tests on b2g emulator KK on TBPL → [meta] Enable tests on b2g emulator KK on Treeherder
emulator-kk and kk tesets have been enabled by default on staging treeherder https://treeherder.allizom.org/#/jobs?repo=try

James' team will be working on make the kk status as good as ics. Then we need every team's assistance in porting kk tests and resolving test failure.
Here is a earlier try push with some changes that are about to land... https://treeherder.allizom.org/#/jobs?repo=try&revision=7b179bb38322
(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #31)
> Let's see what works
> 
>   https://treeherder.mozilla.org/#/jobs?repo=try&revision=c10d67943be4

This had an internal OS error. Let's try again

  https://treeherder.allizom.org/#/jobs?repo=try&revision=a6265865b27a
Priority: -- → P1
Assignee: tzimmermann → nobody
Status: ASSIGNED → NEW
No longer blocks: emu-x86-kk-tests
Whiteboard: [tc-build-support]
Whiteboard: [tc-build-support] → [b2g-build-support]
Thomas or Hsin-Yi, it's been a while since this one was touched and we are trying to prioritize the open build/config requests. Can you provide an update on whether this is still valid and needed, and what help you think we need from RelEng to complete it?
Flags: needinfo?(tzimmermann)
Flags: needinfo?(htsai)
(In reply to Dylan Oliver [:doliver] from comment #35)
> Thomas or Hsin-Yi, it's been a while since this one was touched and we are
> trying to prioritize the open build/config requests. Can you provide an
> update on whether this is still valid and needed, and what help you think we
> need from RelEng to complete it?

Hi Dylan,
Per previous discussion, emu-x86-kk (bug 1144970) is the focus, and the team isn't working on this. Once we have x86-kk enabled on TC then kk (arm-based) isn't necessary.  According to this, I'd say this bug (targeting on arm-based) is not needed.
Flags: needinfo?(htsai)
Please see Hsinyi's answer.
Flags: needinfo?(tzimmermann)
Status: NEW → RESOLVED
Closed: 6 years ago
Priority: P1 → --
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.