B2G Emulator: lollipop build

RESOLVED FIXED in Firefox OS v2.2

Status

RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: vicamo, Assigned: seinlin)

Tracking

unspecified
2.2 S7 (6mar)
ARM
Gonk (Firefox OS)
Dependency tree / graph

Firefox Tracking Flags

(blocking-b2g:2.2+, b2g-v2.2 fixed, b2g-master fixed)

Details

Attachments

(6 attachments, 1 obsolete attachment)

(Reporter)

Description

4 years ago
+++ This bug was initially created as a clone of Bug #957526 +++

Would be nice to have something sweet before the end of year :)
(Reporter)

Comment 3

4 years ago
From Android Lollipop and on, emulator binaries are provided by a new repository android-emulator [1]. Emulator specific products are mini_emulator_{arm,arm64,x86,x86_64}. Only mini_emulator_x86_64 compiles emulator binaries by setting BUILD_EMULATOR to true.

[1]: https://android.googlesource.com/platform/prebuilts/android-emulator/
(Reporter)

Comment 4

4 years ago
Correct: |aosp_arm.mk| includes |full.mk|, and |full.mk| still includes |emulator.mk| in lollipop. So we can still use full-eng as Mozilla's emulator product target, but it won't include emulator binaries built from source :(
(Reporter)

Comment 5

4 years ago
Finished first run external/qemu merge (https://github.com/vicamo/b2g_platform_external_qemu/tree/bugzilla/1116074/emulator-lollipop, and https://github.com/vicamo/b2g_platform_external_qemu/tree/next/aosp/master for aosp/master). Need an lollipop environment to verify it.
(Reporter)

Comment 6

4 years ago
(In reply to Vicamo Yang [:vicamo][:vyang] from comment #5)
> Need an lollipop environment to verify it.

Verified merge commits are kept in this branch: https://github.com/vicamo/b2g_platform_external_qemu/tree/bugzilla/1116074/successfully-built
This was done by building all against existing emulator-kk source tree. However, the next to-be-verified commit failed due to toolchain compatibilities. Need another way to make further progress.
(Reporter)

Comment 7

4 years ago
https://github.com/vicamo/b2g_platform_external_qemu/tree/bugzilla/1116074/emulator-lollipop passes built agains kitkat-dev tree, to try lollipop later.

Updated

4 years ago
Blocks: 905098
(Reporter)

Comment 9

4 years ago
xpcom/ds/TimeStamp_posix.cpp:274:78: error: call to 'sprintf' declared with attribute warning: sprintf is often misused; please use snprintf [-Werror]

A lot of sprintf errors when compiling Gecko :(
(Reporter)

Comment 10

4 years ago
Created attachment 8552344 [details] [review]
Github PR for B2G

This pr add two new emulator targets emulator-l and emulator-x86-l.
(Reporter)

Comment 11

4 years ago
Created attachment 8552350 [details] [review]
Github PR for b2g-manifest

This pr add two new emulator targets emulator-l and emulator-x86-l.

Note that before all necessary heads being created for new projects added in this change, you may need https://github.com/vicamo/b2g_manifest/tree/bugzilla/1116074/use-vicamo-s-repositories-temporarily to fetch the whole source tree.
(Reporter)

Comment 12

4 years ago
(In reply to Vicamo Yang [:vicamo][:vyang] from comment #11)
> Note that before all necessary heads being created for new projects added in
> this change, you may need
> https://github.com/vicamo/b2g_manifest/tree/bugzilla/1116074/use-vicamo-s-
> repositories-temporarily to fetch the whole source tree.

All the necessary new branch heads are:

1. https://github.com/mozilla-b2g/platform_external_qemu/ : "b2g-lollipop" => refs/tags/android-5.0.0_r6 (006a174)
2. https://github.com/mozilla-b2g/platform_hardware_ril/ : "b2g-lollipop" => refs/tags/android-5.0.0_r6 (5ffec08)
3. https://github.com/mozilla-b2g/device_generic_goldfish : "b2g-5.0.0_r6" => refs/tags/android-5.0.0_r6 (82a4dc7)
4. https://github.com/mozilla-b2g/android-sdk : "b2g-5.0.0_r6" => refs/tags/android-5.0.0_r6 (d4ce15e)
(Reporter)

Comment 13

4 years ago
per comment 9, blocked by bug 92064.
Depends on: 92064
(Reporter)

Comment 14

4 years ago
TODO:

0. Figure out how to compile gecko on Mac OS X 10.10, or any other viable platforms.

1. Pull requests for repositories listed in comment 12 can only be sent after the branches being created. So far only external/qemu is ready.

2. Merge hardware/ril. Not really blocked by anything else but item 0. Need a way to test merged RIL functions.

3. Kernel changes. Lollipop x86_64 emulator is using Android goldfish kernel v3.10, but others are still on v3.4. So probably we may reuse v3.4 until we're going to go x86_64, then we'll need manifest change as well as a new branch?
(Reporter)

Comment 15

4 years ago
(In reply to Vicamo Yang [:vicamo][:vyang] from comment #12)
> All the necessary new branch heads are:
> 
> 4. https://github.com/mozilla-b2g/android-sdk : "b2g-5.0.0_r6" =>
> refs/tags/android-5.0.0_r6 (d4ce15e)

I'm not quite sure with this one, but due to bug 92064 I can't not have a successful build to make sure that.
(Reporter)

Comment 16

4 years ago
(In reply to Vicamo Yang [:vicamo][:vyang] from comment #12)
> 1. https://github.com/mozilla-b2g/platform_external_qemu/ : "b2g-lollipop"
> => refs/tags/android-5.0.0_r6 (006a174)
> 2. https://github.com/mozilla-b2g/platform_hardware_ril/ : "b2g-lollipop" =>
> refs/tags/android-5.0.0_r6 (5ffec08)

Depends on b2g-kitkat merge on external/qemu, hardware/ril. See https://bugzilla.mozilla.org/show_bug.cgi?id=1083650#c21
(Reporter)

Comment 17

4 years ago
(In reply to Vicamo Yang [:vicamo][:vyang] from comment #14)
> TODO:
> 
> 0. Figure out how to compile gecko on Mac OS X 10.10, or any other viable
> platforms.

Gecko compiles on Linux.

Besides, it seems we may no longer compile qemu along with lollipop source tree. At least two errors found. The first one, "no input source" when compiling gdbstub-xml.o, can be resolved with a patch to Makefile.common [1]. The second one, "PIC register clobbered" in function no_gs_ioctl, takes some time to analyze what happened. Any way, I think I should go the AOSP way -- prebuild emulator binaries and save them in prebuilts/android-emulator [2] as AOSP does.

[1]: https://github.com/vicamo/b2g_platform_external_qemu/tree/bugzilla/1116074/fix-gdbstub-xml-o-no-input-source
[2]: https://android.googlesource.com/platform/prebuilts/android-emulator/
No longer depends on: 92064

Updated

4 years ago
Blocks: 1124557

Comment 18

4 years ago
(In reply to Vicamo Yang [:vicamo][:vyang] from comment #16)
> (In reply to Vicamo Yang [:vicamo][:vyang] from comment #12)
> > 1. https://github.com/mozilla-b2g/platform_external_qemu/ : "b2g-lollipop"
> > => refs/tags/android-5.0.0_r6 (006a174)
> > 2. https://github.com/mozilla-b2g/platform_hardware_ril/ : "b2g-lollipop" =>
> > refs/tags/android-5.0.0_r6 (5ffec08)
> 
> Depends on b2g-kitkat merge on external/qemu, hardware/ril. See
> https://bugzilla.mozilla.org/show_bug.cgi?id=1083650#c21

I will help on the merging thing. Will keep you posted here. :)

Updated

4 years ago
Status: UNCONFIRMED → NEW
Ever confirmed: true

Comment 19

4 years ago
(In reply to Edgar Chen [:edgar][:echen] from comment #18)
> (In reply to Vicamo Yang [:vicamo][:vyang] from comment #16)
> > (In reply to Vicamo Yang [:vicamo][:vyang] from comment #12)
> > > 1. https://github.com/mozilla-b2g/platform_external_qemu/ : "b2g-lollipop"
> > > => refs/tags/android-5.0.0_r6 (006a174)
> > > 2. https://github.com/mozilla-b2g/platform_hardware_ril/ : "b2g-lollipop" =>
> > > refs/tags/android-5.0.0_r6 (5ffec08)
> > 
> > Depends on b2g-kitkat merge on external/qemu, hardware/ril. See
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1083650#c21
> 
> I will help on the merging thing. Will keep you posted here. :)

Done,
https://github.com/mozilla-b2g/platform_external_qemu/commits/b2g-kitkat
https://github.com/mozilla-b2g/platform_hardware_ril/commits/b2g-kitkat
(Reporter)

Comment 20

4 years ago
(In reply to Edgar Chen [:edgar][:echen] from comment #19)
> Done,
> https://github.com/mozilla-b2g/platform_external_qemu/commits/b2g-kitkat
> https://github.com/mozilla-b2g/platform_hardware_ril/commits/b2g-kitkat

Done? You call this "Done"???? I've told you I've prepare those merged branches in 
https://github.com/vicamo/b2g_platform_external_qemu/tree/next/b2g/b2g-jellybean
https://github.com/vicamo/b2g_platform_external_qemu/tree/next/b2g/b2g-kitkat
https://github.com/vicamo/b2g_platform_hardware_ril/tree/next/b2g/b2g-jellybean
https://github.com/vicamo/b2g_platform_hardware_ril/tree/next/b2g/b2g-kitkat
Now you pushed another commit that has NOTHING changed but just a new commit. What do you mean? Do you know I had to re-merge all since the merged commit is different?
(Reporter)

Comment 21

4 years ago
Since you have other opinion on how this bug should be done, and it has never been assigned to me, good luck.
(Assignee)

Updated

4 years ago
Depends on: 1121378, 1128837
(Assignee)

Comment 22

4 years ago
Created attachment 8562612 [details] [review]
Github PR for b2g-manifest

Thomas, Could you help me review this patch? Thanks!

I think we need to have emulator-l build ok and on try server to avoid b2g build base on lollipop build break by other commit.

As comment in manifest, hardware/ril and qemu need to be updated.
Attachment #8552350 - Attachment is obsolete: true
Attachment #8562612 - Flags: review?(tzimmermann)
(Assignee)

Comment 23

4 years ago
Created attachment 8562615 [details] [review]
Fix qemu compile error

Thomas,

This could be a temporary solution for emulator-l build successfully. I think we need wait bug 1121378 resolved fixed or move all changes on b2g-kitkat to lollipop.

Could you review this patch? Thanks!
Attachment #8562615 - Flags: review?(tzimmermann)
(Assignee)

Comment 24

4 years ago
Created attachment 8562616 [details] [review]
Update goldfish to 5.0.0_r6

Thomas, I rebase our changes to 5.0.0_r6. Could you review this pull request? Thanks!
Attachment #8562616 - Flags: review?(tzimmermann)
(Assignee)

Comment 25

4 years ago
Comment on attachment 8552344 [details] [review]
Github PR for B2G

Looks good to me.
Attachment #8552344 - Flags: review+
(Assignee)

Updated

4 years ago
Assignee: nobody → kli
Oh cool!

I have a bunch of other things on the top of my to-do list. I'll take a look during the next days. Sorry for the delay.
I get the following build error:

Import includes file: out/target/product/generic/obj/SHARED_LIBRARIES/libsqlite_intermediates/import_includes
hardware/ril/reference-ril/atchannel.c:38:29: fatal error: linux/omap_csmi.h: No such file or directory
 #include <linux/omap_csmi.h>
                             ^
compilation terminated.
Comment on attachment 8562615 [details] [review]
Fix qemu compile error

Rubber-stamping.
Attachment #8562615 - Flags: review?(tzimmermann) → review+
Comment on attachment 8562616 [details] [review]
Update goldfish to 5.0.0_r6

Rubber-stamping.
Attachment #8562616 - Flags: review?(tzimmermann) → review+
Comment on attachment 8562612 [details] [review]
Github PR for b2g-manifest

I'm not sure if I can just r+ this, but it works for me with the remaining nit fixed.

Make sure you fix the latest comment in the pull request before landing this change!
Attachment #8562612 - Flags: review?(tzimmermann) → review+
Thanks for working on this bug!

Updated

4 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 32

4 years ago
(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #30)
> Comment on attachment 8562612 [details] [review]
> Github PR for b2g-manifest
> 
> I'm not sure if I can just r+ this, but it works for me with the remaining
> nit fixed.
> 
> Make sure you fix the latest comment in the pull request before landing this
> change!

Thomas, Thanks! 

I updated my latest patch to PR. To use repo from caf, the name should be platform/hardware/ril. As the comment in manifest, let's update qemu and hardware/ril in corresponding bugs.
(Assignee)

Comment 33

4 years ago
v2.2 support lollipop, I think we need emulator-l on it.
blocking-b2g: --- → 2.2?
(Assignee)

Updated

4 years ago
Depends on: 1113054
No longer depends on: 1128837
(Assignee)

Comment 34

4 years ago
Created attachment 8565080 [details] [review]
Build emulators locally

Thomas, We also need to update build to have emulators get built. Could you have a look to this patch? Thanks!
Attachment #8565080 - Flags: review?(tzimmermann)
Comment on attachment 8565080 [details] [review]
Build emulators locally

I don't generally object this patch, but could we pull prebuilds/android-emulator instead? It seems the 'canonical' way of getting an emulator binary.

For future versions of our emulator, we would just update this repository. I guess that's also nice for reducing workload on our test machines.
Comment on attachment 8565080 [details] [review]
Build emulators locally

I left some suggestions in the pull request.
Attachment #8565080 - Flags: review?(tzimmermann) → review+
I was able to build emulator-l locally. Thank you very much for working on this bug.

Comment 38

4 years ago
Triage is taking as a blocker given Lollipop on 2.2. And if it's blocking testing, it will block the release.

Also ni? to Thomas to see if he can resolve this as WORKSFORME.
blocking-b2g: 2.2? → 2.2+
Flags: needinfo?(tzimmermann)
(Assignee)

Comment 40

4 years ago
Now emulator can be built, but it will not function properly before bug 1113054 and bug 1121378 get fixed.
(Assignee)

Updated

4 years ago
Depends on: 1134982
(Assignee)

Comment 41

4 years ago
To correct comment 40, hardware/ril should be upgraded in bug 1134982 and not bug 1113054.
(Assignee)

Comment 42

4 years ago
Created attachment 8566978 [details] [review]
[v2.2]  Github PR for b2g-manifest
(In reply to Kai-Zhen Li [:kli][:seinlin] from comment #40)
> Now emulator can be built, but it will not function properly before bug
> 1113054 and bug 1121378 get fixed.

Bug 1121378 is purely experimental and we should not let anything depend on it. My work on it rather shows that it is probably not possible to use a single QEMU binary, because the guest's OpenGL driver depends on specific QEMU versions.
Flags: needinfo?(tzimmermann)
(In reply to Stephany Wilkes from comment #38)
> Triage is taking as a blocker given Lollipop on 2.2. And if it's blocking
> testing, it will block the release.
> 
> Also ni? to Thomas to see if he can resolve this as WORKSFORME.

I'm not sure what the exact criteria for blockers are. With these patches landed, we can only build emulator-l. If you what to run actual tests on it, that will still be a long way.
'Building' allows us to verify that a specific revision of Gecko compiles in an environment of Android L.
Given that this may soon be blocking Nexus 5-L job visibility, I think blocking status is warranted.
I've had to backout the b2g-manifest change as this broke b2g bumper in production:
https://github.com/mozilla-b2g/b2g-manifest/commit/6d817977d2c5404b03a096be16eae2bb383c6fea

Please note the travis tests failed, which indicates that b2g bumper will break:
https://travis-ci.org/mozilla-b2g/b2g-manifest/builds/51473586

I'm not sure if there is a way to configure the b2g-manifest github project to only allow merges from forks with passing tests, but this might be something we could add to help increase stability.

Thanks,
Pete
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Last lines of log file buildbot-master66.bb.releng.usw2.mozilla.com:/builds/b2g_bumper/master.log:

05:09:43  WARNING - https://git.mozilla.org/external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8:refs/tags/android-5.0.0_r6 - got output: fatal: remote error: FATAL: R any external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8 gitweb DENIED by fallthru
05:09:43  WARNING - (or you mis-spelled the reponame)
05:09:43     INFO - https://git.mozilla.org/external/caf/platform/hardware/qcom/wlan:refs/tags/android-5.0.0_r6 - got output: 2e54754cc0529d26ccac37ed291600048adbf6c0	refs/tags/android-5.0.0_r6
05:09:44     INFO - https://git.mozilla.org/external/caf/platform/hardware/ril:refs/tags/android-5.0.0_r6 - got output: 71dfa8228ad0d6cdf6bac0426ac59404ab74b7f3	refs/tags/android-5.0.0_r6
05:09:45     INFO - https://git.mozilla.org/external/caf/platform/hardware/qcom/sensors:refs/tags/android-5.0.0_r6 - got output: 3724fd91ef5183684d97e2bf1d7ff948faabe090	refs/tags/android-5.0.0_r6
05:09:46     INFO - https://git.mozilla.org/external/caf/platform/hardware/qcom/media:refs/tags/android-5.0.0_r6 - got output: 758a80fbb178b5663d4edbb46944b2dc553cb1ca	refs/tags/android-5.0.0_r6
05:10:14  WARNING - Returned 128 - sleeping and retrying
05:10:14  WARNING - https://git.mozilla.org/external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8:refs/tags/android-5.0.0_r6 - got output: fatal: remote error: FATAL: R any external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8 gitweb DENIED by fallthru
05:10:14  WARNING - (or you mis-spelled the reponame)
05:10:44  WARNING - Returned 128 - sleeping and retrying
05:10:44  WARNING - https://git.mozilla.org/external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8:refs/tags/android-5.0.0_r6 - got output: fatal: remote error: FATAL: R any external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8 gitweb DENIED by fallthru
05:10:44  WARNING - (or you mis-spelled the reponame)
05:11:15  WARNING - Returned 128 - sleeping and retrying
05:11:15  WARNING - https://git.mozilla.org/external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8:refs/tags/android-5.0.0_r6 - got output: fatal: remote error: FATAL: R any external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8 gitweb DENIED by fallthru
05:11:15  WARNING - (or you mis-spelled the reponame)
05:11:45    ERROR - Couldn't resolve reference https://git.mozilla.org/external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8 refs/tags/android-5.0.0_r6
05:11:45     INFO - needs sync? git://codeaurora.org//platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8 -> https://git.mozilla.org/external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8
05:11:45    FATAL - couldn't resolve some refs; exiting
05:11:45    FATAL - Running post_fatal callback...
05:11:45    FATAL - Exiting -1
It is not clear why nagios did not alert, I only discovered this by coincidence. Raising a separate bug for that...
It looks like some CAF-based repo is not available. CAF was down earlier today. Could this be the cause of this problem?

Otherwise, how can this be fixed? Do we need to mirror these repos explicitly before landing the patch set? (Sorry for asking, I don't know enough about the infrastructure.)
Flags: needinfo?(pmoore)
See Also: → bug 1135030
Hi Thomas,

It looks like we are mirroring to:
http://git.mozilla.org/?p=external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.6.git;a=summary

but not to:
http://git.mozilla.org/?p=external/caf/platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8.git;a=summary

I'm guessing the problem is just that we need to set up the mirroring first, before landing the b2g-manifest change.

If the travis tests for b2g-manifest pass, b2g bumper should run fine in production.

Are you ok to request the mirroring? Maybe there is an existing bug for the previous x86_64-linux-glibc2.11-4.6 version which can be adjusted slightly to change to x86_64-linux-glibc2.11-4.8.

Ideally, b2g bumper would mirror a repo that it spotted it is missing, but we're not there yet, so we have to manually set up the mirroring first, unfortunately.

Pete
Flags: needinfo?(pmoore)
Thanks, I'll request mirroring of branches, according to the changes in the manifest.
Depends on: 1135041
See Also: → bug 1135041

Comment 53

4 years ago
(In reply to Kai-Zhen Li [:kli][:seinlin] from comment #40)
> Now emulator can be built, but it will not function properly before bug
> 1113054 and bug 1121378 get fixed.

Correct the dependence based on above comment.
No longer depends on: 1113054
(Assignee)

Comment 54

4 years ago
(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #52)
> Thanks, I'll request mirroring of branches, according to the changes in the
> manifest.

Sorry, I forgot to mirror the new added repos before merge the manifest.

Comment 55

4 years ago
b2g-manifest PRs should not be merged if tests fail. Be sure to check the test status next time.
No longer depends on: 1121378
(Assignee)

Comment 56

4 years ago
bug 1135041 is resolved fixed and test result is green.  Merge again,

https://github.com/mozilla-b2g/b2g-manifest/commit/4f43590df7257b1646c8eb8d74a75faef7897d07
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
v2.2: https://github.com/mozilla-b2g/b2g-manifest/commit/9852a31075443f525bb7d869f7cd1694ebb440cd
status-b2g-v2.2: --- → fixed
status-b2g-master: --- → fixed
Target Milestone: --- → 2.2 S7 (6mar)

Updated

3 years ago
Blocks: 1177234
No longer blocks: 1124557
You need to log in before you can comment on or make changes to this bug.