920198 - generate "FxOS Simulator" builds for B2G

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per meeting w/myk: (He's currently generating these manually on a best-effort-basis, and would be happy to stop once automation takes over.).

1) FxOS Simulator builds are for external third parties to develop apps for B2G, without having a physical device.
2) there is nothing proprietary in these, so these builds are fine to publish publicly on ftp.m.o. Therefore, no legal bug filed.

3) FxOS Simulator builds bundles Simulator addon, B2Gdesktop builds and gaia. See details in bug#920186. Unknown if this needs signing.

4) Locales: Myk's currently using a set of locales at https://github.com/mozilla/r2d2b2g/blob/master/build/languages.json. Note that this list is always a subset of languages_all.json. Whether this should continue as separate list, or be merged into one of the existing lists, is something that we *think* can be figured out later. If using this 3rd list complicates mechanics, please say so.

5) Updates: There's some updating functionality when new versions of addon are posted to addons.m.o, but this is scoped out of this for now. Possible ideas include getting updates from an update.xml file on well known location on ftp.m.o, or maybe using balrog/aus?

6) These builds should be generated for every OS that we currently generate B2G desktop builds.

7) These builds should be on m-c and on try and on the v1.2 (aurora) branch. Once v1.2 moves from aurora to new-to-be-setup-repo, these should be moved from aurora also.

Comment 1

[Ed Morley [:emorley]]

(In reply to John O'Duinn [:joduinn] from comment #0)
> 7) These builds should be on m-c and on try and on the v1.2 (aurora) branch.

Note that this doesn't meet https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy so these jobs will likely not be visible in the default view - if that's fine then ignore this comment :-)

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Ed Morley [:edmorley UTC+1] from comment #1)
> (In reply to John O'Duinn [:joduinn] from comment #0)
> > 7) These builds should be on m-c and on try and on the v1.2 (aurora) branch.
> 
> Note that this doesn't meet
> https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy so these jobs will
> likely not be visible in the default view - if that's fine then ignore this
> comment :-)

emorley: Dont follow - even after reading this policy wiki page. I guess that myk would like these builds to be visible, so can you elaborate?

Comment 3

[Ed Morley [:emorley]]

Comment 0 mentions only running these on m-c and try.

https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#3.29_Runs_on_mozilla-central_and_all_trees_that_merge_into_it says:

3) Runs on mozilla-central and all trees that merge into it 
 * Otherwise job failures when tree X merges into mozilla-central will not be attributable to a single changeset, resulting in either tree closure or backout of the entire merge (see requirement #2).

Requirement #2 being:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#2.29_Breakage_is_expected_to_be_followed_by_tree_closure_or_backout

Comment 4

[Myk Melez [:myk] [@mykmelez]]

(In reply to John O'Duinn [:joduinn] from comment #2)
> emorley: Dont follow - even after reading this policy wiki page. I guess
> that myk would like these builds to be visible, so can you elaborate?

Yes, I would indeed like these builds to be visible, so I guess that means we need to run them "on mozilla-central and all trees that merge into it"!

Comment 5

[Ed Morley [:emorley]]

Sounds good to me :-)

Comment 6

[Alexandre Poirot [:ochameau]]

A quick update about bug 944451.
Once landed, it will allow building simulator xpis from mozilla-central repo.

It isn't r+ yet, so details may change, but you just need to run a regular b2g desktop build, with GAIADIR being set, and, also set FXOS_SIMULATOR=1. You will then find simulator xpis in $(objdir)/dist/fxos-simulator-%(version)-%(platform).xpi

We are planning to build 1.4+ simulator xpis like this. So we do not have any 1.4 builds yet.
We start receiving more and more request for 1.4 builds, so I'd like to know if there is anything we can do to ease building such xpis.

What about adding export FXOS_SIMULATOR=1 to each b2g desktop nightly mozconfigs?
http://mxr.mozilla.org/mozilla-central/source/b2g/config/mozconfigs/linux64_gecko/nightly
And then we would just need to ensure copying xpis from dist directory to the ftp.

Comment 7

[Aki Sasaki (not active)]

(In reply to Alexandre Poirot (:ochameau) from comment #6)
> A quick update about bug 944451.
> Once landed, it will allow building simulator xpis from mozilla-central repo.
> 
> It isn't r+ yet, so details may change, but you just need to run a regular
> b2g desktop build, with GAIADIR being set, and, also set FXOS_SIMULATOR=1.
> You will then find simulator xpis in
> $(objdir)/dist/fxos-simulator-%(version)-%(platform).xpi
> 
> We are planning to build 1.4+ simulator xpis like this. So we do not have
> any 1.4 builds yet.
> We start receiving more and more request for 1.4 builds, so I'd like to know
> if there is anything we can do to ease building such xpis.
> 
> What about adding export FXOS_SIMULATOR=1 to each b2g desktop nightly
> mozconfigs?
> http://mxr.mozilla.org/mozilla-central/source/b2g/config/mozconfigs/
> linux64_gecko/nightly
> And then we would just need to ensure copying xpis from dist directory to
> the ftp.

This would be great.
If we then add the xpis to UPLOAD_EXTRA_FILES in toolkit/mozapps/installer/packager.mk , we would get the xpis uploaded.

Comment 8

[Alexandre Poirot [:ochameau]]

Oh great, I didn't knew about that variable, looks like it should be simple task.

Comment 9

[Alexandre Poirot [:ochameau]]

Attachment: Build simulator xpi and enable phone tweaks for b2g-desktop

Aki, Are you a potential reviewer for such patch?

Looks green so far and I can see the simulator xpi in try build directories:
https://tbpl.mozilla.org/?tree=Try&rev=fa2e462f8b1d

Comment 10

[Alexandre Poirot [:ochameau]]

It is failing on windows... 
It looks like gaia build system can't be built with pymake.
The simulator build script, b2g/simulator/build_xpi.py, uses mozbuild.MozBuildObject.run_make.
Which seems to end up using pymake even if force_pymake=False...
  http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#475

Any idea how to address that? Do I have to patch _make_path?
Anyway, I would like to enable this flag on other platforms and figure out the windows case in parallel.

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 8395673 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

This patch looks good to me, but a) the pymake issue needs fixing and b) this change may require build peer review.

Comment 12

[Gregory Szorc [:gps]]

At this point, I wouldn't deploy new jobs utilizing pymake. Instead, I'd use GNU Make 4.0. If you drop mozmake.exe from MozillaBuild 1.9.0 into $PATH, it should just work.

glandium can provide more info if you want.

Comment 13

[Alexandre Poirot [:ochameau]]

Attachment: Build simulator xpi and enable phone tweaks for b2g-desktop

I factored out enabling windows simulator builds.
Let's not hold on windows for other platforms!

Who can help on addressing the pymake issue?
If I understand it correctly, that's something to do on build slaves?

Comment 14

[Alexandre Poirot [:ochameau]]

Attachment: Build simulator xpi on windows

Comment 15

[Aki Sasaki (not active)]

(In reply to Alexandre Poirot (:ochameau) from comment #13)
> Created attachment 8395855 [details] [diff] [review]
> Build simulator xpi and enable phone tweaks for b2g-desktop
> 
> I factored out enabling windows simulator builds.
> Let's not hold on windows for other platforms!
> 
> Who can help on addressing the pymake issue?
> If I understand it correctly, that's something to do on build slaves?

We could change PATH here http://hg.mozilla.org/build/buildbot-configs/file/20e53905a90f/mozilla/b2g_config.py#l306 and here http://hg.mozilla.org/build/buildbot-configs/file/20e53905a90f/mozilla/b2g_config.py#l567 .

However, it looks like we have mozilla-build 1.5.1 installed on the win64-rev2 boxes.  We could roll out latest, but that would be 1.8.0 (1.9.0 is still -pre).  We can wait for 1.9.0 to get out of -pre, and then the time to prioritize/test/roll out, or we can resolve in some other way.

Comment 16

[Aki Sasaki (not active)]

I think Catlee is suggesting we copy the mozmake.exe from 1.9.0-pre onto the windows build slaves.

Comment 17

[Mike Hommey [:glandium]]

(In reply to Aki Sasaki [:aki] from comment #16)
> I think Catlee is suggesting we copy the mozmake.exe from 1.9.0-pre onto the
> windows build slaves.

Please don't. Deploy the one from the upcoming mozillabuild 1.9 instead. This is bug 927671.

Comment 18

[Aki Sasaki (not active)]

Alexandre, I just got a request in the Tues b2g meeting for a dev-gaia/b2g post when these are available.  Can you do that after you land?

Comment 19

[Alexandre Poirot [:ochameau]]

Sure, the xpi will automatically pop up next to b2g nightly tarballs.
But they may require some tweaks before broad distribution.

Comment 20

[Alexandre Poirot [:ochameau]]

Comment on attachment 8395855 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

Switching reviewer as you look more involved into this ;)

Comment 21

[Alexandre Poirot [:ochameau]]

Comment on attachment 8395856 [details] [diff] [review]
Build simulator xpi on windows

> In bug 927671, you said:
> In the end, we're going to use tooltool to deploy mozmake.

Is there a bug to follow or it is already done?

Comment 22

[Mike Hommey [:glandium]]

(In reply to Alexandre Poirot (:ochameau) from comment #21)
> Comment on attachment 8395856 [details] [diff] [review]
> Build simulator xpi on windows
> 
> > In bug 927671, you said:
> > In the end, we're going to use tooltool to deploy mozmake.
> 
> Is there a bug to follow or it is already done?

It is done for desktop builds, but not b2g builds yet.

Comment 23

[Mike Hommey [:glandium]]

Comment on attachment 8395855 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

Review of attachment 8395855 [details] [diff] [review]:
-----------------------------------------------------------------

::: b2g/config/mozconfigs/linux32_gecko/nightly
@@ +32,5 @@
>  
>  GAIADIR=$topsrcdir/gaia
>  
> +# Build simulator xpi and phone tweaks for b2g-desktop
> +export FXOS_SIMULATOR=1

This likely doesn't need to be exported.

::: b2g/installer/Makefile.in
@@ +38,5 @@
>  include $(topsrcdir)/toolkit/mozapps/installer/packager.mk
>  
> +# Ensure copying Simulator xpi to ftp
> +ifdef FXOS_SIMULATOR
> +UPLOAD_EXTRA_FILES += fxos-simulator-*-*.xpi

are those xpis created in $(DIST)?

Comment 24

[Mike Hommey [:glandium]]

Comment on attachment 8395856 [details] [diff] [review]
Build simulator xpi on windows

Review of attachment 8395856 [details] [diff] [review]:
-----------------------------------------------------------------

::: b2g/config/mozconfigs/win32_gecko/nightly
@@ +27,5 @@
>  
>  GAIADIR=$topsrcdir/gaia
>  
> +# Build simulator xpi and phone tweaks for b2g-desktop
> +export FXOS_SIMULATOR=1

Likewise.

Comment 25

[Alexandre Poirot [:ochameau]]

Attachment: Build simulator xpi and enable phone tweaks for b2g-desktop

I had to tweak the UPLOAD_EXTRA_FILES thing and move the call to build the simulator to /b2g/installer/Makefile.in.
That's because the simulator needs to be build *after* b2g desktop itself (as we ship it into the simulator xpi),
but the simulator has to be built *before* the UPLOAD_EXTRA_FILES is actually used.

Comment 26

[Alexandre Poirot [:ochameau]]

Attachment: Build simulator xpi on windows

Comment 27

[Alexandre Poirot [:ochameau]]

Comment on attachment 8400719 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

(In reply to Mike Hommey [:glandium] from comment #23)
> ::: b2g/installer/Makefile.in
> @@ +38,5 @@
> >  include $(topsrcdir)/toolkit/mozapps/installer/packager.mk
> >  
> > +# Ensure copying Simulator xpi to ftp
> > +ifdef FXOS_SIMULATOR
> > +UPLOAD_EXTRA_FILES += fxos-simulator-*-*.xpi
> 
> are those xpis created in $(DIST)?

Yes

Comment 28

[Mike Hommey [:glandium]]

Comment on attachment 8400719 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

Review of attachment 8400719 [details] [diff] [review]:
-----------------------------------------------------------------

::: b2g/installer/Makefile.in
@@ +78,5 @@
> +simulator: make-package
> +	@echo 'Building simulator addon...'
> +	$(PYTHON) $(topsrcdir)/b2g/simulator/build_xpi.py $(MOZ_PKG_PLATFORM)
> +
> +libs:: simulator

s/libs/default/

Comment 29

[Alexandre Poirot [:ochameau]]

Attachment: Build simulator xpi and enable phone tweaks for b2g-desktop

Carrying over r+.

Another try, to be sure it still build the xpis correctly:
  https://tbpl.mozilla.org/?tree=Try&rev=b14ff31ca0d6

Comment 30

[Alexandre Poirot [:ochameau]]

Green and xpi are in visible in ftp!

Comment 31

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6be95d2bb90f
https://hg.mozilla.org/integration/mozilla-inbound/rev/c7865e76e945

Comment 32

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6be95d2bb90f
https://hg.mozilla.org/mozilla-central/rev/c7865e76e945

Comment 33

[Alexandre Poirot [:ochameau]]

Comment on attachment 8403340 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

See bug 976773 comment 30.
We would like to build 1.4 simulators and need to uplift mostly desktop-specific patches.

This patch modify mostly desktop build system.

Comment 34

[Alexandre Poirot [:ochameau]]

Comment on attachment 8400721 [details] [diff] [review]
Build simulator xpi on windows

See bug 976773 comment 30.
We would like to build 1.4 simulators and need to uplift mostly desktop-specific patches.

This patch modify mostly desktop build system.

Comment 35

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 8403340 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

looks like a=npotb to me, but rubber stamp in any case.

Comment 36

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/f9f60d47dbee
https://hg.mozilla.org/releases/mozilla-aurora/rev/3316ffe2a75a

Comment 37

[Phil Ringnalda (:philor)]

Burning b2g desktop Windows builds on aurora, which is closed because of that.

Comment 38

[Aki Sasaki (not active)]

Backed out of aurora.

Comment 39

[Alexandre Poirot [:ochameau]]

I think we can safely uplift the second patch, attachment 8403340 [details] [diff] [review], while holding the windows one.

I'm tempted to say that windows supports depends on bug 990739.
https://tbpl.mozilla.org/php/getParsedLog.php?id=38097730&tree=Mozilla-Aurora
Exception: Process executed with non-0 exit code: [u'C:/mozilla-build/msys/bin/sh.exe', u'-c', u'c:/builds/moz2_slave/m-aurora-w32_g-000000000000000/build/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-aurora-w32_g-000000000000000/build/build/pymake/make.py -C c:/builds/moz2_slave/m-aurora-w32_g-000000000000000/build/gaia -j1 -w profile']
http://mxr.mozilla.org/mozilla-central/source/b2g/simulator/build_xpi.py#38


mshal, is bug 990739 upliftable? Would it be the only one thing to get mozmake to work?
Otherwise, before you made mozmake to work on windows, I was using this patch:
  https://hg.mozilla.org/try/rev/013fc594f2d7
Do you think we can reasonably land and then uplift it?

Comment 40

[Michael Shal [:mshal]]

(In reply to Alexandre Poirot (:ochameau) from comment #39)
> mshal, is bug 990739 upliftable? Would it be the only one thing to get
> mozmake to work?

Bug 990739 should be upliftable, though all that does is tell tooltool to download mozmake.exe. The code that actually uses it is glandium's patch in bug 927672 - https://bug927672.bugzilla.mozilla.org/attachment.cgi?id=8398210

So you'll need that uplifted as well if you don't already have it.

Comment 41

[Alexandre Poirot [:ochameau]]

Mike, I was wondering which patches are really needed to make attachment 8400721 [details] [diff] [review] to work on windows.
See comment 21 for context. It depends on your efforts to deploy mozmake.
It is not really clear what we would need to uplift and if that's reasonably upliftable or not.

Comment 42

[Mike Hommey [:glandium]]

The bugs listed in comment 40 are supposed to be all that's needed.