generate "FxOS Simulator" builds for B2G

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
5 years ago
13 days ago

People

(Reporter: joduinn, Assigned: ochameau)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox30 fixed, firefox31 fixed, b2g-v1.4 affected, b2g-v2.0 fixed)

Details

(Whiteboard: [b2g])

Attachments

(2 attachments, 4 obsolete attachments)

per meeting w/myk: (He's currently generating these manually on a best-effort-basis, and would be happy to stop once automation takes over.).

1) FxOS Simulator builds are for external third parties to develop apps for B2G, without having a physical device.
2) there is nothing proprietary in these, so these builds are fine to publish publicly on ftp.m.o. Therefore, no legal bug filed.

3) FxOS Simulator builds bundles Simulator addon, B2Gdesktop builds and gaia. See details in bug#920186. Unknown if this needs signing.

4) Locales: Myk's currently using a set of locales at https://github.com/mozilla/r2d2b2g/blob/master/build/languages.json. Note that this list is always a subset of languages_all.json. Whether this should continue as separate list, or be merged into one of the existing lists, is something that we *think* can be figured out later. If using this 3rd list complicates mechanics, please say so.

5) Updates: There's some updating functionality when new versions of addon are posted to addons.m.o, but this is scoped out of this for now. Possible ideas include getting updates from an update.xml file on well known location on ftp.m.o, or maybe using balrog/aus?

6) These builds should be generated for every OS that we currently generate B2G desktop builds.

7) These builds should be on m-c and on try and on the v1.2 (aurora) branch. Once v1.2 moves from aurora to new-to-be-setup-repo, these should be moved from aurora also.

Updated

5 years ago
Whiteboard: [b2g]

Comment 1

5 years ago
(In reply to John O'Duinn [:joduinn] from comment #0)
> 7) These builds should be on m-c and on try and on the v1.2 (aurora) branch.

Note that this doesn't meet https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy so these jobs will likely not be visible in the default view - if that's fine then ignore this comment :-)
(In reply to Ed Morley [:edmorley UTC+1] from comment #1)
> (In reply to John O'Duinn [:joduinn] from comment #0)
> > 7) These builds should be on m-c and on try and on the v1.2 (aurora) branch.
> 
> Note that this doesn't meet
> https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy so these jobs will
> likely not be visible in the default view - if that's fine then ignore this
> comment :-)

emorley: Dont follow - even after reading this policy wiki page. I guess that myk would like these builds to be visible, so can you elaborate?
Flags: needinfo?(myk)
Flags: needinfo?(emorley)

Comment 3

5 years ago
Comment 0 mentions only running these on m-c and try.

https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#3.29_Runs_on_mozilla-central_and_all_trees_that_merge_into_it says:

3) Runs on mozilla-central and all trees that merge into it 
 * Otherwise job failures when tree X merges into mozilla-central will not be attributable to a single changeset, resulting in either tree closure or backout of the entire merge (see requirement #2).

Requirement #2 being:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#2.29_Breakage_is_expected_to_be_followed_by_tree_closure_or_backout
Flags: needinfo?(emorley)
(In reply to John O'Duinn [:joduinn] from comment #2)
> emorley: Dont follow - even after reading this policy wiki page. I guess
> that myk would like these builds to be visible, so can you elaborate?

Yes, I would indeed like these builds to be visible, so I guess that means we need to run them "on mozilla-central and all trees that merge into it"!
Flags: needinfo?(myk)

Comment 5

5 years ago
Sounds good to me :-)
(Assignee)

Updated

5 years ago
Depends on: 944451
(Assignee)

Comment 6

4 years ago
A quick update about bug 944451.
Once landed, it will allow building simulator xpis from mozilla-central repo.

It isn't r+ yet, so details may change, but you just need to run a regular b2g desktop build, with GAIADIR being set, and, also set FXOS_SIMULATOR=1. You will then find simulator xpis in $(objdir)/dist/fxos-simulator-%(version)-%(platform).xpi

We are planning to build 1.4+ simulator xpis like this. So we do not have any 1.4 builds yet.
We start receiving more and more request for 1.4 builds, so I'd like to know if there is anything we can do to ease building such xpis.

What about adding export FXOS_SIMULATOR=1 to each b2g desktop nightly mozconfigs?
http://mxr.mozilla.org/mozilla-central/source/b2g/config/mozconfigs/linux64_gecko/nightly
And then we would just need to ensure copying xpis from dist directory to the ftp.

Comment 7

4 years ago
(In reply to Alexandre Poirot (:ochameau) from comment #6)
> A quick update about bug 944451.
> Once landed, it will allow building simulator xpis from mozilla-central repo.
> 
> It isn't r+ yet, so details may change, but you just need to run a regular
> b2g desktop build, with GAIADIR being set, and, also set FXOS_SIMULATOR=1.
> You will then find simulator xpis in
> $(objdir)/dist/fxos-simulator-%(version)-%(platform).xpi
> 
> We are planning to build 1.4+ simulator xpis like this. So we do not have
> any 1.4 builds yet.
> We start receiving more and more request for 1.4 builds, so I'd like to know
> if there is anything we can do to ease building such xpis.
> 
> What about adding export FXOS_SIMULATOR=1 to each b2g desktop nightly
> mozconfigs?
> http://mxr.mozilla.org/mozilla-central/source/b2g/config/mozconfigs/
> linux64_gecko/nightly
> And then we would just need to ensure copying xpis from dist directory to
> the ftp.

This would be great.
If we then add the xpis to UPLOAD_EXTRA_FILES in toolkit/mozapps/installer/packager.mk , we would get the xpis uploaded.
(Assignee)

Comment 8

4 years ago
Oh great, I didn't knew about that variable, looks like it should be simple task.
(Assignee)

Comment 9

4 years ago
Created attachment 8395673 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

Aki, Are you a potential reviewer for such patch?

Looks green so far and I can see the simulator xpi in try build directories:
https://tbpl.mozilla.org/?tree=Try&rev=fa2e462f8b1d
Attachment #8395673 - Flags: review?(aki)
(Assignee)

Comment 10

4 years ago
It is failing on windows... 
It looks like gaia build system can't be built with pymake.
The simulator build script, b2g/simulator/build_xpi.py, uses mozbuild.MozBuildObject.run_make.
Which seems to end up using pymake even if force_pymake=False...
  http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#475

Any idea how to address that? Do I have to patch _make_path?
Anyway, I would like to enable this flag on other platforms and figure out the windows case in parallel.
Flags: needinfo?(aki)

Comment 11

4 years ago
Comment on attachment 8395673 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

This patch looks good to me, but a) the pymake issue needs fixing and b) this change may require build peer review.
Flags: needinfo?(aki) → needinfo?(gps)
At this point, I wouldn't deploy new jobs utilizing pymake. Instead, I'd use GNU Make 4.0. If you drop mozmake.exe from MozillaBuild 1.9.0 into $PATH, it should just work.

glandium can provide more info if you want.

Updated

4 years ago
Flags: needinfo?(gps)
(Assignee)

Comment 13

4 years ago
Created attachment 8395855 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

I factored out enabling windows simulator builds.
Let's not hold on windows for other platforms!

Who can help on addressing the pymake issue?
If I understand it correctly, that's something to do on build slaves?
Attachment #8395673 - Attachment is obsolete: true
Attachment #8395673 - Flags: review?(aki)
Attachment #8395855 - Flags: review?(gps)
Attachment #8395855 - Flags: review?(aki)
(Assignee)

Comment 14

4 years ago
Created attachment 8395856 [details] [diff] [review]
Build simulator xpi on windows

Updated

4 years ago
Attachment #8395855 - Flags: review?(aki) → review+

Comment 15

4 years ago
(In reply to Alexandre Poirot (:ochameau) from comment #13)
> Created attachment 8395855 [details] [diff] [review]
> Build simulator xpi and enable phone tweaks for b2g-desktop
> 
> I factored out enabling windows simulator builds.
> Let's not hold on windows for other platforms!
> 
> Who can help on addressing the pymake issue?
> If I understand it correctly, that's something to do on build slaves?

We could change PATH here http://hg.mozilla.org/build/buildbot-configs/file/20e53905a90f/mozilla/b2g_config.py#l306 and here http://hg.mozilla.org/build/buildbot-configs/file/20e53905a90f/mozilla/b2g_config.py#l567 .

However, it looks like we have mozilla-build 1.5.1 installed on the win64-rev2 boxes.  We could roll out latest, but that would be 1.8.0 (1.9.0 is still -pre).  We can wait for 1.9.0 to get out of -pre, and then the time to prioritize/test/roll out, or we can resolve in some other way.

Comment 16

4 years ago
I think Catlee is suggesting we copy the mozmake.exe from 1.9.0-pre onto the windows build slaves.
(In reply to Aki Sasaki [:aki] from comment #16)
> I think Catlee is suggesting we copy the mozmake.exe from 1.9.0-pre onto the
> windows build slaves.

Please don't. Deploy the one from the upcoming mozillabuild 1.9 instead. This is bug 927671.

Comment 18

4 years ago
Alexandre, I just got a request in the Tues b2g meeting for a dev-gaia/b2g post when these are available.  Can you do that after you land?

Updated

4 years ago
Depends on: 927671
(Assignee)

Comment 19

4 years ago
Sure, the xpi will automatically pop up next to b2g nightly tarballs.
But they may require some tweaks before broad distribution.
(Assignee)

Comment 20

4 years ago
Comment on attachment 8395855 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

Switching reviewer as you look more involved into this ;)
Attachment #8395855 - Flags: review?(gps) → review?(mh+mozilla)
(Assignee)

Comment 21

4 years ago
Comment on attachment 8395856 [details] [diff] [review]
Build simulator xpi on windows

> In bug 927671, you said:
> In the end, we're going to use tooltool to deploy mozmake.

Is there a bug to follow or it is already done?
Attachment #8395856 - Flags: review?(mh+mozilla)
(In reply to Alexandre Poirot (:ochameau) from comment #21)
> Comment on attachment 8395856 [details] [diff] [review]
> Build simulator xpi on windows
> 
> > In bug 927671, you said:
> > In the end, we're going to use tooltool to deploy mozmake.
> 
> Is there a bug to follow or it is already done?

It is done for desktop builds, but not b2g builds yet.
Comment on attachment 8395855 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

Review of attachment 8395855 [details] [diff] [review]:
-----------------------------------------------------------------

::: b2g/config/mozconfigs/linux32_gecko/nightly
@@ +32,5 @@
>  
>  GAIADIR=$topsrcdir/gaia
>  
> +# Build simulator xpi and phone tweaks for b2g-desktop
> +export FXOS_SIMULATOR=1

This likely doesn't need to be exported.

::: b2g/installer/Makefile.in
@@ +38,5 @@
>  include $(topsrcdir)/toolkit/mozapps/installer/packager.mk
>  
> +# Ensure copying Simulator xpi to ftp
> +ifdef FXOS_SIMULATOR
> +UPLOAD_EXTRA_FILES += fxos-simulator-*-*.xpi

are those xpis created in $(DIST)?
Attachment #8395855 - Flags: review?(mh+mozilla) → feedback+
Comment on attachment 8395856 [details] [diff] [review]
Build simulator xpi on windows

Review of attachment 8395856 [details] [diff] [review]:
-----------------------------------------------------------------

::: b2g/config/mozconfigs/win32_gecko/nightly
@@ +27,5 @@
>  
>  GAIADIR=$topsrcdir/gaia
>  
> +# Build simulator xpi and phone tweaks for b2g-desktop
> +export FXOS_SIMULATOR=1

Likewise.
Attachment #8395856 - Flags: review?(mh+mozilla) → feedback+
(Assignee)

Comment 25

4 years ago
Created attachment 8400719 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

I had to tweak the UPLOAD_EXTRA_FILES thing and move the call to build the simulator to /b2g/installer/Makefile.in.
That's because the simulator needs to be build *after* b2g desktop itself (as we ship it into the simulator xpi),
but the simulator has to be built *before* the UPLOAD_EXTRA_FILES is actually used.
Attachment #8395855 - Attachment is obsolete: true
(Assignee)

Comment 26

4 years ago
Created attachment 8400721 [details] [diff] [review]
Build simulator xpi on windows
Attachment #8395856 - Attachment is obsolete: true
(Assignee)

Comment 27

4 years ago
Comment on attachment 8400719 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

(In reply to Mike Hommey [:glandium] from comment #23)
> ::: b2g/installer/Makefile.in
> @@ +38,5 @@
> >  include $(topsrcdir)/toolkit/mozapps/installer/packager.mk
> >  
> > +# Ensure copying Simulator xpi to ftp
> > +ifdef FXOS_SIMULATOR
> > +UPLOAD_EXTRA_FILES += fxos-simulator-*-*.xpi
> 
> are those xpis created in $(DIST)?

Yes
Attachment #8400719 - Flags: review?(mh+mozilla)
(Assignee)

Updated

4 years ago
Attachment #8400721 - Flags: review?(mh+mozilla)
Comment on attachment 8400719 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

Review of attachment 8400719 [details] [diff] [review]:
-----------------------------------------------------------------

::: b2g/installer/Makefile.in
@@ +78,5 @@
> +simulator: make-package
> +	@echo 'Building simulator addon...'
> +	$(PYTHON) $(topsrcdir)/b2g/simulator/build_xpi.py $(MOZ_PKG_PLATFORM)
> +
> +libs:: simulator

s/libs/default/
Attachment #8400719 - Flags: review?(mh+mozilla) → review+
Attachment #8400721 - Flags: review?(mh+mozilla) → review+
Assignee: nobody → poirot.alex
(Assignee)

Comment 29

4 years ago
Created attachment 8403340 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

Carrying over r+.

Another try, to be sure it still build the xpis correctly:
  https://tbpl.mozilla.org/?tree=Try&rev=b14ff31ca0d6
Attachment #8400719 - Attachment is obsolete: true
Attachment #8403340 - Flags: review+
(Assignee)

Comment 30

4 years ago
Green and xpi are in visible in ftp!
Keywords: checkin-needed
(Assignee)

Comment 33

4 years ago
Comment on attachment 8403340 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

See bug 976773 comment 30.
We would like to build 1.4 simulators and need to uplift mostly desktop-specific patches.

This patch modify mostly desktop build system.
Attachment #8403340 - Flags: approval-mozilla-aurora?
(Assignee)

Comment 34

4 years ago
Comment on attachment 8400721 [details] [diff] [review]
Build simulator xpi on windows

See bug 976773 comment 30.
We would like to build 1.4 simulators and need to uplift mostly desktop-specific patches.

This patch modify mostly desktop build system.
Attachment #8400721 - Flags: approval-mozilla-aurora?
Comment on attachment 8403340 [details] [diff] [review]
Build simulator xpi and enable phone tweaks for b2g-desktop

looks like a=npotb to me, but rubber stamp in any case.
Attachment #8403340 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Attachment #8400721 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
https://hg.mozilla.org/releases/mozilla-aurora/rev/f9f60d47dbee
https://hg.mozilla.org/releases/mozilla-aurora/rev/3316ffe2a75a
status-b2g-v1.4: --- → fixed
status-b2g-v2.0: --- → fixed
status-firefox30: --- → fixed
status-firefox31: --- → fixed
Burning b2g desktop Windows builds on aurora, which is closed because of that.

Comment 38

4 years ago
Backed out of aurora.
status-b2g-v1.4: fixed → affected
(Assignee)

Comment 39

4 years ago
I think we can safely uplift the second patch, attachment 8403340 [details] [diff] [review], while holding the windows one.

I'm tempted to say that windows supports depends on bug 990739.
https://tbpl.mozilla.org/php/getParsedLog.php?id=38097730&tree=Mozilla-Aurora
Exception: Process executed with non-0 exit code: [u'C:/mozilla-build/msys/bin/sh.exe', u'-c', u'c:/builds/moz2_slave/m-aurora-w32_g-000000000000000/build/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-aurora-w32_g-000000000000000/build/build/pymake/make.py -C c:/builds/moz2_slave/m-aurora-w32_g-000000000000000/build/gaia -j1 -w profile']
http://mxr.mozilla.org/mozilla-central/source/b2g/simulator/build_xpi.py#38


mshal, is bug 990739 upliftable? Would it be the only one thing to get mozmake to work?
Otherwise, before you made mozmake to work on windows, I was using this patch:
  https://hg.mozilla.org/try/rev/013fc594f2d7
Do you think we can reasonably land and then uplift it?
Flags: needinfo?(mshal)
(In reply to Alexandre Poirot (:ochameau) from comment #39)
> mshal, is bug 990739 upliftable? Would it be the only one thing to get
> mozmake to work?

Bug 990739 should be upliftable, though all that does is tell tooltool to download mozmake.exe. The code that actually uses it is glandium's patch in bug 927672 - https://bug927672.bugzilla.mozilla.org/attachment.cgi?id=8398210

So you'll need that uplifted as well if you don't already have it.
Flags: needinfo?(mshal)
(Assignee)

Comment 41

4 years ago
Mike, I was wondering which patches are really needed to make attachment 8400721 [details] [diff] [review] to work on windows.
See comment 21 for context. It depends on your efforts to deploy mozmake.
It is not really clear what we would need to uplift and if that's reasonably upliftable or not.
Flags: needinfo?(mh+mozilla)
The bugs listed in comment 40 are supposed to be all that's needed.
Flags: needinfo?(mh+mozilla)
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.