1201451 - [Simulator] Create the Simulator package in order to using from WebIDE.

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Simulator, defect)

Description

[Mantaroh Yoshinaga[:mantaroh]]

We should create the Firefox OS Simulator package in order to provide simulator to everyone.
If we provide this package to everyone, 
The people who want to create TV Apps can check their apps on simulator.
It will benefit for community and contents provider.

This task is composed as follow:
 - Change GAIA_DEVICE_TYPE to tv when build.
 - Create XPI and put the mozilla server.
 - Add the Simulator version for [https://ftp.mozilla.org/pub/mozilla.org/labs/fxos-simulator/index.json]

Comment 1

[Evelyn Hung]

I can follow up this bug. Steal it.

Comment 2

[Evelyn Hung]

(In reply to Mantaroh Yoshinaga[:mantaroh] from comment #0)
> We should create the Firefox OS Simulator package in order to provide
> simulator to everyone.
> If we provide this package to everyone, 
> The people who want to create TV Apps can check their apps on simulator.
> It will benefit for community and contents provider.
> 
> This task is composed as follow:
>  - Change GAIA_DEVICE_TYPE to tv when build.
>  - Create XPI and put the mozilla server.
>  - Add the Simulator version for
> [https://ftp.mozilla.org/pub/mozilla.org/labs/fxos-simulator/index.json]

Hi Ryan,

A few questions need your input:

1. How could I make the Simulator window size to be default in TV size? 
2. How could I access the server hosted xpi files?
3. Should I file a bug for altering the json file?

Thank you. :)

Comment 5

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Evelyn Hung [:evelyn] from comment #2)
> (In reply to Mantaroh Yoshinaga[:mantaroh] from comment #0)
> > We should create the Firefox OS Simulator package in order to provide
> > simulator to everyone.
> > If we provide this package to everyone, 
> > The people who want to create TV Apps can check their apps on simulator.
> > It will benefit for community and contents provider.
> > 
> > This task is composed as follow:
> >  - Change GAIA_DEVICE_TYPE to tv when build.
> >  - Create XPI and put the mozilla server.
> >  - Add the Simulator version for
> > [https://ftp.mozilla.org/pub/mozilla.org/labs/fxos-simulator/index.json]
> 
> Hi Ryan,
> 
> A few questions need your input:
> 
> 1. How could I make the Simulator window size to be default in TV size?

Jan, what seems like a good way to have a special default simulator size for TV?  Since WebIDE now controls simulator startup, the add-on itself can't do much about it.  Should we check for some special TV name?  Not sure.

> 2. How could I access the server hosted xpi files?

Evelyn, the server was recently changed from FTP to using uploads from S3, and only a few of us have access for now.  You could certainly request access, but it's kind of painful anyway.  Do you anticipate releasing many versions?

> 3. Should I file a bug for altering the json file?

Yes, please file a WebIDE bug.  We may need to make a WebIDE code change to show an additional "TV" name somewhere, I am not sure yet.

Comment 6

[Evelyn Hung]

(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #5) 
> > A few questions need your input:
> > 
> > 1. How could I make the Simulator window size to be default in TV size?
> 
> Jan, what seems like a good way to have a special default simulator size for
> TV?  Since WebIDE now controls simulator startup, the add-on itself can't do
> much about it.  Should we check for some special TV name?  Not sure.
> 

We need to launch TV in a correct size otherwise the UI will be a bit broken because we didn't listen to window size change and act correspondingly. So I hope WebIDE could have some way to pre-configure simulator size on WebIDE.

> > 2. How could I access the server hosted xpi files?
> 
> Evelyn, the server was recently changed from FTP to using uploads from S3,
> and only a few of us have access for now.  You could certainly request
> access, but it's kind of painful anyway.  Do you anticipate releasing many
> versions?
> 

Like phone we will update versions and we are riding on Firefox OS releases. In terms of Gecko there is no difference between phone and tv, but Gaia profiles are totally different, so we need to build simulators separately.

I'm glad to have your help to upload this time, and we can discuss a long-term plan.

> > 3. Should I file a bug for altering the json file?
> 
> Yes, please file a WebIDE bug.  We may need to make a WebIDE code change to
> show an additional "TV" name somewhere, I am not sure yet.

Yeah, I'm thinking the same thing.

Thanks!

Comment 7

[Tori Chen [:tori]]

Attachment: device_deck_demo2.png

Comment 8

[Jan Keromnes [:janx]]

(In reply to Evelyn Hung [:evelyn] from comment #6)
> (In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #5) 
> > > A few questions need your input:
> > > 
> > > 1. How could I make the Simulator window size to be default in TV size?
> > 
> > Jan, what seems like a good way to have a special default simulator size for
> > TV?  Since WebIDE now controls simulator startup, the add-on itself can't do
> > much about it.  Should we check for some special TV name?  Not sure.
> > 
> 
> We need to launch TV in a correct size otherwise the UI will be a bit broken
> because we didn't listen to window size change and act correspondingly. So I
> hope WebIDE could have some way to pre-configure simulator size on WebIDE.

To address this, I see a potential solution in two parts:

1. Create the notion of a "Simulator type" (e.g. "phone", "tablet", "television"...) in a Simulator's configuration in WebIDE. They would come from the "TYPES" list present in the devices catalog[1]. This would allow for type-specific default hardware profiles, e.g. the first profile of the matching type will be default (today, the default profile for all Simulators is the first "phone" profile i.e. "Firefox OS Flame").

2. If we build and distribute special TV simulator add-ons (Gecko + Gaia TV-build), we should name them in a special way so that WebIDE can recognize them as "television" type (e.g. "[e-mail]"). As a consequence, when installing such an addon, WebIDE will recognize its type as "television", then set its initial hardware profile to "720p HD Television" (the first hardware profile of type "television") and thus the default screen size will automatically be correct. 

An added benefit of having a "type" in the Simulator configuration is the possibility of showing type-specific features in the WebIDE configuration screen. Take bug 1201456's screenshot[2] as an example: when a Simulator of type "television" is being configured, we'll be able to show an additional "Select TV Dummy Directory" field.

Would that work for you?

[1] https://code.cdn.mozilla.net/devices/devices.json
[2] https://bug1201456.bmoattachments.org/attachment.cgi?id=8670076

Comment 9

[Evelyn Hung]

Sure, adding "type" is a good idea, totally opt for this. :)

To achieve my goal (having WebIDE support TV Simulator), here are to-dos in my mind. Let me know if I missed any pieces.

1. bug 1049704

2. a TV simulator xpi file and name it properly with device type token

3. your help of uploading it to the server

4. bug 1212352

Finish in this order, am I right?

Comment 10

[Jan Keromnes [:janx]]

(In reply to Evelyn Hung [:evelyn] from comment #9)
> Sure, adding "type" is a good idea, totally opt for this. :)

Ok, I'll get to that soon.

> To achieve my goal (having WebIDE support TV Simulator), here are to-dos in
> my mind. Let me know if I missed any pieces.
> 
> 1. bug 1049704

This bug is just about part 1 of comment 8 (creating a Simulator type), but maybe I can also sneak in the WebIDE side of part 2 (detect special simulator add-on names to assign the correct type to them).

> 2. a TV simulator xpi file and name it properly with device type token

For this to work, WebIDE should know what that token will look like (see above). Also, there is a series of steps we use to build XPIs, which we can show you.

> 3. your help of uploading it to the server

Ryan will be able to help you with that once steps 1 and 2 are complete.

> 4. bug 1212352

I think this should just be a JSON-file change, but maybe Ryan is right in suspecting we'll also need some WebIDE change to properly show the name.

> Finish in this order, am I right?

Works for me!

Comment 11

[Evelyn Hung]

(In reply to Jan Keromnes [:janx] from comment #10)
> (In reply to Evelyn Hung [:evelyn] from comment #9)
> > Sure, adding "type" is a good idea, totally opt for this. :)
> 
> Ok, I'll get to that soon.
> 
> > To achieve my goal (having WebIDE support TV Simulator), here are to-dos in
> > my mind. Let me know if I missed any pieces.
> > 
> > 1. bug 1049704
> 
> This bug is just about part 1 of comment 8 (creating a Simulator type), but
> maybe I can also sneak in the WebIDE side of part 2 (detect special
> simulator add-on names to assign the correct type to them).
> 
> > 2. a TV simulator xpi file and name it properly with device type token
> 
> For this to work, WebIDE should know what that token will look like (see
> above). Also, there is a series of steps we use to build XPIs, which we can
> show you.
> 

I already found instructions on MDN, and I am able to build a TV version xpi now. I'm polishing it and will upload the file here. 

For the token, currently how do we name simulators? We could follow the naming token you said above if you also feel it's good enough. Then the name will be "[e-mail]".

Comment 12

[Jan Keromnes [:janx]]

Great! That name works for me, and I think I even had such a thing in mind when I wrote the original addon-name parsing code. I expect the necessary WebIDE changes to support Simulator types will be pretty small.

Comment 13

[Evelyn Hung]

Hi Jan,

I'm wondering how many platforms we need for packaging a version of Simulator? I can think of MacOS, Linux, Windows, anything else? Do you have settled environments to make these builds? (I can get these platforms on my side, but I'd like to confirm it's fine to build from an arbitrary environment. :) 

Furthermore, how to we adjust the file naming rules per our discussion above? Thanks!

Comment 14

[Evelyn Hung]

We use latest m-c Gecko with my Gaia local branch for building Simulator:
https://github.com/evelynhung/gaia/commits/issue-1201451

Comment 15

[Jan Keromnes [:janx]]

Hi Evelyn! Forwarding your questions about packaging a Simulator in comment 13 to Ryan.

Comment 16

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Evelyn Hung [:evelyn] from comment #13)
> I'm wondering how many platforms we need for packaging a version of
> Simulator? I can think of MacOS, Linux, Windows, anything else? Do you have
> settled environments to make these builds? (I can get these platforms on my
> side, but I'd like to confirm it's fine to build from an arbitrary
> environment. :)

There's Mac, Linux, and Windows.  Typically we've produced both a 32-bit and 64-bit build for Linux in the past.

The main simulator has historically been built in automation as changes land in the tree, but we've changed it a bit to be based on Mulet, and for the moment only the Linux version is automated.

Alex has been working on these recent changes, so he's the most up to date on building simulators.  Hopefully between all 3 of us, we can get you the info you need. :) We don't really have anything like shared build environments or anything for this...  Either they are built in automation, or we build them locally on our own machines.  Since yours is a special TV version, I'm guessing you'd need to do those builds yourself, and then we can distribute them via WebIDE.

We'll likely also want to sign the final XPIs, so they work in all Firefox builds (where signing requirements are on).  I can help with the signing and distribution once we reach that stage.

> Furthermore, how to we adjust the file naming rules per our discussion
> above?

I think all that's needed is to adjust the simulator add-on ID to match the format you agreed on with Jan.

Comment 17

[Alexandre Poirot [:ochameau]]

Evelyn, it would be great to make it built by automation.
At least we should make it so that the build are easily reproducible.
Building from a personal branch isn't much. Can't you land your changes to gaia master or at least have an official branch on gaia repo?

It has been a very long time since we stopped building simulators locally.
The worse scenario was to make it build by a try run when automation wasn't running our branch anymore.
Note that a few days ago, you could have built custom simulators from try for all platforms.
(you can even specify a custom gaia via b2g/config/gaia.json within gecko)
but we recently stopped building b2g-desktop for windows and mac.

We are in middle of a mess with automation.
Mulet should now be restored and built on all platforms again, but simulators addon is only automated for linux mulet builds. We still haven't released 2.5 builds, the latest simulators release are from pre-whistler.

I'm planning to do semi-automated releases via a bash script I pushed to bug 1198877.
I'll pull automated mulet builds from:
  https://index.taskcluster.net/v1/task/gecko.v1.mozilla-central.latest.linux.mulet.opt/artifacts/public/build/target.linux-x86_64.tar.bz2
  http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/firefox-44.0a1.en-US.win32.zip
  http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/firefox-44.0a1.en-US.mac64.dmg
And with a local gaia master checkout, use the build-simulator.sh script to craft xpi files.
Then sign and upload them.
This bash script is used to build linux nigthlies and could be also used to build tv simulator nightlies if that helps.

Comment 18

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Just to make sure people are aware, pvtbuild server may be deprecated as early as early november.  It would be good if everyone moved over to using Taskcluster builds asap.

Comment 19

[Alexandre Poirot [:ochameau]]

Everything I do is on taskcluster. See my work in bug 1213011 and bug 1207653.
Simulator is all built from taskcluster tasks.

Otherwise, but that's another subject which is not my scope.
We had to resurect mulet builds for osx and windows which are run only on buildbot.
We would need them to be running on taskcluster, but TC isn't ready for these platforms.

Naoki, are you saying all jobs are going to be run on taskcluster by November?
Even windows and linux ones?!

Comment 20

[Alexandre Poirot [:ochameau]]

Wait, what is pvtbuild exactly? Isn't it some b2g specific servers for device images?
I thought your comment was related to buildbot versus taskcluster.
Here I don't think we care about pvtbuild as simulator is only on desktop.

Comment 21

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

pvtbuild server is the ftp server that holds the builds made by buildbot previous to taskcluster.
We're trying to replace the build system that used to be in place with taskcluster.

ie buildbot + pvtbuild server will be replaced by taskcluster and it's cloud storage.

If you're doing everything on Taskcluster, wonderful!  Just making sure that you're builds will survive past the deprecation of taskcluster.

Comment 22

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Correction, last sentence should read:
Just making sure that you're builds will survive past the deprecation of pvtbuild server.

Comment 23

[Evelyn Hung]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #22)
> Correction, last sentence should read:
> Just making sure that you're builds will survive past the deprecation of
> pvtbuild server.

Thanks for your reminder. I had a chat with Alex last week and learn the whole picture of devtool building. The plan for making a TV daily build will depend on phone's infrastructure ready, and as I know Alex is working on it. :)

Comment 24

[Evelyn Hung]

https://drive.google.com/drive/folders/0ByeEXemdLw-BQXJxT3ZXMUhyVVE

Hi, there are TV Simulators for Mac, Linux and Windows. The files are too big to attach them on bugzilla so I put them in Google drive. As we discussed earlier, I need your help to go through signature process and upload them to Simulator server. Let me know if I missed anything. Thank you so much. :)

More information of this build:
This is built from latest Gaia and Nightly Mulet. Some local patches applied to workaround known issue and polish UI a bit so the user won't get confused on hotkeys.
See my local patches here:
https://github.com/evelynhung/gaia/commits/issue-1201451

Comment 25

[Alexandre Poirot [:ochameau]]

Thanks! Looks great, at least on Windows and linux. Did you gave it a try on Mac?
It would be great to land all your tweaks to gaia!!

Now, we just have to ask some magician to sign the xpis.
I opened bug 1228025 for that.

Then we would need to upload them to the FTP,
this is something for me or :jryans.
Most likely me as he is going to be on PTO.

Comment 26

[Evelyn Hung]

(In reply to Alexandre Poirot [:ochameau] from comment #25)
> Thanks! Looks great, at least on Windows and linux. Did you gave it a try on
> Mac?

Yes, my team helps me try them on their machines, linux, Mac an two Windows.
These builds are not perfect but acceptable. ;)

> It would be great to land all your tweaks to gaia!!
> 

Yes, I've landed some of them, and keep working on it.

> Now, we just have to ask some magician to sign the xpis.
> I opened bug 1228025 for that.
> 
> Then we would need to upload them to the FTP,
> this is something for me or :jryans.
> Most likely me as he is going to be on PTO.

Great, thank you so much. Can't wait to see them on WebIDE!

Comment 27

[Sören Hentzschel]

I tested the Mac build on my Firefox Nightly. It works with the 720p and 4K device settings but not with the 1080p device setting. With 1080p there is the flowers background and nothing else. No problems with 720p and 4K.

Comment 28

[Evelyn Hung]

(In reply to Sören Hentzschel from comment #27)
> I tested the Mac build on my Firefox Nightly. It works with the 720p and 4K
> device settings but not with the 1080p device setting. With 1080p there is
> the flowers background and nothing else. No problems with 720p and 4K.

Yes, someone reported similar issue to me, but it was on Windows. However, my Mac and Windows works good on both 720p and 1080p. I haven't had a clue yet. :(  
Would you mind to attach your screenshot of 1080p? I was guessing the screen got rotated to portrait mode so you can only see a background.

Comment 29

[Sören Hentzschel]

Attachment: screenshot (1080p)

Comment 30

[Sören Hentzschel]

Oh, it works if I launch the simulator on my Macbook display (HiDPI). The bug only happens on my external (no HiDPI) display.

Comment 31

[Evelyn Hung]

Thanks! From the screenshot in comment 29, it did rotate to portrait mode. We've had some findings and file bug 1228899 for follow-up. Feel free to track that bug to get updates. :)

Comment 32

[Evelyn Hung]

(In reply to Alexandre Poirot [:ochameau] from comment #25)
> Thanks! Looks great, at least on Windows and linux. Did you gave it a try on
> Mac?
> It would be great to land all your tweaks to gaia!!
> 
> Now, we just have to ask some magician to sign the xpis.
> I opened bug 1228025 for that.
> 
> Then we would need to upload them to the FTP,
> this is something for me or :jryans.
> Most likely me as he is going to be on PTO.

Alex, I see bug 1228025 has been fixed(yay!), so what's our next step? :)

Comment 33

[Alexandre Poirot [:ochameau]]

I have to upload them to S3 to get them on the ftp. I'll give this a shot tomorrow.
But that won't be enough I think. At least to have them installable easily from WebIDE addons panels.
At least, having them on the ftp would allow installing them manually from ftp listings.

Comment 34

[Evelyn Hung]

(In reply to Evelyn Hung [:evelyn] from comment #32)
> (In reply to Alexandre Poirot [:ochameau] from comment #25)
> > Thanks! Looks great, at least on Windows and linux. Did you gave it a try on
> > Mac?
> > It would be great to land all your tweaks to gaia!!
> > 
> > Now, we just have to ask some magician to sign the xpis.
> > I opened bug 1228025 for that.
> > 
> > Then we would need to upload them to the FTP,
> > this is something for me or :jryans.
> > Most likely me as he is going to be on PTO.
> 
> Alex, I see bug 1228025 has been fixed(yay!), so what's our next step? :)

Okay, I guess besides uploading those xpi files to FTP, the next is doing bug 1212352. I'm working on it.

Comment 35

[Evelyn Hung]

(In reply to Alexandre Poirot [:ochameau] from comment #33)
> I have to upload them to S3 to get them on the ftp. I'll give this a shot
> tomorrow.
> But that won't be enough I think. At least to have them installable easily
> from WebIDE addons panels.

yes, that's bug 1212352. I'm working!

> At least, having them on the ftp would allow installing them manually from
> ftp listings.

Cool, thanks!

Comment 36

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Looks like Alex has this under control.

Comment 37

[Alexandre Poirot [:ochameau]]

Unfortunately, I can't find any valuable information on how to setup my s3 access.
Someone from IT sent me a key.gpg file. I can't find any way to import it via gpg command.
Nor do I have any of the information asked by aws configure:
  AWS Access Key ID [None]: 
  AWS Secret Access Key [None]: 
  Default region name [None]: 
  Default output format [None]: 

Do you remember how you setup your s3 access?

Comment 38

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Alexandre Poirot [:ochameau] from comment #37)
> Unfortunately, I can't find any valuable information on how to setup my s3
> access.
> Someone from IT sent me a key.gpg file. I can't find any way to import it
> via gpg command.
> Nor do I have any of the information asked by aws configure:
>   AWS Access Key ID [None]: 
>   AWS Secret Access Key [None]: 
>   Default region name [None]: 
>   Default output format [None]: 
> 
> Do you remember how you setup your s3 access?

It's an encrypted file that *contains* the answers the `aws` command wants.  To read the file, decrypt it with gpg, such as `gpg -d key.gpg`.  It will print the info you need to give to `aws`.

Comment 39

[Alexandre Poirot [:ochameau]]

Thanks a lot, I would never have figured this out ;)

I updates the tv builds to this folder:
  https://ftp.mozilla.org/pub/labs/fxos-simulator/2.6-tv/
But there is some cache weirdness on S3. All folders don't appear, but files exists!

Here is a listing of all files:
2015-12-08 15:14:21   92675938 pub/labs/fxos-simulator/2.6-tv/linux64/fxos-simulator-2.6.20151125005937_tv-linux64.xpi
2015-12-08 15:15:10   92675938 pub/labs/fxos-simulator/2.6-tv/linux64/fxos_2_6_simulator_tv-linux64-latest.xpi
2015-12-08 15:20:55        749 pub/labs/fxos-simulator/2.6-tv/linux64/update.rdf
2015-12-08 15:05:54  101535123 pub/labs/fxos-simulator/2.6-tv/mac64/fxos-simulator-2.6.20151124150228_tv-mac64.xpi
2015-12-08 15:12:05  101535123 pub/labs/fxos-simulator/2.6-tv/mac64/fxos_2_6_simulator_tv-mac64-latest.xpi
2015-12-08 15:11:13   84180818 pub/labs/fxos-simulator/2.6-tv/mac64/fxos_2_6_simulator_tv-win32-latest.xpi
2015-12-08 15:19:00        745 pub/labs/fxos-simulator/2.6-tv/mac64/update.rdf
2015-12-08 15:07:35   84180818 pub/labs/fxos-simulator/2.6-tv/win32/fxos-simulator-2.6.20151124150228_tv-win32.xpi
2015-12-08 15:09:00   92675938 pub/labs/fxos-simulator/2.6-tv/win32/fxos-simulator-2.6.20151125005937_tv-linux64.xpi
2015-12-08 15:15:19   84180818 pub/labs/fxos-simulator/2.6-tv/win32/fxos_2_6_simulator_tv-win32-latest.xpi
2015-12-08 15:19:35        745 pub/labs/fxos-simulator/2.6-tv/win32/update.rdf

I haven't updated index.json as it relates to what you end up doing in bug 1212352,
but you should now be able to install addons from these urls without any magic pref!

Comment 40

[Evelyn Hung]

Thanks everyone, we are all done. :)