855143 - Adjust Firefox OS engineering build for testing payments

Status: RESOLVED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Kumar McMillan [:kumar]]

Hi.
We'd like to update the unagi eng build with the following for testing payments:
- Dev Marketplace installed - https://marketplace-dev.allizom.org/manifest.webapp
- Stage Marketplace installed - https://marketplace.allizom.org/manifest.webapp
- Payments enabled for those servers. Put this in build/custom-prefs.js so that it is set on device: https://github.com/mozilla/webpay/blob/master/ezboot/custom-prefs.js#L5-L15

Rationale:

As we bring on external QA groups from our partners we'd like to offer them a custom device build so they can test payments. Otherwise, they need to install custom apps and set device settings which is non-trivial for the average tester.

Also, we need this eventually for Marionette automation to test payments.

+++ This bug was initially created as a clone of Bug #853561 +++

Comment 1

[Aki Sasaki (not active)]

How is this different from bug 853561, which was moving towards WONTFIX?

Comment 2

[Jason Smith [:jsmith]]

(In reply to Aki Sasaki [:aki] from comment #1)
> How is this different from bug 853561, which was moving towards WONTFIX?

bug 853561 involves generating a custom build on pvtbuilds. This new ticket was filed to focus on changing the engineering builds to meet the items mentioned in comment 0:

1. Add marketplace dev & staging to eng builds
2. Add preferences for eng builds only to include the payment prefs
3. If possible, enable the marketplace dev certs for eng builds as well

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Ok, lets put this in Boot2Gecko :: Builds since this is likely in the gaia and/or b2g build system. If there are specific tasks for RelEng then please file a dep bug.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

Don't want to conflate or couple work if it should remain separate, but I had also filed bug 854995 to restore the other test apps which are currently missing on Unagi engineering builds.

Comment 5

[Jason Smith [:jsmith]]

Want to double check with Brian Smith on the request for adding certs into engineering builds by default.

Brian - Are you okay or not okay with enabling the marketplace certs by default in engineering development builds only?

Comment 6

[Fernando Jiménez Moreno [:ferjm]]

(In reply to Kumar McMillan [:kumar] from comment #0)
> Hi.
> We'd like to update the unagi eng build with the following for testing
> payments:
> - Dev Marketplace installed -
> https://marketplace-dev.allizom.org/manifest.webapp
> - Stage Marketplace installed -
> https://marketplace.allizom.org/manifest.webapp
> - Payments enabled for those servers. Put this in build/custom-prefs.js so
> that it is set on device:
> https://github.com/mozilla/webpay/blob/master/ezboot/custom-prefs.js#L5-L15
> 

This can also probably be done via GAIA_CUSTOMIZATION mechanisms. Adding Yuren and Fabrice in CC.

Comment 7

[Fernando Jiménez Moreno [:ferjm]]

s/GAIA_CUSTOMIZATION/GAIA_DISTRIBUTION_DIR/

Comment 8

[[:fabrice] Fabrice Desré]

Yes, please use the GAIA_DISTRIBUTION_DIR for that. This is documented there: https://wiki.mozilla.org/B2G/MarketCustomizations

Comment 9

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Hi, GAIA_DISTRIBUTION_DIR haven't support preload apps yet because bug 838098 haven't landed which blocked by bug 848604.

currently you need to copy apps to external-apps directory.

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Yuren Ju [:yurenju] from comment #9)
> Hi, GAIA_DISTRIBUTION_DIR haven't support preload apps yet because bug
> 838098 haven't landed which blocked by bug 848604.
> 
> currently you need to copy apps to external-apps directory.

While we await the resolution of both bug 838098 and bug 848604, is there anything we can do in the interim to get this onto both commercial and Mozilla-RIL Unagi-engineering builds?

Comment 11

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

could we add copy commands in build script out of gaia? something like copy directories first and then |make -C GAIA_DIR|.

Comment 12

[Kumar McMillan [:kumar]]

jsmith, bsmith: please file another bug to sort out certificates if it's going to slow this bug down. We'd really like to get marketplace and payment settings on the eng builds ASAP; we can deal with certs at a later date.

Comment 13

[Jason Smith [:jsmith]]

The dependencies here don't look right. Let me clarify what needs to be done here.

1. Getting marketplace dev/staging in an engineering build by default just needs to add those apps to the test_external_apps directory. No need to introduce the customization complexity here.

2. An engineering build needs to have a set custom preferences that enable to marketplace dev payment provider. That probably requires modifying the build process under an engineering build condition to include a preferences file or not.

3. An engineering build needs to include the dev certs so long as Brian is in alignment. This I know less about the process and will need Brian's input on.

bug 838098 isn't needed here because adding marketplace dev/staging does not need to go through a customization process - we want the apps in the engineering build by default. bug 848604 isn't needed here because we can still go forward with this implementation without the app directory abstraction complete or not. This is higher priority then doing app directory abstraction, so I do not want to hold the work here on that abstraction being completed.

Comment 14

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Jason Smith [:jsmith] from comment #13)
> 3. An engineering build needs to include the dev certs so long as Brian is
> in alignment. This I know less about the process and will need Brian's input
> on.

Sure. I filed bug 855143.

Comment 15

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Er, bug 864820, I mean.

Comment 16

[Kumar McMillan [:kumar]]

It sounds like everyone agrees that we can proceed with the payment settings. Is there an ETA when we can see this on the engineering builds? 

Without this, it is becoming hard to work with QA contractors. It's hard enough to help them flash builds because of their tools. Pushing custom preferences makes the process more complicated.

Comment 17

[Zac C (:zac)]

Doing this manually is a bit of a pain point for us testers. Would be great to have this in place by default.

Comment 18

[Stephen Donner [:stephend] Not actively reading bugmail]

Kumar, Zac: releng is ready to help us, they just need to know *exactly* what should go in these builds, prefs/config-wise - please help them help us; thanks!

Comment 19

[Kumar McMillan [:kumar]]

Great. Here is exactly what we would like to add to the builds.

- We would like the Dev Marketplace pre-installed. You can find its manifest here https://marketplace-dev.allizom.org/manifest.webapp
- We would like the Stage Marketplace pre-installed. You can find its manifest here https://marketplace.allizom.org/manifest.webapp
- We would like payments enabled for those installed apps. Put these prefs in /path/to/gaia/build/custom-prefs.js so that they are pushed to device during `make profile`: https://github.com/mozilla/webpay/blob/master/ezboot/custom-prefs.js#L5-L15

Please let me know if you need more details and sorry if these instructions were not clear in the original bug summary.

Comment 20

[Chris AtLee [:catlee]]

How do we incorporate those manifests into the build?

Comment 21

[Kumar McMillan [:kumar]]

sorry, missed your question somehow. External test apps go here https://github.com/mozilla-b2g/gaia/tree/master/test_external_apps Ah, so it looks like we need a gaia patch. I'll prepare one. However, we're still waiting on bug 877531 to get certs for the Marketplace stage packaged app. The gaia patch will need to add the dev and stage packaged apps similar to this https://github.com/mozilla-b2g/gaia/tree/master/external-apps/marketplace.firefox.com

Comment 22

[John Ford [:jhford] CET/CEST Berlin Time]

Is this still an issue?

Comment 23

[Kumar McMillan [:kumar]]

Yes. It's in my queue to prepare the Gaia patch but I've been too busy with production related payments issues. If anyone else wants to provide the patch that would be nice :) We still need this. Not having payment prefs on engineering builds causes a lot of headaches with QA testers.

Comment 24

[Kumar McMillan [:kumar]]

Attachment: gaia patch for dev/stage Marketplace in eng builds

r? https://github.com/mozilla-b2g/gaia/pull/12838

Comment 25

[Kumar McMillan [:kumar]]

Brian, do you know where I need to patch in the Marketplace dev cert for engineering builds? I don't see any certs in the original patch that made Marketplace a packaged app: bug 856799 (unless I missed it)

Comment 26

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Kumar McMillan [:kumar] from comment #25)
> Brian, do you know where I need to patch in the Marketplace dev cert for
> engineering builds? I don't see any certs in the original patch that made
> Marketplace a packaged app: bug 856799 (unless I missed it)

The preinstalled marketplace cert is in Gecko. Look at security/build/b2g-certdata.mk. The instructions for changing the cert is in the comment at the top of the file. The main change you probably need to make:

#     PATH=$NSS/bin:$NSS/lib addbuiltin -n "b2g-app-root-cert" -t ",,Cu" \
#       < b2g-app-root-cert.der > b2g-certdata.txt

replace "> b2g-certdata.txt" with ">> b2g-certdata.txt".

But, since you only want to do this for ENG builds, you need to add some conditional logic and whatnot.

Comment 27

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

If you want to try this at the Gaia layer, you may be able to do so by adding the certs with the object signing bit to the user NSS certificate database in Gaia. That may allow you to do this from within the Gaia build system.

Comment 28

[Kumar McMillan [:kumar]]

(In reply to Brian Smith (:briansmith, was :[e-mail]) from comment #26)
> But, since you only want to do this for ENG builds, you need to add some
> conditional logic and whatnot.

Do we have a way to add conditional logic for eng builds right now? Or is that what bug 864820 is going to add?

Comment 29

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Kumar McMillan [:kumar] from comment #28)
> (In reply to Brian Smith (:briansmith, was :[e-mail]) from comment
> #26)
> > But, since you only want to do this for ENG builds, you need to add some
> > conditional logic and whatnot.
> 
> Do we have a way to add conditional logic for eng builds right now?

I don't know the answer to that. That's why I was so vague.

Comment 30

[Alexandre Poirot [:ochameau]]

Comment on attachment 816853 [details] [diff] [review]
gaia patch for dev/stage Marketplace in eng builds

Sorry for getting back so late at your patch, it looks good.
Just some nits in order to help distinguish the various marketplace app we will end up with.

Comment 31

[Alexandre Poirot [:ochameau]]

Landed on master:
https://github.com/mozilla-b2g/gaia/commit/85c929f602b6791fcc53c96e30b77145adef3dd9

Comment 32

[Kumar McMillan [:kumar]]

Thanks for the merge, Alexandre. I'm re-opening because we still need a gecko patch to put the Dev cert on eng builds. I'm still trying to wrap my head around that part.

Comment 33

[Kumar McMillan [:kumar]]

on second thought, this might be as close as we'll get so I'm closing the bug :) It's a step in the right direction. Testers only have one manual step now: to install certs. If someone wants to put certs in the eng builds then by all means please do. It can be another bug.