1365372 - releng work for new Leanplum SDK private app tokens

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

see bug 1365089 for details

Similar work was done in bug 1365089 

I know the buildduty folks have a lot on their plate now so just as a fyi this bug doesn't supercede existing work on your plate in terms of priority.

Comment 1

[Nick Alexander :nalexander [he/him]]

(In reply to Nevin Chen [:nechen] from comment #10)
> If you need the app_id and access_key for each channel. Please feel free to
> let me know.

Nevin: we will absolutely need to get kmoir the relevant token files for release, beta, and nightly.  You'll need to GPG encrypt them and send them to kmoir, like mfinkle did in https://bugzilla.mozilla.org/show_bug.cgi?id=1152871#c4.  kmoir's key is available at

https://gpg.mozilla.org/pks/lookup?search=kmoir%40mozilla.com

kmoir: the files can go in /build or wherever all the other tokens are, and then we can update the patches in Bug 1365089.

Comment 2

[Kim Moir [:kmoir] ET]

Probably someone from buildduty will do this work so I'm not sure using my key is the best path until someone is assigned this work.

Comment 3

[Nevin Chen(Not active on Bugzilla)]

I've sent out an email to Kim. If I did it wrong, please feel free to tell me.

Comment 4

[Kim Moir [:kmoir] ET]

Andrei would this be something you could look into once you are done with the security work?

Comment 5

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Kim,sure I can take care of this bug,if is not that urgent I can check it next week after I come from PTO.

I have one question here (https://bugzilla.mozilla.org/show_bug.cgi?id=1152871#c5),where is hiera cache ,where I should add the tokens.

Comment 6

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

(In reply to Andrei Obreja [:aobreja][:buildduty] from comment #5)
> Kim,sure I can take care of this bug,if is not that urgent I can check it
> next week after I come from PTO.
> 
> I have one question here
> (https://bugzilla.mozilla.org/show_bug.cgi?id=1152871#c5),where is hiera
> cache ,where I should add the tokens.

Hey, may I know the ETA of this?
It's quite urgent for us because it's a testing blocker.
w/o this the fennec nightly build doesn't have the LeanPlum SDK (Bug 1361664) functionality working at all.

Comment 7

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

My apology for keeping pushing, but this is essential to unblock testing.
We are aiming to have everything ready for testing by end of this week. Is that achievable ?

Comment 8

[Kim Moir [:kmoir] ET]

Andrei is on PTO until Thursday. 

What are the values for the tokens?

It looks like these are requested here

https://bugzilla.mozilla.org/show_bug.cgi?id=1365089#c17

We need the values before we can implement them in production.

Comment 9

[Nevin Chen(Not active on Bugzilla)]

I've sent out the email to you again. Please help verify. Thank you!

Comment 10

[Kim Moir [:kmoir] ET]

I won't be able to get to this bug, I have too much other work on my plate.  Andrei, can you make writing patches for this bug your first priority when you return to work tomorrow?  Bug 1152871 is an example of previous work that is similar to this request. However, we build Android on taskcluster now so there are probably changes required.

Comment 11

[Nevin Chen(Not active on Bugzilla)]

Hi Andrei
I've sent out an email with tokens to you. Please help verify:) Thanks!

Comment 12

[Kim Moir [:kmoir] ET]

So I talked to catlee about this bug today and mentioned that since we build on taskcluster, these secrets are stored here

https://tools.taskcluster.net/secrets/

You can see examples of the adjust.sdk tokens there, similar to what we will be doing for Leanplum sdk

Comment 13

[Nevin Chen(Not active on Bugzilla)]

Thank you! Kim!

But when I want to create a secret with name "/builds/leanplum-sdk.token"
It said "You do not have sufficient scopes. This request requires you to have one of the following sets of scopes"

What should I do? Thanks!

Comment 14

[Nevin Chen(Not active on Bugzilla)]

(In reply to Nevin Chen [:nechen] from comment #13)
> Thank you! Kim!
> 
btw, do you know what leve-2 and level-3 means? I can't find it in the document.

Comment 15

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

NI to Chris: 
Hey, is it possible to assign someone from RelEng to support this urgent task? It's no ideally efficient for Fennec front-end developer to spend hours trying to handle by their own.

NI Mike Han as he's the major stakeholder. This is a potential risk factor.

Comment 16

[Chris AtLee [:catlee]]

The secrets should be named something like this:

project/releng/gecko/build/level-2/adjust-sdk.token
project/releng/gecko/build/level-3/adjust-sdk-beta.token
project/releng/gecko/build/level-3/adjust-sdk-release.token

Level 2 / 3 refers to the trust level of the HG repository the builds are happening on.

mozilla-central, beta, and release are level3 repositories.

Do you need a separate token for try / inbound / autoland?

Comment 17

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Added leanplum-sdk-nightly-beta.token and leanplum-sdk-release.token for level 2 and level 3 on https://tools.taskcluster.net/secrets/

Comment 18

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1365372_mozharness.patch

The patch to add the tokens on releng_base_android_64_builds.

Comment 19

[Nevin Chen(Not active on Bugzilla)]

Hi Andrei
Thanks for the support. 
According to comment 16, Should it be three entries like below? 
project/releng/gecko/build/level-2/adjust-sdk.token
project/releng/gecko/build/level-3/adjust-sdk-beta.token
project/releng/gecko/build/level-3/adjust-sdk-release.token


Thanks again for your help!

Comment 20

[Nevin Chen(Not active on Bugzilla)]

(In reply to Chris AtLee [:catlee] from comment #16)
> The secrets should be named something like this:
> 
> 
> mozilla-central, beta, and release are level3 repositories.
> 
> Do you need a separate token for try / inbound / autoland?

Hi Chris, thank you for the promot reply. I don't really know why try / inbound / autoland should use separate token. So I guess it's not neccesarry? Since this libray(Leanplum) will charge us by how many use it. So I  want to restric to only let nightly, beta, and release build use those keys.

Comment 21

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

> According to comment 16, Should it be three entries like below? 
> project/releng/gecko/build/level-2/adjust-sdk.token
> project/releng/gecko/build/level-3/adjust-sdk-beta.token
> project/releng/gecko/build/level-3/adjust-sdk-release.token

Hi Nevin

you will have the bellow list:

project/releng/gecko/build/level-2/leanplum-sdk-nightly-beta.token
project/releng/gecko/build/level-2/leanplum-sdk-release.token
project/releng/gecko/build/level-3/leanplum-sdk-nightly-beta.token
project/releng/gecko/build/level-3/leanplum-sdk-release.token

A token for each case(nightly-beta and release)but we also need to push the patch from comment18 in order to work.

Comment 22

[Nevin Chen(Not active on Bugzilla)]

Hi Andrei.
Can you separate project/releng/gecko/build/level-3/leanplum-sdk-nightly-beta.token and make it 
project/releng/gecko/build/level-3/leanplum-sdk-beta.token
and
project/releng/gecko/build/level-3/leanplum-sdk-nightly.token
?
I may want to have the flexibility to separate nightly key and beta key in the future.

Comment 23

[Nevin Chen(Not active on Bugzilla)]

Attachment: Bug 1365372 - Add Leanplum SDK private app tokens

Review commit: https://reviewboard.mozilla.org/r/142866/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/142866/

Comment 24

[Nevin Chen(Not active on Bugzilla)]

Comment on attachment 8871390 [details]
Bug 1365372 - Add Leanplum SDK private app tokens

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/142866/diff/1-2/

Comment 25

[Chris AtLee [:catlee]]

Comment on attachment 8871390 [details]
Bug 1365372 - Add Leanplum SDK private app tokens

https://reviewboard.mozilla.org/r/142866/#review146602

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/297b1dcda856
Add Leanplum SDK private app tokens r=catlee

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/297b1dcda856

Comment 28

[Nevin Chen(Not active on Bugzilla)]

I'm writing down some notes here.

Our previous design is release use one api token, nightly and beta use another token.
pros:
Use Leanplum's "User attribute" to separate the users in Beta and Nightly. The user only need to maintain one message setting between Nightly and Beta in Leanplum backend. 

cons:
1. If we forget to set the target user in Nightly and Beta, there'll be ~200K Beta users affected.
2. Nightly and Beta have different code and bug, same Leanplum marketing setting may not fit everyone.
3. Leanplum message can target different user attribute, but don't know for other features yet (Push Notification/Vairaible/Onboarding UI)
4. I'm also not sure about the data presentation looks like in Leanplum Admin when Nightly and Beta User mixed together. There may be a way to distinquish that but I currently don't have the time to investigate.
5. It's easier to separate now. It'll be hard if we want to separate later.

Another approach is separate each Channel with different Leanplum App token. This will fix above issues but need to maintain the message list. 

In the last patch I provided(copy from Andrei's :p ) uses different secrets for different channels. But the content of those secret (Beta and Nightly) are the same. So the patch provide the flexibility to change them in the future without landing new code. We might want to update the secrets and make them link to different Leanplum App before Bug 1361664 get into beta.

Thank you all for your effort and kind support!

Comment 29

[Nick Alexander :nalexander [he/him]]

Technical note first:

> In the last patch I provided(copy from Andrei's :p ) uses different secrets
> for different channels. But the content of those secret (Beta and Nightly)
> are the same. So the patch provide the flexibility to change them in the
> future without landing new code. We might want to update the secrets and
> make them link to different Leanplum App before Bug 1361664 get into beta.

I expect you will regret doing this.  Fennec users do not upgrade very quickly, so as soon as you switch the ID contents to separate Beta and Nightly, you will have users with {Nightly vOLD, beta-and-nightly-ID} and {Nightly vNEW, nightly-only-ID}.  So you have even more user populations that you need to manage.

jcheng: very shortly you will have three different code bases (one for each release channel), each of which you will want to experiment with (and hotfix!) independently.  I strongly recommend that you figure out how to target each channel independently _before_ you need to do so in a rush (to work around a bug, for example).

I see two options:

1) We allocate one Leanplum ID for all Mozilla products and trust that Leanplum's "User Attribute" or version targeting lets us control each release channel independently.

2) We allocate one Leanplum ID for each Mozilla release channel, and accept that Leanplum's analytics might not be intended to work across IDs in this way.

I do not know how Leanplum expects to be configured in this situation, but I do know you need to understand _exactly_ how you're going to handle this situation before we enable this code in any release channel.

In the meantime, I will ensure that the current patch is technically acceptable.

Comment 30

[Nick Alexander :nalexander [he/him]]

Over in https://treeherder.mozilla.org/#/jobs?repo=try&revision=02b1d0f489ef43815a5754ed66c76a50365672ea&selectedJob=102454766

I see:

[task 2017-05-26T19:37:25.839039Z] 19:37:25     INFO -  checking for the Leanplum SDK key... no
[task 2017-05-26T19:37:25.839215Z] 19:37:25     INFO -  ERROR: '/builds/leanplum-sdk-nightly.token': No such file or directory.

What am I doing wrong?

Comment 31

[Nick Alexander :nalexander [he/him]]

https://hg.mozilla.org/mozilla-central/rev/297b1dcda856#l1.18 has

+        {'filename': '/builds/leanplum-sdk-nightly.token',

and I've just checked that my copy-pasta looks correct in the two places.

Comment 32

[Nevin Chen(Not active on Bugzilla)]

The file name looks correct to me. I don't know what's going on...

Comment 33

[Chris AtLee [:catlee]]

The way we have it set up involves a few components.

mozharness configs [1] specify what secret names to fetch from the TC secret service, and for which SCM level. The patch that landed specified no secrets were required for try.

mozconfig [2] specifies which token to use for which channel name.

Finally, the taskcluster secret service [3] hosts the tokens, protected by scopes appropriate to each SCM level.

Right now we have no level-1 tokens in Try, the mozconfigs aren't looking for try tokens, and mozharness isn't trying to download any tokens on try.

The approach used by the adjust SDK may be appropriate for Try? Have a dummy token in-tree that you can use in the absence of one of the real tokens?


[1] https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/builds/releng_base_android_64_builds.py#41

[2] https://dxr.mozilla.org/mozilla-central/source/mobile/android/config/mozconfigs/common#60

[3] https://tools.taskcluster.net/secrets/

Comment 34

[Nick Alexander :nalexander [he/him]]

> The approach used by the adjust SDK may be appropriate for Try? Have a dummy
> token in-tree that you can use in the absence of one of the real tokens?

nechen: this works for Adjust because the dummy, in-tree token really works with the Adjust sandbox.  (At least, it did when I was testing.)  I'm okay with pushing a bogus default token in-tree (like we had before) if it's OK to have Leanplum enabled but totally broken in every local build.

Comment 35

[Nevin Chen(Not active on Bugzilla)]

Thanks for the help!
Please correct me if I'm wrong. That's my imagination:

Nightly 's       MOZ_ANDRORID_MMA build flat is true,  so the implementation is Leamplum. It has the real token
Local builds's   MOZ_ANDRORID_MMA build flat is false, so the implementation is Stumb   . It has the fake token

Shouldn't try build use the same setting as Nightly?

btw, using a wrong app_id / key in real Leamplum impl won't break anything. Leanplum just won't start.

Hi catlee
What is Level 1 SCM? Thanks!

Comment 36

[Nevin Chen(Not active on Bugzilla)]

(In reply to Nick Alexander :nalexander from comment #34)
> > The approach used by the adjust SDK may be appropriate for Try? Have a dummy
> > token in-tree that you can use in the absence of one of the real tokens?
> 
> nechen: this works for Adjust because the dummy, in-tree token really works
> with the Adjust sandbox.  (At least, it did when I was testing.)  I'm okay
> with pushing a bogus default token in-tree (like we had before) if it's OK
> to have Leanplum enabled but totally broken in every local build.

Sorry I don't understand.
In local build, why Leanplum is enable? Do you mean it's enabled by default? Or enabled by dev's local mozconfig?
If it's enabled by developers's local mozconfig, it should be fine using a fake token. Leanplum won't crash if the token is fake when I were doing the development.
Thanks for your help!

Comment 37

[Nevin Chen(Not active on Bugzilla)]

Hi Chris. Thansk for the help!

Bug 1365089 is backed out for breaking Android L10n nightlies:

Error message is below:
https://hg.mozilla.org/mozilla-central/rev/1815768e6a1f7d9027c3c8400e324fc6dde70879

Push showing failures (previous merge had other bustage):
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=35099b4caec14bf0e3c5e3fed7a17dd3faf51dbe&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable
Failure log:
https://treeherder.mozilla.org/logviewer.html#?job_id=102597476&repo=mozilla-central
[task 2017-05-27T16:47:58.173109Z] 16:47:58     INFO -  checking for the
Leanplum SDK key... no
[task 2017-05-27T16:47:58.173507Z] 16:47:58     INFO -  ERROR:
'/builds/leanplum-sdk-nightly.token': No such file or directory.


May I know why? I thgought the issue in comment 30 only happens in try builds.
Please advise what I did wrong ?

Comment 38

[Nevin Chen(Not active on Bugzilla)]

Add NI

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

I answered that question at https://bugzilla.mozilla.org/show_bug.cgi?id=1365089#c48. The taskcluster secrets seem fine to me, the single locale repacks just their mozconfig adjusted to avoid the configure checks to the leanplum sdk, like we do for adjust already.

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Updated these secrets at nechen's request:
project/releng/gecko/build/level-3/leanplum-sdk-beta.token
project/releng/gecko/build/level-3/leanplum-sdk-nightly.token
project/releng/gecko/build/level-2/leanplum-sdk-beta.token
project/releng/gecko/build/level-2/leanplum-sdk-nightly.token

Comment 41

[Joe Cheng [:jcheng] (please needinfo)]

Hi Nevin, per our discussion offline, we will use separate keys for different channels

Comment 42

[Nevin Chen(Not active on Bugzilla)]

Hi Jean
Should we put Android in separate Leanplum team so that iOS and Android could have separate admin group in Leaplum bakcend? This will affect how we setup the token here.

Comment 43

[[e-mail]]

(In reply to Wesley Huang [:wesley_huang] (EPM) (NI me) from comment #15)
> NI to Chris: 
> Hey, is it possible to assign someone from RelEng to support this urgent
> task? It's no ideally efficient for Fennec front-end developer to spend
> hours trying to handle by their own.
> 
> NI Mike Han as he's the major stakeholder. This is a potential risk factor.