909557 - [project] Marketplace in China (Flatfish)

Status: RESOLVED WONTFIX

(Tracking :: User Story, defect)

Description

[Caitlin Galimidi [:c8o]]

REQUEST:
China would like to stand up a separate instance of Marketplace, leveraging existing, "Global Marketplace" code. Because maintenance of the Marketplace involves significant work for both developers and operations teams, there will be work for the China team(s) to maintain the Marketplace in China


RECOMMENDATION:
- "Global Marketplace" maintains the main code repository (a reference implementation)
- China Marketplace consumes the code base from the main code repository, "preffing off" the payments services features (for now) 
- China may opt to maintain local patches 
- China team is encouraged to submit relevant patches upstream to be approved and integrated into the shared code base

This is the same model Firefox and thousands of other open source projects use.  This lets the Chinese group customize the Marketplace as much as they'd like, and enables improvements to flow both ways in a very common workflow.


EXPECTATIONS:
- the U.S. and China Marketplace environments will get out of sync. once they do will remain that way, unless China is rigorous about submitting their patches for review and updating against the Marketplace "reference build"
- China team can request modifications in bugzilla, assign to the Marketplace product, for consideration and triage into current product priorities and roadmap (please consider cc-ing the team below for proper visibility)


TEAM:
- Caitlin Galimidi : Program Manager Marketplace (to ensure new requests 
- Wil Clouser : Development Manager Global Marketplace
- David Bialer : Sr Product Manager Global Marketplace (for significant new feature requests and to discuss the priority of feature requests) 
- Tony Santos : UX Lead Marketplace
- Mark Mayo : VP Service Operations
- Jeremy Orem : Marketplace Service Operations


COMMUNICATIONS:
- detail technical requirements and procedures : bugzilla bugs
- developer questions in real-time : IRC #marketplace
- twice weekly meetings during initial deployment : Caitlin hosting 
- questions and discussions between China and Global Marketplace teams : mailing list tbd
- conversation tracking : https://etherpad.mozilla.org/mkt-china 


SERVER ENVIRONMENT:
- https://mana.mozilla.org/wiki/display/websites/Network 


CODE DEPLOYMENT:
- https://wiki.mozilla.org/Marketplace/FindingTheCode 


OPEN QUESTIONS:
- Operations / Infrastructure procedures to stand up and deploy servers / network
- Managing developer engagement features and instructions to app developers to create, submit and manage apps in the environment
- Preffing on and managing payments in the China Marketplace

Comment 1

[Caitlin Galimidi [:c8o]]

Setting up meeting to talk through this proposal with the China team:
oremj
clouserw
mmayo
cgalimidi
jguo
swang
...

Comment 2

[Caitlin Galimidi [:c8o]]

Jack to identify IT lead and Development Lead to speak directly with Mark and Wil next week. These teams need to identify the tasks / requirements to stand up China marketplace and provide an estimate of work back to the team.


Open Questions: 

> What is the anticipated frequency of development code changes that will be made to the China Marketplace? 
> Wil Clouser & China Development Lead


> What servers does China currently have up and running? Do they have any marketplace?
> Mark Mayo & China IT Ops / Services Lead

Comment 3

[Caitlin Galimidi [:c8o]]

Open Question:
> Is there an actual legal requirement for the server instance to be inside China?
> From Mark Mayo to China team

Comment 4

[Jack Guo]

(In reply to Caitlin Galimidi from comment #2)
> Jack to identify IT lead and Development Lead to speak directly with Mark
> and Wil next week. These teams need to identify the tasks / requirements to
> stand up China marketplace and provide an estimate of work back to the team.
> 
> 
> Open Questions: 
> 
> > What is the anticipated frequency of development code changes that will be made to the China Marketplace? 
> > Wil Clouser & China Development Lead
> 
> 
> > What servers does China currently have up and running? Do they have any marketplace?
> > Mark Mayo & China IT Ops / Services Lead

Talked with local IT people and found out that AMO mirror site here in China was fully controlled by Shyam Mani ([e-mail]) from Corporate IT. Local people has no right to manage it. So, suggest you guys to talk with Shyam directly. 

AMO mirror site only worked out access issue, which caused by "Great Fire Wall" thing, but has nothing to do with product. For Marketplace, sine China local team will control "everything" from server to content, so add it up on AMO mirror server seems will not work. We shall think something separate, i.e. set up a local site.

Comment 5

[Jack Guo]

(In reply to Caitlin Galimidi from comment #3)
> Open Question:
> > Is there an actual legal requirement for the server instance to be inside China?
> > From Mark Mayo to China team

Cannot provide or contain any sensitive contents. We have people here review our contents manually on a daily basis.

Comment 6

[Jack Guo]

(In reply to Jack Guo from comment #4)
> (In reply to Caitlin Galimidi from comment #2)
> > Jack to identify IT lead and Development Lead to speak directly with Mark
> > and Wil next week. These teams need to identify the tasks / requirements to
> > stand up China marketplace and provide an estimate of work back to the team.
> > 
> > 
> > Open Questions: 
> > 
> > > What is the anticipated frequency of development code changes that will be made to the China Marketplace? 
> > > Wil Clouser & China Development Lead
> > 
> > 
> > > What servers does China currently have up and running? Do they have any marketplace?
> > > Mark Mayo & China IT Ops / Services Lead
> 
> Talked with local IT people and found out that AMO mirror site here in China
> was fully controlled by Shyam Mani ([e-mail]) from Corporate IT.
> Local people has no right to manage it. So, suggest you guys to talk with
> Shyam directly. 
> 
> AMO mirror site only worked out access issue, which caused by "Great Fire
> Wall" thing, but has nothing to do with product. For Marketplace, sine China
> local team will control "everything" from server to content, so add it up on
> AMO mirror server seems will not work. We shall think something separate,
> i.e. set up a local site.

Let me answer you questions below (my people is not good at English)
(1) will not change development code very often. once a quarter we guess
(2) We have a local "marketplace" liked website http://ffmarket.com/. It is a demo version, since no Firefox OS device in this market yet.

Comment 7

[Karen Ward [:kward]]

Does the team need to consider set up and staffing of App Review team, Apps Review tools and Partner Engineering to support this new, unique Marketplace?

Comment 8

[Caitlin Galimidi [:c8o]]

Karen - reviews would be managed in China - and yes, they will have to staff accordingly and develop processes around app approvals. (ie: apps currently approved in our marketplace may not be approved in theirs)

Comment 9

[Caitlin Galimidi [:c8o]]

Jeremy, Shyam and Wil met yesterday to talk through the work to standup Marketplace in China and uncovered all of our risks and challenges.

SUMMARY: 
Our current marketplace more features and infrastructure than what China team will use (payments, solitude etc). Maintaining our complex site requires a small army - China has one, lightly skilled ops person so far confirmed and 0 devs (so far). U.S team maintaining this from the other side of the firewall is next to impossible.

The logistics in China make the timeline to stand up Marketplace infrastructure impossible (i.e.: 3 mos to get servers through customs. otherwise, pay 3x as much for servers) 


RECOMMENDATIONS:
- deliver a "simpler experience" / UI for both consumers and engineers to maintain
- consider an alternative approach for hosting the infrastructure
- - - ChinaCache - caching site in a China data center, hosting in the states
- - - Simplifying the UI for better user interaction, maintaining UI in China

NEXT STEPS:
- Mayo > diagram proposed alternative infrastructure (Caching, firewalls, geographic location of servers)
- Wil & David > diagram out flows, flesh out China stories (developer uploads, app reviewer, regional-only content)
- Caitlin > reach out to Jinghua : need to quantify site performance in country
- Jack > need an alternative for the developer preview date (Oct 15)



DETAILED NOTES ON STANDING UP NEW INFRASTRUCTURE IN CHINA

SITE PERFORMANCE may be degraded (need to quantify) (Jinghua?):
- Bandwidth in China is extremely limited, even for "special cases" like government officials and diplomats. Bandwidth is very expensive. Our site is not going to perform as well in China market
- - - App uploads may be exceptionally slow
- - - social / user feedback features may be very slow

RESOURCES onsite in China:
- Marketplace front end is complex to maintain and needs to be maintained from behind the firewall 
- - - efficiency, not necessarily legal requirement
- - - current marketplace = 8 full time devs / engineers
- - - Need to identify / confirm developers in China with appropriate skill-set to setup / maintain front end and integration

- China does not need the entire Marketplace infrastructure (payments / solitude / user data) 
- - - Need to identify / confirm Server Systems Admins in China with the appropriate skill-set to setup / maintain
- - - Shyam is aware of one resource: Yousheng Li)

SET-UP COSTS:
- Currently have 2 racks supporting China. Mozilla does not own (subcontracted to "Mozilla Online", a vendor.) 
- for the scenario previously listed in this bug we'll need to stand up a whole new environment
- Cost of racks and servers is significant : Getting servers through customs is expensive vs. servers costing 3x as much within China
- we currently assume we need to look for alternatives

LOGISTICS:
- 3 mos for servers to get through customs (must find servers in China)
- power supply to server racks significantly limited, requiring different setup / management 
- Need team on the ground for the duration of setup and team onsite to maintain 

RISKS
- need to understand the user data to be stored in China 

OPTIONS
- Hong Kong server location (cloud based)
- AWS in Singapore (cloud)
*** - ChinaCache - caching site in a China data center, hosting in the states ***
*** - Simplifying the UI for better user interaction ***

Comment 10

[Tony Santos [:tsmuse]]

I feel the need to point out for scoping/budgeting/timelining that a UI re-design for users who have to use an IME to use Chinese/Japanese/Korean/etc characters is not a small task. Designing sites for Asia is not something the existing Firefox Marketplace UX team is really equipped to do, and would require close work with Chinese designers to make it work.

Comment 11

[Thomas Ho [:tho]]

Tony,

I'm not sure what UI re-design issue you're talking about here. If it's on device, both English and Chinese needs a software keyboard to input. For PC/Macs, Chinese people simply use keyboard to input. Nothing fancy here. Please clarify your concern. Thank you very much.

Comment 12

[David Bialer [:dbialer]]

1. Let us not assume that our current UI for tablets is not usable.  It is actually quite simple, though still may not be appropriate for China.  Just need to know if it is culturally appropriate.  Thomas, can you have your team evaluate our current design for tablets if was localized?  Also future design?  It would be *WAY* easier to support a single or similar code base.

2. Regarding the given reasons for hosting in China.  Data and facts simply don't validate the perception.

* our performance data in Marketplace doesn't indicate that the site is slower in China.  It is about average and faster than many of the Latin countries.  So I would not assume we have a performance problem here until we do some further testing.  We have no data that indicates a performance issue.  From our Data, wireless connections in China are relatively slow - having to do with the network, not the point of hosting - from page load times, not server response times.  Sure we would most likely get some performance boost if hosted in China, but the internal wireless networks are largely the bottleneck, not server response times.  Our 'wired' network from China performance data is substantially better and towards the top of all countries.  It may be OK to at least launch with our current Marketplace infrastructure.  It would be great to have some data on the perceived performance issues.

3. all content in China is subject to censorship.  Internal China companies just understand this better.  But all content passes through gateways and 'firewalls' that block IP addresses.  Foreign companies just have central points of routers that can be more easily blocked.  And they are more likely to be blocked since they don't censor their content for China.   There is no single 'great firewall'- that is a bit of a myth.   The issue is about having content that complies with local laws - and Chinese companies understand these better while foreign companies don't screen content for China laws.   In our case, for China content, we will ensure content is appropriate.  The hosting location is largely irrelevant if you have no cause for being blocked. 

It is a lot of effort and expense to host Marketplace in China and all we have are assumptions that (a) we have a problem and that (b) our solution will solve them.  

I suggest we focus on the high priority issues:  ensuring the content is appropriate for China on the the app intake side and the consumer display side.  And that the presentation of the content has a culturally appropriate UI.

Comment 13

[Caitlin Galimidi [:c8o]]

[:dbialer]

Content reviews - The Mozilla China team will manage their own content approvals and reviews (not our team here in the U.S)

Reviewing this week with the China team this week the proposal NOT to host in China (comment 9)

[:tsmuse]
UX : The China team will deliver their own (if any) UI modifications

Comment 14

[Jack Guo]

I have a proposal here, which I think it might be easier to implement. Please comment. 

(1) We copy global marketplace database into China, to solve user access issue that caused by "Great Firewall".
(2) We build up a local UI. This UI will be the same style as global marketplace, but will be modified to fit local market, for example, replace Facebook with Weibo etc. 
(3) China developers submit app to global marketplace, using global marketplace platform.
(4) Local database, we do regular sync with global marketplace, to make sure we have the latest apps.

Comment 15

[Wil Clouser [:clouserw]]

Attachment: china.proposal.pdf

That's possible, but I think your step #2 adds more development and UX time than you need, at least for a first version.  I'm attaching an alternative proposal which I think fulfills the requirements and will use the least amount of time to implement.

Comment 16

[Wil Clouser [:clouserw]]

We had a meeting about this today (Jack, Hong, Yousheng, Caitlin, Wil).  Some bullet point conclusions:

* The attached flow (attachment 803377 [details]) looks like something we can pursue

* We're interested in registering a separate domain for the Chinese Marketplace because China blocks at the DNS level.  The new domain would just front the current one.  That's easy enough to do technically but unknown how this affects app installation (signing/whitelists/etc.).  (TODO:Caitlin: followup on this question)

* We need the equivalent of the AMO Netscalers in China.  Either more Netscalers (very expensive, and slow through customs?) or some other caching service.  (TODO:Wil: followup with Jeremy)

* There is interest in inverting the developer/curator flow so that developers have the option of selecting China and the curator sees a list of all the new opted-in apps and can approve them.  This also answers question #1 on slide 11 for free so worth exploring.  (TODO:Wil: explore this flow)

* We are agreed on the high level flow which means China needs to find 3 resources:  1) A developer to build their UI to interact with our API for curation,  2) Someone who will do the actual curation,  3) A reviewer to help review new apps (particularly, the chinese-only apps) (TODO:Hong: start thinking about resources)

We're planning on meeting again next week with the answers to the above. Thanks!

Comment 17

[Li Gong]

I would like to add a side note here -- in *no* circustances should we plan to *import* hardware and/or software into China for the purpose of standing up this system. We will have to procure everything from China and handle the import issues directly from China. Note it does not work even if someone could hand carry HW/SW into China.

Part of the reason is that our own finance requires that all China expenses be paid by Mozilla China to Mozilla US. In order to do that (paying a foreign entity via legal means), we need all the proper paperwork for the purchase and the importation of these HW/SW, but the purchaser has to be Mozilla China, not Mozilla US (it would be considered *used* equipment if the original purchaser is not Mozilla China). Plus, SW pre-installed on HW would not pass import process. And a lot more hassle. And a lot more tax (in both directions). Last time IT sent us some servers, it took us a year and many many many man-hours and a lot of extra money to get it sorted out.

So please -- no HW/SW import from Mozilla US to Mozilla China, period. We will just buy whatever we decide on, from China, and have the usual importers deal with all the import issues.

Thanks,

Li

Comment 18

[Mark Mayo [:mmayo]]

Wil's flow chart suggests we only need in-country caching in China, in which case I propose we look at services from ChinaCache immediately and not even get into buying/installing/supporting hardware (regardless of origin) if at all possible.

Comment 19

[Yousheng Li [:yli]]

Hi Mark,

Using ChinaCache can't resolve the problem caused by GreatWall Firewall.
marketplace.firefox.com was registered outside of China, and this domain has not getting ICP (Internet Content Provider) record.
Because of this reason, this domain will be blocked from time to time in China.

Comment 20

[Wil Clouser [:clouserw]]

Attachment: china.proposal.pdf

(In reply to Mark Mayo [:mmayo] from comment #18)
> Wil's flow chart suggests we only need in-country caching in China, in which
> case I propose we look at services from ChinaCache immediately and not even
> get into buying/installing/supporting hardware (regardless of origin) if at
> all possible.

From talking to Yousheng last night it sounds like they do DNS-level blocking so we talked about setting up what would essentially be a proxy+cache with a different domain name.  I'm not sure if China Cache can do that or not, but I'm sure someone offers the functionality.  There are still some open questions around whether that will work or not (see attached).

I'm attaching the newest proposal.  I redid the flows to be in line with what we talked about last night.  Open questions around signing/stats are still open, but please review the rest of the document.  Thanks.

Comment 21

[Jeremy Orem [:oremj]]

We can get in contact with chinacache. Unless they have a very non-standard approach, we will be able to use an origin that differs from the distribution name.

Comment 22

[swang]

Just to add the Foxconn tablet project milestone and deliverable, to align with China Marketplace requirement and dev schedule,
Phase I: mid Oct. 2013- focus on stability and performance, for Foxconn to be able to demonstrate with partner/content providers for tablet app/content dev.
Phase II: mid Dec. 2013- final image with tablet UI change should be available, for the commercial launch by Jan.2014.
Launch market(wave 1): China.
The Marketplace dev will need to be available by mid Dec. 2013.

Comment 23

[David Bialer [:dbialer]]

One other thing I was thinking about is search and see if there is anything we need to do to support Chinese language analysis within elasticsearch.

Comment 24

[[:fabrice] Fabrice Desré]

I'm closing this bug since we confirmed that we could install apps from 2 different marketplaces using the same signing key.

Comment 25

[Caitlin Galimidi [:c8o]]

Testing on app install on two domains was resolved successfully : We are using the same server key for .cn as .com

- (Thomas Ho) Prefs will need to be set on device during the OS / Hardware build integration (Thomas Ho)
- (Thomas Ho) Need to set the ADI ping url as part of the custom build
- (Hong) Need to get the .cn domain (Hong)
- (Yousheng / Hong - Jeremy can support) Do we need CDN domain name?
- (Yousheng (Shyam?)) Need to solve for Marketplace updates during blackout (internet down)?
- Need to test how app receipts will work during blackouts : Apps that have rigid, real-time receipt verification will not allow the app to function (ie: banking apps)
- (Jeremy) Identify servers available in China (via Sylvie Veilleux's team) - set up load balancer and cache (uncached url for ADI pings)
- (Hong) Long term: Need to investigate option for ChinaCache, and permanent server will likely be necessary 

Jack and Spinger - 
Please review both the attachment and recommended next steps with your teams.
Ops and Marketplace Dev will hold all activity until we receive confirmation from you. 
Happy to set up a meeting to discuss in detail. 

Caitlin

Comment 26

[Caitlin Galimidi [:c8o]]

Jack, Spinger and Hong - 

Needs info Item #1:
Please review this latest attachment along with the open action items in the comment above. https://bugzilla.mozilla.org/attachment.cgi?id=803992 
Are you aligned with the approach and action items above?


Needs info Item #2:
Our content review team would like to share their process and identify how to best support your team in setting up apps reviews. https://bugzilla.mozilla.org/show_bug.cgi?id=919822 
Who is your point(s) of contact for content reviews and curation?

Comment 27

[David Bialer [:dbialer]]

We should probably also ensure that elasticsearch can support Chinese - there may be plug-ins for this.

Comment 28

[Hong Tang]

(In reply to Caitlin Galimidi from comment #25)
> Testing on app install on two domains was resolved successfully : We are
> using the same server key for .cn as .com
Great, Thanks !

> 
> - (Thomas Ho) Prefs will need to be set on device during the OS / Hardware
> build integration (Thomas Ho)
> - (Thomas Ho) Need to set the ADI ping url as part of the custom build
> - (Hong) Need to get the .cn domain (Hong)
We just get the .cn domain ready today, which is "marketplace.firefox.com.cn"
Now what we might need is the IP address this .cn domain pionting to.
Yousheng will send you the SSL certification via email later today.

> - (Yousheng / Hong - Jeremy can support) Do we need CDN domain name?
use the same domain name "marketplace.firefox.com.cn" can be used.

> - (Yousheng (Shyam?)) Need to solve for Marketplace updates during blackout
> (internet down)?
Since we will use .cn domain name, it won't be blocked by Great Firewall.

> - Need to test how app receipts will work during blackouts : Apps that have
> rigid, real-time receipt verification will not allow the app to function
> (ie: banking apps)
> - (Jeremy) Identify servers available in China (via Sylvie Veilleux's team)
> - set up load balancer and cache (uncached url for ADI pings)
> - (Hong) Long term: Need to investigate option for ChinaCache, and permanent
> server will likely be necessary 
> 
> Jack and Spinger - 
> Please review both the attachment and recommended next steps with your teams.
> Ops and Marketplace Dev will hold all activity until we receive confirmation
> from you. 
> Happy to set up a meeting to discuss in detail. 
> 
> Caitlin

Comment 29

[Yousheng Li [:yli]]

(In reply to Hong Tang from comment #28)
> (In reply to Caitlin Galimidi from comment #25)
> > - (Yousheng / Hong - Jeremy can support) Do we need CDN domain name?
> use the same domain name "marketplace.firefox.com.cn" can be used.
> 
Hi Caitlin,
Do you mean the replace domain name for "marketplace.cdn.mozilla.net"?
If so, I suggest we can use marketplace.cdn.firefox.com.cn.
I'm applying the SSL certification for this domain, and will send to you when it's done.

Comment 30

[Caitlin Galimidi [:c8o]]

SSL Certification received!

Comment 31

[Wil Clouser [:clouserw]]

(In reply to Yousheng Li [:yli] from comment #29)
> (In reply to Hong Tang from comment #28)
> > (In reply to Caitlin Galimidi from comment #25)
> > > - (Yousheng / Hong - Jeremy can support) Do we need CDN domain name?
> > use the same domain name "marketplace.firefox.com.cn" can be used.
> > 
> Hi Caitlin,
> Do you mean the replace domain name for "marketplace.cdn.mozilla.net"?
> If so, I suggest we can use marketplace.cdn.firefox.com.cn.
> I'm applying the SSL certification for this domain, and will send to you
> when it's done.

Yes please. Thank you.

Comment 32

[Yousheng Li [:yli]]

(In reply to Caitlin Galimidi from comment #25)
> - (Jeremy) Identify servers available in China (via Sylvie Veilleux's team)
> - set up load balancer and cache (uncached url for ADI pings)

Hi Jeremy,
Did you identify the servers available in China?
Please let me know whether need I do research in China.

Comment 33

[Jack Guo]

(In reply to Caitlin Galimidi from comment #26)
> Jack, Spinger and Hong - 
> 
> Needs info Item #1:
> Please review this latest attachment along with the open action items in the
> comment above. https://bugzilla.mozilla.org/attachment.cgi?id=803992 
> Are you aligned with the approach and action items above?
> 
> 
> Needs info Item #2:
> Our content review team would like to share their process and identify how
> to best support your team in setting up apps reviews.
> https://bugzilla.mozilla.org/show_bug.cgi?id=919822 
> Who is your point(s) of contact for content reviews and curation?

I am OK with the flow. 

Please add [e-mail] and [e-mail] for content review and curation. 


Thanks,
Jack

Comment 34

[Yousheng Li [:yli]]

(In reply to Wil Clouser [:clouserw] from comment #31)
> (In reply to Yousheng Li [:yli] from comment #29)
> > (In reply to Hong Tang from comment #28)
> > > (In reply to Caitlin Galimidi from comment #25)
> > > > - (Yousheng / Hong - Jeremy can support) Do we need CDN domain name?
> > > use the same domain name "marketplace.firefox.com.cn" can be used.
> > > 
> > Hi Caitlin,
> > Do you mean the replace domain name for "marketplace.cdn.mozilla.net"?
> > If so, I suggest we can use marketplace.cdn.firefox.com.cn.
> > I'm applying the SSL certification for this domain, and will send to you
> > when it's done.
> 
> Yes please. Thank you.

The SSL certification of marketplace.cdn.firefox.com.cn was sent, please check.

Comment 35

[Jeremy Orem [:oremj]]

Yousheng, what kind of blocks do we need to look out for between our backend servers in the US and our load balancers in China?

Comment 36

[Yousheng Li [:yli]]

(In reply to Jeremy Orem [:oremj] from comment #35)
> Yousheng, what kind of blocks do we need to look out for between our backend
> servers in the US and our load balancers in China?

Hi Jeremy,

I think there are 2 kinds of blocks need to look out:
1. Great Firewall issue:
  1.1 DNS poisoning. 
  1.2 Blocking Access to IPs. 
  1.3 Blocking VPNs. 

  This article describes how the Great Firewall works:
  http://www.howtogeek.com/162092/htg-explains-how-the-great-firewall-of-china-works/

2. The quality of the international internet bandwidth. 
The quality and reliability are not good in China: the international network bandwidth is not large enough, network delay and packet-lost happening from time to time.

Comment 37

[Wil Clouser [:clouserw]]

Attachment: screencast of current curation flow

See the attachment for a video of the current approval flow.

Comment 38

[Wanting Xiong]

Wil, this curation flow looks pretty good from the video. thanks!