963155 - User Advocacy Server Request

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Matt Grimes [:Matt_G]]

Hello! I don't know if this is the right place to make this request, but I have not found any information on the appropriate channels. If this not the correct process, apologies in advance.

The User Advocacy team is working on several internal only tools to help us better understand user emotion, sentiment, and motivation. We need a place for these tools to live. We also generate numerous reports that are shared both internally and externally that will require hosting. I've created a set of requirements that should help you identify exactly what we need. 


1) A web server accessible from outside the VPN but with some part
protected by LDAP. This will host reports, front-end for some tools etc.
No PII will be stored on this machine.

2) A backend server (could be the same as above...) that can
   *) Access sumotools1.webapp.phx1 (including MySQL ports)
   *) Have its own standalone MySQL instance (also Mongo DB)
   *) Have Python v3 or higher
   *) Allow installs of python modules
   *) Can push (sanitized) data and reports to the server above
   *) Can handle pretty high processor loads (for machine-learning based
      text analysis tools)
   *) Can connect to github and automatically run pull requests.



I think that about covers it. Let me know if you have any questions.

Matt

Comment 1

[Chris Turra [:cturra]]

our group currently has limited resources available for new project work. to help us prioritize this request against existing tasks in our queue, can you please provide some insight into the following questions?

 - which top level goal(s) does this support?
 - what is the impact of this service to our products and/or users?
 - what is the expected outcome of this request?


additionally, i'd like to clarify why direct access to sumotools1.webapp.phx1 is required. ideally, you'd be able to do rest api calls through sumo directly to get data? 

generally, deployments that do not follow our standard python+django deployments, are designed and deployed to run on stackato (our PaaS). i am not entirely sure this is a good candidate based on your external sumotools1 and high processor load requirements, but that's something i would like to discuss with you.

Comment 2

[Matt Grimes [:Matt_G]]

Hey Chris. Info below:


 - which top level goal(s) does this support?
Fighting attrition for Firefox Desktop and getting us on a solid growth path, continuing to provide an amazing Android browser, and increasing marketshare for firefox OS

 - what is the impact of this service to our products and/or users?
We are building a new toolkit that will allow us to better analyze the data that we collect from our various feedback channels. That will allow us to provide better insights to our product and engineering teams, spot and diagnose products issues earlier in the cycle, and better quantify and prioritize long user pain points across multiple versions of our products. 

 - what is the expected outcome of this request?
The expected outcome is to provide an adequate environment for our backend team to develop and deploy our text analysis tools and for our frontend team to host and deliver more impactful reports to all mozillians.


Looping in the backend folks to answer the technical questions for you.

Comment 3

[[:Cww]]

sumotools1.webapp.phx1 basically mirrors our support and input databases and we pull data from both of those for analysis. This new user advocacy server just needs to be able to access the MySQL port (3306) on that machine so it can pull the needed info. I think we accomplish that by just being behind the VPN but I wanted to call it out in the technical needs explicitly.

Comment 4

[Matt Grimes [:Matt_G]]

Hey Chris. Do you have the information you need? We currently have several tools and reports being hosted on one of the Metrics servers. They've informed us that this server is going away, so that increases the priority for this request. I'm happy to meet with you to discuss this in more detail if needed.

Comment 5

[Matt Grimes [:Matt_G]]

Do you have any status updates for us on this server request? We have two User Sentiment Reports for Firefox 26 ready to publish. Let me know if you need additional information from me.

Comment 6

[Jake Maul [:jakem]]

Hmm... is this the same thing as bug 938895? "User advocacy" vs "user sentiments", but in comment 5 here you said sentiments. The requirements here are slightly different... that bug indicates it's all just static content (very likely suitable for one of our shared clusters), whereas this one is much more complicated.

In any case, as defined here I'd definitely like to see if we can get this into the PaaS. The only hurdles would be, as Chris mentioned, the CPU usage and access to sumotools. I think we can sort both of those out. The biggest advantage is you can move at your own pace and you have greater freedom to install packages. If that can't be made to work, the next option is to see about setting you up with some sort of standalone VM that you would likely wind up managing yourself.

Comment 7

[[:Cww]]

This bug superceeds bug 938895. At that time, we were looking at static content that we generate on the sumotools server and copy over, but now that we need a new processing server, we think that a more efficient solution would be to have the content be dynamic so we don't need to do things like generate jsons that we copy from one place to the next.

Comment 8

[Matt Grimes [:Matt_G]]

Hey C and team. I think you have all the information you need from us. Is that correct? What sort of an ETA are we looking at so that I can allocate my team's time accordingly?

Comment 9

[C. Liang [:cyliang]]

I think the last piece of information I was waiting on was whether or not everyone on the list of users I got from Cheng should also be granted sudo (root).

Comment 10

[[:Cww]]

I think just me, rrayborn, aforsstrom, willkg and any DBAs or IT people are the only ones that need root for now. If we find out we need others, we can add them in the future, I assume.

Thanks

Comment 11

[C. Liang [:cyliang]]

The VM advocacy1.webapp.phx1.mozilla.com should be ready for y'all to use.

Access to the VM requires a connection to the new datacenter VPN (https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=30769829) and membership to a specific VPN group.   User accounts and sudo privileges are being managed through Infra's Puppet.  

Please let me know if you have issues connecting to the server or setting things up.

Comment 12

[C. Liang [:cyliang]]

Chang: 

Going through the user list, it looks like Aimee doesn't have SSH public keys on file.  I would create a bug to get a public key from her but she doesn't appear to have a bugzilla account either. =)  Can you or someone in your group work with her to either get her to generate a set of keys or get her into bugzilla (and I can work with her in a separate bug)?

Comment 13

[[:Cww]]

She's actually a new hire who hasn't started yet, let's leave her off for now, I just figured it'd be easy to do now if she had the info in place.

Comment 14

[C. Liang [:cyliang]]

No worries.  We can make it one of the first bugs Aimee gets to file. =)

Comment 15

[C. Liang [:cyliang]]

Closing out this bug, since it looks like folks are actively using the server.

Comment 16

[Matt Grimes [:Matt_G]]

Thanks C! Much appreciated.