881899 - Deploy a private SymbolicationServer for obfuscated Flash symbols

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Vladan Djeric (:vladan)]

We would like to provide Mozilla engineers with the ability to profile the Flash plugin.

For this, I would like to deploy the "SymbolicationServer" Python script on a Linux server that is only reachable from the Mozilla network, and I would like this server to have access to the Breakpad symbol mount (dm-symbolpush01:/mnt/netapp/breakpad).

The SymbolicationServer script is already deployed on symbolapi.mozilla.org (production server) and breakpad-symbolapi1.dmz.phx1.mozilla.com (testing server), but the production server is accessible publicly (the Flash symbols are confidential) and the testing server cannot be reached directly without going through jump host. 

Deployment instructions (see bug 849708 for an earlier example):

1. Clone the git repo at https://github.com/vdjeric/Snappy-Symbolication-Server/

2. Set "hostname" and "portNumber" in sample.ini. 
Note: You can remove the carriage return characters from the file if you wish.

3. Add the following line to the bottom of the sample.ini file (the "SymbolPaths" section):

Flash = /mnt/netapp/breakpad/symbols_adobe

4. Run the server as a low privilege user, I use:

nohup python symbolicationWebService.py sample.ini &> symbolServer.log

5. Test with:

curl -d '{"symbolSources":["Flash", "Windows"], "stacks":[[[0,10625146],[1,65802]]],"memoryMap":[["NPSWF32.pdb","1E41352BA4254AC2BD7CDDAA4E56C7161"],["wntdll.pdb","D74F79EB1F8D4A45ABCD2F476CCABACC2"]],"version":3}' 127.0.0.1:8000

You should see:

[["_getextendedkeycode (in NPSWF32.pdb)", "KiUserCallbackDispatcher (in wntdll.pdb)"]]

Comment 1

[Jake Maul [:jakem]]

Can we do something simpler here, like some kind of authentication layer blocking access to the adobe symbols? We can easily do an Apache basic auth thing, backed by LDAP and using SSL.

Comment 2

[Benoit Girard (:BenWa)]

Sure, can you deploy the above with LDAP authentication?

Comment 3

[Jake Maul [:jakem]]

So... bear with me on the next couple questions. I don't know how, when, or what calls this app, so I don't really know how those calling things might respond to various scenarios. :)

Would it be feasible to have this new instance of the app respond to queries with a path name in them?

http://symbolapi.mozilla.org/ <-- current, unchanged
http://symbolapi.mozilla.org/adobe/ <-- new ?

This would make it almost trivial to spin this up... we'd install Apache (if it's not already) and set up a simple VirtualHost container... if /adobe/, then proxy to the adobe python socket, otherwise proxy to the main one. In fact, we probably wouldn't even need a new copy/clone of the app... we could use the same one, with a different sample.ini config file.

If this all makes sense to you and sounds okay, then LDAP+SSL is no problem. We would want to force SSL on it, but the app doesn't need to change for this, we can take care of it at the load balancer level so that the app still gets plaintext just like it does now. It might be simplest to have it just redirect all queries to SSL... would there be any problems caused by injecting a simple redirect before things reach the app? In other words, will the thing that is calling this service have any concerns about following redirects?



One more question... do you want this new instance to host *everything*, or just Flash/symbols_adobe ? If the latter, then "/private/" might make better sense as a prefix then "/adobe/".

Comment 4

[Vladan Djeric (:vladan)]

> Would it be feasible to have this new instance of the app respond to queries
> with a path name in them?

I think this would work. The script is dumb, it only handles one type of request so it doesn't even care about the URI the clients use in their request.

> In fact, we probably wouldn't even need a new copy/clone of the app... we could
> use the same one, with a different sample.ini config file.

If I'm understanding correctly, you want to run 2 instances of the same app but with different config files? That would be fine.


> It might be simplest to have it
> just redirect all queries to SSL... would there be any problems caused by
> injecting a simple redirect before things reach the app? In other words,
> will the thing that is calling this service have any concerns about
> following redirects?

The client is a Firefox extension that communicates with the server using XHR. XHR follows redirects so we should be alright.

> One more question... do you want this new instance to host *everything*, or
> just Flash/symbols_adobe ? If the latter, then "/private/" might make better
> sense as a prefix then "/adobe/".

I think it's better if the new instance hosts everything.

Comment 5

[Benoit Girard (:BenWa)]

Sounds good to me.

Comment 6

[Vladan Djeric (:vladan)]

What's the status of this request?

Comment 8

[Chris Turra [:cturra]]

i just gave this bug a quick overview and should be able to grab this. the way i see it right is we need guidance for creating the new Snappy-Symbolication-Server .ini file for the adobe configurations. once that is in place, i should be able to get a second supervisord process running. finally, we will need to get apache on the node and configure it per the comments made by :jakem. i have confirm that apache is /not/ currently on this server.

can someone on this bug ping me on irc to assist with the new .ini configuration?

Comment 9

[Chris Turra [:cturra]]

:vladan was kind enough to help me sort out the necessary configurations. we now have both the public (old) and private (adobe) assets running in separate supervisord processes using different ports. 

the final step here is to sort out how we deliver this content with names ;) i will report back when that is complete.

Comment 10

[Chris Turra [:cturra]]

the goal with this change is we will be able to use a private address for the "adobe" requests. i have just submitted an ACL request bug that once complete will allow private resource requests through: symbolapi.private.phx1.mozilla.com:8001

Comment 11

[Chris Turra [:cturra]]

looks like i forgot to close this bug off. should all be resolved at this point. 

$ curl -d '{"symbolSources":["Flash", "Windows"], "stacks":[[[0,10625146],[1,65802]]],"memoryMap":[["NPSWF32.pdb","1E41352BA4254AC2BD7CDDAA4E56C7161"],["wntdll.pdb","D74F79EB1F8D4A45ABCD2F476CCABACC2"]],"version":3}' symbolapi.private.phx1.mozilla.com:8001
[["_getextendedkeycode (in NPSWF32.pdb)", "KiUserCallbackDispatcher (in wntdll.pdb)"]]