Closed Bug 544912 Opened 15 years ago Closed 15 years ago

Stage socorro HBase collector mod_python deployment

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ozten, Assigned: aravind)

References

Details

Please update the collector on stage so that we start writing crashes to HBase. 1) svn up scripts and socorro directories on stage 2) grab two new config parameters from scripts/config/collectorconfig.py hbaseHost = cm.Option() hbaseHost.default = 'thrift-socorro-hadoop' hbasePort = cm.Option() hbasePort.default = 9090 3) restart apache 4) Test modpython-collector to make sure it's happy 4.1) Go to http://crash-reports.stage.mozilla.com/submit 4.2) Make sure it says "Method Not Allowed" and not python ImportError or other such errors
Blocks: 538206
I forgot one step... 1.1) Install the Apache Thrift Python library, which is a new dependency for Socorro. Info from Thrift.egg-info: Metadata-Version: 1.0 Name: Thrift Version: 0.1 Summary: Thrift Python Libraries Home-page: http://incubator.apache.org/thrift/ Author: ['Thrift Developers'] Author-email: ['thrift-dev@incubator.apache.org'] License: Apache License 2.0 Description: UNKNOWN Platform: UNKNOWN
Should we do this during maintenance window?
Just the opposite, while not critical, it would be better not to have socorro stage pouring crashes into hbase during this maint window
Assignee: server-ops → aravind
Is the hbase server ready and accepting crashes?
Yes, but as I said, I'd like to wait until those three machines are back online before flipping this switch
HBase/HDFS look good after the maintenance, for what it's worth.
Yes. the maint window is over and the cluster is ready. We can flip the switch on this whenever you guys are ready.
Okay, installing non-standard libraries is not easy - do we really need this apache thrift library deployed across the entire generic cluster when this goes to production. Is there anyway for you to accomplish the same using std python libraries?
Can we double check that the thing mentioned in Comment #1 is needed? That is the library that is used to build new Thrift client interfaces. Unless I'm mistaken, the code that I built and supplied in bug 538206 should be all that the collectors need.
If you need this installed it's pretty easy to do in puppet. Let me know if you need help.
(In reply to comment #9) I'm interested in Aravind's opinion, here was the reasoning: I didn't check in the Thrift library, per Python coding standards from the webdev team. I did check in the generated thrift client, which I've named hbasethrift.py, because I didn't think it's available as a library.
(In reply to comment #8) Do we use virtualenvs in production, or is that a dev only thing? With a virtual environment then it would have to be deployed across Socorro boxes, but shouldn't affect all python deployments. If we manage the dependencies in a fine grained way, then it only needs to be deployed to collector boxes.
We are using a virtualenv for Zamboni, but if you just need one library installed I'd rather not create a virtualenv (because they are kind of a pita to move around).
Does the library need to be installed per comment 9? Are we using python 2.6 or 2.4 in production?
We have been using python 2.4 in prod and unless there is a strong reason to change that, I'd prefer we use the python thats shipped with rhel5 (on the collectors). @oremj: I thought Zamboni was the next gen addons? This collector lives on the generic cluster.. so unless we install that everywhere.
(In reply to comment #13) I don't want to introduce virtualenv to the project, just checking. (In reply to comment #14) Comment 9 is a zip file that contains the Thrift library plus Application specific code. We are using Python 2.4.
(In reply to comment #8) Okay I'll include it in the socorro project because: * Thrift is an Apache Incubator project and I found it's packaging was slightly b0rked. * We can unblock metrics Benefits: Freezes code version so we'll be sure dev and prod are in sync Weaknesses: We should be consistent about 3rd party libraries We should be able to use standard package deployment technologies that are part of the python community
I will update this bug when trunk is ready.
I have completed this work. Please deploy but instead of 1.1 in comment #1 do 1.1) Add thirdparty to the PYTHONPATH of the mod_python collector. Tested in development and HBase writes looked good. (Sent 46019ac4-f011-4980-9d6a-500a52100209) Saw some other random errors in an old code path, will investigate, but it shouldn't block deployment.
We currently have simplejson at the same level as the collector.py script. We don't modify pythonpath, nor do we create any kind of extra links etc for this to work. I am okay with modifying pythonpath for and create a thirdpary directory and move simplejson and the thrift library into it. Just wanted to point this out. Will deploy it once I have a final decision from you folks.
Do what is in comment #19 and when Lars gets back we can explore other methods of dealing with third party libraries.
Deployed thrift, but I get this error now. [Wed Feb 10 12:27:43 2010] [error] [client 10.2.81.4] PythonHandler collector: ImportError: No module named hbase [Wed Feb 10 12:27:43 2010] [error] [client 10.2.81.4] PythonHandler collector: Traceback (most recent call last): [Wed Feb 10 12:27:43 2010] [error] [client 10.2.81.4] PythonHandler collector: File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n log=debug) [Wed Feb 10 12:27:43 2010] [error] [client 10.2.81.4] PythonHandler collector: File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n module = imp.load_module(mname, f, p, d) [Wed Feb 10 12:27:43 2010] [error] [client 10.2.81.4] PythonHandler collector: File "/data/www/crash-reports.mozilla.com/collector.py", line 8, in ?\n import socorro.collector.initializer [Wed Feb 10 12:27:43 2010] [error] [client 10.2.81.4] PythonHandler collector: File "/data/www/crash-reports.mozilla.com/socorro/collector/initializer.py", line 11, in ?\n import hbaseClient [Wed Feb 10 12:27:43 2010] [error] [client 10.2.81.4] PythonHandler collector: File "/data/www/crash-reports.mozilla.com/socorro/collector/hbaseClient.py", line 7, in ?\n from hbase import ttypes [Wed Feb 10 12:27:43 2010] [error] [client 10.2.81.4] PythonHandler collector: ImportError: No module named hbase
is thirdparty/ in your pythonpath?
What should stage point to - for hbaseHost and hbasePort?
hbaseHost = cm.Option() hbaseHost.doc = 'Hostname for hbase hadoop cluster. May be a VIP or load balancer' hbaseHost.default = 'thrift-socorro-hadoop' hbasePort = cm.Option() hbasePort.doc = 'hbase port number' hbasePort.default = 9090
I am now seeing this error. 2010-02-10 17:03:54,453 INFO - Time to Create hbase conn 0.00124287605286 2010-02-10 17:03:54,492 INFO - mod-python subinterpreter name: crash-reports.mozilla.com 2010-02-10 17:03:54,493 ERROR - MainThread Caught Error: <type 'exceptions.AttributeError'> 2010-02-10 17:03:54,493 ERROR - 'str' object has no attribute 'read' 2010-02-10 17:03:54,494 INFO - trace back follows: File "/data/www/crash-reports.mozilla.com/application/collector.py", line 86, in handler collectObject.storeDump(dumpData, dumpFileHandle) File "/data/www/crash-reports.mozilla.com/application/socorro/collector/collect.py", line 189, in storeDump data = dumpInputStream.read(4096)
I think this should fix it: svn up socorro/collector/modpython-collector.py
It looks like things are working now, I do see crash ids being returned etc.. I still see this error in the collector.log file. Not sure if it matters. 2010-02-10 18:22:47,473 INFO - about to create ooid {'submitted_timestamp': '2010-02-10T18:22:47.464882', 'Version': '3.6', 'FramePoisonBase': '00000000f0de0000', 'Vendor': 'Mozilla', 'InstallTime': '1265057306', 'EMCheckCompatibility': 'true', 'Add-ons': 'jqs@sun.com:1.0,{64161300-e22b-11db-8314-0800200c9a66}:0.9.5,{972ce4c6-7e08-4474-a285-3208198ce6fd}:3.6,info@djzig.com:1.1.0,{5A170DD3-63CA-4c58-93B7-DE9FF536C2FF}:1.8.57', 'BuildID': '20100115144158', 'SecondsSinceLastCrash': '2919', 'FramePoisonSize': '65536', 'ProductName': 'Firefox', 'URL': 'http://nasza-klasa.pl/#profile/8990202/gallery/album/2/56', 'Theme': 'walnut', 'Throttleable': '1', 'timestamp': 1265854967.4651091, 'CrashTime': '1265722434', 'ReleaseChannel': 'release', 'StartupTime': '1265721954'} 2010-02-10 18:22:47,473 ERROR - MainThread Caught Error: <type 'exceptions.NameError'> 2010-02-10 18:22:47,473 ERROR - global name 'dumpData' is not defined 2010-02-10 18:22:47,474 INFO - trace back follows: File "/data/www/crash-reports.mozilla.com/application/collector.py", line 94, in handler persistentStorage["hbaseConnection"].create_ooid(uuid, str(jsonDataDictionary), dumpData) 2010-02-10 18:22:47,474 INFO - 0.0365610122681 Finished Handler 2010-02-10 18:22:47,611 INFO - 0.00148105621338 Wrote to disk
(In reply to comment #28) Sorry I b0rked this and didn't see that exception when I tested this. svn up socorro/collector/modpython-collector.py
All done.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Awesome, thanks Aravind!
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.