Closed Bug 1015383 Opened 12 years ago Closed 11 years ago

Try out multicorn for hive_fdw

Categories

(Socorro :: Database, task)

Product:
Socorro
Component:
Database
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: selenamarie, Assigned: selenamarie)

References

Details

Set up multicorn on dev and see if we can talk to hive!
With jakem's help, installed python2.7 on socorro1.dev.db. installing multicorn: [root@socorro1.dev.db.phx1 multicorn-1.0.3]# export PYTHON_OVERRIDE=/usr/bin/python2.7 [root@socorro1.dev.db.phx1 multicorn-1.0.3]# make Python version is 2.7 ./preflight-check.sh [root@socorro1.dev.db.phx1 multicorn-1.0.3]# make clean Python version is 2.7 rm -f multicorn.so libmulticorn.a rm -f src/errors.o src/python.o src/query.o src/multicorn.o rm -rf sql/multicorn--1.0.3.sql ./multicorn-1.0.3.zip rm -rf results/ regression.diffs regression.out tmp_check/ log/ [root@socorro1.dev.db.phx1 multicorn-1.0.3]# make Python version is 2.7 ./preflight-check.sh cp sql/multicorn.sql sql/multicorn--1.0.3.sql gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -I/usr/include/python2.7 -I/usr/include/python2.7 -I. -I. -I/usr/pgsql-9.2/include/server -I/usr/pgsql-9.2/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include -c -o src/errors.o src/errors.c gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -I/usr/include/python2.7 -I/usr/include/python2.7 -I. -I. -I/usr/pgsql-9.2/include/server -I/usr/pgsql-9.2/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include -c -o src/python.o src/python.c gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -I/usr/include/python2.7 -I/usr/include/python2.7 -I. -I. -I/usr/pgsql-9.2/include/server -I/usr/pgsql-9.2/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include -c -o src/query.o src/query.c gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -I/usr/include/python2.7 -I/usr/include/python2.7 -I. -I. -I/usr/pgsql-9.2/include/server -I/usr/pgsql-9.2/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include -c -o src/multicorn.o src/multicorn.c gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -shared -o multicorn.so src/errors.o src/python.o src/query.o src/multicorn.o -L/usr/pgsql-9.2/lib -L/usr/lib64 -Wl,--as-needed -lpthread -ldl -lutil -lm -lpython2.7 -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic [root@socorro1.dev.db.phx1 multicorn-1.0.3]# make install Python version is 2.7 ./preflight-check.sh /bin/mkdir -p '/usr/pgsql-9.2/lib' /bin/mkdir -p '/usr/pgsql-9.2/share/extension' /bin/mkdir -p '/usr/pgsql-9.2/share/extension' /bin/mkdir -p '/usr/share/doc/pgsql/extension' cp ./setup.py ./setup--1.0.3.py sed -i -e "s/__VERSION__/1.0.3-dev/g" ./setup--1.0.3.py /usr/bin/python2.7 ./setup--1.0.3.py install running install running bdist_egg running egg_info creating python/multicorn.egg-info writing python/multicorn.egg-info/PKG-INFO writing top-level names to python/multicorn.egg-info/top_level.txt writing dependency_links to python/multicorn.egg-info/dependency_links.txt writing manifest file 'python/multicorn.egg-info/SOURCES.txt' reading manifest file 'python/multicorn.egg-info/SOURCES.txt' writing manifest file 'python/multicorn.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/ldapfdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/xmlfdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/processfdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/gitfdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/csvfdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/sqlalchemyfdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/statefdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/googlefdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/utils.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/imapfdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/rssfdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/compat.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/__init__.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/gcfdw.py -> build/lib.linux-x86_64-2.7/multicorn copying python/multicorn/testfdw.py -> build/lib.linux-x86_64-2.7/multicorn creating build/lib.linux-x86_64-2.7/multicorn/fsfdw copying python/multicorn/fsfdw/test.py -> build/lib.linux-x86_64-2.7/multicorn/fsfdw copying python/multicorn/fsfdw/docutils_meta.py -> build/lib.linux-x86_64-2.7/multicorn/fsfdw copying python/multicorn/fsfdw/structuredfs.py -> build/lib.linux-x86_64-2.7/multicorn/fsfdw copying python/multicorn/fsfdw/__init__.py -> build/lib.linux-x86_64-2.7/multicorn/fsfdw running build_ext building 'multicorn._utils' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/src gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/pgsql-9.2/include -I/usr/pgsql-9.2/include -I/usr/pgsql-9.2/include/server -I/usr/include/python2.7 -c src/utils.c -o build/temp.linux-x86_64-2.7/src/utils.o -shared src/utils.c: In function ‘log_to_postgres’: src/utils.c:45: warning: implicit declaration of function ‘errorCheck’ src/utils.c:55: warning: implicit declaration of function ‘PyUnicode_AsPgString’ src/utils.c:55: warning: cast to pointer from integer of different size src/utils.c:55: warning: cast to pointer from integer of different size src/utils.c:55: warning: cast to pointer from integer of different size src/utils.c:55: warning: passing argument 1 of ‘strlen’ makes pointer from integer without a cast /usr/include/string.h:399: note: expected ‘const char *’ but argument is of type ‘int’ src/utils.c:55: warning: passing argument 2 of ‘memcpy’ makes pointer from integer without a cast /usr/include/bits/string3.h:49: note: expected ‘const void * __restrict__’ but argument is of type ‘int’ src/utils.c:55: warning: passing argument 1 of ‘__strdup’ makes pointer from integer without a cast /usr/include/bits/string2.h:1303: note: expected ‘const char *’ but argument is of type ‘int’ src/utils.c: In function ‘init_utils’: src/utils.c:156: warning: unused variable ‘st’ gcc -pthread -shared build/temp.linux-x86_64-2.7/src/utils.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/multicorn/_utils.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/ldapfdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/xmlfdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/processfdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/gitfdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/csvfdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/sqlalchemyfdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/statefdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/googlefdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/utils.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/imapfdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/_utils.so -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/rssfdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/compat.py -> build/bdist.linux-x86_64/egg/multicorn creating build/bdist.linux-x86_64/egg/multicorn/fsfdw copying build/lib.linux-x86_64-2.7/multicorn/fsfdw/test.py -> build/bdist.linux-x86_64/egg/multicorn/fsfdw copying build/lib.linux-x86_64-2.7/multicorn/fsfdw/docutils_meta.py -> build/bdist.linux-x86_64/egg/multicorn/fsfdw copying build/lib.linux-x86_64-2.7/multicorn/fsfdw/structuredfs.py -> build/bdist.linux-x86_64/egg/multicorn/fsfdw copying build/lib.linux-x86_64-2.7/multicorn/fsfdw/__init__.py -> build/bdist.linux-x86_64/egg/multicorn/fsfdw copying build/lib.linux-x86_64-2.7/multicorn/__init__.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/gcfdw.py -> build/bdist.linux-x86_64/egg/multicorn copying build/lib.linux-x86_64-2.7/multicorn/testfdw.py -> build/bdist.linux-x86_64/egg/multicorn byte-compiling build/bdist.linux-x86_64/egg/multicorn/ldapfdw.py to ldapfdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/xmlfdw.py to xmlfdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/processfdw.py to processfdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/gitfdw.py to gitfdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/csvfdw.py to csvfdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/sqlalchemyfdw.py to sqlalchemyfdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/statefdw.py to statefdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/googlefdw.py to googlefdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/utils.py to utils.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/imapfdw.py to imapfdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/rssfdw.py to rssfdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/compat.py to compat.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/fsfdw/test.py to test.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/fsfdw/docutils_meta.py to docutils_meta.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/fsfdw/structuredfs.py to structuredfs.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/fsfdw/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/gcfdw.py to gcfdw.pyc byte-compiling build/bdist.linux-x86_64/egg/multicorn/testfdw.py to testfdw.pyc creating stub loader for multicorn/_utils.so byte-compiling build/bdist.linux-x86_64/egg/multicorn/_utils.py to _utils.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying python/multicorn.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying python/multicorn.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying python/multicorn.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying python/multicorn.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... multicorn.fsfdw.test: module references __file__ creating dist creating 'dist/multicorn-1.0.3-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing multicorn-1.0.3-py2.7-linux-x86_64.egg creating /usr/lib/python2.7/site-packages/multicorn-1.0.3-py2.7-linux-x86_64.egg Extracting multicorn-1.0.3-py2.7-linux-x86_64.egg to /usr/lib/python2.7/site-packages Adding multicorn 1.0.3 to easy-install.pth file Installed /usr/lib/python2.7/site-packages/multicorn-1.0.3-py2.7-linux-x86_64.egg Processing dependencies for multicorn==1.0.3 Finished processing dependencies for multicorn==1.0.3 rm ./setup--1.0.3.py /bin/sh /usr/pgsql-9.2/lib/pgxs/src/makefiles/../../config/install-sh -c -m 755 multicorn.so '/usr/pgsql-9.2/lib/multicorn.so' /bin/sh /usr/pgsql-9.2/lib/pgxs/src/makefiles/../../config/install-sh -c -m 644 ./multicorn.control '/usr/pgsql-9.2/share/extension/' /bin/sh /usr/pgsql-9.2/lib/pgxs/src/makefiles/../../config/install-sh -c -m 644 ./sql/multicorn--1.0.3.sql '/usr/pgsql-9.2/share/extension/' /bin/sh /usr/pgsql-9.2/lib/pgxs/src/makefiles/../../config/install-sh -c -m 644 ./doc/multicorn.md '/usr/share/doc/pgsql/extension/' [root@socorro1.dev.db.phx1 multicorn-1.0.3]#
Blocks: 970406
More detailed install logs for hive dependencies: [root@socorro1.dev.db.phx1 ~]# mkdir hive-install [root@socorro1.dev.db.phx1 ~]# cd hive-install/ [root@socorro1.dev.db.phx1 hive-install]# mv ../multicorn-1.0.3* . [root@socorro1.dev.db.phx1 hive-install]# ls -tlr total 132 -rw-r--r-- 1 root root 129806 Apr 24 07:35 multicorn-1.0.3.zip drwxr-xr-x 12 root root 4096 May 23 13:02 multicorn-1.0.3 [root@socorro1.dev.db.phx1 hive-install]# wget https://github.com/youngwookim/hive-thrift-py/archive/master.zip --2014-05-23 13:21:54-- https://github.com/youngwookim/hive-thrift-py/archive/master.zip Resolving github.com... 192.30.252.128 Connecting to github.com|192.30.252.128|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://codeload.github.com/youngwookim/hive-thrift-py/zip/master [following] --2014-05-23 13:21:54-- https://codeload.github.com/youngwookim/hive-thrift-py/zip/master Resolving codeload.github.com... 192.30.252.147 Connecting to codeload.github.com|192.30.252.147|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 92015 (90K) [application/zip] Saving to: “master” 100%[====================================================================================================================>] 92,015 229K/s in 0.4s 2014-05-23 13:21:55 (229 KB/s) - “master” saved [92015/92015] [root@socorro1.dev.db.phx1 hive-install]# ls -tlr total 224 -rw-r--r-- 1 root root 129806 Apr 24 07:35 multicorn-1.0.3.zip drwxr-xr-x 12 root root 4096 May 23 13:02 multicorn-1.0.3 -rw-r--r-- 1 root root 92015 May 23 13:21 master [root@socorro1.dev.db.phx1 hive-install]# file master master: Zip archive data, at least v1.0 to extract [root@socorro1.dev.db.phx1 hive-install]# mv master hive-thrift-py.zip [root@socorro1.dev.db.phx1 hive-install]# unzip hive-thrift-py.zip Archive: hive-thrift-py.zip 3d86467ca00c4429ed8a639281f71135bc4a7206 creating: hive-thrift-py-master/ inflating: hive-thrift-py-master/.gitignore inflating: hive-thrift-py-master/README.md inflating: hive-thrift-py-master/setup.py creating: hive-thrift-py-master/src/ creating: hive-thrift-py-master/src/fb303/ inflating: hive-thrift-py-master/src/fb303/FacebookBase.py inflating: hive-thrift-py-master/src/fb303/FacebookService-remote inflating: hive-thrift-py-master/src/fb303/FacebookService.py inflating: hive-thrift-py-master/src/fb303/__init__.py inflating: hive-thrift-py-master/src/fb303/constants.py inflating: hive-thrift-py-master/src/fb303/ttypes.py creating: hive-thrift-py-master/src/fb303_scripts/ inflating: hive-thrift-py-master/src/fb303_scripts/__init__.py inflating: hive-thrift-py-master/src/fb303_scripts/fb303_simple_mgmt.py creating: hive-thrift-py-master/src/hive_metastore/ inflating: hive-thrift-py-master/src/hive_metastore/ThriftHiveMetastore-remote inflating: hive-thrift-py-master/src/hive_metastore/ThriftHiveMetastore.py inflating: hive-thrift-py-master/src/hive_metastore/__init__.py inflating: hive-thrift-py-master/src/hive_metastore/constants.py inflating: hive-thrift-py-master/src/hive_metastore/ttypes.py creating: hive-thrift-py-master/src/hive_serde/ extracting: hive-thrift-py-master/src/hive_serde/__init__.py inflating: hive-thrift-py-master/src/hive_serde/constants.py inflating: hive-thrift-py-master/src/hive_serde/ttypes.py creating: hive-thrift-py-master/src/hive_service/ inflating: hive-thrift-py-master/src/hive_service/ThriftHive-remote inflating: hive-thrift-py-master/src/hive_service/ThriftHive.py inflating: hive-thrift-py-master/src/hive_service/__init__.py inflating: hive-thrift-py-master/src/hive_service/constants.py inflating: hive-thrift-py-master/src/hive_service/ttypes.py creating: hive-thrift-py-master/src/queryplan/ extracting: hive-thrift-py-master/src/queryplan/__init__.py inflating: hive-thrift-py-master/src/queryplan/constants.py inflating: hive-thrift-py-master/src/queryplan/ttypes.py creating: hive-thrift-py-master/src/thrift/ inflating: hive-thrift-py-master/src/thrift/TSCons.py inflating: hive-thrift-py-master/src/thrift/Thrift.py inflating: hive-thrift-py-master/src/thrift/__init__.py creating: hive-thrift-py-master/src/thrift/protocol/ inflating: hive-thrift-py-master/src/thrift/protocol/TBinaryProtocol.py inflating: hive-thrift-py-master/src/thrift/protocol/TProtocol.py inflating: hive-thrift-py-master/src/thrift/protocol/__init__.py inflating: hive-thrift-py-master/src/thrift/protocol/fastbinary.c creating: hive-thrift-py-master/src/thrift/reflection/ inflating: hive-thrift-py-master/src/thrift/reflection/__init__.py creating: hive-thrift-py-master/src/thrift/reflection/limited/ inflating: hive-thrift-py-master/src/thrift/reflection/limited/__init__.py inflating: hive-thrift-py-master/src/thrift/reflection/limited/constants.py inflating: hive-thrift-py-master/src/thrift/reflection/limited/ttypes.py creating: hive-thrift-py-master/src/thrift/server/ inflating: hive-thrift-py-master/src/thrift/server/THttpServer.py inflating: hive-thrift-py-master/src/thrift/server/TNonblockingServer.py inflating: hive-thrift-py-master/src/thrift/server/TServer.py inflating: hive-thrift-py-master/src/thrift/server/__init__.py creating: hive-thrift-py-master/src/thrift/transport/ inflating: hive-thrift-py-master/src/thrift/transport/THttpClient.py inflating: hive-thrift-py-master/src/thrift/transport/TSocket.py inflating: hive-thrift-py-master/src/thrift/transport/TTransport.py inflating: hive-thrift-py-master/src/thrift/transport/TTwisted.py inflating: hive-thrift-py-master/src/thrift/transport/__init__.py [root@socorro1.dev.db.phx1 hive-install]# cd hive-thrift-py-master/ [root@socorro1.dev.db.phx1 hive-thrift-py-master]# ls -tlr total 12 drwxr-xr-x 9 root root 4096 Jun 5 2012 src -rw-r--r-- 1 root root 474 Jun 5 2012 setup.py -rw-r--r-- 1 root root 72 Jun 5 2012 README.md [root@socorro1.dev.db.phx1 hive-thrift-py-master]# cd ../ [root@socorro1.dev.db.phx1 hive-install]# virtualenv hive New python executable in hive/bin/python Installing setuptools............done. Installing pip...............done. [root@socorro1.dev.db.phx1 hive-install]# source hive/bin/activate (hive)[root@socorro1.dev.db.phx1 hive-install]# python --version Python 2.7.3 (hive)[root@socorro1.dev.db.phx1 hive-install]# cd hive-thrift-py-master/ (hive)[root@socorro1.dev.db.phx1 hive-thrift-py-master]# python setup.py usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: no commands supplied (hive)[root@socorro1.dev.db.phx1 hive-thrift-py-master]# python setup.py install running install running bdist_egg running egg_info creating src/hive_thrift_py.egg-info writing src/hive_thrift_py.egg-info/PKG-INFO writing top-level names to src/hive_thrift_py.egg-info/top_level.txt writing dependency_links to src/hive_thrift_py.egg-info/dependency_links.txt writing manifest file 'src/hive_thrift_py.egg-info/SOURCES.txt' reading manifest file 'src/hive_thrift_py.egg-info/SOURCES.txt' writing manifest file 'src/hive_thrift_py.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib creating build/lib/queryplan copying src/queryplan/__init__.py -> build/lib/queryplan copying src/queryplan/ttypes.py -> build/lib/queryplan copying src/queryplan/constants.py -> build/lib/queryplan creating build/lib/hive_service copying src/hive_service/ThriftHive.py -> build/lib/hive_service copying src/hive_service/__init__.py -> build/lib/hive_service copying src/hive_service/ttypes.py -> build/lib/hive_service copying src/hive_service/constants.py -> build/lib/hive_service creating build/lib/hive_metastore copying src/hive_metastore/ThriftHiveMetastore.py -> build/lib/hive_metastore copying src/hive_metastore/__init__.py -> build/lib/hive_metastore copying src/hive_metastore/ttypes.py -> build/lib/hive_metastore copying src/hive_metastore/constants.py -> build/lib/hive_metastore creating build/lib/fb303 copying src/fb303/FacebookService.py -> build/lib/fb303 copying src/fb303/FacebookBase.py -> build/lib/fb303 copying src/fb303/__init__.py -> build/lib/fb303 copying src/fb303/ttypes.py -> build/lib/fb303 copying src/fb303/constants.py -> build/lib/fb303 creating build/lib/fb303_scripts copying src/fb303_scripts/fb303_simple_mgmt.py -> build/lib/fb303_scripts copying src/fb303_scripts/__init__.py -> build/lib/fb303_scripts creating build/lib/thrift copying src/thrift/Thrift.py -> build/lib/thrift copying src/thrift/TSCons.py -> build/lib/thrift copying src/thrift/__init__.py -> build/lib/thrift creating build/lib/hive_serde copying src/hive_serde/__init__.py -> build/lib/hive_serde copying src/hive_serde/ttypes.py -> build/lib/hive_serde copying src/hive_serde/constants.py -> build/lib/hive_serde creating build/lib/thrift/protocol copying src/thrift/protocol/TProtocol.py -> build/lib/thrift/protocol copying src/thrift/protocol/__init__.py -> build/lib/thrift/protocol copying src/thrift/protocol/TBinaryProtocol.py -> build/lib/thrift/protocol creating build/lib/thrift/server copying src/thrift/server/TServer.py -> build/lib/thrift/server copying src/thrift/server/TNonblockingServer.py -> build/lib/thrift/server copying src/thrift/server/THttpServer.py -> build/lib/thrift/server copying src/thrift/server/__init__.py -> build/lib/thrift/server creating build/lib/thrift/reflection copying src/thrift/reflection/__init__.py -> build/lib/thrift/reflection creating build/lib/thrift/transport copying src/thrift/transport/TSocket.py -> build/lib/thrift/transport copying src/thrift/transport/TTwisted.py -> build/lib/thrift/transport copying src/thrift/transport/THttpClient.py -> build/lib/thrift/transport copying src/thrift/transport/__init__.py -> build/lib/thrift/transport copying src/thrift/transport/TTransport.py -> build/lib/thrift/transport creating build/lib/thrift/reflection/limited copying src/thrift/reflection/limited/__init__.py -> build/lib/thrift/reflection/limited copying src/thrift/reflection/limited/ttypes.py -> build/lib/thrift/reflection/limited copying src/thrift/reflection/limited/constants.py -> build/lib/thrift/reflection/limited creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/queryplan copying build/lib/queryplan/__init__.py -> build/bdist.linux-x86_64/egg/queryplan copying build/lib/queryplan/ttypes.py -> build/bdist.linux-x86_64/egg/queryplan copying build/lib/queryplan/constants.py -> build/bdist.linux-x86_64/egg/queryplan creating build/bdist.linux-x86_64/egg/hive_service copying build/lib/hive_service/ThriftHive.py -> build/bdist.linux-x86_64/egg/hive_service copying build/lib/hive_service/__init__.py -> build/bdist.linux-x86_64/egg/hive_service copying build/lib/hive_service/ttypes.py -> build/bdist.linux-x86_64/egg/hive_service copying build/lib/hive_service/constants.py -> build/bdist.linux-x86_64/egg/hive_service creating build/bdist.linux-x86_64/egg/hive_metastore copying build/lib/hive_metastore/ThriftHiveMetastore.py -> build/bdist.linux-x86_64/egg/hive_metastore copying build/lib/hive_metastore/__init__.py -> build/bdist.linux-x86_64/egg/hive_metastore copying build/lib/hive_metastore/ttypes.py -> build/bdist.linux-x86_64/egg/hive_metastore copying build/lib/hive_metastore/constants.py -> build/bdist.linux-x86_64/egg/hive_metastore creating build/bdist.linux-x86_64/egg/fb303 copying build/lib/fb303/FacebookService.py -> build/bdist.linux-x86_64/egg/fb303 copying build/lib/fb303/FacebookBase.py -> build/bdist.linux-x86_64/egg/fb303 copying build/lib/fb303/__init__.py -> build/bdist.linux-x86_64/egg/fb303 copying build/lib/fb303/ttypes.py -> build/bdist.linux-x86_64/egg/fb303 copying build/lib/fb303/constants.py -> build/bdist.linux-x86_64/egg/fb303 creating build/bdist.linux-x86_64/egg/fb303_scripts copying build/lib/fb303_scripts/fb303_simple_mgmt.py -> build/bdist.linux-x86_64/egg/fb303_scripts copying build/lib/fb303_scripts/__init__.py -> build/bdist.linux-x86_64/egg/fb303_scripts creating build/bdist.linux-x86_64/egg/thrift creating build/bdist.linux-x86_64/egg/thrift/protocol copying build/lib/thrift/protocol/TProtocol.py -> build/bdist.linux-x86_64/egg/thrift/protocol copying build/lib/thrift/protocol/__init__.py -> build/bdist.linux-x86_64/egg/thrift/protocol copying build/lib/thrift/protocol/TBinaryProtocol.py -> build/bdist.linux-x86_64/egg/thrift/protocol copying build/lib/thrift/Thrift.py -> build/bdist.linux-x86_64/egg/thrift creating build/bdist.linux-x86_64/egg/thrift/server copying build/lib/thrift/server/TServer.py -> build/bdist.linux-x86_64/egg/thrift/server copying build/lib/thrift/server/TNonblockingServer.py -> build/bdist.linux-x86_64/egg/thrift/server copying build/lib/thrift/server/THttpServer.py -> build/bdist.linux-x86_64/egg/thrift/server copying build/lib/thrift/server/__init__.py -> build/bdist.linux-x86_64/egg/thrift/server creating build/bdist.linux-x86_64/egg/thrift/reflection creating build/bdist.linux-x86_64/egg/thrift/reflection/limited copying build/lib/thrift/reflection/limited/__init__.py -> build/bdist.linux-x86_64/egg/thrift/reflection/limited copying build/lib/thrift/reflection/limited/ttypes.py -> build/bdist.linux-x86_64/egg/thrift/reflection/limited copying build/lib/thrift/reflection/limited/constants.py -> build/bdist.linux-x86_64/egg/thrift/reflection/limited copying build/lib/thrift/reflection/__init__.py -> build/bdist.linux-x86_64/egg/thrift/reflection copying build/lib/thrift/TSCons.py -> build/bdist.linux-x86_64/egg/thrift creating build/bdist.linux-x86_64/egg/thrift/transport copying build/lib/thrift/transport/TSocket.py -> build/bdist.linux-x86_64/egg/thrift/transport copying build/lib/thrift/transport/TTwisted.py -> build/bdist.linux-x86_64/egg/thrift/transport copying build/lib/thrift/transport/THttpClient.py -> build/bdist.linux-x86_64/egg/thrift/transport copying build/lib/thrift/transport/__init__.py -> build/bdist.linux-x86_64/egg/thrift/transport copying build/lib/thrift/transport/TTransport.py -> build/bdist.linux-x86_64/egg/thrift/transport copying build/lib/thrift/__init__.py -> build/bdist.linux-x86_64/egg/thrift creating build/bdist.linux-x86_64/egg/hive_serde copying build/lib/hive_serde/__init__.py -> build/bdist.linux-x86_64/egg/hive_serde copying build/lib/hive_serde/ttypes.py -> build/bdist.linux-x86_64/egg/hive_serde copying build/lib/hive_serde/constants.py -> build/bdist.linux-x86_64/egg/hive_serde byte-compiling build/bdist.linux-x86_64/egg/queryplan/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/queryplan/ttypes.py to ttypes.pyc byte-compiling build/bdist.linux-x86_64/egg/queryplan/constants.py to constants.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_service/ThriftHive.py to ThriftHive.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_service/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_service/ttypes.py to ttypes.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_service/constants.py to constants.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_metastore/ThriftHiveMetastore.py to ThriftHiveMetastore.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_metastore/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_metastore/ttypes.py to ttypes.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_metastore/constants.py to constants.pyc byte-compiling build/bdist.linux-x86_64/egg/fb303/FacebookService.py to FacebookService.pyc byte-compiling build/bdist.linux-x86_64/egg/fb303/FacebookBase.py to FacebookBase.pyc byte-compiling build/bdist.linux-x86_64/egg/fb303/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/fb303/ttypes.py to ttypes.pyc byte-compiling build/bdist.linux-x86_64/egg/fb303/constants.py to constants.pyc byte-compiling build/bdist.linux-x86_64/egg/fb303_scripts/fb303_simple_mgmt.py to fb303_simple_mgmt.pyc byte-compiling build/bdist.linux-x86_64/egg/fb303_scripts/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/protocol/TProtocol.py to TProtocol.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/protocol/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/protocol/TBinaryProtocol.py to TBinaryProtocol.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/Thrift.py to Thrift.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/server/TServer.py to TServer.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/server/TNonblockingServer.py to TNonblockingServer.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/server/THttpServer.py to THttpServer.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/server/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/reflection/limited/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/reflection/limited/ttypes.py to ttypes.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/reflection/limited/constants.py to constants.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/reflection/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/TSCons.py to TSCons.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/transport/TSocket.py to TSocket.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/transport/TTwisted.py to TTwisted.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/transport/THttpClient.py to THttpClient.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/transport/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/transport/TTransport.py to TTransport.pyc byte-compiling build/bdist.linux-x86_64/egg/thrift/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_serde/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_serde/ttypes.py to ttypes.pyc byte-compiling build/bdist.linux-x86_64/egg/hive_serde/constants.py to constants.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying src/hive_thrift_py.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying src/hive_thrift_py.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying src/hive_thrift_py.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying src/hive_thrift_py.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying src/hive_thrift_py.egg-info/zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO creating dist creating 'dist/hive_thrift_py-0.0.1-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing hive_thrift_py-0.0.1-py2.7.egg Copying hive_thrift_py-0.0.1-py2.7.egg to /usr/lib/python2.7/site-packages Adding hive-thrift-py 0.0.1 to easy-install.pth file Installed /usr/lib/python2.7/site-packages/hive_thrift_py-0.0.1-py2.7.egg Processing dependencies for hive-thrift-py==0.0.1 Finished processing dependencies for hive-thrift-py==0.0.1 (hive)[root@socorro1.dev.db.phx1 hive-thrift-py-master]# cd ../ (hive)[root@socorro1.dev.db.phx1 hive-install]# wget https://github.com/youngwookim/hive-fdw-for-postgresql/archive/master.zip --2014-05-23 13:23:22-- https://github.com/youngwookim/hive-fdw-for-postgresql/archive/master.zip Resolving github.com... 192.30.252.129 Connecting to github.com|192.30.252.129|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://codeload.github.com/youngwookim/hive-fdw-for-postgresql/zip/master [following] --2014-05-23 13:23:22-- https://codeload.github.com/youngwookim/hive-fdw-for-postgresql/zip/master Resolving codeload.github.com... 192.30.252.146 Connecting to codeload.github.com|192.30.252.146|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/zip] Saving to: “master” [ <=> ] 2,760 --.-K/s in 0s 2014-05-23 13:23:22 (39.2 MB/s) - “master” saved [2760] (hive)[root@socorro1.dev.db.phx1 hive-install]# mv master hive-fdw-for-postgresql.zip (hive)[root@socorro1.dev.db.phx1 hive-install]# unzip hive unzip: cannot find or open hive, hive.zip or hive.ZIP. (hive)[root@socorro1.dev.db.phx1 hive-install]# unzip hive-fdw-for-postgresql.zip Archive: hive-fdw-for-postgresql.zip 23813061332a015d586b20d8897eee80399bcf61 creating: hive-fdw-for-postgresql-master/ extracting: hive-fdw-for-postgresql-master/.gitignore extracting: hive-fdw-for-postgresql-master/README inflating: hive-fdw-for-postgresql-master/README.md inflating: hive-fdw-for-postgresql-master/hivefdw.py inflating: hive-fdw-for-postgresql-master/setup.py (hive)[root@socorro1.dev.db.phx1 hive-install]# cd hive-fdw-for-postgresql-master/ (hive)[root@socorro1.dev.db.phx1 hive-fdw-for-postgresql-master]# ls hivefdw.py README README.md setup.py (hive)[root@socorro1.dev.db.phx1 hive-fdw-for-postgresql-master]# python setup.py sdist running sdist running check warning: sdist: manifest template 'MANIFEST.in' does not exist (using default file list) writing manifest file 'MANIFEST' creating HiveFDW-0.0.2 making hard links in HiveFDW-0.0.2... hard linking README -> HiveFDW-0.0.2 hard linking hivefdw.py -> HiveFDW-0.0.2 hard linking setup.py -> HiveFDW-0.0.2 creating dist Creating tar archive removing 'HiveFDW-0.0.2' (and everything under it) (hive)[root@socorro1.dev.db.phx1 hive-fdw-for-postgresql-master]# pythonn setup.py install -bash: pythonn: command not found (hive)[root@socorro1.dev.db.phx1 hive-fdw-for-postgresql-master]# python setup.py install running install running build running build_py creating build creating build/lib copying hivefdw.py -> build/lib running install_lib copying build/lib/hivefdw.py -> /usr/lib/python2.7/site-packages byte-compiling /usr/lib/python2.7/site-packages/hivefdw.py to hivefdw.pyc running install_egg_info Writing /usr/lib/python2.7/site-packages/HiveFDW-0.0.2-py2.7.egg-info Window width too large, truncated ve-fdw-for-postgresql-master]# breakpad:5432# create extension multicorn; CREATE EXTENSION Time: 227.100 ms breakpad:5432# CREATE SERVER multicorn_hive FOREIGN DATA WRAPPER multicorn breakpad-# OPTIONS ( breakpad(# wrapper 'hivefdw.HiveForeignDataWrapper' breakpad(# ); ERROR: Error in python: ImportError DETAIL: No module named hive_service Time: 8.801 ms breakpad:5432# \q [postgres@socorro1 ~]$ psql breakpad psql (9.2.8) Type "help" for help. breakpad:5432# CREATE SERVER multicorn_hive FOREIGN DATA WRAPPER multicorn OPTIONS ( wrapper 'hivefdw.HiveForeignDataWrapper' ); CREATE SERVER Time: 54.709 ms breakpad:5432#
Assignee: nobody → sdeckelmann
Blocks: 1011648
Stalled waiting for credentials.
I can reach the systems now! (hive)[sdeckelmann@socorro1.dev.db.phx1 hive-test]$ python test-script.py [['default', '']] ['bing_logs'] ['boo'] ['fhr_meta_20140208'] ['v2_processed_logs'] ['v2_raw_logs'] ['v2_raw_logs_m'] ['vertica_amo_download_requests_by_day'] ['vertica_aus_requests_by_day'] ['vertica_blocklist_requests_by_day'] ['vertica_blocklist_requests_by_day_th'] ['vertica_download_requests_by_day'] ['vertica_download_requests_by_day_th'] ['vertica_opt_f_weblog_sync']
Schema is: (hive)[sdeckelmann@socorro1.dev.db.phx1 hive-test]$ python test-script.py [['default', '']] ['ip_address', 'string', ''] ['server_host', 'string', ''] ['empty_string', 'string', ''] ['request_time', 'string', ''] ['request_type', 'string', ''] ['request_url', 'string', ''] ['http_status_code', 'string', ''] ['byte_size', 'string', ''] ['empty_string_1', 'string', ''] ['user_agent', 'string', ''] ['cookie', 'string', ''] ['custom_field_1', 'string', ''] ['custom_field_2', 'string', ''] ['ds', 'string', ''] ['domain', 'string', '']
I think multicorn + this fdw isn't going to work. First couple attempts result in a SELECT that never returns :/ In the interest of time, closing this PR and working on a pure python solution instead.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.