1257615 - Hive metastore can't import Parquet tables with binary column

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Roberto Agostino Vitillo (:rvitillo)]

Even though Hive 1.0 supports binary columns the following statements fails:

create external table KPI(activityDate string, normalizedChannel string, country string, version string, e10sEnabled boolean, e10sCohort string, hll binary) stored as parquet location 's3://telemetry-test-bucket/KPI/v2016030120160313';


FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.UnsupportedOperationException: Unknown field type: binary

Comment 1

[Roberto Agostino Vitillo (:rvitillo)]

Apparently Hive 1.0 doesn't support the binary data type for Parquet, but 1.1 does: https://issues.apache.org/jira/browse/HIVE-7073.

Wesley, could you upgrade our Hive metastore? That might also fix Bug 1255457.

Comment 2

[Wesley Dawson [:whd]]

I've got hive updated on the current "prod" presto instance. With 1.1 I ran into https://issues.apache.org/jira/browse/HIVE-10831 but 1.2 seems to be working so far without issue. The example create from comment 1 for instance succeeded.

It's hacked together right now, I'll file a PR to "operationalize" it assuming it continues to work as intended.

Comment 3

[Thomas Huelbert]

not happening this sprint, back to  p2 until :whd comes up for air after ops related activities.

Comment 4

[Wesley Dawson [:whd]]

https://github.com/vitillo/emr-bootstrap-presto/pull/2