Use new Presto parquet reader

NEW
Assigned to

Status

P3
normal
a year ago
5 months ago

People

(Reporter: frank, Assigned: robotblake)

Tracking

Details

(Whiteboard: [DataOps])

(Reporter)

Description

a year ago
This new reader will make schema evolution of ROW types work. See [0].

Update it with the following option: hive.parquet-optimized-reader.enabled=true

[0] https://github.com/fbertsch/schema_evolution_exploration/blob/master/parquet-optimized-reader%3Dtrue/use-column-names%3Dtrue/Schema_Evolution_Exploration.ipynb
Whiteboard: [SvcOps]
(Assignee)

Updated

a year ago
Points: 1 → 2
Priority: P1 → P2
Depends on: 1358232
Component: Metrics: Pipeline → Presto
Product: Cloud Services → Data Platform and Tools
(Assignee)

Comment 1

a year ago
This is on hold as it breaks some queries due to what appears to be a bug in upstream Presto. Currently working on a repro and will drop the link in here once I get that bug filed.

Updated

7 months ago
Whiteboard: [SvcOps] → [DataOps]
(Assignee)

Comment 2

6 months ago
I'm trying to remember if this is still broken (or even a valid bug anymore).
Flags: needinfo?(fbertsch)
(Reporter)

Comment 3

6 months ago
Schema evolution of ROW types is still broken. If you haven't started using the new one then we aren't.
Flags: needinfo?(fbertsch)
(Assignee)

Updated

5 months ago
Priority: P2 → P3
You need to log in before you can comment on or make changes to this bug.