1304421 - Parquet2hive fails on non-parquet files

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Frank Bertsch [:frank]]

parquet2hive s3://net-mozaws-prod-us-west-2-pipeline-analysis/mobile/android_events -dv v1
fails with:

Analyzing dataset android_events, v1
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/mnt1/ -Xmx15000M -Xms1000M
Could not read footer: java.lang.RuntimeException: file:/tmp/tmp9XOS_v is not a Parquet file (too small)
Failure to parse dataset, 'NoneType' object has no attribute 'group'

Ideally, p2h should just ignore these files and continue processing.

Comment 1

[Frank Bertsch [:frank]]

Implemented a fix which igores all directories that begin with '_'. While this solves our current problem, it then fails on a later file: mobile/android_events/v1/channel=aurora/submission=20160919_$folder$

While I'm keeping that original change, I'm also going to implement a fix that will skip over non-parquet files. They will still need to be downloaded in order to be verified, but at least it won't break the import.

Comment 2

[Frank Bertsch [:frank]]

We've decided that it's better to have a blacklist of files to exclude. We're going to include all files and directories that end with $folder$, which should solve this problem. 

Parquet2hive WILL break on non-parquet files, in which case we'll need to edit the blacklist again in the future.

Comment 3

[Frank Bertsch [:frank]]

PR here: https://github.com/mozilla/parquet2hive/pull/16