1458256 - Investigate pioneer d2p dataset size oddities

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, enhancement, P3)

Description

[bugzilla]

After running into constant OOM issues in the online news log entry deduplication job, breaking the job into phases where the first phase explodes out the logs and writes to hdfs (instead of trying to that all at once) dramatically reduced the size of that data in memory (previously, ~60gb persisted after exploding in memory -- reading the data back after writing to hdfs, on the other hand, dropped the size in memory to ~15GB.)

This seems like a very odd result that's not consistent with my mental model of how spark caches datasets, and the implications seem important enough to investigate further. My initial thought is that this might be related to snappy compression when writing that intermediate to disk.