1382673 - Parquet Writer should write NULL values for MAP types

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Frank Bertsch [:frank]]

Currently, the parquet writer doesn't write a key/value in a MAP if the value is NULL. Parquet schema allows for optional values, in which case the value should be written as NULL.

Comment 1

[Allen Short [:ashort]]

This is a consequence of representing Parquet NULL as Lua's "nil". Lua tables cannot have nil as a value for a key, it's equivalent to the key being absent.

Some other representation must be found for nulls; cjson uses a null-pointer lightuserdata object and we could do that here too.

Comment 2

[Rob Miller [:rmiller]]

This is nearly done... there has been a round of implementation / code-review back-and-forth that you can see in the pull request (https://github.com/mozilla-services/lua_sandbox_extensions/pull/154) if you're interested in the gritty details. Currently we (where "we" is :ashort) *think* the code is complete, and are waiting for :trink to review the latest changes.

Comment 3

[Mike Trinkala [:trink]]

Reviewed and commented still r-

Comment 4

[Mike Trinkala [:trink]]

Fix in https://github.com/mozilla-services/lua_sandbox_extensions/commit/ffadf3b401e01e55461d2a93363ff21b76baea04