Currently, the parquet writer doesn't write a key/value in a MAP if the value is NULL. Parquet schema allows for optional values, in which case the value should be written as NULL.
Assignee: trink → ashort
Status: NEW → ASSIGNED
Points: --- → 2
This is a consequence of representing Parquet NULL as Lua's "nil". Lua tables cannot have nil as a value for a key, it's equivalent to the key being absent. Some other representation must be found for nulls; cjson uses a null-pointer lightuserdata object and we could do that here too.
This is nearly done... there has been a round of implementation / code-review back-and-forth that you can see in the pull request (https://github.com/mozilla-services/lua_sandbox_extensions/pull/154) if you're interested in the gritty details. Currently we (where "we" is :ashort) *think* the code is complete, and are waiting for :trink to review the latest changes.
Reviewed and commented still r-
Fix in https://github.com/mozilla-services/lua_sandbox_extensions/commit/ffadf3b401e01e55461d2a93363ff21b76baea04
Status: ASSIGNED → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.