1304439 - Create a parquet lua module

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Comment 2

[Mike Trinkala [:trink]]

The high level tasks for the initial version of the parquet module follow: 

- define writer API
- implement schema parser (data output by the parquet-dump-schema utility)
- implement schema object
    - record creation
    - column creation
    - finalize
- implement a loader to iterate the schema parse table and create the corresponding schema object 
- implement writer property builder
    - support for the following properties
        -  global
            - enable_dictionary (bool)
            - dictionary_pagesize_limit (size_t)
            - write_batch_size (size_t)
            - data_pagesize (size_t)
            - version ("1.0", "2.0")
            - created_by (string)
            - encoding ("plain", "rle")
            - compression ("uncompressed", "snappy", "gzip", "lzo", "brotli")
            - enable_statistics (bool)
        - column_properties
            - col_name1
                - enable_dictionary (bool)
                - encoding ("plain", "rle")
                - compression ("uncompressed", "snappy", "gzip", "lzo", "brotli")
                - enable_statistics (bool)
            - col_nameN
- implement dremel column striping
- implement writer append_row (column output from each message)
- implement writer close (outputs the row group and closes the file)
    - Note: this design will only allow one row group per file
- testing/documentation

Comment 3

[Mike Trinkala [:trink]]

https://github.com/mozilla-services/lua_sandbox_extensions/pull/48

Comment 4

[Mike Trinkala [:trink]]

Tracking a segfault
https://issues.apache.org/jira/browse/PARQUET-780?filter=-2