1365737 - Generic Longitudinal should aggregate nested type elements as arrays

Status: RESOLVED WONTFIX

(Data Platform and Tools Graveyard :: Datasets: Longitudinal, enhancement, P4)

Description

[Frank Bertsch [:frank]]

Currently, the Generic Longitudinal aggregates Struct Types by returning an array of Struct Types. The desired output would actually be a single struct, where each element of that struct was an array (unless that element is also a struct, in which case it should recursively create a single struct).

Optionally, Maps could be treated in a similar fashion. Instead of Array[Map[Any, Any]], the schema should contain a Map[Any, Array[Any]] with the values of all keys merged as an array. 

This is a more difficult scenario, however - if the key is not present in one of the pings, do we add a value of NULL, or simply not add a value at all? The former results in NULLs where keys were empty, the latter in confusion between a key being present with NULL value and no key at all. (I'm leaning towards not doing this.)