1876219 - Flag records with attachments that contain unknown fields in `UnparsableRecords`

Status: RESOLVED WONTFIX

(Application Services :: Suggest, task, P1)

Description

[Lina Butler [:lina]]

The UnparsableRecords scheme that Tif added in https://github.com/mozilla/application-services/pull/5786 flags suggestion types that our component doesn’t understand at all. Ben's PR (https://github.com/mozilla/application-services/pull/6096) expands that to also flag records with attachments that fail to parse.

I think there’s a third case that we aren’t handling yet, though: when we successfully parse an attachment, but don't understand all its fields. For example, let’s say we ship our Rust component in Firefox 120, and then add a shortDescription field to Pocket suggestions sometime during Firefox 121. New 121 clients will know what to do with that shortDescription, but 120 clients will silently drop shortDescription when they ingest Pocket suggestions.

We can work around this by having Firefox 121 bump the schema version, throw the old database away, and re-ingest all suggestions. This is what we've done for all our migrations so far.

But I think it would be nice to move to a future where we don't need to throw everything away each time we add a new suggestion type, or a new field to an existing suggestion type.

I think we could expand UnparsableRecords to support this case. Alternatively, we could store any unknown fields into a separate JSON column, like Sync does—but my gut feeling is that this would be overkill. UnparsableRecords is already an optimization over throwing away and re-ingesting all suggestions; I'm not sure that going further buys us much.

Comment 1

[Ben Dean-Kawamura [:bdk]]

I'm going to close this one because we decided to stop tracking unparsable records. The current system where we throw away all suggestion records and re-download them when the schema changes is working well enough and the unparsable records code adds more complexity than we want.