1335704 - DSMO-RS download_stats table has null values

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Romain Testard [:RT]]

The download_stats table has a significant number of rows where build_id, version, download_ip or old_version can be set to null.

'build_id' has 2,745,784 entries where it's set to null
'version' has 2,745,772 entries where it's set to null
'download_ip' has 2,748,231 entries where it's set to null
'old_version' has 2,623,340 entries where it's set to null

When talking to Matt who manages the stub installer, the installer initializes everything to some value, even if it's just zero so it looks like a potential data pipeline issue. Could someone look into this?

Comment 1

[Blake Imsland [:robotblake]]

I'll be looking into this this week along with work on the other DSMO-RS bugs.

Comment 2

[Blake Imsland [:robotblake]]

So I thought this was fixed, but it still appears to be occurring sporadically so I'll need to dig more.

Comment 3

[Wesley Dawson [:whd]]

I've identified the root cause of this, and addressed it here: https://github.com/whd/dsmo_load/commit/c4ba3ba412bc29203ae2e9ba80da8878a9745a9e. Data from 20170408 forward will not have this issue. The grammar assumed that there weren't any empty entries, and would stop populating fields for entries after an empty one. This lead to a large set of NULL fields for any entry that had a single NULL value. For example, if download_size was empty then all later fields such as download_ip, version etc. would all be NULL.

If the installer should never return an empty string for anything then there is a bug in the installer.

Marking this as fixed. I filed bug #1354716 for doing the backfill to remove these values from historical data.