1271640 - Calculate the exact number of Telemetry records on S3

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mark Reid [:mreid]]

When querying for a relatively large amount of records using different methods, I keep seeing slightly different record counts.

I'd like to use Heka/Hindsight to confirm the actual record counts on S3 (as well as check for any corrupt streams) so that I can validate counts that come from the Scala and Python code.

Ideally, I'd like to count the exact number of records under the following S3 prefixes:

telemetry-2/20160401/telemetry/4/main/Firefox
telemetry-2/20160402/telemetry/4/main/Firefox

That will make it easy to compare against the counts in the "main_summary" derived dataset.

Comment 1

[Mike Trinkala [:trink]]

Attachment: Counts per file for each day

schema.json was the file selection criteria

tsv: column 1 filename, column 2 number of messages in the file
cnts-20160401.tsv
  total files    = 166338
  total messages = 396151215
cnts-20160402.tsv
  total files    = 152984
  total messages = 303040220

Comment 2

[Roberto Agostino Vitillo (:rvitillo)]

Mark checked that the Scala implementation returns the correct number of records while I did the same for the Python one. We can both confirm that the number of records match for the given dates.