1491330 - Hardware report failing on improperly formatted timestamp

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[Frank Bertsch [:frank]]

Stacktrace:

Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1536883313642_0001/container_1536883313642_0001_01_000009/pyspark.zip/pyspark/worker.py", line 229, in main
    process()
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1536883313642_0001/container_1536883313642_0001_01_000009/pyspark.zip/pyspark/worker.py", line 224, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1536883313642_0001/container_1536883313642_0001_01_000009/pyspark.zip/pyspark/serializers.py", line 372, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 845, in func
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1254, in countPartition
  File "/mnt/analyses/python_mozetl/mozetl/hardware_report/summarize_json.py", line 652, in <lambda>
  File "build/bdist.linux-x86_64/egg/mozetl/hardware_report/summarize_json.py", line 244, in get_latest_valid_per_client
    pkt_date, "%Y-%m-%dT%H:%M:%S.%fZ").date()
  File "/mnt/anaconda2/lib/python2.7/_strptime.py", line 332, in _strptime
    (data_string, format))
ValueError: time data '2018-03-07T00:00:00Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/python_mozetl/pull/258