1381801 - Task payload timestamps have inconsistent precision with top level task timestamps

Status: RESOLVED FIXED

(Taskcluster :: Services, enhancement)

Description

[Pete Moore [:pmoore][:pete]]

When we generate tasks in the gecko decision task, timestamps are given microsecond precision.

The top level task timestamps (deadline, created, expires) are downgraded to millisecond provision in the Queue, however, the Queue does not interpret the timestamps included in the task payload.

This leads to unwanted behaviour. For example:

A worker will consider that if a task has an artifact which expires later than the task, the task is invalid, and will raise a malformed-payload exception.

In the case that the artifact expiry is set to equal the task expiry, the queue will truncated the task expiry to millisecond precision, but not the artifact expiry, and therefore result in a task definition where the artifact expires some number of microseconds after the task expiry. This then leads to tasks being resolved as malformed-payload, due to the different treatment of timestamps in the payload by the taskcluster systems.

To work around this limitation, I propose we reduce generated timestamps to millisecond precision, to match the Queue's precision.

Comment 1

[Pete Moore [:pmoore][:pete]]

Try push:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e9dc567b73e3423a6002c83d59099a95466eb51b

Comment 2

[Pete Moore [:pmoore][:pete]]

Explanation of implementation choice
====================================

According to https://docs.python.org/2/library/datetime.html:

 datetime.isoformat([sep])

    Return a string representing the date and time in ISO 8601 format, YYYY-MM-DDTHH:MM:SS.mmmmmm or, if microsecond is 0, YYYY-MM-DDTHH:MM:SS



Therefore, trivially appending [:-3] to the timestamp expression to strip the last three microseconds will fail 1 time in a million fail (when microseconds == 0).

However, appending [:23] should do the trick, as this won't truncate in the case the expression is 23 characters or less already.

Comment 3

[Pete Moore [:pmoore][:pete]]

Attachment: bug1381801_gecko_v1.patch

My editor did some funky tab/space replacement in my edit which caused a flake8 failure in my task run, but I've fixed that manually and created this patch (which should be tab free).

Thanks!

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7554ee085d2c
use millisecond precision in taskcluster task definition timestamps;r=dustin

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/taskcluster/generic-worker

https://github.com/taskcluster/generic-worker/commit/87c3b37aca6c16c33426b72aabcb9640056b0f8c
Bug 1381801 - be less strict with timestamp comparisons in generic-worker to account for timestamp rounding in upstream systems

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/7554ee085d2c

Comment 7

[Pete Moore [:pmoore][:pete]]

Released in https://github.com/taskcluster/generic-worker/releases/tag/v10.1.6