When we generate tasks in the gecko decision task, timestamps are given microsecond precision. The top level task timestamps (deadline, created, expires) are downgraded to millisecond provision in the Queue, however, the Queue does not interpret the timestamps included in the task payload. This leads to unwanted behaviour. For example: A worker will consider that if a task has an artifact which expires later than the task, the task is invalid, and will raise a malformed-payload exception. In the case that the artifact expiry is set to equal the task expiry, the queue will truncated the task expiry to millisecond precision, but not the artifact expiry, and therefore result in a task definition where the artifact expires some number of microseconds after the task expiry. This then leads to tasks being resolved as malformed-payload, due to the different treatment of timestamps in the payload by the taskcluster systems. To work around this limitation, I propose we reduce generated timestamps to millisecond precision, to match the Queue's precision.
Explanation of implementation choice ==================================== According to https://docs.python.org/2/library/datetime.html: datetime.isoformat([sep]) Return a string representing the date and time in ISO 8601 format, YYYY-MM-DDTHH:MM:SS.mmmmmm or, if microsecond is 0, YYYY-MM-DDTHH:MM:SS Therefore, trivially appending [:-3] to the timestamp expression to strip the last three microseconds will fail 1 time in a million fail (when microseconds == 0). However, appending [:23] should do the trick, as this won't truncate in the case the expression is 23 characters or less already.
My editor did some funky tab/space replacement in my edit which caused a flake8 failure in my task run, but I've fixed that manually and created this patch (which should be tab free). Thanks!
Assignee: nobody → pmoore
Status: NEW → ASSIGNED
Attachment #8887456 - Flags: review?(dustin)
Attachment #8887456 - Flags: review?(dustin) → review+
Pushed by email@example.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/7554ee085d2c use millisecond precision in taskcluster task definition timestamps;r=dustin
Commit pushed to master at https://github.com/taskcluster/generic-worker https://github.com/taskcluster/generic-worker/commit/87c3b37aca6c16c33426b72aabcb9640056b0f8c Bug 1381801 - be less strict with timestamp comparisons in generic-worker to account for timestamp rounding in upstream systems
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Component: Integration → Services
Product: Taskcluster → Taskcluster
You need to log in before you can comment on or make changes to this bug.