1413204 - No trunk nightlies scheduled today due to decision task failure

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

It was noticed that no m-c nightlies ran today at 3am PT. I tried manually triggering them via the hook, but that also failed.

[task 2017-10-31T14:27:54.774494Z] force-running cron job nightly-desktop
[task 2017-10-31T14:27:54.811088Z] Traceback (most recent call last):
[task 2017-10-31T14:27:54.811139Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/cron/__init__.py", line 70, in run_job
[task 2017-10-31T14:27:54.811201Z]     tasks = JOB_TYPES[job_type](job['job'], params)
[task 2017-10-31T14:27:54.811252Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/cron/decision.py", line 27, in run_decision_task
[task 2017-10-31T14:27:54.811280Z]     arguments=arguments),
[task 2017-10-31T14:27:54.811326Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/cron/decision.py", line 77, in make_decision_task
[task 2017-10-31T14:27:54.811363Z]     rendered = jsone.render(taskcluster_yml, context)
[task 2017-10-31T14:27:54.811411Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/__init__.py", line 19, in render
[task 2017-10-31T14:27:54.811439Z]     rv = renderValue(template, full_context)
[task 2017-10-31T14:27:54.811482Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/render.py", line 279, in renderValue
[task 2017-10-31T14:27:54.811504Z]     return dict(updated())
[task 2017-10-31T14:27:54.811550Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/render.py", line 270, in updated
[task 2017-10-31T14:27:54.811577Z]     v = renderValue(v, context)
[task 2017-10-31T14:27:54.811620Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/render.py", line 291, in renderValue
[task 2017-10-31T14:27:54.811644Z]     return list(updated())
[task 2017-10-31T14:27:54.811690Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/render.py", line 285, in updated
[task 2017-10-31T14:27:54.811714Z]     v = renderValue(e, context)
[task 2017-10-31T14:27:54.811758Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/render.py", line 261, in renderValue
[task 2017-10-31T14:27:54.811789Z]     return operators[matches[0]](template, context)
[task 2017-10-31T14:27:54.811828Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/render.py", line 135, in let
[task 2017-10-31T14:27:54.812023Z]     return renderValue(in_expression, subcontext)
[task 2017-10-31T14:27:54.812176Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/render.py", line 279, in renderValue
[task 2017-10-31T14:27:54.812241Z]     return dict(updated())
[task 2017-10-31T14:27:54.812302Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/render.py", line 270, in updated
[task 2017-10-31T14:27:54.812338Z]     v = renderValue(v, context)
[task 2017-10-31T14:27:54.812410Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/render.py", line 261, in renderValue
[task 2017-10-31T14:27:54.812469Z]     return operators[matches[0]](template, context)
[task 2017-10-31T14:27:54.812527Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/render.py", line 102, in fromNow
[task 2017-10-31T14:27:54.812575Z]     return shared.fromNow(offset, reference)
[task 2017-10-31T14:27:54.812664Z]   File "/home/worker/checkouts/gecko/third_party/python/json-e/jsone/shared.py", line 92, in fromNow
[task 2017-10-31T14:27:54.812717Z]     return stringDate(reference + delta if future else reference - delta)
[task 2017-10-31T14:27:54.812790Z] TypeError: coercing to Unicode: need string or buffer, datetime.timedelta found
[task 2017-10-31T14:27:54.812835Z] cron job nightly-desktop run failed; continuing to next job

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

My manual Nightly request from last European night is missing, no cron jobs for https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=083a9c84fbd09a6ff9bfecabbf773650842fe1c0&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=success&filter-searchStr=cron

Comment 2

[Rok Garbas [:garbas]]

looks like this is a problem with recent update of json-e in Bug 1412136. :dustin would you have any insight what is going wrong?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

So the issue here is that $fromNow is looking at `now` in the context, and that's a string, but it expects a datetime object.  Coincidentally I just fixed a similar issue in the queue - https://github.com/taskcluster/taskcluster-queue/pull/218

In this csae, though, I think jsone needs to take a string in context, since Date objects are not JSON.

Comment 4

[Nick Alexander :nalexander [he/him]]

So Bug 1412136 is almost cosmetic.  Can we back that out until we address this?

Comment 5

[Nick Alexander :nalexander [he/him]]

bstack: ^

Comment 6

[Brian Stack [:bstack]]

I believe that should be fine to back out for the time being if it is more expedient to do so. I'm not too sure of what is best in this case.

Comment 7

[Marcia Knous [:marcia]]

Brian: We are two days away from the 58 merge and we really need to get nightlies back on track. I suggest backing this out as soon as possible, and then manually triggering new builds (RyanVM - can you help here)?

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Bug 1413304 just landed on m-c, which should resolve it. We'll find out for sure in ~2hr when the 3pm nightlies start.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

We have nightlies running on m-c again :)