Closed Bug 1423182 Opened 7 years ago Closed 6 years ago

Retriggering build signing tasks fails with jsonschema.exceptions.ValidationError: [] is too short

Categories

(Release Engineering :: Release Automation: Other, defect)

Product:
Release Engineering
Component:
Release Automation: Other
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1424482

People

(Reporter: aryx, Assigned: jlorenzo)

References

Details

(Whiteboard: [stockwell infra]) 

See the Windows build signing exceptions e.g. in https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=fe49764e750b4fe5e133a4075bea613ad5008b6e&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

Build log: https://treeherder.mozilla.org/logviewer.html#?job_id=149831048&repo=autoland

2017-12-05 12:32:54,230 - asyncio - DEBUG - Using selector: EpollSelector
2017-12-05 12:32:54,232 - signingscript.script - INFO - validating task
2017-12-05 12:32:54,232 - signingscript.task - DEBUG - {'properties': {'scopes': {'items': {'type': 'string'}, 'uniqueItems': True, 'type': 'array', 'minItems': 1}, 'payload': {'properties': {'upstreamArtifacts': {'uniqueItems': True, 'items': {'properties': {'paths': {'items': {'type': 'string'}, 'uniqueItems': True, 'type': 'array', 'minItems': 1}, 'taskId': {'type': 'string'}, 'taskType': {'type': 'string'}, 'formats': {'items': {'type': 'string'}, 'uniqueItems': True, 'type': 'array'}}, 'required': ['taskId', 'taskType', 'paths', 'formats'], 'type': 'object'}, 'type': 'array', 'minItems': 1}}, 'required': ['upstreamArtifacts'], 'type': 'object'}, 'dependencies': {'items': {'type': 'string'}, 'uniqueItems': True, 'type': 'array', 'minItems': 1}}, 'title': 'Taskcluster signing task minimal schema', 'required': ['scopes', 'payload'], 'type': 'object'}
Traceback (most recent call last):
  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/client.py", line 52, in validate_json_schema
    jsonschema.validate(data, schema)
  File "/builds/scriptworker/lib/python3.5/site-packages/jsonschema/validators.py", line 541, in validate
    cls(schema, *args, **kwargs).validate(instance)
  File "/builds/scriptworker/lib/python3.5/site-packages/jsonschema/validators.py", line 130, in validate
    raise error
jsonschema.exceptions.ValidationError: [] is too short

Failed validating 'minItems' in schema['properties']['dependencies']:
    {'items': {'type': 'string'},
     'minItems': 1,
     'type': 'array',
     'uniqueItems': True}

On instance['dependencies']:
    []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builds/scriptworker/lib/python3.5/site-packages/signingscript/script.py", line 140, in main
    loop.run_until_complete(async_main(context, conn=conn))
  File "/tools/python35/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/tools/python35/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/tools/python35/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/builds/scriptworker/lib/python3.5/site-packages/signingscript/script.py", line 51, in async_main
    validate_task_schema(context)
  File "/builds/scriptworker/lib/python3.5/site-packages/signingscript/task.py", line 71, in validate_task_schema
    scriptworker.client.validate_json_schema(context.task, task_schema)
  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/client.py", line 56, in validate_json_schema
    exit_code=STATUSES['malformed-payload']
scriptworker.exceptions.ScriptWorkerTaskException: Can't validate task schema!
[] is too short

Failed validating 'minItems' in schema['properties']['dependencies']:
    {'items': {'type': 'string'},
     'minItems': 1,
     'type': 'array',
     'uniqueItems': True}

On instance['dependencies']:
    []
exit code: 3
Component: Hooks → Release Automation
Product: Taskcluster → Release Engineering
QA Contact: catlee
There had been 100 failures in the last week.
This started to occur today on android-4-0-armv7-api16 / opt

Looking at the entire history of the bug, I can see that this started to occur on December 4th 2017 and had failures on windows2012-32, Windows 2012 x64 on debug and opt.

Here is a recent log file and a snippet with the failure:
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=165100300&lineNumber=10

   jsonschema.validate(data, schema)
  File "/builds/scriptworker/lib/python3.5/site-packages/jsonschema/validators.py", line 541, in validate
    cls(schema, *args, **kwargs).validate(instance)
  File "/builds/scriptworker/lib/python3.5/site-packages/jsonschema/validators.py", line 130, in validate
    raise error
jsonschema.exceptions.ValidationError: [] is too short

Failed validating 'minItems' in schema['properties']['payload']['properties']['upstreamArtifacts']['items']['properties']['paths']:
    {'items': {'type': 'string'},
     'minItems': 1,
     'type': 'array',
     'uniqueItems': True}
Flags: needinfo?(catlee)
Whiteboard: [stockwell needswork]
Fallout from bug 1424482. 

mtabara|mergeduty> jlorenzo: I wonder if we should have more of the https://hg.mozilla.org/mozilla-central/rev/d69d6fe4a1b8#l4.55 along all the beetmover-related transforms?
12:53:51 <mtabara|mergeduty> aaaaaah
12:54:04 <mtabara|mergeduty> jlorenzo: we haven't hit this on maple because we ran staging releases with no-single locales
12:54:25 <mtabara|mergeduty> and for the multi build, there's other artifacts that stay in the build, hence BM don't fail
12:54:28 <jlorenzo> ah, right! 
12:54:31 <mtabara|mergeduty> https://tools.taskcluster.net/groups/SFB-maizTbm7glUPt16A6g/tasks/XIPl5d7LTtSUoRccrvZtLw/details
12:54:45 <jlorenzo> I think I see how to fix it quickly 
12:54:45 <mtabara|mergeduty> but for mozilla-central nightlies, we have all the single-locales enabled
12:54:58 <mtabara|mergeduty> ah, we missed this corner case
12:55:00 <jlorenzo> worst case scenario, I'll backout 1424482  
12:55:06 <mtabara|mergeduty> yeah
12:55:10 <mtabara|mergeduty> but I think we can fix it easily
12:55:17 <jlorenzo> I agree. I'm on the fix 
12:55:40 <mtabara|mergeduty> we need to tweak the beetmover transforms for the `if locale in payload` case
12:55:59 <mtabara|mergeduty> happy to review
12:55:59 <mtabara|mergeduty> thanks
Assignee: nobody → jlorenzo
Status: NEW → ASSIGNED
Flags: needinfo?(catlee)
See Also: → 1424482
I agree, comment 2 is fallout of bug 1424482. I don't think comment 0 is still happening. Closing bug.
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → DUPLICATE
Whiteboard: [stockwell needswork] → [stockwell infra]
is it safe to assume that this is really fixed by bug 1424482?
Flags: needinfo?(jlorenzo)
The fact that Orange factor didn't report anything between comment 1 and comment 2, which are 2.5+ months apart, makes me think comment 0 was fixed more than 2 months ago. I looked more closely to what happened at that time. The task definitions didn't list any dependencies, which is wrong. My guess the error came up from the Taskcluster side. Based on the number of times such such mechanism is exercised, I believe it's safe to assume somebody fixed it. Otherwise most Taskcluster users would have noticed the dep mechanism is broken.

If my assumption is incorrect, please reopen this bug :)
Flags: needinfo?(jlorenzo)
For the record, comment 8 is an echo of comment 5: 
* it's a summary of the last 7 days (whereas comment 5 was for just the day before)
* the number of failures didn't increase
* The graph shows a flat line starting March 2nd.
You need to log in before you can comment on or make changes to this bug.