Failure output of JS shell tests is lost when running in automation

ASSIGNED
Assigned to

Status

()

enhancement
ASSIGNED
Last year
10 months ago

People

(Reporter: sfink, Assigned: sfink, NeedInfo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

We don't log it anywhere.
This is a little clunky, but seems to do the job.
Attachment #8983911 - Flags: review?(nicolas.b.pierron)
Comment on attachment 8983911 [details] [diff] [review]
Upload JS shell test failure output

Review of attachment 8983911 [details] [diff] [review]:
-----------------------------------------------------------------

::: js/src/devtools/automation/autospider.py
@@ +475,5 @@
>  
> +# Store all failure output to an uploaded log file. This could be done in each
> +# individual variant, but it's verbose and we want it across the board.
> +if AUTOMATION:
> +    env['JSTESTS_EXTRA_ARGS'] = "--show-output --failed-only --output-file={MOZ_UPLOAD_DIR}/jstests.fail.txt {old}".format(

nit: Add --show-cmd.
Attachment #8983911 - Flags: review?(nicolas.b.pierron) → review+
(In reply to Nicolas B. Pierron [:nbp] {backlog: 36} from comment #2)
> nit: Add --show-cmd.

Thanks! You're right.
Pushed by sfink@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4ede0210acdb
Upload JS shell test failure output, r=nbp
Backed out changeset 4ede0210acdb (bug 1467249) for linting failures in /builds/worker/checkouts/gecko/js/src/devtools/automation/autospider.py on a CLOSED TREE

Problematic push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=4ede0210acdbfeeaf7ec13e445618f94540eccf1&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified
Failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified&selectedJob=182618927
Backout: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=0ffe269f5e96c70120a935cf322397e3baf6103b&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified
Log: https://treeherder.mozilla.org/logviewer.html#?job_id=182618927&repo=mozilla-inbound&lineNumber=255

[task 2018-06-09T00:20:35.821Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2018-06-09T00:21:17.104Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/js/src/devtools/automation/autospider.py:492:100 | line too long (134 > 99 characters) (E501)
[task 2018-06-09T00:21:17.104Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/js/src/devtools/automation/autospider.py:494:12 | unexpected spaces around keyword / parameter equals (E251)
[task 2018-06-09T00:21:17.104Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/js/src/devtools/automation/autospider.py:494:14 | unexpected spaces around keyword / parameter equals (E251)
[task 2018-06-09T00:21:17.104Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/js/src/devtools/automation/autospider.py:495:100 | line too long (123 > 99 characters) (E501)
[task 2018-06-09T00:21:17.104Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/js/src/devtools/automation/autospider.py:497:12 | unexpected spaces around keyword / parameter equals (E251)
[task 2018-06-09T00:21:17.104Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/js/src/devtools/automation/autospider.py:497:14 | unexpected spaces around keyword / parameter equals (E251)
[taskcluster 2018-06-09 00:21:17.548Z] === Task Finished ===
[taskcluster 2018-06-09 00:21:17.548Z] Unsuccessful task run with exit code: 1 completed in 298.403 seconds
Flags: needinfo?(sphink)
Pushed by sfink@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/383b08391cb8
Upload JS shell test failure output, r=nbp
Pushed by sfink@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9fb5a7ee3f35
followup - syntax error on a CLOSED TREE
Backout by sfink@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fddd866cc32c
Backed out changesets 9fb5a7ee3f35 and 383b08391cb8
Pushed by sfink@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1fc7ea54cf4c
Upload JS shell test failure output, r=nbp
Flags: needinfo?(sphink)
Backed out for jstests check failures.

backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/def346f39b1d0427cc62676dfd470e95d6ed015f

push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=1fc7ea54cf4cf6ff25bc009c6062f1b40b8aadfd

failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=182867515&repo=mozilla-inbound&lineNumber=92593

mozmake -C js/src check-jstests
mozmake[1]: Entering directory 'Z:/task_1528755942/src/obj-spider/js/src'
Z:/task_1528755942/src/obj-spider/_virtualenvs/init/Scripts/python.exe -u z:/task_1528755942/src/js/src/tests/jstests.py \
	--no-progress --format=automation --timeout 300 \
	--show-output --show-cmd --failed-only --output-file=./public/build/jstests.fail.txt --jitflags=all \
	../../dist/bin/js.exe
Failed to open output file: [Errno 2] No such file or directory: './public/build/jstests.fail.txt'
mozmake[1]: *** [Makefile:73: check-jstests] Error 1
mozmake[1]: Leaving directory 'Z:/task_1528755942/src/obj-spider/js/src'
Makefile:331: recipe for target 'check-jstests' failed
mozmake: *** [check-jstests] Error 2
+ BUILD_STATUS=2
+ mkdir -p ./public/build
+ cp -rL ./src/obj-spider/dist/bin/js ./src/obj-spider/dist/bin/jsapi-tests ./src/obj-spider/dist/bin/js-gdb.py ./public/build
+ gzip -c ./clang/bin/llvm-symbolizer
gzip: ./clang/bin/llvm-symbolizer: No such file or directory
+ true
+ exit 2
[taskcluster 2018-06-11T23:26:59.715Z]    Exit Code: 2
[taskcluster 2018-06-11T23:26:59.715Z]    User Time: 0s
[taskcluster 2018-06-11T23:26:59.715Z]  Kernel Time: 15.625ms
[taskcluster 2018-06-11T23:26:59.715Z]    Wall Time: 33m59.5879985s
[taskcluster 2018-06-11T23:26:59.715Z]       Result: FAILED
[taskcluster 2018-06-11T23:26:59.715Z] === Task Finished ===
Flags: needinfo?(sphink)
Ignore this review request; it doesn't work. I'm actually testing bzexport. Sorry for the spam.
Attachment #8985407 - Flags: review?(nicolas.b.pierron)
Attachment #8985407 - Flags: review?(nicolas.b.pierron)
You need to log in before you can comment on or make changes to this bug.