1311991 - Stand up "cram" test harness in mozilla-central

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

Cram [1] is mercurial's run-tests.py framework pulled into a standalone python package. It uses .t style tests that run a command and compares whether the output is expected.

I'd like to stand up a simple test runner based on cram so we have a better option for testing things like mach_commands.py, test harnesses and other cli based applications. Probably following these steps:

1) Vendor in cram to /python
2) Create /testing/cramtest.py which will be a thin wrapper around "cram" that provides stuff like manifestparser/mozlog support, maybe even mozrunner capabilities.
3) Create a taskcluster task that runs only when appropriate files are touched.

The jury is still out on whether this can be "good enough" for something like selftests for mochitest or not.. but I figure if we at least have this option maybe we can figure out a way to use it for that purpose. At the very least it will be useful for simpler mach commands.

[1] https://github.com/brodie/cram

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1311991 - Vendor in cram 0.7 to /third_party/python,

Review commit: https://reviewboard.mozilla.org/r/109650/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/109650/

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1311991 - Add CRAMTEST_MANIFEST moz.build variable,

Review commit: https://reviewboard.mozilla.org/r/109652/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/109652/

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1311991 - Add a basic cramtest mach command,

Review commit: https://reviewboard.mozilla.org/r/109654/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/109654/

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Create example cramtest

Review commit: https://reviewboard.mozilla.org/r/109656/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/109656/

Comment 5

[Andrew Halberstadt [:ahal]]

I started a very basic prototype with the previous 4 commits. The cramtest runner will need to be improved a bit more, and I didn't touch the taskcluster side of things yet. That part will be blocked on bug 1320194.

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1311991 - Add testing/mach_commands.py to flake8 linter,

This was mostly generated with autopep8 v1.3.2. A few left over errors
were fixed by hand.

Review commit: https://reviewboard.mozilla.org/r/170220/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/170220/

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Example cramtest for demonstration purposes

Review commit: https://reviewboard.mozilla.org/r/170222/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/170222/

Comment 8

[Andrew Halberstadt [:ahal]]

Comment on attachment 8833420 [details]
Bug 1311991 - Vendor in cram 0.7 to /third_party/python,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/109650/diff/1-2/

Comment 9

[Andrew Halberstadt [:ahal]]

Comment on attachment 8833421 [details]
Bug 1311991 - Add CRAMTEST_MANIFEST moz.build variable,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/109652/diff/1-2/

Comment 10

[Andrew Halberstadt [:ahal]]

Comment on attachment 8833422 [details]
Bug 1311991 - Add a basic cramtest mach command,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/109654/diff/1-2/

Comment 11

[Andrew Halberstadt [:ahal]]

Comment on attachment 8833422 [details]
Bug 1311991 - Add a basic cramtest mach command,

https://reviewboard.mozilla.org/r/109654/#review175368

This is a pretty terrible runner, I was aiming to get the minimum viable product working and improve it over time. Feel free to r- if you think there are features it needs to support right out of the gate.

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8833420 [details]
Bug 1311991 - Vendor in cram 0.7 to /third_party/python,

https://reviewboard.mozilla.org/r/109650/#review175374

Comment 13

[Gregory Szorc [:gps]]

Comment on attachment 8833421 [details]
Bug 1311991 - Add CRAMTEST_MANIFEST moz.build variable,

https://reviewboard.mozilla.org/r/109652/#review175376

Comment 14

[Gregory Szorc [:gps]]

Comment on attachment 8898853 [details]
Bug 1311991 - Add testing/mach_commands.py to flake8 linter,

https://reviewboard.mozilla.org/r/170220/#review175378

Comment 15

[Gregory Szorc [:gps]]

Comment on attachment 8833422 [details]
Bug 1311991 - Add a basic cramtest mach command,

https://reviewboard.mozilla.org/r/109654/#review175380

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8898854 [details]
Example cramtest for demonstration purposes

https://reviewboard.mozilla.org/r/170222/#review175384

Thank you for finally adding this.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ce8a7ae1772f
Vendor in cram 0.7 to /third_party/python, r=gps
https://hg.mozilla.org/integration/autoland/rev/64deedb9a531
Add CRAMTEST_MANIFEST moz.build variable, r=gps
https://hg.mozilla.org/integration/autoland/rev/eeca48a18e59
Add testing/mach_commands.py to flake8 linter, r=gps
https://hg.mozilla.org/integration/autoland/rev/275840003417
Add a basic cramtest mach command, r=gps

Comment 18

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/ce8a7ae1772f
https://hg.mozilla.org/mozilla-central/rev/64deedb9a531
https://hg.mozilla.org/mozilla-central/rev/eeca48a18e59
https://hg.mozilla.org/mozilla-central/rev/275840003417