Closed Bug 1311991 Opened 3 years ago Closed 2 years ago

Stand up "cram" test harness in mozilla-central

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(firefox57 fixed)

RESOLVED FIXED
mozilla57
Tracking Status
firefox57 --- fixed

People

(Reporter: ahal, Assigned: ahal)

References

Details

Attachments

(4 files, 2 obsolete files)

Cram [1] is mercurial's run-tests.py framework pulled into a standalone python package. It uses .t style tests that run a command and compares whether the output is expected.

I'd like to stand up a simple test runner based on cram so we have a better option for testing things like mach_commands.py, test harnesses and other cli based applications. Probably following these steps:

1) Vendor in cram to /python
2) Create /testing/cramtest.py which will be a thin wrapper around "cram" that provides stuff like manifestparser/mozlog support, maybe even mozrunner capabilities.
3) Create a taskcluster task that runs only when appropriate files are touched.

The jury is still out on whether this can be "good enough" for something like selftests for mochitest or not.. but I figure if we at least have this option maybe we can figure out a way to use it for that purpose. At the very least it will be useful for simpler mach commands.

[1] https://github.com/brodie/cram
I started a very basic prototype with the previous 4 commits. The cramtest runner will need to be improved a bit more, and I didn't touch the taskcluster side of things yet. That part will be blocked on bug 1320194.
Depends on: 1320194
Blocks: 1391675
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
Attachment #8833423 - Attachment is obsolete: true
Comment on attachment 8833422 [details]
Bug 1311991 - Add a basic cramtest mach command,

https://reviewboard.mozilla.org/r/109654/#review175368

This is a pretty terrible runner, I was aiming to get the minimum viable product working and improve it over time. Feel free to r- if you think there are features it needs to support right out of the gate.
Comment on attachment 8833420 [details]
Bug 1311991 - Vendor in cram 0.7 to /third_party/python,

https://reviewboard.mozilla.org/r/109650/#review175374
Attachment #8833420 - Flags: review?(gps) → review+
Comment on attachment 8833421 [details]
Bug 1311991 - Add CRAMTEST_MANIFEST moz.build variable,

https://reviewboard.mozilla.org/r/109652/#review175376
Attachment #8833421 - Flags: review?(gps) → review+
Comment on attachment 8898853 [details]
Bug 1311991 - Add testing/mach_commands.py to flake8 linter,

https://reviewboard.mozilla.org/r/170220/#review175378
Attachment #8898853 - Flags: review?(gps) → review+
Comment on attachment 8833422 [details]
Bug 1311991 - Add a basic cramtest mach command,

https://reviewboard.mozilla.org/r/109654/#review175380
Attachment #8833422 - Flags: review?(gps) → review+
Comment on attachment 8898854 [details]
Example cramtest for demonstration purposes

https://reviewboard.mozilla.org/r/170222/#review175384

Thank you for finally adding this.
Attachment #8898854 - Attachment is obsolete: true
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ce8a7ae1772f
Vendor in cram 0.7 to /third_party/python, r=gps
https://hg.mozilla.org/integration/autoland/rev/64deedb9a531
Add CRAMTEST_MANIFEST moz.build variable, r=gps
https://hg.mozilla.org/integration/autoland/rev/eeca48a18e59
Add testing/mach_commands.py to flake8 linter, r=gps
https://hg.mozilla.org/integration/autoland/rev/275840003417
Add a basic cramtest mach command, r=gps
You need to log in before you can comment on or make changes to this bug.