Marionette should support bisection ability

NEW
Unassigned

Status

Testing
Marionette
P5
normal
4 years ago
2 days ago

People

(Reporter: vaibhav1994, Unassigned)

Tracking

({ateam-marionette-runner})

Trunk
ateam-marionette-runner
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
I have done some work for adding bisection for tests for mochitests in bug 1014125 . This ability can also be useful for marionette for faster debugging.
I would like to know:
* does marionette use the same manifestparser as mochitest
* is there support for running marionette tests in a loop (inside of runtests.py)
* where is the log parsing done for marionette
* do we rely on returncodes for tbpl status as we do for mochitests
(In reply to Joel Maher (:jmaher) from comment #1)
> * does marionette use the same manifestparser as mochitest
Yes.

> * is there support for running marionette tests in a loop (inside of
> runtests.py)
I don't think so, but I'll defer to :mdas

> * where is the log parsing done for marionette
:chmanchester recently landed structured logging for marionette, so I guess it doesn't? cc'ing him to answer this one.

> * do we rely on returncodes for tbpl status as we do for mochitests
Not sure, I think so but again will defer to :mdas
(In reply to Joel Maher (:jmaher) from comment #1)
> I would like to know:
> * does marionette use the same manifestparser as mochitest
> * is there support for running marionette tests in a loop (inside of
> runtests.py)

Yes, there's a --repeat option.

> * where is the log parsing done for marionette

In mozharness/TBPL, if I understand the question correctly.

> * do we rely on returncodes for tbpl status as we do for mochitests

Yes, along with counts of passed/failed tests.
Regarding log parsing, is there a way to get access to the log from inside the runtests.py?
(In reply to Joel Maher (:jmaher) from comment #4)
> Regarding log parsing, is there a way to get access to the log from inside
> the runtests.py?

Chris already landed the structured logging patch for marionette.. so please don't parse any logs :). I think what you want to do is add a new output handler. Needinfo'ing him to help point you in the right direction.
Flags: needinfo?(cmanchester)
(In reply to Joel Maher (:jmaher) from comment #4)
> Regarding log parsing, is there a way to get access to the log from inside
> the runtests.py?

From skimming the mochitests patch, it looks like you need the test name and status (for failures) as the tests are running. There isn't a formatted log in the runner, but there are various places you could hook into to get this. I might suggest the add_test_result function[1], which runs at the end of each test and has access to the status of the result, although I don't know the exact requirements, and someone closer to the implementation might have a better suggestion.

[1]http://hg.mozilla.org/mozilla-central/annotate/f6e46d1fc903/testing/marionette/client/marionette/runner/base.py#l108
Flags: needinfo?(cmanchester)
Keywords: ateam-marionette-runner
[mass update] Setting Harness bugs to all P3
Priority: -- → P3
Why would we have to implement bisecting capabilities into each harness, when mozregression could be used in combination with the `--command` argument?
the work done for mochitest was to allow us to bisect failures inside a test job, not on revisions- often we get many failures for leaks/crashes that are hard to pinpoint and automating that is useful.  To be fair we don't use the built in bisection inside of mochitest much anymore.
Priority: P3 → P5
You need to log in before you can comment on or make changes to this bug.