789488 - allow talos to run some portion of pageloader tests

Status: RESOLVED DUPLICATE of bug 795172

(Testing :: Talos, defect)

Description

[Jeff Hammel]

Currently, talos has a bunch of pageloader tests that specify the
manifest:

http://hg.mozilla.org/build/talos/file/1c5976f92643/talos/test.py#l154

Currently the only way of changing where the manifest file points to
is to run PerfConfigurator and edit the resulting file.  This is
stnank #1.

Stnank #2 is that there is no easy way of running a subset of the
manifest.  You can't really run a particular location without copying
the manifest, editing it, and then repeating the steps for stnank #1.
We should make this easier.

One thing that would help is going to manifestparser for manifests.
Obviously this is a big change.  We can make this a smaller change by
fixin manifestparser to support e.g. text manifests.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

what type of interface would we have for this?  Why would we need this?

If we could pass in a manifest via the command line that would help.

Comment 2

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #1)
> what type of interface would we have for this?  

That is an open question.  Perhaps `--manifest` for a test manifest and `--page`/`--pages` for specifying pages.  We should be careful, though....currently options like this are global overrides for all tests.  We can change this behaviour but it will involve drastically changing talos and how it is run.

> Why would we need this?

So this is useful if you're debugging a subset of pages.  Right now, since a test suite is the atom, we don't need this.  Once datazilla becomes a reality, however, if you're debugging an individual page....presumedly you don't want to run 100 (e.g.) pages for the one(s) you care about.

> If we could pass in a manifest via the command line that would help.

Yes.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

we have --tpstart and --tpend to give us a start index in the manifest and an end index in the manifest.

Comment 4

[Jeff Hammel]

See also https://bugzilla.mozilla.org/show_bug.cgi?id=795172 ; I'm tempted to close this bug as a dupe of that