584753 - runtests.py: do all environment inspection checking before initiating any benchmarks

Status: RESOLVED WONTFIX

(Tamarin Graveyard :: Tools, defect)

Description

[Chris Peyer]

From Felix:
Another suggestion: do all environment inspection checking before initiating
any benchmarks.

I just ran into this, where the runtests script eagerly started running all the
compiled asmicro benchmarks, but then upon finishing that, it died with a
message that "ERROR: shell .abc does not exist, ..."

I understand that we may not want to unconditionally require that all runs set
SHELLABC or pass --shellabc argument, since it is not necessary in all cases
(namely if all the object code has been compiled, right?)  But it should be
easy and quick to first do a prepass checking if all of the necessary object
code is in place, and if any are missing, *then* check if the environment is
properly configured to do a compile.  (You could even compile a dummy file to
really make sure the whole pipeline is clean...)

Comment 1

[Chris Peyer]

One issue with this is we sometimes run the tests on platforms that do not have the compiler available (we copy pre-built .abc files).  A command line switch (--checkenv?) could be added to verify that all nec. files are there before starting the run.

Overall though I don't think this bites too many people once they are setup.  I suggest resolve wontfix unless more people are affected.