Since --jitflags defaults to m,mn now, we no longer look at the --ion flag. A related issue is that we run each test 2^5 = 32 times now with --ion. This number doubles whenever we add a new flag (-m, -n?). It's necessary to run tests with different optimizations and register allocators, but this exponential behavior seems a bit excessive.
I agree, in addition some tests are only relevant for a specific set of options. Last example I have is a test which is only useful when ran with "--ion -n" options. Most of the tests we have should run with "--ion --ion-eager" otherwise we are only testing the interpreter.
Once we have OSR, we could remove --ion-eager from the test matrix. We do catch valid bugs with all the options, so I think it's valuable for TBPL if nothing else. When testing locally we don't have to use --ion (I use --jitflags=, --args="--ion-eager", so we could have --ion-tbpl and --ion as shorthand for the defaults).
Yes, --ion-tbpl sounds good.
Created attachment 577210 [details] [diff] [review] Fix Changes --ion to run tests with the --ion flag and add --ion-tbpl to run tests with all IM options. These options ignore --jitflags so this should fix the tinderbox problem. I considered supporting "--jitflags=n --ion", but it may be confusing since jitflags defaults to m,mn. We could change the default --jitflags value, but we probably don't want to diverge from mozilla-central too much. Once we have on stack invalidation, --ion should probably run tests with both --ion and --ion -n.