Hrrrm, this bug may not be an issue - might be fixed with the new regex engine. Looks from other bugs like firefox might not just be adopting YARR JIT, but YARR interpreter too. If so, this should not be a bug with YARR interpreter enabled.
Thanks for the report, Gavin! Will check it out -- we're not using YARR-interp, just YARR-JIT and PCRE. We had to make some modifications to PCRE in order to set nested capture groups to become undefined in subsequent iterations, so the difference may stem from that. (For example, our reftests state that in /(?:(f)(o)(o)|(b)(a)(r))*/.exec('foobar') capture groups 1-3 wind up as undefined, which I don't believe is part of the spec.)
Created attachment 537314 [details] [diff] [review] Test examples we've identified. All good with the current regexp engine state.
This didn't make it into mozilla-central. I'll land it now.