In practice I think this would mean getting a list of failures that are starred with a comment "fixed by <some commit>", where <some commit> is the backout. It would be super cool to update the scripts we use for backing out to automatically add a link to this list of failure to the bug.
Yes, I think we need a more deterministic way to track the causes of backouts. This may or may not be inside Treeherder.
we have failures at the job level which treeherder knows more about and the starring is done on, and there are actual test failures. Technically treeherder has all the raw test failures, it is just sorting out the intermittent ones in those starred jobs vs the real failures.