1720230 - Gtest update changed the gtest reports, losing gtest details in all.sh reports.

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P1)

Description

[Robert Relyea]

Attachment: nss-3.66-fix-gtest-parsing.patch

gtests.sh and ssl_gtest.sh parses the report outputted from gtests and reflects that to the all.sh reporting mechanism. Mostly this results in properly counting each gtest as one test, rather than the entire ssl_gtest as one test. This looked like we lost 10's of thousands of tests in NSS to our qa team.

The attached patch updated the sed script which parses the output. It should be short enough to review, but I'm willing to push it to phabricator if the reviewer would prefer.

Comment 1

[Martin Thomson [:mt:]]

Any reason not to try to make the ignored result and timestamp attributes to the other regexes? I find the regex handling in sed incomprehensible, but I think that you can use \(timestamp="[^"]*"\ )* to ignore an attribute; the only trick is in ignoring the extra value that produces ((?:...) groupings would be ideal, but we can't get nice things).

Comment 2

[Robert Relyea]

I would be totally fine with a more robust regex for these, but like you, I'm pretty much at the limit of my comprehension on exactly on the sed script works. Intuitively it seems like we should be able to come up with something that can handle new additions and changes in order, but I'm not sure how to do it with sed.

Comment 3

[Robert Relyea]

Attachment: Bug 1720230 Gtest update changed the gtest reports, losing gtest details in all.sh reports.

This patch includes the updated .sed script, and an experiment using bash instead to see how hard it would be to make a more robust parser.

The robust parser generates identical output as sed, but takes about 30x longer, so instead of subsecond operations, it takes almost half a minute. With that result, I think we can stay with sed and continue to update when we get new versions of gtests. (sigh).

time cat report.xml.0 | sed -f parsegtestreport.sed > r1

real 0m0.710s
user 0m0.705s
sys 0m0.008s

time cat report.xml.0 | sh parsegtestreport.sh > r2

real 0m25.066s
user 0m17.759s
sys 0m9.506s
[rrelyea@localhost common]$ diff r1 r2

Comment 4

[Benjamin Beurdouche [:beurdouche]]

https://hg.mozilla.org/projects/nss/rev/f12856d5d2c2fefd9be075e280cd55efdb4dfed2