Closed Bug 1407799 Opened 8 years ago Closed 8 years ago

|mach test-info| sometimes reports duplicate results for chunked and e10s run.type

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gbrown, Assigned: gbrown)

Details

gbrown@mozpad:~/src$ ./mach test-info test_console_binding.html ===== test_console_binding.html ===== Found dom/console/tests/test_console_binding.html in source control. dom/console/tests/test_console_binding.html found in manifest dom/console/tests/mochitest.ini flavor: mochitest Querying ActiveData... Found records matching 'dom/console/tests/test_console_binding.html' in ActiveData. Test results for dom/console/tests/test_console_binding.html on mozilla-central,mozilla-inbound,autoland between 2017-10-04 and 2017-10-11 android-4-3-armv7-api16/debug-chunked: 0 failures in 165 runs android-4-3-armv7-api16/opt-chunked: 0 failures in 156 runs linux32-nightly/opt-chunked: 0 failures in 11 runs linux32-nightly/opt-e10s: 0 failures in 11 runs linux32-stylo-disabled/debug-chunked: 0 failures in 165 runs linux32-stylo-disabled/debug-e10s: 0 failures in 165 runs linux32-stylo-disabled/opt-chunked: 0 failures in 159 runs linux32-stylo-disabled/opt-e10s: 0 failures in 159 runs linux32/debug-chunked: 0 failures in 163 runs linux32/debug-e10s: 0 failures in 163 runs linux32/opt-chunked: 0 failures in 157 runs linux32/opt-e10s: 0 failures in 157 runs linux64-nightly/opt-chunked: 0 failures in 12 runs linux64-nightly/opt-e10s: 0 failures in 12 runs linux64-stylo-disabled/debug-chunked: 0 failures in 165 runs linux64-stylo-disabled/debug-e10s: 0 failures in 165 runs linux64-stylo-disabled/opt-chunked: 0 failures in 165 runs linux64-stylo-disabled/opt-e10s: 0 failures in 165 runs linux64/asan-chunked: 0 failures in 161 runs linux64/asan-e10s: 0 failures in 161 runs linux64/ccov-chunked: 0 failures in 22 runs linux64/ccov-e10s: 0 failures in 22 runs linux64/debug-chunked: 0 failures in 171 runs linux64/debug-e10s: 0 failures in 171 runs linux64/jsdcov-chunked: 0 failures in 20 runs linux64/opt-chunked: 0 failures in 213 runs linux64/opt-e10s: 0 failures in 193 runs linux64/pgo-chunked: 0 failures in 161 runs linux64/pgo-e10s: 0 failures in 161 runs Total: 0 failures in 3771 runs or 0.000 failures/run Test durations for dom/console/tests/test_console_binding.html on mozilla-central,mozilla-inbound,autoland between 2017-10-04 and 2017-10-11 android-4-3-armv7-api16/debug-chunked: 4.55 s (2.69 s - 14.98 s over 165 runs) android-4-3-armv7-api16/opt-chunked: 2.11 s (1.16 s - 4.79 s over 156 runs) linux32-nightly/opt-chunked: 0.16 s (0.11 s - 0.21 s over 11 runs) linux32-nightly/opt-e10s: 0.16 s (0.11 s - 0.21 s over 11 runs) linux32-stylo-disabled/debug-chunked: 0.59 s (0.38 s - 1.78 s over 165 runs) linux32-stylo-disabled/debug-e10s: 0.59 s (0.38 s - 1.78 s over 165 runs) linux32-stylo-disabled/opt-chunked: 0.18 s (0.13 s - 0.38 s over 159 runs) linux32-stylo-disabled/opt-e10s: 0.18 s (0.13 s - 0.38 s over 159 runs) linux32/debug-chunked: 0.63 s (0.39 s - 1.76 s over 163 runs) linux32/debug-e10s: 0.63 s (0.39 s - 1.76 s over 163 runs) linux32/opt-chunked: 0.17 s (0.13 s - 0.38 s over 157 runs) linux32/opt-e10s: 0.17 s (0.13 s - 0.38 s over 157 runs) linux64-nightly/opt-chunked: 0.15 s (0.13 s - 0.19 s over 12 runs) linux64-nightly/opt-e10s: 0.15 s (0.13 s - 0.19 s over 12 runs) linux64-stylo-disabled/debug-chunked: 0.54 s (0.33 s - 3.43 s over 165 runs) linux64-stylo-disabled/debug-e10s: 0.54 s (0.33 s - 3.43 s over 165 runs) linux64-stylo-disabled/opt-chunked: 0.17 s (0.13 s - 1.78 s over 165 runs) linux64-stylo-disabled/opt-e10s: 0.17 s (0.13 s - 1.78 s over 165 runs) linux64/asan-chunked: 0.35 s (0.24 s - 1.26 s over 161 runs) linux64/asan-e10s: 0.35 s (0.24 s - 1.26 s over 161 runs) linux64/ccov-chunked: 0.35 s (0.20 s - 0.67 s over 22 runs) linux64/ccov-e10s: 0.35 s (0.20 s - 0.67 s over 22 runs) linux64/debug-chunked: 0.57 s (0.32 s - 1.33 s over 171 runs) linux64/debug-e10s: 0.57 s (0.32 s - 1.33 s over 171 runs) linux64/jsdcov-chunked: 1.41 s (1.34 s - 1.60 s over 20 runs) linux64/opt-chunked: 0.30 s (0.12 s - 1.60 s over 213 runs) linux64/opt-e10s: 0.19 s (0.12 s - 0.67 s over 193 runs) linux64/pgo-chunked: 0.15 s (0.10 s - 0.51 s over 161 runs) linux64/pgo-e10s: 0.15 s (0.10 s - 0.51 s over 161 runs) Why is there linux64/pgo-chunked and linux64/pgo-e10s when we only run e10s now? Why are the results identical for those two platforms? Notice that the linux64/opt-chunked and linux64/opt-e10s results are different from one another. Again, curious that opt-chunked results are not 0. I think these are the results from activedata, with the chunked/e10s part coming from run.type, but I need to verify that.
I checked test-info results for several other tests. xpcshell tests (which are never e10s) are OK: results have no chunked or e10s type, as expected. mochitest-chrome tests are OK: results have no chunked or e10s type, which seems OK. reftest results are OK: results are (almost) all e10s, as expected. But most/all plain mochitests and browser-chrome mochitests show the same anomaly: both chunked and e10s types, with often-identical results.
Simplified query that confirms the problem: { "from":"unittest", "format":"list", "where":{"and":[ {"eq":{"result.test":"dom/console/tests/test_console_binding.html"}}, {"eq":{"build.branch":"mozilla-inbound"}}, {"gt":{"run.timestamp":{"date":"2017-10-17"}}} ]}, "groupby":["build.platform","build.type","run.type"] } Results for this query show that most (build.platform,build.type) platforms have records with both run.type=="chunked" and run.type=="e10s", even though we should be running only e10s on most platforms (only chunked on Android, perhaps both on Windows 7). :ekyle -- Do you know what's going wrong? Where does activedata get the run.type field from?
Flags: needinfo?(klahnakoski)
I am looking at this now.
For my notes... ES is double counting documents when using "terms" aggregation on multivalue properties: > platform run_type build_type > -------------------------------------------------------- > macosx64-stylo-disabled [e10s,chunked] debug > android-4-3-armv7-api16 chunked debug > linux64 [e10s,chunked] debug > linux64 [e10s,chunked] asan > linux32-stylo-disabled [e10s,chunked] opt > windows7-32-stylo-disabled [e10s,chunked] debug > linux64 [e10s,chunked] debug > macosx64-stylo-disabled [e10s,chunked] debug > osx-10-10 [e10s,chunked] debug > osx-10-10 [e10s,chunked] opt In this case, all the [e10s,chunked] are counted under both the "e10s" and the "chunked" buckets.
Flags: needinfo?(klahnakoski)
Flags: needinfo?(klahnakoski)
More pontification..... :gbrown's query is correct; it is the result that is logically wrong. In this case, we should be returning 2 run_types (one for each combination): [chunked,e10s], [chunked]. At most there are 2^N combinations: [chunked,e10s], [chunked], [e10s], []; but only the realized combinations need to be returned. The issue is getting this type of reponse out of groupby would really chafe ES. This may have an easier solution with the new version of ES. I added a test to capture this desired behaviour.
In the meantime we could use this { "from":"unittest", "format":"list", "where":{"and":[ {"eq":{"result.test":"dom/console/tests/test_console_binding.html"}}, {"eq":{"build.branch":"mozilla-inbound"}}, {"gt":{"run.timestamp":{"date":"2017-10-17"}}} ]}, "groupby":[ "build.platform", "build.type", { "name":"e10s", "value":{ "when":{"eq":{"run.type":"e10s"}}, "then":{"literal":"e10s"} } }, { "name":"chunked", "value":{ "when":{"eq":{"run.type":"chunked"}}, "then":{"literal":"chunked"} } } ] } it also give us a hint of what should be happening in the background for the original query.
Flags: needinfo?(klahnakoski)
So far, so good! It seems a small amount of work is required to get the original query to work in the existing version. I have a working version now. I will continue this tonight and get back to you on the level of success.
Flags: needinfo?(klahnakoski)
Fixed via terrible hack! But since this version of ActiveData soon expires, this is not a problem
Status: NEW → RESOLVED
Closed: 8 years ago
Flags: needinfo?(klahnakoski)
Resolution: --- → FIXED
Thanks! ./mach test-info dom/console/tests/test_console_binding.html ===== dom/console/tests/test_console_binding.html ===== Found dom/console/tests/test_console_binding.html in source control. Build configuration changed. Regenerating backend. dom/console/tests/test_console_binding.html found in manifest dom/console/tests/mochitest.ini flavor: mochitest Querying ActiveData... Found records matching 'dom/console/tests/test_console_binding.html' in ActiveData. Test results for dom/console/tests/test_console_binding.html on mozilla-central,mozilla-inbound,autoland between 2017-10-27 and 2017-11-03 android-4-3-armv7-api16/debug-chunked: 0 failures in 218 runs android-4-3-armv7-api16/opt-chunked: 0 failures in 214 runs linux32-nightly/opt-[u'e10s', u'chunked']: 0 failures in 14 runs linux32-stylo-disabled/debug-[u'e10s', u'chunked']: 0 failures in 227 runs linux32-stylo-disabled/opt-[u'e10s', u'chunked']: 0 failures in 211 runs linux32/debug-[u'e10s', u'chunked']: 0 failures in 229 runs linux32/opt-[u'e10s', u'chunked']: 0 failures in 216 runs linux64-nightly/opt-[u'e10s', u'chunked']: 0 failures in 13 runs linux64-nightly/opt: 0 failures in 5 runs linux64-stylo-disabled/debug-[u'e10s', u'chunked']: 0 failures in 222 runs linux64-stylo-disabled/opt-[u'e10s', u'chunked']: 0 failures in 215 runs linux64/asan-[u'e10s', u'chunked']: 0 failures in 219 runs linux64/ccov-[u'e10s', u'chunked']: 0 failures in 38 runs linux64/debug-[u'e10s', u'chunked']: 0 failures in 860 runs linux64/jsdcov-chunked: 0 failures in 40 runs linux64/opt-[u'e10s', u'chunked']: 0 failures in 897 runs linux64/opt-chunked: 0 failures in 40 runs linux64/opt: 0 failures in 5 runs linux64/pgo-[u'e10s', u'chunked']: 0 failures in 218 runs macosx64-nightly/opt-[u'e10s', u'chunked']: 0 failures in 14 runs macosx64-stylo-disabled/debug-[u'e10s', u'chunked']: 0 failures in 222 runs macosx64-stylo-disabled/opt-[u'e10s', u'chunked']: 0 failures in 200 runs osx-10-10/debug-[u'e10s', u'chunked']: 0 failures in 229 runs osx-10-10/opt-[u'e10s', u'chunked']: 0 failures in 223 runs windows10-64-nightly/opt-[u'e10s', u'chunked']: 0 failures in 10 runs windows10-64-stylo-disabled/debug-[u'e10s', u'chunked']: 0 failures in 207 runs windows10-64-stylo-disabled/opt-[u'e10s', u'chunked']: 0 failures in 220 runs windows10-64/debug-[u'e10s', u'chunked']: 0 failures in 829 runs windows10-64/opt-[u'e10s', u'chunked']: 0 failures in 814 runs windows10-64/pgo-[u'e10s', u'chunked']: 0 failures in 224 runs windows7-32-nightly/opt-[u'e10s', u'chunked']: 0 failures in 10 runs windows7-32-stylo-disabled/debug-[u'e10s', u'chunked']: 0 failures in 223 runs windows7-32-stylo-disabled/opt-[u'e10s', u'chunked']: 0 failures in 224 runs windows7-32/debug-[u'e10s', u'chunked']: 0 failures in 716 runs windows7-32/debug-chunked: 0 failures in 224 runs windows7-32/opt-[u'e10s', u'chunked']: 0 failures in 225 runs windows7-32/pgo-[u'e10s', u'chunked']: 0 failures in 219 runs Total: 0 failures in 9134 runs or 0.000 failures/run Test durations for dom/console/tests/test_console_binding.html on mozilla-central,mozilla-inbound,autoland between 2017-10-27 and 2017-11-03 android-4-3-armv7-api16/debug-chunked: 4.08 s (2.63 s - 9.40 s over 218 runs) android-4-3-armv7-api16/opt-chunked: 1.98 s (1.15 s - 4.53 s over 214 runs) linux32-nightly/opt-[u'e10s', u'chunked']: 0.09 s (0.07 s - 0.12 s over 14 runs) linux32-stylo-disabled/debug-[u'e10s', u'chunked']: 0.38 s (0.21 s - 1.09 s over 227 runs) linux32-stylo-disabled/opt-[u'e10s', u'chunked']: 0.08 s (0.06 s - 0.11 s over 211 runs) linux32/debug-[u'e10s', u'chunked']: 0.37 s (0.22 s - 1.05 s over 229 runs) linux32/opt-[u'e10s', u'chunked']: 0.08 s (0.06 s - 0.12 s over 216 runs) linux64-nightly/opt-[u'e10s', u'chunked']: 0.07 s (0.06 s - 0.11 s over 13 runs) linux64-nightly/opt: 1.04 s (0.96 s - 1.17 s over 5 runs) linux64-stylo-disabled/debug-[u'e10s', u'chunked']: 0.29 s (0.18 s - 0.65 s over 222 runs) linux64-stylo-disabled/opt-[u'e10s', u'chunked']: 0.08 s (0.06 s - 0.20 s over 215 runs) linux64/asan-[u'e10s', u'chunked']: 0.18 s (0.13 s - 0.29 s over 219 runs) linux64/ccov-[u'e10s', u'chunked']: 0.32 s (0.18 s - 0.83 s over 38 runs) linux64/debug-[u'e10s', u'chunked']: 0.27 s (0.18 s - 0.73 s over 860 runs) linux64/jsdcov-chunked: 1.32 s (1.20 s - 1.41 s over 40 runs) linux64/opt-[u'e10s', u'chunked']: 0.09 s (0.05 s - 0.83 s over 897 runs) linux64/opt-chunked: 1.32 s (1.20 s - 1.41 s over 40 runs) linux64/opt: 1.18 s (0.93 s - 1.50 s over 5 runs) linux64/pgo-[u'e10s', u'chunked']: 0.07 s (0.05 s - 0.14 s over 218 runs) macosx64-nightly/opt-[u'e10s', u'chunked']: 0.06 s (0.05 s - 0.07 s over 14 runs) macosx64-stylo-disabled/debug-[u'e10s', u'chunked']: 0.14 s (0.08 s - 0.27 s over 222 runs) macosx64-stylo-disabled/opt-[u'e10s', u'chunked']: 0.06 s (0.03 s - 0.08 s over 200 runs) osx-10-10/debug-[u'e10s', u'chunked']: 0.14 s (0.09 s - 0.27 s over 229 runs) osx-10-10/opt-[u'e10s', u'chunked']: 0.06 s (0.04 s - 0.09 s over 223 runs) windows10-64-nightly/opt-[u'e10s', u'chunked']: 0.04 s (0.04 s - 0.05 s over 10 runs) windows10-64-stylo-disabled/debug-[u'e10s', u'chunked']: 0.09 s (0.08 s - 0.10 s over 207 runs) windows10-64-stylo-disabled/opt-[u'e10s', u'chunked']: 0.05 s (0.04 s - 0.07 s over 220 runs) windows10-64/debug-[u'e10s', u'chunked']: 0.09 s (0.07 s - 0.17 s over 829 runs) windows10-64/opt-[u'e10s', u'chunked']: 0.05 s (0.04 s - 0.08 s over 814 runs) windows10-64/pgo-[u'e10s', u'chunked']: 0.05 s (0.04 s - 0.07 s over 224 runs) windows7-32-nightly/opt-[u'e10s', u'chunked']: 0.03 s (0.03 s - 0.04 s over 10 runs) windows7-32-stylo-disabled/debug-[u'e10s', u'chunked']: 0.09 s (0.07 s - 0.19 s over 223 runs) windows7-32-stylo-disabled/opt-[u'e10s', u'chunked']: 0.04 s (0.03 s - 0.06 s over 224 runs) windows7-32/debug-[u'e10s', u'chunked']: 0.09 s (0.08 s - 0.20 s over 716 runs) windows7-32/debug-chunked: 0.11 s (0.10 s - 0.15 s over 224 runs) windows7-32/opt-[u'e10s', u'chunked']: 0.04 s (0.03 s - 0.06 s over 225 runs) windows7-32/pgo-[u'e10s', u'chunked']: 0.03 s (0.03 s - 0.05 s over 219 runs)
You need to log in before you can comment on or make changes to this bug.