(In reply to Daniel Holbert [:dholbert] from comment #3) > Though also: this seems like it turned out not to be a perma-fail, right? There are some pushes where it looks perma-faily, but other pushes where we've got some green and some orange, and others (older and newer than the fail-prone ones) that look fully green. Ah, it looks like maybe the green logs are all just **logs that ran hardly any tests at all (and not these tests in particular**. e.g. compare these two runs (a33f8b42c8f35579dd1e9f74036641b094bd3f4d where this is green vs. its parent c0f325a58cd9f31aafbe33c36e8d5963cfc50511 where this looks perma-orange, on "Android 7.0 x86-64 Lite WebRender opt R-nofis and R-nofis-ship R6): https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=android%2C7.0%2Copt%2Creftests%2Ctest-android-em-7.0-x86_64-lite-qr%2Fopt-geckoview-reftest-nofis%2Cr6&tochange=a33f8b42c8f35579dd1e9f74036641b094bd3f4d&fromchange=c0f325a58cd9f31aafbe33c36e8d5963cfc50511 The green logs only run 47 tests, and they're 4 minutes & 6 minutes long (respectively) -- here's the link and a quote from the first one: https://firefoxci.taskcluster-artifacts.net/IRK2uhq-SqSpEgm275e4FQ/0/public/logs/live_backing.log ``` [task 2024-08-26T17:50:46.701Z] 17:50:46 INFO - REFTEST SUITE-START | Running 47 tests [task 2024-08-26T17:50:46.701Z] 17:50:46 INFO - REFTEST TEST-START | gfx/tests/reftest/468496-1.html == gfx/tests/reftest/468496-1-ref.html [...] [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST TEST-END | gfx/tests/reftest/1912431-emoji-globalAlpha.html == gfx/tests/reftest/1912431-emoji-globalAlpha-ref.html [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Result summary: [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Successful: 40 (40 pass, 0 load only) [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception) [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Known problems: 7 (0 known fail, 0 known asserts, 1 random, 6 skipped, 0 slow) [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST SUITE-END | Shutdown ``` vs. the orange tests look like this and run 3163 tests, taking 25-40min: https://firefoxci.taskcluster-artifacts.net/BDpzYP1bS8KsLAexwIIxOQ/0/public/logs/live_backing.log ``` [task 2024-08-26T17:10:03.297Z] 17:10:03 INFO - REFTEST SUITE-START | Running 3163 tests [task 2024-08-26T17:10:03.298Z] 17:10:03 INFO - REFTEST TEST-START | layout/reftests/bugs/105-1.html == layout/reftests/bugs/105-1-ref.html [...] [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST TEST-END | layout/reftests/position-dynamic-changes/horizontal/fromauto-leftA-widthA-rightN.html?border_parent == layout/reftests/position-dynamic-changes/horizontal/leftA-widthA-rightN-ref.html?border_parent [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST INFO | Result summary: [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST INFO | Successful: 2993 (2993 pass, 0 load only) [task 2024-08-26T17:31:22.723Z] 17:31:22 WARNING - REFTEST INFO | Unexpected: 6 (6 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception) [task 2024-08-26T17:31:22.723Z] 17:31:22 WARNING - One or more unittests failed. [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST INFO | Known problems: 164 (47 known fail, 0 known asserts, 23 random, 94 skipped, 0 slow) [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST SUITE-END | Shutdown ``` jmaher, is there any reason why we'd be picking such a vastly different set of tests for the same configuration from one run to the next? Note that the only commit that differs between these two runs is https://hg.mozilla.org/integration/autoland/rev/a33f8b42c8f35579dd1e9f74036641b094bd3f4d which didn't add or remove any tests & hence doesn't seem like it should have changed rebucketing?
Bug 1915025 Comment 8 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
(In reply to Daniel Holbert [:dholbert] from comment #3) > Though also: this seems like it turned out not to be a perma-fail, right? There are some pushes where it looks perma-faily, but other pushes where we've got some green and some orange, and others (older and newer than the fail-prone ones) that look fully green. Ah, it looks like maybe the green logs are all just **logs that ran hardly any tests at all (and not these tests in particular**. e.g. compare these two runs (a33f8b42c8f35579dd1e9f74036641b094bd3f4d where this is green vs. its parent c0f325a58cd9f31aafbe33c36e8d5963cfc50511 where this looks perma-orange, on "Android 7.0 x86-64 Lite WebRender opt R-nofis and R-nofis-ship R6): https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=android%2C7.0%2Copt%2Creftests%2Ctest-android-em-7.0-x86_64-lite-qr%2Fopt-geckoview-reftest-nofis%2Cr6&tochange=a33f8b42c8f35579dd1e9f74036641b094bd3f4d&fromchange=c0f325a58cd9f31aafbe33c36e8d5963cfc50511 The green logs only run 47 tests, and they're 4 minutes & 6 minutes long (respectively) -- here's the link and a quote from the first one: https://firefoxci.taskcluster-artifacts.net/IRK2uhq-SqSpEgm275e4FQ/0/public/logs/live_backing.log ``` [task 2024-08-26T17:50:46.701Z] 17:50:46 INFO - REFTEST SUITE-START | Running 47 tests [task 2024-08-26T17:50:46.701Z] 17:50:46 INFO - REFTEST TEST-START | gfx/tests/reftest/468496-1.html == gfx/tests/reftest/468496-1-ref.html [...] [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST TEST-END | gfx/tests/reftest/1912431-emoji-globalAlpha.html == gfx/tests/reftest/1912431-emoji-globalAlpha-ref.html [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Result summary: [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Successful: 40 (40 pass, 0 load only) [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception) [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Known problems: 7 (0 known fail, 0 known asserts, 1 random, 6 skipped, 0 slow) [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST SUITE-END | Shutdown ``` vs. the orange tests look like this and run 3163 tests (including the ones in question here), taking 25-40min: https://firefoxci.taskcluster-artifacts.net/BDpzYP1bS8KsLAexwIIxOQ/0/public/logs/live_backing.log ``` [task 2024-08-26T17:10:03.297Z] 17:10:03 INFO - REFTEST SUITE-START | Running 3163 tests [task 2024-08-26T17:10:03.298Z] 17:10:03 INFO - REFTEST TEST-START | layout/reftests/bugs/105-1.html == layout/reftests/bugs/105-1-ref.html [...] [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST TEST-END | layout/reftests/position-dynamic-changes/horizontal/fromauto-leftA-widthA-rightN.html?border_parent == layout/reftests/position-dynamic-changes/horizontal/leftA-widthA-rightN-ref.html?border_parent [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST INFO | Result summary: [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST INFO | Successful: 2993 (2993 pass, 0 load only) [task 2024-08-26T17:31:22.723Z] 17:31:22 WARNING - REFTEST INFO | Unexpected: 6 (6 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception) [task 2024-08-26T17:31:22.723Z] 17:31:22 WARNING - One or more unittests failed. [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST INFO | Known problems: 164 (47 known fail, 0 known asserts, 23 random, 94 skipped, 0 slow) [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST SUITE-END | Shutdown ``` jmaher, is there any reason why we'd be picking such a vastly different set of tests for the same configuration from one run to the next? Note that the only commit that differs between these two runs is https://hg.mozilla.org/integration/autoland/rev/a33f8b42c8f35579dd1e9f74036641b094bd3f4d which didn't add or remove any tests & hence doesn't seem like it should have changed rebucketing?
(In reply to Daniel Holbert [:dholbert] from comment #3) > Though also: this seems like it turned out not to be a perma-fail, right? There are some pushes where it looks perma-faily, but other pushes where we've got some green and some orange, and others (older and newer than the fail-prone ones) that look fully green. Ah, it looks like maybe the green logs are all just **logs that ran hardly any tests at all (and not these tests in particular**). e.g. compare these two runs (a33f8b42c8f35579dd1e9f74036641b094bd3f4d where this is green vs. its parent c0f325a58cd9f31aafbe33c36e8d5963cfc50511 where this looks perma-orange, on "Android 7.0 x86-64 Lite WebRender opt R-nofis and R-nofis-ship R6): https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=android%2C7.0%2Copt%2Creftests%2Ctest-android-em-7.0-x86_64-lite-qr%2Fopt-geckoview-reftest-nofis%2Cr6&tochange=a33f8b42c8f35579dd1e9f74036641b094bd3f4d&fromchange=c0f325a58cd9f31aafbe33c36e8d5963cfc50511 The green logs only run 47 tests, and they're 4 minutes & 6 minutes long (respectively) -- here's the link and a quote from the first one: https://firefoxci.taskcluster-artifacts.net/IRK2uhq-SqSpEgm275e4FQ/0/public/logs/live_backing.log ``` [task 2024-08-26T17:50:46.701Z] 17:50:46 INFO - REFTEST SUITE-START | Running 47 tests [task 2024-08-26T17:50:46.701Z] 17:50:46 INFO - REFTEST TEST-START | gfx/tests/reftest/468496-1.html == gfx/tests/reftest/468496-1-ref.html [...] [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST TEST-END | gfx/tests/reftest/1912431-emoji-globalAlpha.html == gfx/tests/reftest/1912431-emoji-globalAlpha-ref.html [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Result summary: [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Successful: 40 (40 pass, 0 load only) [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception) [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST INFO | Known problems: 7 (0 known fail, 0 known asserts, 1 random, 6 skipped, 0 slow) [task 2024-08-26T17:51:16.966Z] 17:51:16 INFO - REFTEST SUITE-END | Shutdown ``` vs. the orange tests look like this and run 3163 tests (including the ones in question here), taking 25-40min: https://firefoxci.taskcluster-artifacts.net/BDpzYP1bS8KsLAexwIIxOQ/0/public/logs/live_backing.log ``` [task 2024-08-26T17:10:03.297Z] 17:10:03 INFO - REFTEST SUITE-START | Running 3163 tests [task 2024-08-26T17:10:03.298Z] 17:10:03 INFO - REFTEST TEST-START | layout/reftests/bugs/105-1.html == layout/reftests/bugs/105-1-ref.html [...] [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST TEST-END | layout/reftests/position-dynamic-changes/horizontal/fromauto-leftA-widthA-rightN.html?border_parent == layout/reftests/position-dynamic-changes/horizontal/leftA-widthA-rightN-ref.html?border_parent [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST INFO | Result summary: [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST INFO | Successful: 2993 (2993 pass, 0 load only) [task 2024-08-26T17:31:22.723Z] 17:31:22 WARNING - REFTEST INFO | Unexpected: 6 (6 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception) [task 2024-08-26T17:31:22.723Z] 17:31:22 WARNING - One or more unittests failed. [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST INFO | Known problems: 164 (47 known fail, 0 known asserts, 23 random, 94 skipped, 0 slow) [task 2024-08-26T17:31:22.723Z] 17:31:22 INFO - REFTEST SUITE-END | Shutdown ``` jmaher, is there any reason why we'd be picking such a vastly different set of tests for the same configuration from one run to the next? Note that the only commit that differs between these two runs is https://hg.mozilla.org/integration/autoland/rev/a33f8b42c8f35579dd1e9f74036641b094bd3f4d which didn't add or remove any tests & hence doesn't seem like it should have changed rebucketing?