Intermittent devtools/server/tests/browser/browser_storage_dynamic_windows.js | Test timed out -
Categories
(DevTools :: General, defect, P5)
Tracking
(firefox-esr78 unaffected, firefox89 unaffected, firefox90 unaffected)
Tracking | Status | |
---|---|---|
firefox-esr78 | --- | unaffected |
firefox89 | --- | unaffected |
firefox90 | --- | unaffected |
People
(Reporter: intermittent-bug-filer, Assigned: ochameau)
References
(Blocks 1 open bug, Regression)
Details
(Keywords: intermittent-failure, leave-open, regression, Whiteboard: [stockwell disabled])
Attachments
(1 file)
Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=342442520&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/CpBSjua5TmyEhkmtt8k3Aw/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/CpBSjua5TmyEhkmtt8k3Aw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
INFO - TEST-START | devtools/server/tests/browser/browser_storage_dynamic_windows.js
[task 2021-06-10T22:09:06.799Z] 22:09:06 INFO - TEST-INFO | started process screentopng
[task 2021-06-10T22:09:07.051Z] 22:09:07 INFO - TEST-INFO | screentopng: exit 0
[task 2021-06-10T22:09:07.052Z] 22:09:07 INFO - Buffered messages logged at 22:08:21
[task 2021-06-10T22:09:07.053Z] 22:09:07 INFO - Entering test bound
[task 2021-06-10T22:09:07.054Z] 22:09:07 INFO - Adding a new tab with URL: http://test1.example.org/browser/devtools/server/tests/browser/storage-dynamic-windows.html
[task 2021-06-10T22:09:07.055Z] 22:09:07 INFO - Buffered messages logged at 22:08:22
[task 2021-06-10T22:09:07.056Z] 22:09:07 INFO - Console message: [JavaScript Warning: "Cookie “c1” will be soon rejected because it has the “SameSite” attribute set to “None” or an invalid value, without the “secure” attribute. To know more about the “SameSite“ attribute, read https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite" {file: "http://test1.example.org/browser/devtools/server/tests/browser/storage-dynamic-windows.html" line: 19}]
[task 2021-06-10T22:09:07.057Z] 22:09:07 INFO - Console message: [JavaScript Warning: "Cookie “cs2” will be soon rejected because it has the “SameSite” attribute set to “None” or an invalid value, without the “secure” attribute. To know more about the “SameSite“ attribute, read https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite" {file: "http://test1.example.org/browser/devtools/server/tests/browser/storage-dynamic-windows.html" line: 20}]
[task 2021-06-10T22:09:07.058Z] 22:09:07 INFO - Console message: [JavaScript Warning: "Cookie “c3” will be soon rejected because it has the “SameSite” attribute set to “None” or an invalid value, without the “secure” attribute. To know more about the “SameSite“ attribute, read https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite" {file: "http://test1.example.org/browser/devtools/server/tests/browser/storage-dynamic-windows.html" line: 22}]
[task 2021-06-10T22:09:07.062Z] 22:09:07 INFO - Tab added a URL http://test1.example.org/browser/devtools/server/tests/browser/storage-dynamic-windows.html loaded
[task 2021-06-10T22:09:07.062Z] 22:09:07 INFO - Console message: [JavaScript Warning: "Cookie “uc1” has been rejected because a non-HTTPS cookie can’t be set as “secure”." {file: "http://sectest1.example.org/browser/devtools/server/tests/browser/storage-unsecured-iframe.html" line: 13}]
[task 2021-06-10T22:09:07.063Z] 22:09:07 INFO - Console message: [JavaScript Error: "Unknown Collection "main/partitioning-exempt-urls"" {file: "resource://services-settings/RemoteSettingsClient.jsm" line: 160}]
[task 2021-06-10T22:09:07.064Z] 22:09:07 INFO - UnknownCollectionError@resource://services-settings/RemoteSettingsClient.jsm:160:5
[task 2021-06-10T22:09:07.065Z] 22:09:07 INFO - sync@resource://services-settings/RemoteSettingsClient.jsm:475:13
[task 2021-06-10T22:09:07.066Z] 22:09:07 INFO - async*get/this._importingPromise<@resource://services-settings/RemoteSettingsClient.jsm:375:26
[task 2021-06-10T22:09:07.066Z] 22:09:07 INFO - async*get@resource://services-settings/RemoteSettingsClient.jsm:377:13
[task 2021-06-10T22:09:07.067Z] 22:09:07 INFO - async*lazyInit@resource://gre/modules/PartitioningExceptionListService.jsm:111:26
[task 2021-06-10T22:09:07.068Z] 22:09:07 INFO - registerAndRunExceptionListObserver@resource://gre/modules/PartitioningExceptionListService.jsm:135:10
[task 2021-06-10T22:09:07.069Z] 22:09:07 INFO -
[task 2021-06-10T22:09:07.069Z] 22:09:07 INFO - Buffered messages logged at 22:08:23
<...>
INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Got updates for indexed-db -
[task 2021-06-10T22:09:07.087Z] 22:09:07 INFO - Buffered messages finished
[task 2021-06-10T22:09:07.087Z] 22:09:07 INFO - TEST-UNEXPECTED-FAIL | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Test timed out -
[task 2021-06-10T22:09:07.088Z] 22:09:07 INFO - Removing tab.
[task 2021-06-10T22:09:07.088Z] 22:09:07 INFO - Waiting for event: 'TabClose' on [object XULElement].
[task 2021-06-10T22:09:07.090Z] 22:09:07 INFO - Got event: 'TabClose' on [object XULElement].
[task 2021-06-10T22:09:07.090Z] 22:09:07 INFO - GECKO(22781) | console.error: "Tried to send a 'target-destroyed-form' event on an already destroyed actor 'watcher'"
[task 2021-06-10T22:09:07.090Z] 22:09:07 INFO - Tab removed and finished closing
[task 2021-06-10T22:09:07.091Z] 22:09:07 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | The main process DevToolsServer has no pending connection when the test ends -
[task 2021-06-10T22:09:07.127Z] 22:09:07 INFO - GECKO(22781) | MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
[task 2021-06-10T22:09:07.127Z] 22:09:07 INFO - GECKO(22781) | MEMORY STAT | vsize 3202MB | residentFast 443MB | heapAllocated 193MB
[task 2021-06-10T22:09:07.127Z] 22:09:07 INFO - TEST-OK | devtools/server/tests/browser/browser_storage_dynamic_windows.js | took 45551ms
[task 2021-06-10T22:09:07.168Z] 22:09:07 INFO - checking window state
[task 2021-06-10T22:09:07.229Z] 22:09:07 INFO - TEST-START | devtools/server/tests/browser/browser_storage_dynamic_windows.js
Comment 1•3 years ago
|
||
Recent log:
https://treeherder.mozilla.org/logviewer?job_id=342475294&repo=autoland&lineNumber=24267
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Assignee | ||
Updated•3 years ago
|
Assignee | ||
Comment 3•3 years ago
|
||
I see two distinct failure:
devtools/server/tests/browser/browser_storage_dynamic_windows.js
FAIL We get an update after the resource cookies, with the host values - {"https://sectest1.example.org":["cs2{9d414cc5-8319-0a04-0586-c0a6ae01670a}.example.org{9d414cc5-8319-0a04-0586-c0a6ae01670a}/"]} deepEqual {"https://sectest1.example.org":["cs2{9d414cc5-8319-0a04-0586-c0a6ae01670a}.example.org{9d414cc5-8319-0a04-0586-c0a6ae01670a}/","sc1{9d414cc5-8319-0a04-0586-c0a6ae01670a}sectest1.example.org{9d414cc5-8319-0a04-0586-c0a6ae01670a}/browser/devtools/server/tests/browser"],"http://sectest1.example.org":["sc1{9d414cc5-8319-0a04-0586-c0a6ae01670a}sectest1.example.org{9d414cc5-8319-0a04-0586-c0a6ae01670a}/browser/devtools/server/tests/browser"]} - JS frame :: chrome://mochitests/content/browser/devtools/server/tests/browser/browser_storage_dynamic_windows.js :: testAddIframe :: line 312
Stack trace:
chrome://mochitests/content/browser/devtools/server/tests/browser/browser_storage_dynamic_windows.js:testAddIframe:312
chrome://mochitests/content/browser/devtools/server/tests/browser/browser_storage_dynamic_windows.js:null:118
But this isn't the timeout reported here.
This happens because we get two single-store-update events on slow debug builds, while the test helper only wait for one.
This issue disappear if we increate the batch delay for parent process watchers as it seems to only occur for cookies.
The other issue, reported here is about this timeout:
0:23.22 INFO Testing if new iframe addition works properly
0:23.28 INFO Wait for all resources
0:24.24 PASS Got updates for indexed-db -
0:24.58 PASS Got updates for cookies -
0:24.61 PASS Got resource for local-storage -
0:24.61 PASS Got resource for session-storage -
1:51.18 FAIL Test timed out -
The issue is that we are missing single-store-updates for content process storages (local and session storage).
I tracked it down to a race condition between server and client.
We do correctly emit the single-store-updates from the server, but the front never receive it.
I suspect that we emit the event before the client is able to instantiate the front, but I'm not 100%.
I imagine using onResourceUpdate should solve this. It is meant to solve issues of early event against an existing resource.
In the meantime, this doesn't look deeply broken. It looks more like test not being super resilient to slow runtimes.
We may disable the test on debug builds.
Comment hidden (Intermittent Failures Robot) |
Comment 5•3 years ago
|
||
Update:
There have been 35 failures within the last 7 days, all of them on Linux 18.04 x64 debug.
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=343976121&repo=mozilla-central&lineNumber=24527
[task 2021-06-27T10:42:43.957Z] 10:42:43 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Got resource for session-storage -
[task 2021-06-27T10:42:43.958Z] 10:42:43 INFO - Buffered messages finished
[task 2021-06-27T10:42:43.960Z] 10:42:43 INFO - TEST-UNEXPECTED-FAIL | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Test timed out -
[task 2021-06-27T10:42:43.961Z] 10:42:43 INFO - Removing tab.
[task 2021-06-27T10:42:43.962Z] 10:42:43 INFO - Waiting for event: 'TabClose' on [object XULElement].
[task 2021-06-27T10:42:43.964Z] 10:42:43 INFO - Got event: 'TabClose' on [object XULElement].
[task 2021-06-27T10:42:43.965Z] 10:42:43 INFO - GECKO(6076) | console.error: "Tried to send a 'target-destroyed-form' event on an already destroyed actor 'watcher'"
[task 2021-06-27T10:42:43.967Z] 10:42:43 INFO - Tab removed and finished closing
[task 2021-06-27T10:42:43.968Z] 10:42:43 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | The main process DevToolsServer has no pending connection when the test ends -
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment 9•3 years ago
|
||
There are 31 total failures in the last 7 days on linux1804-64 debug
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=344525822&repo=autoland&lineNumber=7561
task 2021-07-07T00:36:52.966Z] 00:36:52 INFO - TEST-START | devtools/server/tests/browser/browser_storage_dynamic_windows.js
[task 2021-07-07T00:36:52.994Z] 00:36:52 INFO - GECKO(1547) | [Child 3292: Main Thread]: I/DocShellAndDOMWindowLeak --DOCSHELL 7f862f025400 == 0 [pid = 3292] [id = 0] [url = http://test1.example.org/browser/devtools/server/tests/browser/storage-cookies-same-name.html]
[task 2021-07-07T00:36:53.086Z] 00:36:53 INFO - GECKO(1547) | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /tmp/tmplaboovfc.mozrunner/runtests_leaks_tab_pid3318.log
[task 2021-07-07T00:38:23.169Z] 00:38:23 INFO - TEST-INFO | started process screentopng
[task 2021-07-07T00:38:23.517Z] 00:38:23 INFO - TEST-INFO | screentopng: exit 0
[task 2021-07-07T00:38:23.518Z] 00:38:23 INFO - Buffered messages logged at 00:36:53
[task 2021-07-07T00:38:23.518Z] 00:38:23 INFO - Entering test bound
[task 2021-07-07T00:38:23.519Z] 00:38:23 INFO - Adding a new tab with URL: http://test1.example.org/browser/devtools/server/tests/browser/storage-dynamic-windows.html
[task 2021-07-07T00:38:23.519Z] 00:38:23 INFO - Buffered messages logged at 00:36:54
[task 2021-07-07T00:38:23.519Z] 00:38:23 INFO - Console message: [JavaScript Warning: "Cookie “c1” will be soon rejected because it has the “SameSite” attribute set to “None” or an invalid value, without the “secure” attribute. To know more about the “SameSite“ attribute, read https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite" {file: "http://test1.example.org/browser/devtools/server/tests/browser/storage-dynamic-windows.html" line: 19}]
[task 2021-07-07T00:38:23.525Z] 00:38:23 INFO - Console message: [JavaScript Warning: "Cookie “cs2” will be soon rejected because it has the “SameSite” attribute set to “None” or an invalid value, without the “secure” attribute. To know more about the “SameSite“ attribute, read https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite" {file: "http://test1.example.org/browser/devtools/server/tests/browser/storage-dynamic-windows.html" line: 20}]
[task 2021-07-07T00:38:23.525Z] 00:38:23 INFO - Console message: [JavaScript Warning: "Cookie “c3” will be soon rejected because it has the “SameSite” attribute set to “None” or an invalid value, without the “secure” attribute. To know more about the “SameSite“ attribute, read https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite" {file: "http://test1.example.org/browser/devtools/server/tests/browser/storage-dynamic-windows.html" line: 22}]
[task 2021-07-07T00:38:23.525Z] 00:38:23 INFO - Console message: [JavaScript Warning: "Cookie “uc1” has been rejected because a non-HTTPS cookie can’t be set as “secure”." {file: "http://sectest1.example.org/browser/devtools/server/tests/browser/storage-unsecured-iframe.html" line: 13}]
[task 2021-07-07T00:38:23.525Z] 00:38:23 INFO - Tab added a URL http://test1.example.org/browser/devtools/server/tests/browser/storage-dynamic-windows.html loaded
[task 2021-07-07T00:38:23.526Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Got all the storage resources -
[task 2021-07-07T00:38:23.529Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | cookies storage actor is present -
[task 2021-07-07T00:38:23.530Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Number of hosts for cookies match -
[task 2021-07-07T00:38:23.531Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Host http://test1.example.org is present -
[task 2021-07-07T00:38:23.532Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Host http://sectest1.example.org is present -
[task 2021-07-07T00:38:23.533Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | indexed-db storage actor is present -
[task 2021-07-07T00:38:23.534Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Number of hosts for indexed-db match -
[task 2021-07-07T00:38:23.535Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Host http://test1.example.org is present -
[task 2021-07-07T00:38:23.542Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Host http://sectest1.example.org is present -
[task 2021-07-07T00:38:23.549Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | local-storage storage actor is present -
[task 2021-07-07T00:38:23.551Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Number of hosts for local-storage match -
[task 2021-07-07T00:38:23.552Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Host http://test1.example.org is present -
[task 2021-07-07T00:38:23.552Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Host http://sectest1.example.org is present -
[task 2021-07-07T00:38:23.554Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | session-storage storage actor is present -
[task 2021-07-07T00:38:23.555Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Number of hosts for session-storage match -
[task 2021-07-07T00:38:23.564Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Host http://test1.example.org is present -
[task 2021-07-07T00:38:23.565Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Host http://sectest1.example.org is present -
[task 2021-07-07T00:38:23.566Z] 00:38:23 INFO - Testing if new iframe addition works properly
[task 2021-07-07T00:38:23.566Z] 00:38:23 INFO - Wait for all resources
[task 2021-07-07T00:38:23.568Z] 00:38:23 INFO - Buffered messages logged at 00:36:55
[task 2021-07-07T00:38:23.570Z] 00:38:23 INFO - Console message: [JavaScript Warning: "Cookie “sc1” will be soon rejected because it has the “SameSite” attribute set to “None” or an invalid value, without the “secure” attribute. To know more about the “SameSite“ attribute, read https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite" {file: "https://sectest1.example.org/browser/devtools/server/tests/browser/storage-secured-iframe.html" line: 12}]
[task 2021-07-07T00:38:23.572Z] 00:38:23 INFO - Console message: [JavaScript Warning: "Cookie “cs2” has been treated as cross-site against “https://sectest1.example.org/browser/devtools/server/tests/browser/storage-secured-iframe.html” because the scheme does not match." {file: "https://sectest1.example.org/browser/devtools/server/tests/browser/storage-secured-iframe.html" line: 0}]
[task 2021-07-07T00:38:23.574Z] 00:38:23 INFO - Console message: [JavaScript Warning: "Cookie “cs2” has been treated as cross-site against “https://sectest1.example.org/browser/devtools/server/tests/browser/storage-secured-iframe.html” because the scheme does not match." {file: "https://sectest1.example.org/browser/devtools/server/tests/browser/storage-secured-iframe.html" line: 0}]
[task 2021-07-07T00:38:23.575Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Got updates for indexed-db -
[task 2021-07-07T00:38:23.576Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Got updates for cookies -
[task 2021-07-07T00:38:23.577Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Got resource for local-storage -
[task 2021-07-07T00:38:23.579Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Got resource for session-storage -
[task 2021-07-07T00:38:23.583Z] 00:38:23 INFO - Buffered messages finished
[task 2021-07-07T00:38:23.584Z] 00:38:23 INFO - TEST-UNEXPECTED-FAIL | devtools/server/tests/browser/browser_storage_dynamic_windows.js | Test timed out -
[task 2021-07-07T00:38:23.585Z] 00:38:23 INFO - Removing tab.
[task 2021-07-07T00:38:23.586Z] 00:38:23 INFO - Waiting for event: 'TabClose' on [object XULElement].
[task 2021-07-07T00:38:23.588Z] 00:38:23 INFO - Got event: 'TabClose' on [object XULElement].
[task 2021-07-07T00:38:23.589Z] 00:38:23 INFO - GECKO(1547) | console.error: "Tried to send a 'target-destroyed-form' event on an already destroyed actor 'watcher'"
[task 2021-07-07T00:38:23.590Z] 00:38:23 INFO - Tab removed and finished closing
[task 2021-07-07T00:38:23.595Z] 00:38:23 INFO - TEST-PASS | devtools/server/tests/browser/browser_storage_dynamic_windows.js | The main process DevToolsServer has no pending connection when the test ends -
[task 2021-07-07T00:38:23.596Z] 00:38:23 INFO - GECKO(1547) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-07-07T00:38:23.601Z] 00:38:23 INFO - GECKO(1547) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-07-07T00:38:23.603Z] 00:38:23 INFO - GECKO(1547) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-07-07T00:38:23.604Z] 00:38:23 INFO - GECKO(1547) | MEMORY STAT | vsize 3022MB | residentFast 401MB | heapAllocated 158MB
[task 2021-07-07T00:38:23.605Z] 00:38:23 INFO - TEST-OK | devtools/server/tests/browser/browser_storage_dynamic_windows.js | took 90470ms
Alexandre are you working on this or should we disable the test like you mentioned above?
Assignee | ||
Comment 10•3 years ago
|
||
Slow runtimes highlight race conditions explain in bugzilla comment 3,
which aren't trivial to address. So better disable this test on debug builds for now.
And at least keep it running on opt.
Updated•3 years ago
|
Comment 11•3 years ago
|
||
Comment 12•3 years ago
|
||
bugherder |
Updated•3 years ago
|
Assignee | ||
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Updated•3 years ago
|
Assignee | ||
Updated•3 years ago
|
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Description
•