1517837 - Make it easy to disable Web Replay tests

Status: RESOLVED FIXED

(Core Graveyard :: Web Replay, enhancement)

Description

[Brian Hackett [Laid off!]]

Web Replay and its tests have been in the tree for nearly six months, and while I don't feel it has been much of a drag on development in other parts of the browser, it would be good to make sure it doesn't interfere with new browser features, particularly the changes being made for Fission.

To that end, it should be simple to disable Web Replay's tests.  If a new feature breaks those tests and fixing Web Replay so that it works again is annoying or difficult, people should feel free to disable the tests, and I can go in afterwards and fix things up so that they work again.

Comment 1

[Brian Hackett [Laid off!]]

Attachment: patch

This adds a gDisableWebReplayTests variable to devtools/client/debugger/new/test/mochitest/head.js which disables all functionality in all Web Replay tests.  I think it's better to limit this to testing functionality so that code is still compiled and UI elements don't disappear.

Comment 2

[Logan Smyth [:loganfsmyth]]

Comment on attachment 9034529 [details] [diff] [review]
patch

Review of attachment 9034529 [details] [diff] [review]:
-----------------------------------------------------------------

::: devtools/client/debugger/new/test/mochitest/browser_dbg_rr_breakpoints-01.js
@@ +9,2 @@
>  // Test basic breakpoint functionality in web replay.
> +add_task(async function() {

Would it be worth using the existing test-skip infrastructure for this by making a wrapper, e.g.

```
function add_webreplay_task(fn) {
  const fn = add_task(fn);

  if (gDisableWebReplayTests) fn.skip();
  return fn;
}
```

Comment 3

[Jason Laster [:jlast]]

I think it would be nice to try doing this through browser.ini. One thing i was considering was moving the web replay tests into their own browser.ini, which would mean that it is easier to skip the replay tests. Also, locally it would be easier to run just the debugger or just the replay tests...

Comment 4

[Brian Hackett [Laid off!]]

Attachment: patch

This moves all web replay tests to devtools/client/webreplay/mochitest. The default skip-if line in its browser.ini can be used to turn off all web replay tests, and this file and the tests include instructions for doing so.

Comment 5

[Logan Smyth [:loganfsmyth]]

Comment on attachment 9035806 [details] [diff] [review]
patch

Review of attachment 9035806 [details] [diff] [review]:
-----------------------------------------------------------------

This makes sense to me!

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a0b8e8c08c8d
Move web replay tests to their own directory, r=lsmyth.

Comment 7

[Cristina Coroiu [:ccoroiu]]

Backed out changeset a0b8e8c08c8d (bug 1517837) for devtools failures at devtools/client/webreplay/mochitest/browser_dbg_rr_breakpoints-01.js

Backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/5e5b06dcc0c021a16df0a9a57d93a4b14b895e8b

Failure push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=a0b8e8c08c8d2fe95c24c26e2160ce57077355fe

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=222000949&repo=mozilla-inbound&lineNumber=4463

08:58:49 INFO - TEST-PASS | devtools/client/webreplay/mochitest/browser_dbg_rr_breakpoints-01.js | Source is doc_rr_basic.html -
08:58:49 INFO - Buffered messages logged at 08:58:11
08:58:49 INFO - Console message: [JavaScript Error: "remote browser crashed while on http://example.com/browser/devtools/client/webreplay/mochitest/examples/doc_rr_basic.html
08:58:49 INFO - " {file: "chrome://mochikit/content/mochitest-e10s-utils.js" line: 8}]
08:58:49 INFO - e10s_init/<@chrome://mochikit/content/mochitest-e10s-utils.js:8:5
08:58:49 INFO - EventListener.handleEvente10s_init@chrome://mochikit/content/mochitest-e10s-utils.js:6:3
08:58:49 INFO - testInit@chrome://mochikit/content/browser-test.js:104:5
08:58:49 INFO - setTimeout handler@chrome://mochikit/content/browser-test.js:22:1
08:58:49 INFO - loadChromeScripts@jar:file:///var/folders/kj/x1zrj9mx5q3bjwjjhbddg90m00000x/T/tmppGXKwU.mozrunner/extensions/[e-mail]!/api.js:15:3
08:58:49 INFO - loadMochitest@jar:file:///var/folders/kj/x1zrj9mx5q3bjwjjhbddg90m00000x/T/tmppGXKwU.mozrunner/extensions/[e-mail]!/api.js:111:3
08:58:49 INFO - @tests/mochitest/runtests.py:1733:1
08:58:49 INFO - @tests/mochitest/runtests.py:1712:2
08:58:49 INFO - evaluate.sandbox/promise<@chrome://marionette/content/evaluate.js:130:13
08:58:49 INFO - evaluate.sandbox@chrome://marionette/content/evaluate.js:97:17
08:58:49 INFO - GeckoDriver.prototype.execute_@chrome://marionette/content/driver.js:1017:19
08:58:49 INFO - asyncGeckoDriver.prototype.executeScript@chrome://marionette/content/driver.js:898:24
08:58:49 INFO - Asyncdespatch@chrome://marionette/content/server.js:290:20
08:58:49 INFO - asyncexecute@chrome://marionette/content/server.js:263:11
08:58:49 INFO - asynconPacket/<@chrome://marionette/content/server.js:236:15
08:58:49 INFO - async*onPacket@chrome://marionette/content/server.js:235:8
08:58:49 INFO - _onJSONObjectReady/<@chrome://marionette/content/transport.js:493:9
08:58:49 INFO -
08:58:49 INFO - Buffered messages logged at 08:58:12
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: version." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Buffered messages logged at 08:58:13
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xlink." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
08:58:49 INFO - Buffered messages finished
08:58:49 INFO - TEST-UNEXPECTED-FAIL | devtools/client/webreplay/mochitest/browser_dbg_rr_breakpoints-01.js | Test timed out -
08:58:49 INFO - Removing tab.
08:58:49 INFO - Waiting for event: 'TabClose' on [object XULElement].
08:58:49 INFO - Got event: 'TabClose' on [object XULElement].
08:58:49 INFO - Tab removed and finished closing
08:58:49 INFO - GECKO(2086) | MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
08:58:49 INFO - GECKO(2086) | MEMORY STAT | vsize 4467MB | residentFast 415MB | heapAllocated 125MB

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a441a085b516
Move web replay tests to their own directory, r=lsmyth.

Comment 9

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/a441a085b516