Closed Bug 1774476 Opened 2 years ago Closed 2 years ago

CDP does not report initial tab as a target on the root window with Firefox 101 on Windows

Categories

(Remote Protocol :: CDP, defect, P1)

Firefox 101
defect
Points:
1

Tracking

(firefox-esr91 unaffected, firefox-esr102 fixed, firefox102 wontfix, firefox103 fixed, firefox104 fixed)

RESOLVED FIXED
104 Branch
Tracking Status
firefox-esr91 --- unaffected
firefox-esr102 --- fixed
firefox102 --- wontfix
firefox103 --- fixed
firefox104 --- fixed

People

(Reporter: tbiethman, Assigned: whimboo)

References

(Regression)

Details

(Keywords: regression, Whiteboard: [webdriver:m4])

Attachments

(2 files)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36

Steps to reproduce:

Starting with Firefox 101, I noticed that a launched window's initial tab does not appear to be registered as an available target and does not get returned by Target.getTargets. This only appears to be a problem in Windows, with my repro environment being on Windows 10.

To reproduce:

  1. Start Firefox 101 with remote debugging enabled:

start 'C:\Program Files (x86)\Mozilla Firefox\firefox.exe' 'about:blank', '--remote-debugging-port=51959'

  1. Connect to launched instance and issue Target.getTargets message (simple script provided here: https://gist.github.com/tbiethman/9add1e991efbecdbe3e6eddec9fb4b46)

Actual results:

With Firefox 101, the Target.getTargets message resolves with no found targets, with script output:

found targets
{ targetInfos: [] }

Expected results:

The initial tab target should be returned, with the script output looking something like:

found targets
{
  targetInfos: [{
    targetId: 'xxxxxxx-xxxxxx-xxxxxxx',
    type: 'page',
    title: '',
    url: 'about:blank',
    attached: false,
    browserContextId: null,
  }]
}

This is what returns when running the repro steps against Firefox 100 in Windows. This is also the behavior exhibited by the macOS version of Firefox 101, with the initial tab target returning as expected.


A data couple points that might be relevant:

  • With Firefox 101, if I spawn another window (with Ctrl+N) from the initial window, the script will start returning targets upon subsequent executions.
  • Also with Firefox 101, if I spawn the initial window in troubleshooting mode (with the -safe-mode option), the tab does return as a target for the initial window.
  • I thought that this may be due to differences in handling of 'about:blank' in v101, but launching to an actual location (say, 'https://www.mozilla.org') still results in no targets being returned.

If I can provide can other data points that could be helpful, or if I've missed something completely in the protocol and this behavior is now expected, please let me know.

The Bugbug bot thinks this bug should belong to the 'Core::Widget: Cocoa' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Widget: Cocoa
Product: Firefox → Core
Component: Widget: Cocoa → General
OS: Unspecified → Windows 10
Hardware: Unspecified → Desktop

Following up, I tried to use Foxdriver and Marionette instead to see if I could massage the browser into a connectable state, but I was not successful.

I noticed the decent number of changes to the Remote Protocol in the 101 release, but nothing sticks out to me as a reason for a Windows-specific failure here.

I noticed the "Start the Remote Agent earlier during the startup of Firefox" enhancement and some related bug fixes released with 101; could there be a regression with Windows here? https://bugzilla.mozilla.org/show_bug.cgi?id=1759169

cc-ing @whimboo as the assignee for that feature, any insights here would be greatly appreciated.

Flags: needinfo?(hskupin)

Interesting. Our existing tests don't have an issue. Before I can dig into this issue could you maybe run your test with a profile where the remote.log.level preference is set to Trace? I would be kinda interested in the log output. Thanks!

Component: General → CDP
Flags: needinfo?(hskupin) → needinfo?(tbiethman)
Product: Core → Remote Protocol

Log output for both FF101 and FF100 included here: https://gist.github.com/tbiethman/2bb900635b186747e2ebf5474af1a5a1

The tab open event is noticeably absent from the FF101 log.

Flags: needinfo?(tbiethman)

That's interesting. So we have at least the following test for this scenario where we check the loaded tab and that it is correctly reported:
https://searchfox.org/mozilla-central/source/remote/cdp/test/browser/target/browser_getTargets.js

That's definitely passing across all platforms, but yes it's not run right after the startup of Firefox but some other tabs have already been opened and closed. Could you maybe check if in your case the initial tab is reported after you open one or more tabs? Or is the first tab always missing?

Just to verify on other platforms like Linux and MacOS you aren't able to reproduce this different behavior?

I pushed a try build to check what happens when we run the above mentioned test in isolation right after Firefox has been started up:
https://treeherder.mozilla.org/jobs?repo=try&revision=fccf9d4c02cde8811078fd0ff6cef36738dc447f

The CI jobs didn't show any failure. As such we would need more information (for the questions that I asked before) to further get it investigated. Thanks.

Could you maybe check if in your case the initial tab is reported after you open one or more tabs? Or is the first tab always missing?

It seems related to the first window that is opened. As in I can launch Firefox with the command above, open multiple subsequent tabs in that window, and have that script still return no targets. But when I open a new window from the first, the target for the new window's tab returns. Opening more tabs in that new window results in more targets returning from the script. If your test attaches to an existing window that has maybe gone through a different lifecycle, that might explain the discrepancy?

Just to verify on other platforms like Linux and MacOS you aren't able to reproduce this different behavior?

I am unable to reproduce with Firefox 101 on macOS (12.4). I will confirm whether or not I can repro in Linux and post back here. For Windows, I've been able to repro on Windows 10 and Windows Server 2022.

I appreciate your time spent investigating this; let me know if there's any other data points I can provide. This was first reported as an issue to Cypress, which cannot connect to FF101 on its currently released versions. Posting that here in case any of the comments are helpful.

I can confirm that I cannot reproduce this issue on Linux with FF101, using the same repro steps in the OP. The target returns for the launched window's first tab, just as I see in macOS. I've attached a screenshot showing the commands I used for posterity.

Can you check the values of the preferences mentioned in this documentation: https://firefox-source-docs.mozilla.org/remote/cdp/RequiredPreferences.html ?

It feels like you might be affected by process changes related to Fission?

Edit: Just saw that you attached a log and the error message we normally display for invalid preferences is not visible there, so maybe not related. Still worth checking.

Flags: needinfo?(tbiethman)

(In reply to tbiethman from comment #8)

Could you maybe check if in your case the initial tab is reported after you open one or more tabs? Or is the first tab always missing?

It seems related to the first window that is opened. As in I can launch Firefox with the command above, open multiple subsequent tabs in that window, and have that script still return no targets. But when I open a new window from the first, the target for the new window's tab returns. Opening more tabs in that new window results in more targets returning from the script. If your test attaches to an existing window that has maybe gone through a different lifecycle, that might explain the discrepancy?

This is very strange! So it doesn't look like to be a race but more that we seem to completely miss to register this window and as such also all other events like for opening and closing of targets. But when I check the following code I can see that we clearly start the target watcher before waiting for the initial window:

https://searchfox.org/mozilla-central/rev/26a6a38fb515dbab0bb459c40ec4b877477eefef/remote/cdp/CDP.jsm#102-109

Regarding the trace logs, did you cut parts off? For the 101 one I cannot find the line starting with DevTools listening. It would mean that we actually never announce the CDP port to clients and it just works because you make use of the default port. I wonder if we should actually deny any connection attempt until the CDP protocol has been finished its initialization.

It feels like that you may hit some kind of bug 1382162 constantly which we never were able to reproduce.

I would suggest that you try with the following two Nightly builds so that we can verify if that is a regression from my patch.

Nightly without the changes: https://archive.mozilla.org/pub/firefox/nightly/2022/04/2022-04-25-09-42-17-mozilla-central/
Nightly with the changes: https://archive.mozilla.org/pub/firefox/nightly/2022/04/2022-04-25-21-36-55-mozilla-central/

Pick the firefox-101.0a1.en-US.win64.zip archive, download and extract. Let me know if there is a difference. Thanks.

Also please check without creating a CDP connection how long it takes until the DevTools listening... log line appears, or if it appears at all.

Note that you basically should not connect before that line is printed. Alternatively you can also check the Firefox profile directory for a DevToolActivePort file, which contains the connection details once CDP is ready.

@jdescottes Thank you for calling this out. I can confirm that those Fission prefs were not set, which is something we should remedy within Cypress. However, they don't seem to have an impact here, set or not.

@whimboo I tried the nightly build without those changes first, and it worked! The initial tab returned as a target. I then tried the nightly build with the changes....which also worked, and indicated to me that your patch didn't cause a regression.

But having that working nightly version of 101, I figured "hey, I can binary search with the best of them" and grabbed the last 101 nightly release, thinking that this version would surely fail as the release version does. However, that version also works. Which leaves me quite confused, unless there are functional changes within the release version that aren't in the last nightly build.

I'm including the log output from the working nightly build I linked, and a log from the released version. Both show DevTools listening... prior to my CDP connection being created, but there are a few differences between the two. It appears as though the nightly build is doing some logic around waiting for window restoration that is absent from the release build, but I'm launching both the same way.

101.0.1 Release:

xxxxxxxx       RemoteAgent     DEBUG   WebDriver BiDi enabled
xxxxxxxx       RemoteAgent     DEBUG   CDP enabled
xxxxxxxx       RemoteAgent     TRACE   Received observer notification final-ui-startup
WebDriver BiDi listening on ws://localhost:54121
xxxxxxxx       RemoteAgent     DEBUG   Setting recommended pref datareporting.policy.dataSubmissionPolicyAccepted to false
xxxxxxxx       CDP     DEBUG   Waiting for initial application window to be loaded
EMITTING: emit(ADD_SECTION, topstories, [object Object]) from addSection() -> resource://activity-stream/lib/SectionsManager.jsm:269
EMITTING: emit(ADD_SECTION, highlights, [object Object]) from addSection() -> resource://activity-stream/lib/SectionsManager.jsm:269
EMITTING: emit(INIT) from init() -> resource://activity-stream/lib/SectionsManager.jsm:213
xxxxxxxx       RemoteAgent     TRACE   Received observer notification browser-idle-startup-tasks-finished
EMITTING: emit(target-created, [object MainProcessTarget]) from registerTarget() -> chrome://remote/content/cdp/targets/TargetList.jsm:93
DevTools listening on ws://localhost:54121/devtools/browser/554f9233-4434-4b9c-b5d1-72fae8241f3a
xxxxxxxx       RemoteAgent     DEBUG   CDPConnection f502e4b3-7967-4c8c-a908-9e0437545c86 accepted
EMITTING: emit(packet, [object Object]) from onMessage() -> chrome://remote/content/server/WebSocketTransport.jsm:87
xxxxxxxx       CDP     TRACE   CDPConnection f502e4b3-7967-4c8c-a908-9e0437545c86 -> {"id":1,"method":"Target.getTargets","params":{}}
xxxxxxxx       CDP     TRACE   CDPConnection f502e4b3-7967-4c8c-a908-9e0437545c86 <- {"id":1,"result":{"targetInfos":[]}}

2022-05-02-09-43-29 Nightly:

xxxxxxxx       RemoteAgent     DEBUG   WebDriver BiDi enabled
xxxxxxxx       RemoteAgent     DEBUG   CDP enabled
xxxxxxxx       RemoteAgent     TRACE   Received observer notification final-ui-startup
xxxxxxxx       WebDriver BiDi  DEBUG   Awaiting all initial windows to be restored before enabling the protocol
xxxxxxxx       CDP     DEBUG   Awaiting all initial windows to be restored before enabling the protocol
[GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt
xxxxxxxx       RemoteAgent     TRACE   Received observer notification sessionstore-windows-restored
WebDriver BiDi listening on ws://localhost:54315
xxxxxxxx       RemoteAgent     DEBUG   Setting recommended pref datareporting.policy.dataSubmissionPolicyAccepted to false
EMITTING: emit(open, [object XULElement] (tab.tabbrowser-tab)) from onTabOpen() -> chrome://remote/content/cdp/observers/TargetObserver.jsm:72
EMITTING: emit(target-created, [object Target 2134c710-ab10-41ae-a12b-43c1441c472b]) from registerTarget() -> chrome://remote/content/cdp/targets/TargetList.jsm:93
EMITTING: emit(target-created, [object MainProcessTarget]) from registerTarget() -> chrome://remote/content/cdp/targets/TargetList.jsm:93
DevTools listening on ws://localhost:54315/devtools/browser/5d192446-ae37-4108-9dd3-a729bf95d7c5
EMITTING: emit(ADD_SECTION, topstories, [object Object]) from addSection() -> resource://activity-stream/lib/SectionsManager.jsm:269
EMITTING: emit(ADD_SECTION, highlights, [object Object]) from addSection() -> resource://activity-stream/lib/SectionsManager.jsm:269
EMITTING: emit(INIT) from init() -> resource://activity-stream/lib/SectionsManager.jsm:213
xxxxxxxx       RemoteAgent     DEBUG   CDPConnection 080c574e-9b57-45b8-ab4f-b46f98c3a99f accepted
EMITTING: emit(packet, [object Object]) from onMessage() -> chrome://remote/content/server/WebSocketTransport.jsm:87
xxxxxxxx       CDP     TRACE   CDPConnection 080c574e-9b57-45b8-ab4f-b46f98c3a99f -> {"id":1,"method":"Target.getTargets","params":{}}
xxxxxxxx       CDP     TRACE   CDPConnection 080c574e-9b57-45b8-ab4f-b46f98c3a99f <- {"id":1,"result":{"targetInfos":[{"targetId":"2134c710-ab10-41ae-a12b-43c1441c472b","type":"page","title":"","url":"about:blank","attached":false,"browserContextId":null}]}}
Flags: needinfo?(tbiethman)

Very interesting. DIfferences between the last Nightly 101 and release 101 can be either:

  • a patch which has been uplifted from 102 to 101
  • a preference which has a different default in release/beta/nightly

If you test with a more recent Nightly (a 103 to be sure) do you reproduce the issue? If yes, it might be an uplifted patch. Otherwise, more likely to be a preference.

Regarding the symptoms, our target observer should listen for new windows (which seems to work) but also register existing ones, via:

    Services.wm.addListener(this);

    if (this.registerExisting) {
      // Start listening for events on already open windows
      for (const win of Services.wm.getEnumerator("navigator:browser")) {
        this._registerDOMWindow(win);
      }
    }

Maybe you are hitting a race where the initial window is not yet returned by Services.wm.getEnumerator, but will not be captured by the listener either?

Very good thinking @jdescotte! I bisected the full set of nightly builds between the ones tested above and the current 103 builds and found the diverging build.

This build works with my repro steps: https://archive.mozilla.org/pub/firefox/nightly/2022/05/2022-05-04-09-37-59-mozilla-central/
The following build does not: https://archive.mozilla.org/pub/firefox/nightly/2022/05/2022-05-04-23-45-51-mozilla-central/

Thanks for checking! That gives us the following pushlog: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1a63d68454d11cb0ed8b08e31ce00ad8d5eefddf&tochange=38697602446fd8cb367d8b25078c99b8d3313562

So I would suspect Bug 1767387, the only other patch for remote/ is Bug 1754132, but it's very unlikely it had any impact.

(In reply to Julian Descottes [:jdescottes] from comment #17)

Maybe you are hitting a race where the initial window is not yet returned by Services.wm.getEnumerator, but will not be captured by the listener either?

IMHO very unlikely given that we start the window listener before we actually enter this loop to process existing windows. In such a case the registered listener should initialize the window.

(In reply to Julian Descottes [:jdescottes] from comment #19)

Thanks for checking! That gives us the following pushlog: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1a63d68454d11cb0ed8b08e31ce00ad8d5eefddf&tochange=38697602446fd8cb367d8b25078c99b8d3313562

So I would suspect Bug 1767387, the only other patch for remote/ is Bug 1754132, but it's very unlikely it had any impact.

Yes, I would also tend to agree with bug 1767387. But to be fully sure it's the causing change would you mind to also check with the two following builds from autoland?

https://treeherder.mozilla.org/jobs?repo=autoland&tochange=621def30243df2e5e2390296202c0beae4a5fdc5&fromchange=a62abba5bf3dc02f8998b3115b8fa9cb6347501c&searchStr=build%2Cwindows%2Cship%2Cx64

With the B job selected switch to the Artifacts and Debugging Tools panel at the bottom and look for the target.zip file to download. Please do this with both listed builds.

If it's clear that this bug caused this problem for you I might go ahead and prepare a special build with a lot more trace log entries around the affected lines of code.

I pulled both of those builds and gave them a shot.

Thanks a lot for testing these two builds. Now it's clear that this other bug regressed the behavior on your machine.

I'll have a look tomorrow in creating a custom build for you with additional logging information. Lets hope that this helps with identifying the problem.

The bug has a release status flag that shows some version of Firefox is affected, thus it will be considered confirmed.

Status: UNCONFIRMED → NEW
Ever confirmed: true

One additional question, does it make any difference if you "wait" before connecting to Firefox? From your steps to reproduce, we have:

  1. Start Firefox 101 with remote debugging enabled:
  2. Connect to launched instance and issue Target.getTargets message

Is step 2 starting automatically after step 1 (and then maybe you are hitting a very specific race condition?) or are you triggering the script manually. If you are doing it manually, did you try to wait a bit before initiating the connection? And if it's not manual, can you try to introduce some delay just to see if it would help?

Flags: needinfo?(tbiethman)

Set release status flags based on info from the regressing bug 1767387

For my reproduction I am executing the script manually, after manually launching Firefox, so there's generally a 3-5s delay between the two. I did confirm that DevTools listening on... was logged prior to my attempts to connect.

Within Cypress itself, the connection is attempted much more quickly, but even adding an arbitrary 5s delay there between launching firefox and connecting to CDP has the same result (no targets returned).

Flags: needinfo?(tbiethman)

I've pushed a try build which includes a couple of extra log output around the affected code. It would be great if you can run the binary once it has been finished building:

https://treeherder.mozilla.org/jobs?repo=try&revision=d4c49a59d28d939c0615dd9ebbd89a77ec9e1ccb

Flags: needinfo?(tbiethman)

Apologies for the delay, the logs I get from your build are here: https://gist.github.com/tbiethman/7b27dde4830a1d903e641f896033c3d3

I also ran this build with the proper fission prefs set, which removed the CDP pref error from the logs but had no functional impact on target retrieval. Those logs are here: https://gist.github.com/tbiethman/e371f58fbfb882301ccdd31061901b6a

Flags: needinfo?(tbiethman)

Thank you for the trace log! As it can be seen we are entering the code which starts registering existing windows. But given that there is no window open at this point there is nothing to register.
Then around 2s later the initial browser window has been opened and finished loading, and I can see that the MainProcessTarget gets registered. But that's all and is also not tied to our window detection code.

Our window detection code is based on the nsIWindowMediatorListener implementation and our onOpenWindow callback should be called. Checking some other code in mozilla-central that uses the same logic I found:

https://searchfox.org/mozilla-central/rev/1ce190047b9556c3c10ab4de70a0e61d893e2954/toolkit/components/formautofill/FormAutofillParent.jsm#99-112

As it looks like in some cases the windowType doesn't seem to be set yet during startup. And as such we skip this not yet finished initializing browser window.

I've made new try server builds that are accepting all chrome window types now just to check if this would work for you:
https://treeherder.mozilla.org/jobs?repo=try&revision=f4810fae8892512a6dc7be0977c257e960e52656

Please paste the trace logs again. Thanks.

Flags: needinfo?(tbiethman)

Sorry, the last builds had a problem and didn't actually log something. Please use the following try build:
https://treeherder.mozilla.org/jobs?repo=try&revision=57c94cba68ca9c7b743c41ced69a9813a09d3755

I don't seem to be getting any logs from your second linked build either.

Functionally, the window will open, but It doesn't look like the remote agent is starting up (no RemoteAgent/:robot: indicator in URL bar).

Flags: needinfo?(tbiethman)

This is strange. Are you sure you used the correct build? Even if you start Firefox and the Remote Agent doesn't start you should at least always see trace output from Firefox as long as the preference remote.log.level=Trace is set. Also this output will help to determine why the Remote Agent doesn't start.

Indeed, the previous builds rendered the log output in a separate terminal; this one does not, and doesn't output CDP logs to the browser console. Windows isn't my normal dev environment, so if I'm missing something obvious here, my apologies.

However, I was able to pull some logs when running through Cypress itself: https://gist.github.com/tbiethman/873c58cbbc5bf9ebf051035e1aac079c

Startup error seems to be:

RemoteAgent     ERROR   Unable to start remote agent: can't access property "tabs", win.gBrowser is undefined: TypeError: can't access property "tabs", win.gBrowser is undefined(chrome://remote/content/cdp/observers/TargetObserver.jsm:93:36) 

I was actually able to reproduce today while working on a patch for bug 1769994. With a bit of debugging (note that it doesn't work with --jsdebugger given that this is a race condition) I found out that the initial window that we get when registering already open window is the pre-rendered browser window which is blank and has no tabbrowser attached. As such it's useless for us and we cannot attach any listener for TabOpen and TabClose. At a later time the real gBrowser will be inserted but that we do not notice and as such our CDP implementation cannot track any tab targets.

The only option that I can actually see here is to defer registering our JSON handlers and all the initialization of CDP until the browserStartupFinished promise has been resolved.

I'm going to attach a patch in a moment and here is a try build to verify the fix:
https://treeherder.mozilla.org/jobs?repo=try&revision=fe9bbd91157b0a6fbb10492a94ea41b98d97c564

Assignee: nobody → hskupin
Status: NEW → ASSIGNED
OS: Windows 10 → All
Hardware: Desktop → All

Initializing CDP before the first window has been opened can break
the registration of the first browser window because a pre-loaded
window gets opened first before the gBrowser is attached.

This also prevents any CDP client to connect to any the JSONHandler
end-points before CDP is actually ready.

Points: --- → 1
Priority: -- → P1
Whiteboard: [webdriver:m4]

The build with the fix works great, nice find!

Thanks for confirming the fix. I will try to get it landed soon and also uplifted to 103 beta and 102ESR.

Blocks: 1769994
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/941414f045ab
[cdp] Delay initialization of CDP until browserStartupFinished. r=webdriver-reviewers,jgraham
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 104 Branch

Comment on attachment 9285510 [details]
Bug 1774476 - [cdp] Delay initialization of CDP until browserStartupFinished.

Beta/Release Uplift Approval Request

  • User impact if declined: On some machines our CDP implementation will fail to register the initial browser window and tabs. As result clients won't be able to interact with any of the actually visible tabs.
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The patch only moves the code block to initialize the CDP protocol until the initial browser window has been fully loaded.
  • String changes made/needed: No
  • Is Android affected?: Unknown

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration:
  • User impact if declined: On some machines our CDP implementation will fail to register the initial browser window and tabs. As result clients won't be able to interact with any of the actually visible tabs.
  • Fix Landed on Version: 104
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The patch only moves the code block to initialize the CDP protocol until the initial browser window has been fully loaded.
Attachment #9285510 - Flags: approval-mozilla-esr102?
Attachment #9285510 - Flags: approval-mozilla-beta?

Comment on attachment 9285510 [details]
Bug 1774476 - [cdp] Delay initialization of CDP until browserStartupFinished.

Approved beta uplift request, available on the beta channel with 103.0 RC1.

Attachment #9285510 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Just wanted to report back that the 103.0 release works great. Thank you all for your help getting this resolved!

Comment on attachment 9285510 [details]
Bug 1774476 - [cdp] Delay initialization of CDP until browserStartupFinished.

Approved for 102.2esr.

Attachment #9285510 - Flags: approval-mozilla-esr102? → approval-mozilla-esr102+

(In reply to tbiethman from comment #43)

Just wanted to report back that the 103.0 release works great. Thank you all for your help getting this resolved!

Perfect. Thanks for your valuable input as well!

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: