Closed Bug 1573051 Opened 3 months ago Closed 3 months ago

Links with custom URIs to external applications are losing their fragments

Categories

(External Software Affecting Firefox :: Other, defect, P1, critical)

defect

Tracking

(firefox-esr60 unaffected, firefox-esr6869+ verified, firefox68 wontfix, firefox69+ verified, firefox70+ verified)

VERIFIED FIXED
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 69+ verified
firefox68 --- wontfix
firefox69 + verified
firefox70 + verified

People

(Reporter: jorgk, Assigned: toshi)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: regression)

Attachments

(2 files)

Receive an e-mail with a link containing a reference, for example:
https://bugzilla.mozilla.org/show_bug.cgi?id=1552922#c9

Click on the link. In TB 69, the page opens to https://bugzilla.mozilla.org/show_bug.cgi?id=1552922 and not https://bugzilla.mozilla.org/show_bug.cgi?id=1552922#c9

Alice, can you please find the regression for us?

Flags: needinfo?(alice0775)

Oh, mid-air collision, thank you so much, Alice. And it's the same for bug 1572970.

So the Daily build on 24 July 2019 should already have been broken since that was based on M-C 5585edba8fdb83267722e62241952272c8:
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=bec4c1774b736a6f7fc62a7f3f5c75a924f17b5f

But hold on: I'm using TB 69 and clicking on https://bugzilla.mozilla.org/show_bug.cgi?id=1552922#c9 doesn't work there either. The range we're looking at is in the 70 cycle.

Flags: needinfo?(alice0775)

Or is it something that was backported?

Hmm, well, FF are very conservative when it comes to backporting, and this seems to be a pretty destructive change. But nothing is impossible.

Aha, here we go, bingo: Bug 1567614.

Flags: needinfo?(alice0775)

(In reply to Jorg K (GMT+2) from comment #7)

Aha, here we go, bingo: Bug 1567614.

Which is also on ESR since four days ago. :-(

And backported to FF 68.0.2, terrible, so if I move our branch forward, it will be busted in TB 68 :-(

Aaron and Jim: Bug 1567614 is affecting TB badly, here and in bug 1572970.

Flags: needinfo?(jmathies)
Flags: needinfo?(aklotz)
Regressed by: 1567614

Which is also on ESR since four days ago. :-(

Yes, but I haven't merged that yet, and I will of course back it out.

Summary: Clicking on links with a reference doesn't work → Clicking on links with a reference doesn't work in a Thunderbird message

This bug might cause problems in Firefox too if URLs are losing references, so we're definitely taking this.

Over to Toshihito for investigation.

Assignee: nobody → tkikuchi
Status: NEW → ASSIGNED
Component: General → Other
Flags: needinfo?(jmathies)
Flags: needinfo?(aklotz)
Product: Thunderbird → External Software Affecting Firefox
Version: 69 → unspecified

[Tracking Requested - why for this release]: Opening external URIs may fail if the URI contains a fragment (ie, the stuff after the #)

68.0.2 gtb is tomorrow, so it's pretty likely to ship with this regression. Tracking for Fx69/68.1esr, however.

Aaron's theory was right. In ShellExecuteByExplorer, PIDL has a reference (a string following #) , but the return value from IShellFolder::GetAttributesOf does not contain the reference.

Can you fix it, this is pretty important (sorry for messing with your flags).

Severity: normal → critical
Priority: -- → P1

Can you fix it, this is pretty important (sorry for messing with your flags).

Of course! Thank you for your patience. I'll be able to start a code review by the end of this week.

Microsoft said this behavior of IShellFolder::GetAttributesOf is expected and no way to get a fragment from PIDL. Our fix plan is to use CreateUri for validation.

As a note, we have this validation because of Bug 394974, where there was discussion about SHParseDisplayName vs CreateUri, and SHParseDisplayName was chosen because it better mimics ShellExecute. I try to keep the current validation logic as much as possible.

For launching with an external protocol handler on Windows, we validate a uri
before sending it to ShellExecute, by converting a string into PIDL using
SHParseDisplayName and extract a string back from PIDL using
IShellFolder::GetDisplayNameOf. The problem was that if a fragment, a
string following a hash mark (#), is always dropped after this validation.
This is caused by the intended design of Windows.

A proposed fix is to use CreateUri for validation, which is used behind
IShellFolder::GetDisplayNameOf. However, we also keep SHParseDisplayName
because there are cases where CreateUri succeeds while SHParseDisplayName
fails such as a non-existent file: uri and we want to keep the same
validation result for those cases.

This patch adds a new unittest to make sure the new validation logic
behaves the same as the old one except the fragment issue.

(In reply to Aaron Klotz [:aklotz] from comment #11)

This bug might cause problems in Firefox too if URLs are losing references, so we're definitely taking this.

Btw, this is true. For example, microsoft-edge:https://bugzilla.mozilla.org/show_bug.cgi?id=1573051#c11 (to launch Edge on Win10 via the custom protocol) does not navigate to the comment.

On Thunderbird, it looks more critical because the issue happens for http/https.

Blocks: 1574209
Summary: Clicking on links with a reference doesn't work in a Thunderbird message → Links with custom URIs to external applications are losing their fragments
Keywords: checkin-needed

We'll look into uplift requests on Monday.

Pushed by rmaries@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/722bc0469e8e
Use both SHParseDisplayName and CreateUri to validate a uri. r=aklotz

Keywords: checkin-needed

I'll look into this as soon as I'm able also.

Flags: needinfo?(tom)
Flags: needinfo?(tkikuchi)

(In reply to Toshihito Kikuchi [:toshi] from comment #23)

Thanks Tom! Here's a try after adding oleaut32 and ole32.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1fbd475953b63a3ca908528671c69f2de5617917

I triggered a x86 build. If I'm correct, the MinGW def file for urlmon needs updating for x64.

(In reply to Tom Ritter [:tjr] from comment #24)

I triggered a x86 build. If I'm correct, the MinGW def file for urlmon needs updating for x64.

Seems you're right! WMC32 has passed. How can I update the MinGW def file for x64?

Pushed by aklotz@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4eca0f08c43b
Use both SHParseDisplayName and CreateUri to validate a uri. r=aklotz
Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Flags: needinfo?(tom)
No longer blocks: 1574209
Duplicate of this bug: 1574209
Flags: qe-verify+
Flags: in-testsuite+

Comment on attachment 9085603 [details]
Bug 1573051 - Use both SHParseDisplayName and CreateUri to validate a uri. r=aklotz

Beta/Release Uplift Approval Request

  • User impact if declined: This is a fix for a functionality regression. Without this patch, a link with an external protocol does not work correctly if it contains a fragment. For example, a link microsoft-edge:https://bugzilla.mozilla.org/show_bug.cgi?id=1573051#c18 should launch Microsoft Edge on Windows 10 and navigate to the comment on Bugzilla, but the fragment "#c18" is dropped due to this bug, thus Edge is not navigated to the comment. On Thunderbird, this issue could be more common because http(s) is considered as an external protocol.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The change is not risky because the impacted scenario is limited, a link uses an external uri and a fragment. We changed the validation logic for a uri in that scenario. This change added a unittest to verify the same result from both of the old/new uri validation logics.
  • String changes made/needed: None

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: This bug is critical for an enterprise customer using an application using custom URIs. We already had Bug 1574209 complaining that SSO was broken due to this bug.
  • User impact if declined: This is a fix for a functionality regression. Without this patch, a link with an external protocol does not work correctly if it contains a fragment. For example, a link microsoft-edge:https://bugzilla.mozilla.org/show_bug.cgi?id=1573051#c18 should launch Microsoft Edge on Windows 10 and navigate to the comment on Bugzilla, but the fragment "#c18" is dropped due to this bug, thus Edge is not navigated to the comment. On Thunderbird, this issue could be more common because http(s) is considered as an external protocol.
  • Fix Landed on Version: 70.0a1
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The change is not risky because the impacted scenario is limited, a link uses an external uri and a fragment. We changed the validation logic for a uri in that scenario. This change added a unittest to verify the same result from both of the old/new uri validation logics.
  • String or UUID changes made by this patch: None
Attachment #9085603 - Flags: approval-mozilla-esr68?
Attachment #9085603 - Flags: approval-mozilla-beta?

Comment on attachment 9085603 [details]
Bug 1573051 - Use both SHParseDisplayName and CreateUri to validate a uri. r=aklotz

Fixes a widely-reported regression from bug 1567614. Approved for 69.0b16 and 68.1esr.

Attachment #9085603 - Flags: approval-mozilla-esr68?
Attachment #9085603 - Flags: approval-mozilla-esr68+
Attachment #9085603 - Flags: approval-mozilla-beta?
Attachment #9085603 - Flags: approval-mozilla-beta+

Backed out changeset 894ca270cd7e (Bug 1573051) for build bustage at build/src/obj-firefox/dist/include/mozilla/UniquePtr.h

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=262636739&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=0a5ba27e416309d1d68b1b2fc539496731d52ae3

Backout link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=ebf7ea8d8848f8b7ac895b550ebbb1dac1ff1529

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=262636745&repo=mozilla-beta&lineNumber=18954

0:49:48     INFO -  In file included from z:/build/build/src/obj-firefox/dist/include/jsapi.h:19:
10:49:48     INFO -  z:/build/build/src/obj-firefox/dist/include\mozilla/Variant.h(542,33): error: call to deleted constructor of 'mozilla::UniquePtr<_ITEMIDLIST, mozilla::CoTaskMemFreeDeleter>'
10:49:48     INFO -      ::new (KnownNotNull, ptr()) T(std::forward<RefT>(aT));
10:49:48     INFO -                                  ^ ~~~~~~~~~~~~~~~~~~~~~~
10:49:48     INFO -  z:/build/build/src/obj-firefox/dist/include\mozilla/Result.h(50,52): note: in instantiation of function template specialization 'mozilla::Variant<mozilla::UniquePtr<_ITEMIDLIST, mozilla::CoTaskMemFreeDeleter>, mozilla::WindowsError>::Variant<const mozilla::UniquePtr<_ITEMIDLIST, mozilla::CoTaskMemFreeDeleter> &, mozilla::UniquePtr<_ITEMIDLIST, mozilla::CoTaskMemFreeDeleter> >' requested here
10:49:48     INFO -    explicit ResultImplementation(const V& aValue) : mStorage(aValue) {}
10:49:48     INFO -                                                     ^
10:49:48     INFO -  z:/build/build/src/obj-firefox/dist/include\mozilla/Result.h(303,42): note: in instantiation of member function 'mozilla::detail::ResultImplementation<mozilla::UniquePtr<_ITEMIDLIST, mozilla::CoTaskMemFreeDeleter>, mozilla::WindowsError, mozilla::detail::PackingStrategy::Variant>::ResultImplementation' requested here
10:49:48     INFO -    MOZ_IMPLICIT Result(const V& aValue) : mImpl(aValue) { MOZ_ASSERT(isOk()); }
10:49:48     INFO -                                           ^
10:49:48     INFO -  z:/build/build/src/obj-firefox/dist/include\mozilla/ShellHeaderOnlyUtils.h(167,10): note: in instantiation of member function 'mozilla::Result<mozilla::UniquePtr<_ITEMIDLIST, mozilla::CoTaskMemFreeDeleter>, mozilla::WindowsError>::Result' requested here
10:49:48     INFO -    return UniqueAbsolutePidl(rawAbsPidl);
10:49:48     INFO -           ^
10:49:48     INFO -  z:/build/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h(329,3): note: 'UniquePtr' has been explicitly marked deleted here
10:49:48     INFO -    UniquePtr(const UniquePtr& aOther) = delete;  // construct using std::move()!
10:49:48     INFO -    ^
10:49:48     INFO -  2 errors generated.
10:49:48     INFO -  z:/build/build/src/config/rules.mk:801: recipe for target 'Unified_cpp_uriloader_exthandler0.i_o' failed
10:49:48     INFO -  mozmake.EXE[5]: *** [Unified_cpp_uriloader_exthandler0.i_o] Error 1
10:49:48     INFO -  mozmake.EXE[5]: Leaving directory 'z:/build/build/src/obj-firefox/uriloader/exthandler'
10:49:48     INFO -  z:/build/build/src/config/recurse.mk:74: recipe for target 'uriloader/exthandler/target' failed
10:49:48     INFO -  mozmake.EXE[4]: *** [uriloader/exthandler/target] Error 2
10:49:48     INFO -  mozmake.EXE[4]: *** Waiting for unfinished jobs....
10:49:48     INFO -  mozmake.EXE[5]: Entering directory 'z:/build/build/src/obj-firefox/mozglue/build'
10:49:48     INFO -  mozglue/build/SSE.i_o
10:49:48     INFO -  mozmake.EXE[5]: Leaving directory 'z:/build/build/src/obj-firefox/mozglue/build'
10:49:49     INFO -  mozmake.EXE[5]: Entering directory 'z:/build/build/src/obj-firefox/mozglue/build'
10:49:49     INFO -  z:/build/build/src/clang/bin/clang.exe --driver-mode=cl -m32 -FoSSE.i_o -c  -Iz:/build/build/src/obj-firefox/dist/stl_wrappers -DNDEBUG=1 -DTRIMMED=1 -DMOZ_HAS_MOZGLUE -DIMPL_MFBT -Iz:/build/build/src/mozglue/build -Iz:/build/build/src/obj-firefox/mozglue/build -Iz:/build/build/src/memory/build -Iz:/build/build/src/obj-firefox/dist/include -Iz:/build/build/src/obj-firefox/dist/include/nspr -Iz:/build/build/src/obj-firefox/dist/include/nss -MD -FI z:/build/build/src/obj-firefox/mozilla-config.h -DMOZILLA_CLIENT -Qunused-arguments -guard:cf -Qunused-arguments -fcrash-diagnostics-dir=z:/build/public/build -TP -nologo -Zc:sizedDealloc- -D_HAS_EXCEPTIONS=0 -guard:cf -W3 -Gy -Zc:inline -arch:SSE2 -Gw -Wno-inline-new-delete -Wno-invalid-offsetof -Wno-microsoft-enum-value -Wno-microsoft-include -Wno-unknown-pragmas -Wno-ignored-pragmas -Wno-deprecated-declarations -Wno-invalid-noreturn -Wno-inconsistent-missing-override -Wno-implicit-exception-spec-mismatch -Wno-unused-local-typedef -Wno-ignored-attributes -Wno-used-but-marked-unused -D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING -GR- -Z7 -Xclang -load -Xclang z:/build/build/src/obj-firefox/build/clang-plugin/clang-plugin.dll -Xclang -add-plugin -Xclang moz-check -O2 -Oy- -WX -DNS_FREE_PERMANENT_DATA=1 /clang:-fprofile-generate -Xclang -MP -Xclang -dependency-file -Xclang .deps/SSE.i_o.pp -Xclang -MT -Xclang SSE.i_o   z:/build/build/src/mozglue/build/SSE.cpp
10:49:49     INFO -  mozmake.EXE[5]: Leaving directory 'z:/build/build/src/obj-firefox/mozglue/build'
10:49:49     INFO -  mozmake.EXE[5]: Entering directory 'z:/build/build/src/obj-firefox/mozglue/build'
10:49:49     INFO -  mozglue/build/UntrustedDllsHandler.i_o
10:49:49     INFO -  mozmake.EXE[5]: Leaving directory 'z:/build/build/src/obj-firefox/mozglue/build'
10:49:49     INFO -  mozmake.EXE[5]: Entering directory 'z:/build/build/src/obj-firefox/parser/html'
Flags: needinfo?(tkikuchi)

The central patch effectively moved the skip-if = os != 'win' from [TestNativeNt] to [TestUriValidation] but Linux Cpp runs don't mention NativeNt, e.g. https://treeherder.mozilla.org/logviewer.html#?job_id=262602075&repo=mozilla-central

Should there be two skip-if = so!= 'win' lines, one for each test? And why is NativeNt not mentioned in the Linux log?

Blocks: tb68found

I cannot reproduce this issue on a build before the fix (Nightly v70.0a1 from 2019-08-18) on Windows 10.

These were my steps:

  1. I sent an email to my primary email with a link containing a reference: "https://bugzilla.mozilla.org/show_bug.cgi?id=1552922#c9"
  2. I set the browser as the default system browser.
  3. I open the email with the link above.
  4. I click the link.
    The issue that should reproduce: The link should redirect to the "https://bugzilla.mozilla.org/show_bug.cgi?id=1552922" page (without the reference).
    What actually happened: The link redirected correctly to the comment 9 of bug 1552922: "https://bugzilla.mozilla.org/show_bug.cgi?id=1552922#c9".

What am I missing from reproducing this bug? Which is the OS it got reproduced on, in the first place?

Thanks!

Flags: needinfo?(jorgk)

(In reply to Daniel Varga [:dvarga] from comment #34)

Backed out changeset 894ca270cd7e (Bug 1573051) for build bustage at build/src/obj-firefox/dist/include/mozilla/UniquePtr.h

Oh, sorry, my fix is depending on a recent change https://hg.mozilla.org/mozilla-central/rev/9bc4747798bc. I think it's too much to port. Let me create a patch for Beta/ESR.

Flags: needinfo?(tkikuchi)

(In reply to Bodea Daniel [:danibodea] from comment #37)

I cannot reproduce this issue on a build before the fix (Nightly v70.0a1 from 2019-08-18) on Windows 10.

Your repro steps look correct. I can repro the issue on Thunderbird of 2019-08-18. Here's an installer I used.
https://ftp.mozilla.org/pub/thunderbird/nightly/2019/08/2019-08-18-07-48-02-comm-central/thunderbird-70.0a1.en-US.win64.installer.exe

Are you using Thunderbird as e-mail client? It doesn't reproduce in webmail in the browser. The problem occurs when TB or FF launch another application. TB launches FF or FF launches Edge or another browser.

(In reply to Jorg K (GMT+2) from comment #40)

Are you using Thunderbird as e-mail client? It doesn't reproduce in webmail in the browser. The problem occurs when TB or FF launch another application. TB launches FF or FF launches Edge or another browser.

The problem can occur on both TB and FF. To repro the issue, you need to use an external protocol. On Firefox, http[s] is not an external protocol which does not repro the issue, while http[s] is an external protocol on Thunderbird. So your repro steps will not reproduce the issue on webmail Firefox. To repro the issue on Firefox, the easiest way is to use a link of microsoft-edge: protocol to launch Edge (or probably mailto: to launch a mail client also works).

Flags: needinfo?(jorgk)

For launching with an external protocol handler on Windows, we validate a uri
before sending it to ShellExecute, by converting a string into PIDL using
SHParseDisplayName and extract a string back from PIDL using
IShellFolder::GetDisplayNameOf. The problem was that if a fragment, a
string following a hash mark (#), is always dropped after this validation.
This is caused by the intended design of Windows.

A proposed fix is to use CreateUri for validation, which is used behind
IShellFolder::GetDisplayNameOf. However, we also keep SHParseDisplayName
because there are cases where CreateUri succeeds while SHParseDisplayName
fails such as a non-existent file: uri and we want to keep the same
validation result for those cases.

Adding CreateUri broke MinGW build because of our toolkit issue. We use
dynamic linking for MinGW build in the meantime.

This patch adds a new unittest to make sure the new validation logic
behaves the same as the old one except the fragment issue.

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #36)

The central patch effectively moved the skip-if = os != 'win' from [TestNativeNt] to [TestUriValidation] but Linux Cpp runs don't mention NativeNt, e.g. https://treeherder.mozilla.org/logviewer.html#?job_id=262602075&repo=mozilla-central

Should there be two skip-if = so!= 'win' lines, one for each test? And why is NativeNt not mentioned in the Linux log?

Good catch! I filed Bug 1575655. I'm not sure why this did not break CPPUnitTest on non-Windows platforms.

I'm adding checkin-needed tag with a new patch D42943 for Beta/ESR. The uplift request has already been approved in the other revision that did not fit Beta/ESR. Please land a newer patch Beta/ESR.

Keywords: checkin-needed

Comment on attachment 9085603 [details]
Bug 1573051 - Use both SHParseDisplayName and CreateUri to validate a uri. r=aklotz

Moving the approvals over to the rebased patch for posterity's sake.

Attachment #9085603 - Flags: approval-mozilla-esr68+
Attachment #9085603 - Flags: approval-mozilla-beta+
Attachment #9087181 - Flags: approval-mozilla-esr68+
Attachment #9087181 - Flags: approval-mozilla-beta+

I have managed to reproduce the issue using TB69.0b3 and Fx 69.0b15.
Please note that while using TB69.0b3 with Fx69.0b16 and Fx68.1.0ESR (where the uplifts were made) the issue still could be reproduced - Firefox would not open the entire link.
Also note that the issue cannot be reproduced with another external program, like libre office - the full link is opened in Fx69.0b15.

Note that the issue is verified fixed only using TB 70.0a1 with either Fx69.0b16, Fx68.1.0ESR, Fx69.0b15 or Fx70.0a1.

Looks like this was a TB issue, not a Firefox issue.

Status: RESOLVED → VERIFIED
Flags: qe-verify+

Looks like this was a TB issue, not a Firefox issue.

As detailed in comment #41, this was a problem in the Mozilla platform code affecting both FF and TB.

I have finally managed to reproduce on Windows 10, but not in Ubuntu 18. I have used Nightly v70.0a1 (2019-08-18) and Thunderbird v70.0a1 (2019-08-18). and used these 2 sets of steps:

  • Set 1:
  1. Open Thunderbird.
  2. Receive an email that has a link with a reference.
  3. Set the browser to test as the system default browser.
  4. Open the received email and click the link with a reference.
    The result that reproduces issue: The link is being opened, but the reference is lost.
    The expected result after the fix: The link is being opened at the referenced paragraph/page anchor.
  • Set 2:
  1. Open browser.
  2. Open test page with a link to be opened in EDGE browser and with a reference.
    The result that reproduces issue: The link is being opened, but the reference is lost.
    The expected result after the fix: The link is being opened at the referenced paragraph/page anchor.

I have verified both the fix for Thunderbird Daily v70.0a1 (2019-08-21) and the one for Firefox Nightly v70.0a1 (2019-08-21), by the 2 test sets.
Furthermore, I have also verified the fix in Firefox Beta v69.0b16 (2019-08-21) and FirefoxESR68 v68.1.0esr (2019-08-21) using the second set.

Duplicate of this bug: 1577350
You need to log in before you can comment on or make changes to this bug.