Closed Bug 1788684 Opened 2 years ago Closed 2 years ago

Access to router.asus.com broken as of version 104

Categories

(Core :: Networking: DNS, defect, P3)

Product:
Core
Component:
Networking: DNS
Version:
Firefox 104
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
107 Branch
Tracking Flags:
Tracking Status
firefox-esr102 --- verified
firefox105 --- verified
firefox106 --- verified
firefox107 --- verified

People

(Reporter: andyross, Assigned: RyanVM, NeedInfo)

References

Details

(Whiteboard: [necko-triaged])

Attachments

(2 files, 1 obsolete file)

Bug 1788684 - Remove asus.com from HSTS preload list. r=dveditz
48 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-release+
Details | Review
Backed out changeset c7514250b068 (bug 1788684)
48 bytes, text/x-phabricator-request
Details | Review

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0

Steps to reproduce:

Attempt to access the router's web pages at http://router.asus.com

Actual results:

It appears to force https://router.assus.com, which fails. No changes to HTTPS mode settings were made before the 104 update. There is a thread about this here:
https://www.snbforums.com/threads/router-asus-com-vs-192-168-50-1.80629/

Expected results:

It should access the routers web interface. This affects both desktop and Android. Using 192.168.1.1 does work.

The Bugbug bot thinks this bug should belong to the 'Core::Widget: Win32' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Widget: Win32
Product: Firefox → Core

Could you run mozregression to see when this started happening?

A number of Firefox versions will open in succession to narrow down when this started occurring. Simply answer "good" or "bad" based on whether or not a build reproduces the bug. Once finished, please post the output from the last run. It should give a last good and first bad revision as well as a link to look at the changesets in that range. Thank you!

Severity: -- → S3
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(andyross)
Priority: -- → P3

All of the regression test versions worked. I tested 103-104. Not sure if it also covers 104.0.1.

This is the end of the log file:
2022-09-01T10:00:34.527000: DEBUG : Found commit message:
Bug 1778184 - Allow QuickAction commands to be translated. r=flod,adw,fluent-reviewers

Differential Revision: https://phabricator.services.mozilla.com/D152194

2022-09-01T10:00:34.527000: DEBUG : Did not find a branch, checking all integration branches
2022-09-01T10:00:34.527000: INFO : The bisection is done.
2022-09-01T10:00:34.527000: INFO : Stopped

Flags: needinfo?(andyross)

Then this indicates that there is a local configuration that might interfere with this. Can you confirm that this works in Troubleshoot Mode (Help > Troubleshoot Mode...)?

Flags: needinfo?(andyross)

Still doesn't work in Troubleshoot mode. As noted, my Android Firefox also doesn't work, and that has no customization compared to the desktop.

Flags: needinfo?(andyross)

Some have reported issues with Edge, too, if it's forcing https. The router intercepts that particular name and gives it's local address. Either 192.168.1.1 or 192.168.50.1. It bypasses normal DNS. There have been no updates to firmware in awhile. I have checked my Firefox settings, and HTTPS only and DoH are OFF.
https://www.dslreports.com/forum/r33484605-Firefox-Firefox-104-0-1-released

More info: If I try to type in http://router.asus.com, the failed connection screen shows "https://" for some reason. HTTPS always mode is OFF. The 192.168.1.1 or 192.168.50.1 address depends on the model router being used. Directly using the IP address will have the small notification about being insecure on both Firefox and MS Edge. Even using the URL in Edge uses plain http and has the warning note on the address bar.

There are a few similar-sounding bugs in the Networking: DNS component. Reassigning for further triage.

Component: Widget: Win32 → Networking: DNS

I have this issue, too. It says "Unable to connect

An error occurred during a connection to router.asus.com.

The site could be temporarily unavailable or too busy. Try again in a few moments.
If you are unable to load any pages, check your computer’s network connection.
If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web."

and it spontaneously changes the prefix of the URL from http to https.

This works fine in Chromium (I'm on Ubuntu 22.04.1 LTS), and my version of Firefox is 104.0 (64-bit)

and my HTTPS-only mode is off. I just discovered, though, that http://192.168.50.1 works! Yay!

Sorry, I meant router.asus.com does not work for me in Firefox 104.0, but 192.168.50.1 seems to do the same thing.

This bug is caused by https://hg.mozilla.org/integration/autoland/rev/c68760840d71a54f3ffea60824e245decce580da which added asus.com to the preloaded HSTS list. That means the load will always attempt to use HTTPS.

@RyanVM - presumably it's asus that requested to be included in the pinned list? ( Through https://hstspreload.org/? )
What should we do now? This is breaking their own clients' experience.

Flags: needinfo?(ryanvm)

Hi Andrew, can you check if you can access your router by using this URL instead? https://router.asus.com:8443

Flags: needinfo?(andyross)

Still get the same error.

Flags: needinfo?(andyross)

In the router's settings, there is an option to enable HTTPS for local access. I'm a bit hesitant to try that right now as I'm not sure if that might cause other problems. As it is, that usually causes a local certificate which drives browsers crazy with warnings and outright blocks.

Right, it seems that option has to be turned on first for it to work.
I've reached out to customer support at asus.com - though I'm not sure how long that will take for them to respond.

Whiteboard: [necko-triaged]

Is there a way to override the forced HTTPS without also enabling HTTPS only option?

I think you can disable the preload list by setting network.stricttransportsecurity.preloadlist=false in about:config

That does work and restores the normal access. For now, I will keep it enabled as the direct IP works and the list probably provides more security than worrying about a minor issue.

Looking closer at the list, it looks like "asus.com" is added, probably for their own website, but ends up forcing all subdomains, too. I'm not sure of Firefox or the list itself has the ability to exclude a subdomain like router.asus.com.

(In reply to Valentin Gosu [:valentin] (he/him) from comment #12)

@RyanVM - presumably it's asus that requested to be included in the pinned list? ( Through https://hstspreload.org/? )
What should we do now? This is breaking their own clients' experience.

We use Google's HSTS list. Not sure on the exact mechanics involved with how sites get added to it.
https://searchfox.org/mozilla-central/source/taskcluster/docker/periodic-updates/scripts/getHSTSPreloadList.js#20

Dana would know better.

Flags: needinfo?(ryanvm) → needinfo?(dkeeler)

Google's HSTS list includes this entry:

{ "name": "asus.com", "policy": "bulk-1-year", "mode": "force-https", "include_subdomains": true },

which indicates that Asus (or someone on their behalf) requested "asus.com" and all subdomains to be preloaded as HSTS in Chrome. Indeed, asus.com responds with the header:

strict-transport-security: max-age=31536000; includeSubDomains; preload

which achieves the same effect.
So either Asus needs to get removed from the list (and they need to change the header they send) or they need to use a different domain hierarchy for people to talk to their routers.

Flags: needinfo?(dkeeler)

I got a response from ASUS support:

Thank you for details. We have managed to reproduce on some router models.
Here in Nordic office we cannot directly offer solution to such issue, We have passed this on to our HQ to resolve.
Solution/action plan unfortunately will be delayd due to Mid-Autum festival.

Would it make sense to remove the asus.com entry from nsSTSPreloadList.inc and respin the 105 RCs as a short-term workaround? We don't automatically update the pinning lists on mozilla-release, so it shouldn't get clobbered if we do.

Flags: needinfo?(dkeeler)

We could. That wouldn't help people who have visited asus.com itself, though.

Flags: needinfo?(dkeeler)

This is very-much a band-aid fix and won't prevent users from hitting the
bug if they visit asus.com (until they fix the header being served). But
there's not a lot of risk to doing this as a short-term fix.

Assignee: nobody → ryanvm
Status: NEW → ASSIGNED

Comment on attachment 9294917 [details]
Bug 1788684 - Remove asus.com from HSTS preload list. r=dveditz

Approved for 105.0 RC2.

For those following at home, we're going to respin the RC build for Firefox 105 with asus.com removed from the HSTS preload list that's baked in to the browser. This will hopefully be enough for most users to avoid hitting the problem for now. That said, until asus.com stops serving the header from comment 22 from their website, this is prone to re-breaking in the future any time you visit asus.com. Also, for affected users running Firefox 105 after its release, you may need to clear all site data for asus.com to get rid of the buggy HSTS entry.
https://support.mozilla.org/kb/clear-cookies-and-site-data-firefox

Attachment #9294917 - Flags: approval-mozilla-release+

I tested the issue on Fenix 105.1.0 and Focus 105.1.0 and the router web interface can be accessed by going to http://router.asus.com/.
Routers used:

  • RT-N12+ 3-in-1 Router Wireless-N300
  • AC1200 Dual Band RT-AC1200

Devices used:

  • Google Pixel 6 (Android 13)
  • Huawei MediaPad M3 (Android 7)

Hello,

I have reproduced this issue on 106.0a1(20220901215206) using a macOS 12.6.

Confirming this issue as verified fixed on 105.0-build2 (20220915150737) using macOS 12.6, Windows11 and Ubuntu22 and an Asus RT-AC65P router.

However , this issue is still reproducible on the latest Nightly 106.0a1(20220915171049) with the same repro steps.

status-firefox105: fixed → verified
status-firefox106: --- → affected

I have reproduced this issue on 106.0a1(20220901215206)

Yes, it's expected that Nightly still has the problem: this bug is not in a FIXED state yet. We did a manual tweak and re-spin to fix the Firefox 105 we're about to ship as an urgent measure. For our ongoing builds we need to fix the preload import script first or it will overwrite any manual changes like we did for 105.

This is very-much a band-aid fix and won't prevent users from hitting the bug if they visit asus.com (until they fix the header being served).

If people do a search to get there (e.g. type "asus" in the awesomebar) they'll end up on www.asus.com and be OK. If they type "asus.com" in the awesomebar then they'll get the problematic HSTS before it redirects to www.asus.com. Let's hope more people do the former. As noted, once it's no longer in the preload list people can unbreak themselves through "forget about this site". That is way way better than advising people to turn off preloads!

Pushed by jcristau@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/c7514250b068 exclude asus.com from HSTS preload list. r=dveditz DONTBUILD

Comment on attachment 9294922 [details]
Bug 1788684 - exclude asus.com from HSTS preload list. r?keeler,dveditz

Beta/Release Uplift Approval Request

  • User impact if declined: asus.com is included on bi-weekly hsts preload list refreshes, locking people out of their router's admin interface, unless we keep manually removing it
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): pretty straightforward change to the hsts preload list's update script
  • String changes made/needed:
  • Is Android affected?: Unknown

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: see beta request
  • User impact if declined:
  • Fix Landed on Version: 107
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky):
Attachment #9294922 - Flags: approval-mozilla-esr102?
Attachment #9294922 - Flags: approval-mozilla-beta?
Attachment #9294917 - Flags: approval-mozilla-esr102?
Attachment #9294917 - Flags: approval-mozilla-esr102?

https://hg.mozilla.org/mozilla-central/rev/c7514250b068

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox107: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 107 Branch

Comment on attachment 9294922 [details]
Bug 1788684 - exclude asus.com from HSTS preload list. r?keeler,dveditz

Approved for 106.0b3, thanks.

Attachment #9294922 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Comment on attachment 9294922 [details]
Bug 1788684 - exclude asus.com from HSTS preload list. r?keeler,dveditz

Approved for ESR102.

Attachment #9294922 - Flags: approval-mozilla-esr102? → approval-mozilla-esr102+

Confirmed that today's pinning updates for Nightly and Beta dropped the asus.com entries, so we should be all set for avoiding re-introducing this problem in 106+ until Asus properly reverts this change on their end.

Not sure just how that new code for the special exclusion to the list works, but can it be limited to just router.asus.com, or must it always be the base domain?

No, all our workaround is doing is filtering out the asus.com entry from the upstream list when we pull down updates.

Looking at https://hstspreload.org/?domain=asus.com, I wonder if they've actually removed the broken header already, though.

Hello,

Confirming this issue as verified fixed.

Verification was done on the following builds and configurations:

  • macOS 12, Ubuntu 22, Windows 11 with 107.0a1(20220922214429) and 106.0b3(20220922185808) [Asus RT-AC65P]

However , this issue still occurs with the same repro steps on the Treeherder build 102.4.0esr(20220921185845) for macOS 12, Ubuntu 22, Windows 11[Asus RT-AC65P]

status-firefox106: fixed → verified
status-firefox107: fixed → verified

Hi all, we received confirmation from Asus Security to proceed with manually removing their HSTS preload list entry at this time, and landed that removal in https://chromium-review.googlesource.com/c/chromium/src/+/3910704. We're merging that removal back to Chrome 106 and 107 ahead of releases next week to hopefully prevent any breakage for Chrome clients. For downstream consumers of the HSTS preload list this means that you should be able to stop manually patching out this entry soon, once you pull in the latest version.

Asus will separately stop serving the dynamic HSTS header soon; meanwhile there may still be some clients that visit the bare asus.com domain and pick up the dynamic entry. Such clients may continue to have issues with the router subdomain, but from what I've read this case seems rare (which may be why it didn't serve as an early breakage signal before Asus requested preloading).

asus.com is no longer returning a HSTS header with includeSubdomains so
we don't need to override the check anymore.

A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)

Backout by rvandermeulen@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/11670770f994 Backed out changeset c7514250b068 r=RyanVM
Attachment #9294922 - Attachment is obsolete: true
Flags: qe-verify+

Hello,

Confirming this issue as verified fixed.

Verification was done on the following build and configurations:

macOS 12, Ubuntu 22, Windows 11 with 102.4.0esr(20221010114559) and an Asus RT-AC65P router.

Status: RESOLVED → VERIFIED
status-firefox-esr102: fixed → verified
Flags: qe-verify+

Sadly, this has come back with Firefox Android 123.1. It is forcing https again. No problems with Edge or Chrome. FF 123.0.1 on Windows still works. The alternative addresses for the router do work with Android 123.1.

It appears that a request to https://asus.com returns a response with strict-transport-security: max-age=31536000; includeSubDomains
That means requests too all subdomains will be treated as requiring STS.
I suggest:

  • Filing an issue with asus
  • You can possibly bypass this issue, by going to asus.com, and using Forget this site to clear the HSTS cache. I haven't tested it but it was suggested here.

Hope this helps.

Flags: needinfo?(andyross)

Chris, do you still have a contact at Asus? It would be great if they could stop doing this (again).

Flags: needinfo?(cthomp)

I can ping our thread with them. We initially just reached out to their security@asus.com, which worked well. They at least aren't including the preload directive, so this isn't currently at risk of getting re-added to the preload list.

Flags: needinfo?(cthomp)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: