1483193 - Weather.com website cannot be loaded: "The page isn't redirecting properly" in EU countries

Status: NEW

(NSPR :: NSPR, defect, P2)

Description

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Attachment: Weather.com cannot be loaded.JPG

[Affected versions]:
-62.0b17, 63.0a1 (2018-08-13)

[Affected platforms]:
- Windows 10 x64, Windows 8.1 x64

[Steps to reproduce]:
1.Launch FF.
2. Go to weather.com

[Expected result]:
-the page is loaded without any issues.

[Actual result]:
-Page is loading a "The page isn't redirecting properly" error

[Regression range]:
-https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=23c2ec5544b9e0a74a047b87b594e4c36a8fe95c&tochange=f97a056ae6235de7855fd8aaa04fb1c8d183bd06

[Additional notes]:
-No errors are displayed inside the DevTools Console
-This issue is not reproducible on macOS or Ubuntu platforms.
-This issue is reproducible on both x32 and x64 builds.
-This issue reproduces even with fresh profiles.

Comment 1

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

After further investigation, we have discovered that this issue is reproducible on certain builds of Windows:

This issue is reproducible with 100% repro rate on:

-Windows 8.1 x64 - 6.3 Build 9600
-Windows 10  x64 - Version 1709 (16299.431)

This issue is NOT reproducible on :

-Windows 10 x64 - Version 1803 (17134.165)
-Windows 10 x64 - Version 1803 (17134.191)

Another thing worth mentioning, is that this issue does not reproduce on the affected platforms with longer url's from the same domain. Example:

https://weather.com/weather/radar/interactive/l/USOH1375:1:US?layer=radar&overlays=currtrafficspeed

Comment 2

[Chris Peterson [:cpeterson]]

(In reply to Vlad Lucaci (:vlucaci) from comment #0)
> [Regression range]:
> -https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=23c2ec5544b9e0a74a047b87b594e4c36a8fe95c&tochange=f97a056ae6235de7855fd8aaa04fb1c8d183bd06

The regression range is surprising because it points to Firefox 51 from 2016. I don't see any obvious suspects in the regression range.

Comment 3

[Chris Peterson [:cpeterson]]

Honza, I think this is a networking bug that only affects some Windows versions. How do we get this bug reviewed by the Networking team's bug triage?

Vlad tested weather.com using a VPN and found that, on Windows 8.1 and Windows 10 version 1709 (build 16299.431):

1. weather.com loads correctly when VPN'ing from Romania into the UK or US.
2. weather.com shows a "The page isn't redirecting properly" error when VPN'ing from Romania into Sweden, Germany, France, Netherlands, Switzerland, Italy, or Romania.

weather.com loads correctly when using Windows 10 version 1803 (build 17134.165) or version 1803 (build 17134.191).

Comment 4

[Romain Testard [:RT]]

For info I tested on High Sierra 10.13.6 and reproduced the issue with Firefox 61.0.2 from France
Screencap: http://recordit.co/1AvGPThfKT
https://weather.com/weather/today/l/48.86,2.35?par=google loads fine but https://weather.com does not

Comment 5

[Honza Bambas (:mayhemer)]

Sorry for late reply here.

I will ask anyone able to reproduce - please provide logs:
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

don't forget to note the URL that is failing for you.  the more details the better.  thanks.

Comment 6

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello Honza,

The URL that is failing is : weather.com

This issue is reproducible with 100% repro rate on:

-Windows 8.1 x64 - 6.3 Build 9600
-Windows 10  x64 - Version 1709 (16299.431)
-macOS 10.13.6

This issue is NOT reproducible on :

-Windows 10 x64 - Version 1803 (17134.165)
-Windows 10 x64 - Version 1803 (17134.191)

Another thing worth mentioning, is that this issue does not reproduce on the affected platforms with longer url's from the same domain. Example:

https://weather.com/weather/radar/interactive/l/USOH1375:1:US?layer=radar&overlays=currtrafficspeed

Attached you can find the debugging logs:
https://drive.google.com/file/d/1YB65HKqWYE3EZc4yoMO9I31BborI3qKz/view?usp=sharing

Comment 7

[Honza Bambas (:mayhemer)]

This is a cookie timing issue and it's a server side issue -> invalid

Not locally reproducible for me.

I can see the following in the provided (failing) log:
- navigation to http://weather.com/, no cached entry -> 301 Moved Permanently, Location: https://weather.com/
- redirecting to https://weather.com/, no cached entry -> 302 Found, location: https://weather.com/
- redirecting again to https://weather.com/ (same url), having an existing cache entry with no-cache, hence revalidating with the server -> 302 Found, location: https://weather.com/

And since then we just loop.

This is a cookie bug, I can see Bug 368964 in the regression range, and I can also see a difference in cookies and their expiration timings we get from the server (or a balancer) and that we then send back on the second redirect to https version of the page:



* Good case (no loop, local attempt log):
2018-08-23 15:56:30.725000 UTC - [Parent 8576: Socket Thread]: I/nsHttp http response [
  date: Thu, 23 Aug 2018 14:42:04 GMT
  set-cookie: speedpin=4G; expires=Thu, 23-Aug-2018 16:26:30 GMT; path=/; domain=.weather.com; secure
  set-cookie: Goto=Redirected; expires=Thu, 23-Aug-2018 15:56:35 GMT; path=/; domain=.weather.com; secure
  set-cookie: o=deleted; path=/; domain=.weather.com; secure
  set-cookie: ci=TWC-Locale-Group=US&X-Origin-Hint=Goto-Prod&TWC-GeoIP-Country=CZ&TWC-Privacy=gdpr; path=/; domain=.weather.com; secure

we then send
  Cookie: speedpin=4G; Goto=Redirected; o=deleted; ci=TWC-Locale-Group=US&X-Origin-Hint=Goto-Prod&TWC-GeoIP-Country=CZ&TWC-Privacy=gdpr


>>> 2018-08-23 15:56:30.725000 UTC < 23-Aug-2018 15:56:35 GMT



* Broken (your log):
2018-08-23 15:42:05.268000 UTC - [2600:Socket Thread]: I/nsHttp http response [
  date: Thu, 23 Aug 2018 15:56:30 GMT
  set-cookie: speedpin=4G; expires=Thu, 23-Aug-2018 15:12:04 GMT; path=/; domain=.weather.com; secure
  set-cookie: Goto=Redirected; expires=Thu, 23-Aug-2018 14:42:09 GMT; path=/; domain=.weather.com; secure
  set-cookie: o=deleted; path=/; domain=.weather.com; secure
  set-cookie: ci=TWC-Locale-Group=US&X-Origin-Hint=Goto-Prod&TWC-GeoIP-Country=RO&TWC-Privacy=gdpr; path=/; domain=.weather.com; secure

we send back
  Cookie: o=deleted; ci=TWC-Locale-Group=US&X-Origin-Hint=Goto-Prod&TWC-GeoIP-Country=RO&TWC-Privacy=gdpr


>>> 2018-08-23 15:42:05.268000 UTC > 23-Aug-2018 14:42:09 GMT  <--- we expire the "Goto=Redirected" cookie and don't send it back


Note that the Date response header is in the past relative to the actual time for the good case and in the future for the broken case.  The server/balancer(s) has(ve) clearly time sync issues.

Comment 8

[Chris Peterson [:cpeterson]]

Is this redirect problem reproducible in Chrome and Edge? If not, is weather.com serving them different content or do they handle cookie expiration differently?

If the site didn't work in any browser in Sweden, Germany, France, Netherlands, Switzerland, Italy, or Romania, you would think someone at weather.com would notice. :)

Should we reach out to weather.com?

Comment 9

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

:cpeterson 

This issue does not reproduce on: 
Edge 41.16299,
Internet Explorer 11.16299, 
Google Chrome 68.0 
on the same affected machine on which FF does not work.

The content seems to be the same one on all browsers.

Comment 10

[Honza Bambas (:mayhemer)]

(In reply to Vlad Lucaci (:vlucaci) from comment #9)
> :cpeterson 
> 
> This issue does not reproduce on: 
> Edge 41.16299,
> Internet Explorer 11.16299, 
> Google Chrome 68.0 
> on the same affected machine on which FF does not work.
> 
> The content seems to be the same one on all browsers.

Can you please check the date, set-cookie response headers and cookie request headers in those browsers and post them here?  I can't repro the issue.  If the timing issue is there as well and they treat it differently than us, we may want to back bug 368964 out or do further workarounds.

cc'ing Ehsan who reviewed that patch.

Comment 11

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello Honza

I have accessed weather.com on I Edge 41 and Chrome 68 and pulled a HAR log file for both browsers when accessing the website. Hope it helps:

https://drive.google.com/file/d/1OY--hvdbkLTSKIvvVVUnaEwJQ0HM_Br8/view?usp=sharing
https://drive.google.com/file/d/1o08Ly1GD9UIHhEc3_PYVFAruiAlYlrKE/view?usp=sharing

Comment 12

[Honza Bambas (:mayhemer)]

(In reply to Vlad Lucaci (:vlucaci) from comment #11)
> Hello Honza
> 
> I have accessed weather.com on I Edge 41 and Chrome 68 and pulled a HAR log
> file for both browsers when accessing the website. Hope it helps:
> 
> https://drive.google.com/file/d/1OY--hvdbkLTSKIvvVVUnaEwJQ0HM_Br8/
> view?usp=sharing
> https://drive.google.com/file/d/1o08Ly1GD9UIHhEc3_PYVFAruiAlYlrKE/
> view?usp=sharing

Thanks.

The Edge HAR can't be loaded on [1].

Can you please tell me at what EXACT time (GMT or your local) have you captured these logs?

Next time would be good to provide Chrome and Firefox HARs from nearly the same time, for compare.

Other thing I would like you to try is to change [2] Chrome's UA to Firerox UA for which you can reproduce the problem.  (UA = the User-Agent request header)

Thanks!


[1] https://toolbox.googleapps.com/apps/har_analyzer/
[2] https://winaero.com/blog/change-user-agent-chrome/

Comment 13

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello Honza,

I have re-taken the HARs and this time I will also provide time logs:

Chrome Archive: https://drive.google.com/open?id=1J6DA8Ic8qOQXeP-AVybsH7Qq8dRAEEZ1
Accessed at 17:18:20 Romania Time

Mozilla Archive: https://drive.google.com/open?id=1IrMP53nurpebCTaUktxc0THSMB0kHhxT
Accessed at 17:18:40 Romania Time

Chrome's UA to FF UA: https://drive.google.com/open?id=1JSLWcLRLDbPbn0spoUVnzdAFAI0W420w
Accessed at 3:04 Romania Time

Worth mentioning that for the latter, changing the UA of Chrome to FF loaded weather.com without any issues .

Hope it helps.

Comment 14

[Honza Bambas (:mayhemer)]

(In reply to Vlad Lucaci (:vlucaci) from comment #13)
> Chrome's UA to FF UA:
> https://drive.google.com/open?id=1JSLWcLRLDbPbn0spoUVnzdAFAI0W420w
> Accessed at 3:04 Romania Time

Are you sure about the time?

Comment 15

[Honza Bambas (:mayhemer)]

Another discrepancy I've found is that you use for UA simulation a wrong header value:

Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0

It has to be:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0


The reason this may be a cause you still don't see the bug is that the Firefox behavior change (bug 368964, 51) that they may be trying to workaround will only trigger for version 51+.  You are claiming to be 46.

There is definitely some UA sniffing involved because Firefox receive a different set of cookies, as well as Chrome when claiming to be Firefox (the speedpin=4G cookie)

Can you please retest Chrome with the correct Firefox UA and submit a HAR again?

Comment 16

[Honza Bambas (:mayhemer)]

Chrome Archive: https://drive.google.com/open?id=1J6DA8Ic8qOQXeP-AVybsH7Qq8dRAEEZ1

request made at 14:18:20.921
date Thu, 30 Aug 2018 14:18:19 GMT

set-cookie Goto=Redirected; expires=Thu, 30-Aug-2018 14:18:24 GMT; path=/; domain=.weather.com; secure

==> the cookie expires in 5 secs in the future.

------------------

Mozilla Archive: https://drive.google.com/open?id=1IrMP53nurpebCTaUktxc0THSMB0kHhxT

request made at 14:18:27.497
date Thu, 30 Aug 2018 14:18:26 GMT

set-cookie speedpin=4G; expires=Thu, 30-Aug-2018 14:48:26 GMT; path=/; domain=.weather.com; secure
set-cookie Goto=Redirected; expires=Thu, 30-Aug-2018 14:18:31 GMT; path=/; domain=.weather.com; secure

==> the cookie expires in 5 secs in the future.  *we calculate the expiration wrong!*

------------------

Chrome's UA to FF UA: https://drive.google.com/open?id=1JSLWcLRLDbPbn0spoUVnzdAFAI0W420w

request made at 12:30:13.479
date Thu, 30 Aug 2018 12:30:12 GMT

set-cookie speedpin=4G; expires=Thu, 30-Aug-2018 13:00:12 GMT; path=/; domain=.weather.com; secure
set-cookie Goto=Redirected; expires=Thu, 30-Aug-2018 12:30:17 GMT; path=/; domain=.weather.com; secure

==> same as before.



Vlad, please disregard my previous comment.  Thanks for the logs, looking more carefully inside them seems like a confirmation we have a bug.



Ehsan, the problem is following (all times UTC):

- we send a request at 30 Aug 2018 14:18:27.497 to https://weather.com/

- we get the following response + headers:

302 Found
location https://weather.com/ (the same URL)
date: Thu, 30 Aug 2018 14:18:26 GMT
set-cookie: speedpin=4G; expires=Thu, 30-Aug-2018 14:48:26 GMT; path=/; domain=.weather.com; secure
set-cookie: Goto=Redirected; expires=Thu, 30-Aug-2018 14:18:31 GMT; path=/; domain=.weather.com; secure
set-cookie: o=deleted; path=/; domain=.weather.com; secure
set-cookie: ci=TWC-Locale-Group=US&X-Origin-Hint=Goto-Prod&TWC-GeoIP-Country=RO&TWC-Privacy=gdpr; path=/; domain=.weather.com; secure

- we do another request (redirected) to https://weather.com/ (same URL) at 15:18:27.671, with headers:
Cookie o=deleted; ci=TWC-Locale-Group=US&X-Origin-Hint=Goto-Prod&TWC-GeoIP-Country=RO&TWC-Privacy=gdpr

apparently, we treat the Goto=Redirected cookie as expired, even though it's expiration time against the request time and the date: header value is 5 seconds in the future.

other issue could be we discard the cookies with expiration.

note that this is strongly dependent on the timezone.  I can't reproduce in Prague (gmt+2) but Vlad can repro in Romania (gmt+3)

- we get:

302 Found and a loop starts because we never send back the Goto=Redirected cookie

Comment 17

[Honza Bambas (:mayhemer)]

Vlad, can you please produce whatever MOZ_LOG and look at the timestamp and compare against the local time setting of the machine?


It somewhat looks like DST or time zone is badly handled here (probably by us?).  Looking again into the log from comment 6, PR_Now used for writing timestamps to the log (with params {GMT offset=0,DST offset=0}) says 15:42:05 (UTC) but the set-cookie header expiration is 14:42:09 GMT.  Hence, it may appear to us as in the past.  the date: resp header is shifted the same way.

With a local test on Win10 I can see:
- PR_Now() writes to the log UTC (16:23:41) while my local time is UTC+2, all of it is correct (PRG is UTC+1 and we are on the summer time, hence also DST+1)
- the server responses with:
 1) Goto=Redirected; expires=Mon, 03-Sep-2018 16:23:45 GMT, which is the correct time
 2) date: Mon, 03 Sep 2018 16:23:40 GMT, also correct

In the log from comment 6, the server dates are shifted by one hour to the past against what PR_Now() returns.

Why PR_Now?  Because that is what we are using [1] to compare cookie absolute expiration time.

This could also be a clue to why only some (older platforms) manifest this issue.  There could be a system API bug wrongly returning the UTC time.

[1] https://searchfox.org/mozilla-central/rev/721842eed881c7fcdccb9ec0fe79e4e6d4e46604/netwerk/cookie/nsCookieService.cpp#3450
[2] https://searchfox.org/mozilla-central/rev/721842eed881c7fcdccb9ec0fe79e4e6d4e46604/nsprpub/pr/src/md/windows/ntmisc.c#323-325

Comment 18

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello Honza,

I have produced another log and compared the timestamp from the log and the local time setting of the machine.

I have noticed that the log entries were generated at 12:06 UTC while on the local machine I have UTC+02:00-Athens,Bucharest (14:06PM)so your suspicion seem to check out. 

https://drive.google.com/open?id=15aXlIsNsNR8NcXJTmoLax7hMcWn1ZtSQ

Comment 19

[(no longer active)]

I think Honza's suspicion is completely correct!  That code should really be using GMT time.  It is astonishing that we have not discovered this problem so far...

Comment 20

[u408661]

Honza, you seem to understand what's going on here & the urgency pretty well. Can you triage appropriately?

Comment 21

[Honza Bambas (:mayhemer)]

According comment 1 (thanks for it, Vlad!) this may have been a Windows API bug that has been recently fixed.  

I somewhat tend to close this again as WONTFIX as this has been fixed in Win10 1803.  Users should update their machines, unless this manifests on the non-targeted windows update branches (for organizations).

Vlad, do can you test this on non-targeted Windows branches?


Something tells me to work this around would mean to 1) really understand the bug, 2) likely to detect Windows version (range(s)?) and use some kind of a correction.

OTOH, some assessment of what the hell is/was going on here may be appropriate, just in case.

P2, -> me.

Comment 22

[Honza Bambas (:mayhemer)]

comment 21

Comment 23

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello Honza,

Unfortunately I am unable to test this issue on a non-targeted windows branch because we do not own such a work station.

However, we must also take into account that a large majority of users avoid updating the Windows version to 1709 from 1803 due to different performance/security reasons and may encounter this issue.

Comment 24

[David Balažic]

An info point: I have this issue under Windows 7 Enterprise SP1 64 bit

Affected:
 Firefox 62.0.2 (64 bit)
 IE 11

Not affected:
Chrome Version 69.0.3497.100 (Official Build) (64-bit)


FF and IE fail to load weather.com, but do load the longer URL from comment 1

Chrome loads both.

Location: Slovenia

Comment 25

[Pascal Chevrel:pascalc]

Jan, any progress on this one? Should we reprioritize since this is marked as a P2 but we have shipped multiple versions of Firefox with this bug? Thanks

Comment 26

[Honza Bambas (:mayhemer)]

No progress, sorry.  This is not that much a Firefox but a bug in Windows API regarding wall clock, it's been fixed on newer version of the OS.

I won't have time to look deeply into it, even though we still may support that Windows version.  Also, this is more a NSPR bug than a Necko bug, actually.

Comment 27

[Liz Henry (:lizzard) (relman/hg->git project)]

Too late to fix in 64. Marking this issue as fix-optional for 65; if you land a patch in nightly and think it's low-risk for beta, please request uplift.

Comment 28

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:KaiE, could you have a look please?

For more information, please visit BugBot documentation.

Comment 30

[Kai Engert [:KaiE:]]

I don't understand what solution is suggested.

AFAIU comment 17 suggested that
https://searchfox.org/mozilla-central/rev/721842eed881c7fcdccb9ec0fe79e4e6d4e46604/nsprpub/pr/src/md/windows/ntmisc.c#323-325
should be changed to use UTC.

But according to
https://learn.microsoft.com/en-us/windows/win32/api/sysinfoapi/nf-sysinfoapi-getsystemtime
GetSystemTime does use UTC.

If anyone sees clear what exactly needs to be done, please explain.