1575934 - Application cache regression in version 67.0

Status: VERIFIED FIXED

(Core :: Networking: HTTP, defect, P3)

Description

[chuckd]

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36

Steps to reproduce:

Set up Offline mail in IBM Verse (https://wonka-01.cwp.pnp-hcl.com/verse). Contact [e-mail] for credentials to a test user account.

1.) Login in with credentials provided separately
2.) Click on user name in top right corner and select Setup Offline Mail
3.) Enter Password and Confirm password fields and click on Set up

Actual results:

Get Error message saying Failed to download application files....

At the Appcache API level, the listener receives the following appcache events
appcache - online: yes, event: checking, status: uncached
appcache - online: yes, event: error, status: uncached

This is a regression that was introduced in Firefox 67.0. The problem reproduces in the latest version of Firefox (68.0.2) as well as the latest beta release (69.0b15). The problem does not reproduce in versions of Firefox prior to 67.0, nor in current version of other browsers including Chrome, Safari, IE and Edge.

I'm a developer on the IBM Verse product and am reaching out to Mozilla support because we are unable to come up with a workaround for this issue.

Expected results:

Offline setup should have completed successfully.

Comment 1

[chuckd]

Some more details on the implementation. When Offline Setup is performed, we load the app into a off-screen child iframe. A cookie is set which tells the server to add the manifest attribute to the html element so that the browser will load the appcache. The appcache is loaded in the iframe, and appcache events that occur in the iframe are posted to the parent window. The parent window monitors the progress of the appcache and reports errors or completion to the user. The iframe url is the same as the parent url so that the appcache resources will be used when the parent is reloaded as well. The appcache error event is occurring during the appcache load of the child iframe.

Comment 2

[Florencia Di Ciocco, NI to vbalducci]

Hi Charles,

Thanks for providing me with the credentials and for the details when opening the bug. I was able to reproduce the bug on Windows 10 on the following versions:

release 68.0.2 (64-bit), nightly 70.0a1 (2019-08-26) (64-bit) and Beta 69.0b16 (64-bit).

I've chosen a component. If you consider that there's another component that's more proper for this case you may change it.

Best regards, Flor.

Comment 3

[Marco Bonardo [:mak]]

I think AppCache bugs are tracked in networking. Though note https://www.fxsitecompat.dev/en-CA/docs/2016/application-cache-support-will-be-removed/

Comment 4

[Honza Bambas (:mayhemer)]

This could be a regression from bug 1525900. AFAIK we are still running tests for appcache, so I'm curious how we could miss a major functionality regression.

For bit more details what could cause the appcache update error might be found int he Browser Console (Shift-Ctrl-J).

Comment 5

[Kershaw Chang [:kershaw]]

(In reply to Honza Bambas (:mayhemer) from comment #4)

This could be a regression from bug 1525900. AFAIK we are still running tests for appcache, so I'm curious how we could miss a major functionality regression.

For bit more details what could cause the appcache update error might be found int he Browser Console (Shift-Ctrl-J).

Michal, would you like to take a look?
I think this is P3, since we want to unship AppCache.

Comment 6

[Michal Novotny [:michal]]

(In reply to Kershaw Chang [:kershaw] from comment #5)

(In reply to Honza Bambas (:mayhemer) from comment #4)

This could be a regression from bug 1525900. AFAIK we are still running tests for appcache, so I'm curious how we could miss a major functionality regression.

For bit more details what could cause the appcache update error might be found int he Browser Console (Shift-Ctrl-J).

Michal, would you like to take a look?
I think this is P3, since we want to unship AppCache.

I don't think it's a regression from bug 1525900. I'll have a look at it once I get the credential.

Comment 7

[Michal Novotny [:michal]]

Parsing of the manifest fails because it's not a valid manifest. Cookies are missing in the request fetching the manifest and the server returns a login page. I'm not sure which bug caused this regression. Ehsan, I guess you might know.

Comment 8

[(no longer active)]

(In reply to Michal Novotny [:michal] from comment #7)

Parsing of the manifest fails because it's not a valid manifest. Cookies are missing in the request fetching the manifest and the server returns a login page. I'm not sure which bug caused this regression. Ehsan, I guess you might know.

Not off hand, sorry, 67 is pretty old. Since I don't have the login credentials mentioned in comment 2 I cannot even test this bug...

Florencia, do you mind please running a bisection using mozregression and finding the exact commit that has caused this regression? Thanks!

Comment 9

[Florencia Di Ciocco, NI to vbalducci]

Hi Ehsan,

Sorry for the delay, here's the mozregresssion: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=54ed5eac2abca2519704c74bc5c421b846031504&tochange=3cddc7cd4da5fcd26b93becb55a028621ab68f64

Regards, Flor.

Comment 10

[Honza Bambas (:mayhemer)]

Looks like bug 1525245 is the cause.

baku, how easy this would be to fix if we even want to?

Comment 11

[chuckd]

Any progress on this? It's been a couple of weeks since the last update. This bug is impacting our customers and we'd like to get a fix. We know AppCache is deprecated, but is it still supported?

Comment 12

[Michal Novotny [:michal]]

I don't know. It's a cookie bug and we're still waiting for feedback from baku.

Comment 13

[Liz Henry (:lizzard) (relman/hg->git project)]

Too late for 70 but we could still take a patch for 71/72.

Comment 14

[chuckd]

I notice that no one is assigned to this bug. Who's the right developer to take this?

Comment 15

[Nhi Nguyen (:nhi)]

Christoph, could someone from your team take a look, esp. at comment #10? Thanks!

Comment 16

[(no longer active)]

I can have a look. I contacted the email address in comment 0 to get access to the test server.

Comment 17

[(no longer active)]

In nsOfflineCacheUpdateItem::OpenChannel() we download the cache item with the content policy type TYPE_OTHER. That results in us selecting the "blocking all" variant of cookie settings, which means anything loaded by these channels won't have access to the cookie database.

I have a fix for this part, but that doesn't seem to fix the error on the website, so there is probably more to look into...

Comment 18

[(no longer active)]

OK the second problem was the channel that we create for checking the appcache maifest. With that fixed, the site works as expected (as long as the browser.cache.offline.storage.enable pref is set to true).

Comment 19

[(no longer active)]

Attachment: Bug 1575934 - Create the channels used to download and verify appcache entries with the cookie settings belonging to the document which created the appcache;

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bc3714b56887
Create the channels used to download and verify appcache entries with the cookie settings belonging to the document which created the appcache; r=baku

Comment 21

[Cristina Coroiu [:ccoroiu]]

Backed out changeset bc3714b56887 (bug 1575934) for wpt failures at /html/browsers/offline/application-cache-api/api_update_error.https.html

Backout: https://hg.mozilla.org/integration/autoland/rev/f2a55f2baaf87100626bd1594a376093691ca007

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=bc3714b568870dd7f1cc26cb4036eff3ffeba8e4

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=271304127&repo=autoland&lineNumber=2628

task 2019-10-15T13:55:10.538Z] 13:55:10 INFO - TEST-START | /html/browsers/offline/application-cache-api/api_update_error.https.html
[task 2019-10-15T13:55:10.538Z] 13:55:10 INFO - Closing window 22
[task 2019-10-15T13:55:11.333Z] 13:55:11 INFO - IOError on command, setting status to CRASH
[task 2019-10-15T13:55:12.644Z] 13:55:12 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpVxtxOo/15341576-1949-6fd8-fb40-fb2fe00eda45.dmp /builds/worker/workspace/build/symbols
[task 2019-10-15T13:55:19.744Z] 13:55:19 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/15341576-1949-6fd8-fb40-fb2fe00eda45.dmp
[task 2019-10-15T13:55:19.744Z] 13:55:19 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/15341576-1949-6fd8-fb40-fb2fe00eda45.extra
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - PROCESS-CRASH | /html/browsers/offline/application-cache-api/api_update_error.https.html | application crashed [@ nsOfflineCacheUpdate::SetCookieSettingsArgs(mozilla::net::CookieSettingsArgs const&)]
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - Crash dump filename: /tmp/tmpVxtxOo/15341576-1949-6fd8-fb40-fb2fe00eda45.dmp
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - Operating system: Android
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - 0.0.0 Linux 3.10.0+ #260 SMP PREEMPT Fri May 19 12:48:14 PDT 2017 x86_64
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - CPU: amd64
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - family 6 model 6 stepping 3
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - 4 CPUs
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO -
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - GPU: UNKNOWN
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO -
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - Crash address: 0x0
[task 2019-10-15T13:55:19.896Z] 13:55:19 INFO - Process uptime: not available
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO -
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - Thread 11 (crashed)
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - 0 libxul.so!nsOfflineCacheUpdate::SetCookieSettingsArgs(mozilla::net::CookieSettingsArgs const&) [nsOfflineCacheUpdate.cpp:5392bdcd9d7d66bddc8b0ce43ab084e64a5c49d5 : 2310 + 0x29]
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rax = 0x000070b0aefba75a rdx = 0x0000000000000001
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rcx = 0x000070b0b1dc7a80 rbx = 0x000070b0b26cad30
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rsi = 0x000070b0b26cac80 rdi = 0x000070b0b26ca9d0
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rbp = 0x000070b0b26cace0 rsp = 0x000070b0b26cacc0
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - r8 = 0x0000000000003a27 r9 = 0x000070b0b26ce450
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - r10 = 0x000070b0a9ac0c7a r11 = 0x0000000000000000
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - r12 = 0x0000000080530012 r13 = 0x000070b09b1260f8
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - r14 = 0x000070b0b26cad30 r15 = 0x000070b09ce70060
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rip = 0x000070b0a9ac0c8c
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - Found by: given as instruction pointer in context
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - 1 libxul.so!mozilla::docshell::OfflineCacheUpdateParent::Schedule(mozilla::ipc::URIParams const&, mozilla::ipc::URIParams const&, mozilla::ipc::PrincipalInfo const&, bool const&, mozilla::net::CookieSettingsArgs const&) [OfflineCacheUpdateParent.cpp:5392bdcd9d7d66bddc8b0ce43ab084e64a5c49d5 : 129 + 0x17]
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rbp = 0x000070b0b26cadd0 rsp = 0x000070b0b26cacf0
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rip = 0x000070b0a9ac08da
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - Found by: previous frame's frame pointer
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - 2 libxul.so!mozilla::dom::ContentParent::RecvPOfflineCacheUpdateConstructor(mozilla::docshell::POfflineCacheUpdateParent*, mozilla::ipc::URIParams const&, mozilla::ipc::URIParams const&, mozilla::ipc::PrincipalInfo const&, bool const&, mozilla::net::CookieSettingsArgs const&) [ContentParent.cpp:5392bdcd9d7d66bddc8b0ce43ab084e64a5c49d5 : 4556 + 0x20]
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rbp = 0x000070b0b26cae30 rsp = 0x000070b0b26cade0
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rip = 0x000070b0ab3ad18d
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - Found by: previous frame's frame pointer
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - 3 libxul.so!mozilla::dom::PContentParent::OnMessageReceived(IPC::Message const&) [PContentParent.cpp: : 8890 + 0x33]
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rbp = 0x000070b0b26ccbc0 rsp = 0x000070b0b26cae40
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rip = 0x000070b0a965a3a1
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - Found by: previous frame's frame pointer
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - 4 libxul.so!mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&) [MessageChannel.cpp:5392bdcd9d7d66bddc8b0ce43ab084e64a5c49d5 : 2185 + 0xd]
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rbp = 0x000070b0b26ccc00 rsp = 0x000070b0b26ccbd0
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - rip = 0x000070b0a959ea99
[task 2019-10-15T13:55:19.897Z] 13:55:19 INFO - Found by: previous frame's frame pointer

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ae8dbea0d35a
Create the channels used to download and verify appcache entries with the cookie settings belonging to the document which created the appcache; r=baku

Comment 23

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/ae8dbea0d35a

Comment 24

[(no longer active)]

Ryan, I understand the decision of not shipping a fix for this in 70 and waiting for 71. But about ESR, I'm not sure if wontfix is a good decision? Without this, app cache is completely broken... I know that we're about to unship the feature and the breakage caused by it is probably minimal, but enterprise environments are exactly where I would expect the last deployment(s) of app cache to be found. Do you think you'd be open to reconsidering after some bake time?

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Sure, we can revisit during the next cycle. Note that the patch needs rebasing for ESR68 too.

Comment 26

[chuckd]

Hi. I just tried this with the latest nightly, 72.0a1 (2019-11-04) (64-bit), an it's still not working for us. Should I be using a different build?

Comment 27

[(no longer active)]

(In reply to chuckd from comment #26)

Hi. I just tried this with the latest nightly, 72.0a1 (2019-11-04) (64-bit), an it's still not working for us. Should I be using a different build?

Please see comment 18. On Nightly you need to set the browser.cache.offline.storage.enable preference to true in about:config, because the app cache API is about to be removed from Firefox and has been disabled in Nightly for now.

Comment 28

[chuckd]

Thanks. I set browser.cache.offline.storage.enable to true but it's still not working. I think this message in the browser console may have something to do with it.

The Notification permission may only be requested from inside a short running user-generated event handler. I've not seen that before.

Comment 29

[chuckd]

Never mind. Seems I needed to restart the browser after making that config change. It's working now. Thanks.

Comment 30

[Ryan VanderMeulen [:RyanVM]]

(In reply to :ehsan akhgari from comment #24)

Hi Ehsan, go-to-build for 68.3esr is next week. If you want to nominate a rebased patch for uplift, we'll need it pretty soon.

Comment 31

[chuckd]

We would certainly like to see this fix go into the esr since many of our customers use esr releases.

Comment 32

[(no longer active)]

Attachment: Bug 1575934 - Create the channels used to download and verify appcache entries with the cookie settings belonging to the document which created the appcache

Comment 33

[(no longer active)]

Comment on attachment 9110135 [details]
Bug 1575934 - Create the channels used to download and verify appcache entries with the cookie settings belonging to the document which created the appcache

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: This patch fixes app cache which broke in bug 1525245 (Firefox 67).
• User impact if declined: Web apps relying on app cache fail to function correctly (e.g. see comment 0).
• Fix Landed on Version: 71
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): This change is fairly large, but it is not super complex. It pipes the cookie settings objects to more places. Since that object is exposed to many existing places in the code before that in itself isn't very risky. It has also received some testing on trunk though app cache is disabled on trunk/early beta by default so that testing is of limited usefulness.

The reason why I evaluate the riskiness as medium is that we have a preference that would allow the app cache storage to be turned off in the case of a disaster situation (e.g. this patch causing an unexpected crash spike).

• String or UUID changes made by this patch: None

Comment 34

[(no longer active)]

Unfortunately I'm unable to push this to try due to bug 1597858. I'm also unable to build the patch locally due to bug 1597860 as well as bug 1597864. So I have no way of testing this patch... :-( I would appreciate some help if you've dealt with this before?

Comment 35

[Ryan VanderMeulen [:RyanVM]]

Not sure why you're hitting bug 1597858, but I can try pushing it to Try for you if you want. Any particular builds/tests you wanted to run?

Comment 36

[Bogdan Maris, Desktop Test Engineering]

(In reply to chuckd from comment #29)

Never mind. Seems I needed to restart the browser after making that config change. It's working now. Thanks.

Would you mind testing Firefox 71 as well to see if this is also fixed? Or should I ask for a test account (not sure how many have been given and don't want to abuse that).

https://archive.mozilla.org/pub/firefox/candidates/71.0b11-candidates/build1/

Comment 37

[(no longer active)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #35)

Not sure why you're hitting bug 1597858, but I can try pushing it to Try for you if you want. Any particular builds/tests you wanted to run?

All builds/tests on all platforms please.

Comment 38

[(no longer active)]

(In reply to Bogdan Maris [:bogdan_maris], Release Desktop QA from comment #36)

(In reply to chuckd from comment #29)

Never mind. Seems I needed to restart the browser after making that config change. It's working now. Thanks.

Would you mind testing Firefox 71 as well to see if this is also fixed? Or should I ask for a test account (not sure how many have been given and don't want to abuse that).

https://archive.mozilla.org/pub/firefox/candidates/71.0b11-candidates/build1/

I just emailed you the test account information, FWIW.

Comment 39

[Ryan VanderMeulen [:RyanVM]]

(In reply to :ehsan akhgari from comment #37)

All builds/tests on all platforms please.

Busted build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d9e3d8302f860b64a9c133f2487239cc98bf92c5

Comment 40

[chuckd]

Tested on 71.0b11 and I see the same failure that was reported. I checked browser.cache.offline.storage.enable in about:config and the value is true. I didn't have to change it.

Contact [e-mail] if you need credentials to the test user account.

Comment 41

[(no longer active)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #39)

(In reply to :ehsan akhgari from comment #37)

All builds/tests on all platforms please.

Busted build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d9e3d8302f860b64a9c133f2487239cc98bf92c5

Hmm, not sure what that error means. But now I succeeded to build locally, so I'll look into it...

Comment 42

[(no longer active)]

OK, fixed the build failure. Updated the patch, and re-tested to ensure the fix works properly on the ESR68 branch as well.

Comment 43

[(no longer active)]

This is as green as things can get on try. Good to go on my end.

Comment 44

[chuckd]

Could someone please take a look at why this problem still reproduces on 71.0b11? Thanks.

Comment 45

[(no longer active)]

(In reply to chuckd from comment #44)

Could someone please take a look at why this problem still reproduces on 71.0b11? Thanks.

Comment 46

[Liz Henry (:lizzard) (relman/hg->git project)]

Let's let this bake in 71 before we evaluate this for uplift to esr. That would mean either for a 68.3.1 if we end up with one for some other reason, or 68.4.0.

Comment 47

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9110135 [details]
Bug 1575934 - Create the channels used to download and verify appcache entries with the cookie settings belonging to the document which created the appcache

I'd like to hold off on uplift here but am adding an esr tracking flag to re-evaluate for the next esr.

Comment 48

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Attachment: Offline server + console.png

Hi Chuckd, I verified both on Firefox 71.0b11 and 71.0b12 and couldn't encounter the issue at all. Maybe there could be something wrong with the profile you are using? Here's an image from 71.0b12 with the offline server created + the message from console.

Comment 49

[chuckd]

Hi Catalin. You are correct. When I switch to a new profile, I can no longer reproduce the problem. Not sure what's wrong with my original profile. I tried clearing cookies and site data, but that didn't help. I also checked browser.cache.offline.enable and browser.cache.offline.storage.enable in about:config and they're both true.

I tried upgrading to the beta on a different machine that had 70 installed and had previously failed to setup offline in Verse. After upgrading, I was able to successfully setup offline, so I think we're ok.

Comment 50

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Thanks Chuckd for the fast reply! Glad that it worked out after all. Good luck!

Comment 51

[(no longer active)]

(In reply to Liz Henry (:lizzard) from comment #47)

Comment on attachment 9110135 [details]
Bug 1575934 - Create the channels used to download and verify appcache entries with the cookie settings belonging to the document which created the appcache

I'd like to hold off on uplift here but am adding an esr tracking flag to re-evaluate for the next esr.

Liz, it looks like this decision was made based on incorrect testing results on the 71 branch. Turns out the fix was working fine there. Do you think we should re-evaluate for the current ESR or are you still comfortable with punting until the next one? Thanks!

Comment 52

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9110135 [details]
Bug 1575934 - Create the channels used to download and verify appcache entries with the cookie settings belonging to the document which created the appcache

OK, now that we're more sure of the verification in 71, let's go ahead and uplift this for 68.3esr.

Comment 53

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/d7488a0118d1

Comment 54

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Seems to be working as intended on 68.3.0esr too.