1558070 - Nightly Won't Update While Connected to VPN

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect, P1)

Description

[Mike]

Attachment: nightly capture.PNG

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

Connect to VPN, open Nightly and click on Menu - Help - About Nightly - Check for Updates

Actual results:

Nightly hangs up at "0 bytes Downloaded" and won't progress any further until VPN is turned off. When VPN is off Nightly will download updates as expected.

Expected results:

Nightly should check for, and download it's update as it usually does.

Comment 1

[Mike]

Attachment: mozregression.PNG

Comment 2

[Mike]

Mozregression ran starting June 6, 2019. Good build until 2019-06-09 10:31:22.658000 when updater stopped working. Tested updating Nightly on Linux while connected to VPN without issue, possibly only affects Windows?

Comment 3

[Asif Youssuff]

Guessing regressing bug based on pushlog: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=b5355bd2162ba5740469820ee518649e7fc0b953&tochange=c5ddef8c691be9a24a568d349af7b36c81507b9c

Comment 4

[Robert Strong (they/them - no direct email)]

Please install a non-working version, change the app.update.log preference to true, and open Tools -> Web Developer -> Browser Console. Clear the console output using the trashcan image in the upper left, open the Help -> About Nightly dialog, and try to update. Copy and past the entries that start with "AUS:" and paste them into this bug. Thanks!

Comment 5

[Mike]

Here are the results I received. I let it try 1500 repeats to no avail before I copied the info.

AUS:SVC Downloader:onProgress - progress: 0/-1 22
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC UpdateService:downloadUpdate - no support for downloading more than one update at a time
AUS:SVC readStatusFile - status: downloading, path: C:\ProgramData\Mozilla\updates\6F193CCC56814779\updates\0\update.status
AUS:SVC Downloader:_maybeStartActiveNotifications - Starting active notifications
AUS:SVC Downloader:onProgress - progress: 0/-1 212
AUS:SVC Downloader:onProgress - progress: 0/-1 5
AUS:SVC Downloader:onProgress - progress: 0/-1 1500

Comment 6

[Robert Strong (they/them - no direct email)]

There is some info missing because there is already an update in progress. Could you do the following and then try again?

1. Type about:config in the url bar and press return
2. Click the "I accept the risk" button if it is displayed
3. Type devtools.chrome.enabled in the search box
4. Toggle the value from false to true by either double clicking it or via the context menu.
5. Open Tools -> Web Developer -> Browser Console
6. Copy and paste the following into the text box at the bottom

var fileLocator = Cc["@mozilla.org/file/directory_service;1"].getService(Ci.nsIProperties); var dir = fileLocator.get("UpdRootD", Ci.nsIFile); dir.reveal();

7. Exit Firefox
8. In the directory that was opened delete the active-update.xml file and the updates directory.

Then open Tools -> Web Developer -> Browser Console, clear the console output using the trashcan image in the upper left, open the Help -> About Nightly dialog, and try to update. Copy and past the entries that start with "AUS:" and paste them into this bug. Thanks!

Comment 7

[Robert Strong (they/them - no direct email)]

It is very doubtful that bug 1557963 regressed this since bug 1557963 only changed the check for updates code and this bug affects the download of the mar file so removing regressed by.

Comment 8

[Robert Strong (they/them - no direct email)]

Attaching the updates.xml file might help illuminate what is going on as well so please attach it to this bug. If you still have the existing active-update.xml please attach it as well and if you deleted it already the one that is created when you try again will be just as worthwhile. Thanks!

Comment 9

[Mike]

AUS:SVC Creating UpdateService
AUS:SVC Logging current UpdateService status:
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Elevation required: false
AUS:SVC Update being handled by other instance: false
AUS:SVC Downloading: false
AUS:SVC End of UpdateService status
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC UpdateManager:_loadXMLFileIntoArray - XML file does not exist. path: C:\ProgramData\Mozilla\updates\6F193CCC56814779\active-update.xml
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Checker: checkForUpdates, force: true
AUS:SVC Creating UpdateService
AUS:SVC Logging current UpdateService status:
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Elevation required: false
AUS:SVC Update being handled by other instance: false
AUS:SVC Downloading: false
AUS:SVC End of UpdateService status
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
UpdateUtils.getAppUpdateAutoEnabled - Unable to read app update configuration file. Exception: Win error 2 during operation open on file C:\ProgramData\Mozilla\updates\6F193CCC56814779\update-config.json (The system cannot find the file specified.

)
AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/69.0a1/20190609214350/WINNT_x86_64-msvc-x64/en-US/nightly/Windows_NT%2010.0.0.0.18362.145%20(x64)/ISET:SSE4_2,MEM:7982/default/default/update.xml?force=1
AUS:SVC Checker:checkForUpdates - sending request to: https://aus5.mozilla.org/update/6/Firefox/69.0a1/20190609214350/WINNT_x86_64-msvc-x64/en-US/nightly/Windows_NT%2010.0.0.0.18362.145%20(x64)/ISET:SSE4_2,MEM:7982/default/default/update.xml?force=1
AUS:SVC Checker:onLoad - request completed downloading document
AUS:SVC Checker:onLoad - Getting sslStatus failed.
AUS:SVC Checker:onLoad - number of updates available: 0
UTM:SVC TimerManager:registerTimer - timerID: telemetry_modules_ping interval: 604800
UTM:SVC TimerManager:registerTimer - timerID: telemetry_untrustedmodules_ping interval: 86400

​

Comment 10

[Mike]

Attachment: updates.xml

I attached the old .xml file, the one that I deleted.

Comment 11

[Robert Strong (they/them - no direct email)]

I believe that shows you have successfully updated using app update to the latest Nightly available and we need this information using a build that is failing. Are you still able to get it to fail?

Comment 12

[Robert Strong (they/them - no direct email)]

For the downloads that failed in the log there are both NS_ERROR_UNKNOWN_HOST and NS_ERROR_CONNECTION_REFUSED errors.

Comment 13

[Robert Strong (they/them - no direct email)]

When you can reproduce this please try manually downloading the following from within Firefox and report back the results
https://archive.mozilla.org/pub/firefox/nightly/2019/06/2019-06-09-21-43-50-mozilla-central/firefox-69.0a1.en-US.win64.complete.mar

Comment 14

[Mike]

I'm honestly not sure why it's not failing anymore. The only thing I've done differently to my computer since we started was delete the 2 update files.

Comment 15

[Robert Strong (they/them - no direct email)]

The connection refused and unknown host errors are typically due to issues outside of app update either in the system's network configuration or the networking stack which is outside my area of expertise. One thing that I found surprising is the length of the updates.xml file since it should be deleted by the installer everytime it is used to install in a specific location. Did you use the installer? If not, that isn't supported.

Comment 16

[Robert Strong (they/them - no direct email)]

This is actually very similar to bug 304454 (the oldest app update bug) which I was investigating this weekend and I suspect it has the same root cause.

Comment 17

[Mike]

Yes, I downloaded Nightly directly from the official Mozilla page, probably around a month and a half ago initially. I'm not sure if this is at all helpful to you but I also have Stable 67.0.1, and Stable has never had a problem updating. Just Nightly.

Comment 18

[Robert Strong (they/them - no direct email)]

At this point it would be helpful if you tried to download the following file if it happens again
https://archive.mozilla.org/pub/firefox/nightly/2019/06/2019-06-09-21-43-50-mozilla-central/firefox-69.0a1.en-US.win64.complete.mar

The Nightly builds don't use the same servers as release which could very well be why it hasn't happened on stable. For example:
http://download.mozilla.org/?product=firefox-66.0.3-complete&os=win64&lang=en-US

Comment 19

[Robert Strong (they/them - no direct email)]

I just verified that I get the exact same errors when I reproduced bug 304454 as you had in your updates.xml file and the UI was also the same between this bug and bug 304454.

Comment 20

[timidbit]

I can confirm this. I use VPN permanently. It started a few days ago. Firefox Nightly can't update while connected to VPN.
Just "Downloading update - 0 bytes" until VPN turned off.

Here is the log:

AUS:SVC Creating UpdateService
AUS:SVC Logging current UpdateService status:
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Elevation required: false
AUS:SVC Update being handled by other instance: false
AUS:SVC Downloading: false
AUS:SVC End of UpdateService status
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC UpdateManager:_loadXMLFileIntoArray - XML file does not exist. path: C:\ProgramData\Mozilla\updates\6F193CCC56814779\active-update.xml
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Checker: checkForUpdates, force: true
AUS:SVC Creating UpdateService
AUS:SVC Logging current UpdateService status:
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Elevation required: false
AUS:SVC Update being handled by other instance: false
AUS:SVC Downloading: false
AUS:SVC End of UpdateService status
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
UpdateUtils.getAppUpdateAutoEnabled - Unable to read app update configuration file. Exception: Win error 2 during operation open on file C:\ProgramData\Mozilla\updates\6F193CCC56814779\update-config.json (The system cannot find the file specified.

)
AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/69.0a1/20190701215217/WINNT_x86_64-msvc-x64/en-US/nightly/Windows_NT%2010.0.0.0.18362.207%20(x64)/ISET:SSE4_2,MEM:16119/default/default/update.xml?force=1
AUS:SVC Checker:checkForUpdates - sending request to: https://aus5.mozilla.org/update/6/Firefox/69.0a1/20190701215217/WINNT_x86_64-msvc-x64/en-US/nightly/Windows_NT%2010.0.0.0.18362.207%20(x64)/ISET:SSE4_2,MEM:16119/default/default/update.xml?force=1
AUS:SVC Checker:onLoad - request completed downloading document
AUS:SVC Checker:onLoad - Getting sslStatus failed.
AUS:SVC Checker:onLoad - number of updates available: 1
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC Creating Downloader
AUS:SVC UpdateService:_downloadUpdate
AUS:SVC readStringFromFile - file doesn't exist: C:\ProgramData\Mozilla\updates\6F193CCC56814779\updates\0\update.status
AUS:SVC readStatusFile - status: null, path: C:\ProgramData\Mozilla\updates\6F193CCC56814779\updates\0\update.status
AUS:SVC getCanUseBits - BITS can be used to download updates
AUS:SVC Downloader:_canUseBits - Patch is able to use BITS download
AUS:SVC Downloader:downloadUpdate - Starting BITS download with url: https://archive.mozilla.org/pub/firefox/nightly/partials/2019/07/2019-07-02-09-39-17-mozilla-central/firefox-mozilla-central-69.0a1-win64-en-US-20190701215217-20190702093917.partial.mar, updateDir: C:\ProgramData\Mozilla\updates\6F193CCC56814779\updates\0, filename: update.mar
AUS:SVC Downloader:downloadUpdate - BITS download running. BITS ID: {D1E65910-8E1E-4AC2-A4D5-FAFD1A339CC2}
AUS:SVC Downloader:_maybeStartActiveNotifications - Starting active notifications
AUS:SVC Downloader:onProgress - progress: 0/-1
AUS:SVC Downloader:onProgress - progress: 0/-1
AUS:SVC Downloader:onProgress - progress: 0/-1 4
AUS:SVC Downloader:onProgress - progress: 0/-1
AUS:SVC Downloader:onProgress - progress: 0/-1
AUS:SVC Downloader:onProgress - progress: 0/-1
AUS:SVC Downloader:onProgress - progress: 0/-1 7
AUS:SVC Downloader:onProgress - progress: 0/-1 2
AUS:SVC Downloader:onProgress - progress: 0/-1 2
AUS:SVC Downloader:onProgress - progress: 0/-1
AUS:SVC Downloader:onProgress - progress: 0/-1 4
AUS:SVC Downloader:onProgress - progress: 0/-1
AUS:SVC Downloader:onProgress - progress: 0/-1
etc...

Comment 21

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

exgens (or others), could you share what VPN provider you use? It might be easier to reproduce this and test fixes if we can get an equivalent setup.

Comment 22

[timidbit]

I use NordVPN through OpenVPN, UDP config.
https://downloads.nordcdn.com/configs/files/ovpn_udp/servers/lv31.nordvpn.com.udp.ovpn

Comment 23

[Mike]

I use NordVPN also, through their official app. UDP config as well.

Comment 24

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

Thanks! No luck reproducing the issue yet, unfortunately.

My setup:
Windows 10 Home 64-bit Version 1809 (Build 17763.437) (in a Hyper-V VM)
Firefox Nightly 64-bit (build 20190701215217)
OpenVPN 2.4.7.0 (GUI v11.13.0.0) (for Windows 10)
NordVPN using the lv31.nordvpn.com.udp.ovpn config linked in comment 22 (SHA1 96EC5D567228657E491BD686D4E01E1BD70B0230)

I tried starting the BITS service both before and after connecting to the VPN, and I tried adding block-outside-dns to the config. I started with the same build as reported in exgens' log (from the installer .exe), reinstalling it anew for each test. It's getting a complete MAR now instead of a partial, but I think the mechanics should be the same.

I also tried NordVPN's own client 6.22.6.0 (which uses OpenVPN 2.4.6.0), couldn't detect breakage. I tried the auto-start, starting BITS before and after connecting, and the "internet kill switch".

I welcome suggestions but I'm out of ideas for the moment.

Comment 25

[Robert Strong (they/them - no direct email)]

Bug 1563790 might mitigate this bug

Comment 26

[BugBot [:suhaib / :marco/ :calixte]]

Changing the priority to p1 as the bug is tracked by a release manager for the current release.
See What Do You Triage for more information

Comment 27

[Mike Taylor [:miketaylr]]

exgens, can you try the build from https://bugzilla.mozilla.org/show_bug.cgi?id=1563790#c2 and see if that fixes it? thanks

Comment 28

[Robert Strong (they/them - no direct email)]

That build won't be ready to test until the next build finishes so there is an update available for it

Comment 29

[Robert Strong (they/them - no direct email)]

The build should be ready to test now and are available here
https://archive.mozilla.org/pub/firefox/nightly/2019/07/2019-07-23-21-01-46-oak/

Here is a direct link to the Windows 64 bit installer in that directory
https://archive.mozilla.org/pub/firefox/nightly/2019/07/2019-07-23-21-01-46-oak/firefox-70.0a1.en-US.win64.installer.exe

Comment 30

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

It might be better to extract and run the oak build from the .zip, so as not to mess with your normal Nightly install and profile:

https://archive.mozilla.org/pub/firefox/nightly/2019/07/2019-07-23-21-01-46-oak/firefox-70.0a1.en-US.win64.zip

Comment 31

[Robert Strong (they/them - no direct email)]

Adam is entirely correct and it would definitely be better his way. Thanks Adam!

Comment 32

[timidbit]

Yes, it works for me!
Maybe Mike (Reporter) should try it too?

Comment 33

[Mike Taylor [:miketaylr]]

Good idea. :) Mike, please see Comment 30 and test if that fix works for you. Thanks!

Comment 34

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

Thanks very much exgens!

Could you also try one more build, this is without the proposed fix and should fail to update in the same way as Nightly; I'd just like to confirm the before now that we've seen it fixed in the after: https://archive.mozilla.org/pub/firefox/nightly/2019/07/2019-07-23-03-31-50-oak/firefox-70.0a1.en-US.win64.zip

Comment 35

[timidbit]

Yes, that build fails to update!

Comment 36

[Serge Montagnac]

I am from the very beginning a Firefox afficionado and pro Linux developer ;)
The whole automatic update and release check system appears broken for Linux.

From release 69.0b5 Automatic Update is broken on LINUX-x86-64 : beta 69.0, nightly 70.0, release 68.0 ...

Description: (from Preferences) Firefox Updates:
Version = 69.0b7 : Always right !
Show Update History : Replies always "No updates installed yet"
(Smiley happy) is up to date when or without pressing "Check updates" (even if you install 69.0b1 today !)

All suggested checks, about:config parameters suggested are vain today

Thanks for coop,
Serge.

Comment 37

[Serge Montagnac]

(Complement to Serge Montagnac from comment #36)
I forgot to indicate that all day long we got spurious panel which says
that "Firefox was unable to Install update ..."
Even if we have the last release, beta or nightly ...
(... and sure, all permissions are allowed to the firefox dedicated repository)
Thanks.

Comment 38

[Robert Strong (they/them - no direct email)]

Serge, please file a new bug

Comment 39

[Serge Montagnac]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #38)

Serge, please file a new bug
It'done ! Thanks Robert.

Comment 40

[Ryan VanderMeulen [:RyanVM]]

Per Adam in bug 1563790, that patch is going to need more bake time than we can give it for an Fx68 dot release. Re-targeting this bug for Fx69/68.1esr instead.

Comment 41

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

Now that bug 1563790 has landed in Nightly, this should no longer block updates. Nightly should now be able to update even if BITS encounters a persisten "transient" error. By default it will retry for up to an hour (instead of the BITS default of 2 weeks), but if the About dialog or the Options tab is open it will drop that to 5 seconds (as these trigger an immediate update check). When BITS fails it will immediately fall back to the original incremental downloader within Firefox.

Exgens and others, please try out this Nightly build or a later version (build date 2019-08-02 or later).

Comment 42

[Ryan VanderMeulen [:RyanVM]]

This should be fixed by way of bug 1563790 for Firefox 69 (b14 and later) and 68.1esr shipping in early September. If people want to try out a Nightly build (or Beta after tomorrow), some early feedback would be greatly appreciated!