Closed Bug 1233679 Opened 4 years ago Closed 4 years ago

UpdateTestcase.setUp() should remove any remaining trace of an already downloaded update.mar file

Categories

(Testing :: Firefox UI Tests, defect)

45 Branch
defect
Not set

Tracking

(firefox48 fixed, firefox49 fixed)

RESOLVED FIXED
mozilla49
Tracking Status
firefox48 --- fixed
firefox49 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

Details

Attachments

(1 file)

We have test failures for both direct (bug 1216738) and fallback (bug 1216739) updates, which state:

TEST-UNEXPECTED-FAIL | test_direct_update.py TestDirectUpdate.test_update | AssertionError: <firefox_puppeteer.ui.about_window.deck.DownloadingPanel object at 0x02DEAA90> != <firefox_puppeteer.ui.about_window.deck.CheckForUpdatesPanel object at 0x02DEA8F0>

To get this fixed we should check that no partially downloaded update.mar file is present in the update staging folder.
Product: Mozilla QA → Testing
So that is a really interesting issue!! Not sure yet which layer of Marionette or Firefox to blame, but that's definitely not a problem in our tests.

So what is happening:

For each update test we make a copy of the application into a temporary directory under /tmp. That way we can ensure to have a fresh copy. It means when we start Firefox there should be no update already downloaded or in download progress. But interestingly that happens and manifests in the test failures as listed in the blocking field.

 0:02.88 LOG: MainThread INFO Removing copy of the application at "/tmp/tmpYtOf4G.application.copy"
 0:02.90 LOG: MainThread INFO Creating a copy of the application at "/tmp/tmpSHybHf.application.copy".
*** UTM:SVC TimerManager:registerTimer - id: xpi-signature-verification
*** AUS:SVC Creating UpdateService
*** AUS:SVC gCanCheckForUpdates - able to check for updates
*** AUS:SVC getCanApplyUpdates - testing write access /tmp/tmpSHybHf.application.copy/update.test
*** AUS:SVC getCanApplyUpdates - able to apply updates
*** AUS:SVC readStatusFile - status: downloading, path: /tmp/tmpSHybHf.application.copy/updates/0/update.status
*** AUS:SVC readStatusFile - status: downloading, path: /tmp/tmpSHybHf.application.copy/updates/0/update.status
*** AUS:SVC UpdateService:_postUpdateProcessing - patch found in downloading state
*** AUS:SVC Creating Downloader
*** AUS:SVC UpdateService:_downloadUpdate
*** AUS:SVC readStatusFile - status: downloading, path: /tmp/tmpSHybHf.application.copy/updates/0/update.status
*** AUS:SVC Downloader:_selectPatch - found existing patch with state: downloading
*** AUS:SVC Downloader:_selectPatch - resuming download

What? This path has been created right now. How can there be an already partially downloaded update?

Now the strange part which I fully don't understand:

*** AUS:SVC Downloader:onProgress - progress: 7800000/8911898

Check this size of the download mar file and compare it to the following output:

> $ ls -l /mozilla/bin/nightly/updates/0/
> total 8964
> -rw------- 1 henrik henrik 7800000 May 20 20:34 update.mar

This is my main instance of the browser I'm working with throughout the whole day. There is indeed an update in the queue. But why is that status read by the Firefox instance Marionette brings up?

When I use Marionette to retrieve the staging update dir it reports the correct path:

> ** list dir: /tmp/tmpSHybHf.application.copy/updates/0

So not sure what's going on. Need to dive into deeper.
On Windows we do not download the mar file into the application/update folder but under %appdata%/local/Mozilla/updates. When I check one of the machines where updates are regularly failing, this folder if full of half-wise downloaded updates. I will have to check where Firefox actually points to for the update folder. But I could imagine that this behavior is used due to the Maintenance service. Maybe we should disable its installation via the silent installer if possible.

Robert, if you have the minute mind giving me a quick feedback? Thanks.
Flags: needinfo?(robert.strong.bugs)
On Windows we use the Mozilla\updates\<hash> when installed via the installer with or without the maintenance service.

You can find the directory with the following
const Cc = Components.classes; const Ci = Components.interfaces; var fileLocator = Cc["@mozilla.org/file/directory_service;1"].getService(Ci.nsIProperties); var dir = fileLocator.get("UpdRootD", Ci.nsIFile); dir.reveal();

What files are in what directory?
Flags: needinfo?(robert.strong.bugs)
Interesting. So far I thought that on Windows we also store the downloaded mar files under the application folder. Good to know, and that means we would have to definitely delete everything beneath the folder you pointed out above or via getUpdatesDirectory() as what we currently use.

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #3)
> What files are in what directory?

update/0/update.mar     (~586kb)
update/0/update.status  (downloading)
update/*.log
active-update.xml
active-update-1.xml
updates.xml
First the link to the Treeherder results:

https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=16663eb3dcfa759f25b5e27b101bc79270c156f2&filter-searchStr=firefox%20ui%20windows%208%20x64%20opt&filter-tier=1&filter-tier=2&filter-tier=3&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable&filter-classifiedState=unclassified&selectedJob=3563162

This is a single run for updates. Lets have a look at the 8.1 x64 ones... When we trigger those we have a 64bit and a 32bit build to test. That's why two results per locale. When I now check the %appdata% folder I can only see 2 update folder but not 4. It looks like that we construct the hash based on some specific attributes and maybe the locale included. 

495DDF42545E25B1
FCE21B573705EDF7

Not sure which of those two folders are related to which updates. I would have to test this deeper tomorrow. Sadly due to bug 1008135 we can still not get the AUS logging printed to the console, so it makes it a bit harder to retrieve the details.
Indeed, all the jobs as listed above were using the hash "FCE21B573705EDF7" for the storage of downloaded mar files.
The above is strange because switching the locale from en-US to fr and back to en-US should have stopped downloading the mar file, and triggered a new download with the mar file for the new locale.

13:52:13     INFO -  *** AUS:SVC readStatusFile - status: downloading, path: C:\Users\mozauto\AppData\Local\Mozilla\updates\FCE21B573705EDF7\updates\0\update.status
13:52:13     INFO -  *** AUS:SVC readStatusFile - status: downloading, path: C:\Users\mozauto\AppData\Local\Mozilla\updates\FCE21B573705EDF7\updates\0\update.status
13:52:13     INFO -  *** AUS:SVC UpdateService:_postUpdateProcessing - patch found in downloading state
13:52:13     INFO -  *** AUS:SVC Creating Downloader
13:52:13     INFO -  *** AUS:SVC UpdateService:_downloadUpdate
13:52:13     INFO -  *** AUS:SVC readStatusFile - status: downloading, path: C:\Users\mozauto\AppData\Local\Mozilla\updates\FCE21B573705EDF7\updates\0\update.status
13:52:13     INFO -  *** AUS:SVC Downloader:_selectPatch - found existing patch with state: downloading
13:52:13     INFO -  *** AUS:SVC Downloader:_selectPatch - resuming download
13:52:13     INFO -  *** AUS:SVC Downloader:downloadUpdate - downloading from https://mozilla-nightly-updates.s3.amazonaws.com/mozilla-central/20160523030225/Firefox-mozilla-central-49.0a1-win64-en-US-20160522030240-20160523030225.partial.mar?versionId=xGvrk6Q9Qg2Mi8D0h7dAMkDCC03G49vM to C:\Users\mozauto\AppData\Local\Mozilla\updates\FCE21B573705EDF7\updates\0\update.mar

This downloads the win64-en-US mar file for the french locale.
build and update info as retrieved via appinfo and update service interface:

13:52:17     INFO -   'build_pre': {'buildid': u'20160522030240',
13:52:17     INFO -                 'channel': u'nightly',
13:52:17     INFO -                 'disabled_addons': None,
13:52:17     INFO -                 'locale': u'fr',
13:52:17     INFO -                 'mar_channels': set([u'firefox-mozilla-central']),
13:52:17     INFO -                 'url_aus': u'https://aus5.mozilla.org/update/3/Firefox/49.0a1/20160522030240/WINNT_x86_64-msvc/fr/nightly/Windows_NT%206.3/default/default/update.xml?force=1',
13:52:17     INFO -                 'user_agent': u'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:49.0) Gecko/20100101 Firefox/49.0',
13:52:17     INFO -                 'version': u'49.0a1'},
13:52:17     INFO -   'fallback': False,
13:52:17     INFO -   'patch': {'buildid': u'20160523030225',
13:52:17     INFO -             'channel': u'nightly',
13:52:17     INFO -             'download_duration': None,
13:52:17     INFO -             'is_complete': False,
13:52:17     INFO -             'size': 5934228,
13:52:17     INFO -             'type': u'minor',
13:52:17     INFO -             'url_mirror': u'https://mozilla-nightly-updates.s3.amazonaws.com/mozilla-central/20160523030225/Firefox-mozilla-central-49.0a1-win64-en-US-20160522030240-20160523030225.partial.mar?versionId=xGvrk6Q9Qg2Mi8D0h7dAMkDCC03G49vM',
13:52:17     INFO -             'version': u'49.0a1'},
13:52:17     INFO -   'success': False}]
(In reply to Henrik Skupin (:whimboo) from comment #6)
> First the link to the Treeherder results:
> 
> https://treeherder.allizom.org/#/jobs?repo=mozilla-
> central&revision=16663eb3dcfa759f25b5e27b101bc79270c156f2&filter-
> searchStr=firefox%20ui%20windows%208%20x64%20opt&filter-tier=1&filter-
> tier=2&filter-tier=3&filter-resultStatus=testfailed&filter-
> resultStatus=busted&filter-resultStatus=exception&filter-
> resultStatus=retry&filter-resultStatus=usercancel&filter-
> resultStatus=running&filter-resultStatus=pending&filter-
> resultStatus=runnable&filter-classifiedState=unclassified&selectedJob=3563162
> 
> This is a single run for updates. Lets have a look at the 8.1 x64 ones...
> When we trigger those we have a 64bit and a 32bit build to test. That's why
> two results per locale. When I now check the %appdata% folder I can only see
> 2 update folder but not 4. It looks like that we construct the hash based on
> some specific attributes and maybe the locale included. 
> 
> 495DDF42545E25B1
> FCE21B573705EDF7
> 
> Not sure which of those two folders are related to which updates. I would
> have to test this deeper tomorrow. Sadly due to bug 1008135 we can still not
> get the AUS logging printed to the console, so it makes it a bit harder to
> retrieve the details.
It is a hash of the install dir path and a separate one per install dir is expected.

I don't think there is a locale check in app update and we haven't had a bug filed for the locale changing out from under an in progress update likely because it is extremely edgecasey
Ok, so in this case our tests will make sure that the appropriate folder will be removed during setup and teardown. That way we can ensure that no already started download interferes with the test itself, and we do not leave behind any partial files if a test fails and aborts early. Thanks Robert.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #3)
> You can find the directory with the following
> const Cc = Components.classes; const Ci = Components.interfaces; var
> fileLocator =
> Cc["@mozilla.org/file/directory_service;1"].getService(Ci.nsIProperties);
> var dir = fileLocator.get("UpdRootD", Ci.nsIFile); dir.reveal();

On Linux this actually opens the application directory so we would have to figure out ourselves where the downloaded updates are located. I will keep the code we currently have to determine this folder.
Attachment #8755797 - Flags: review?(mjzffr) → review+
Comment on attachment 8755797 [details]
MozReview Request: Bug 1233679 - UpdateTestcase should remove any partially downloaded update. r?maja_zf

https://reviewboard.mozilla.org/r/54818/#review51488

r+wc

::: testing/firefox-ui/harness/firefox_ui_harness/testcases.py:62
(Diff revision 1)
>          self.download_duration = None
>  
>          # Bug 604364 - Preparation to test multiple update steps
>          self.current_update_index = 0
>  
> -        self.staging_directory = self.software_update.staging_directory
> +        # Ensure that there exists no already partially downloaded update

You have removed the identifier `self.staging_directory`, so I assume the tests don't refer to it at all?
Yes, no test or any piece of code makes use of it. If it would the new way to access it is via the software_update instance.

http://mxr.mozilla.org/mozilla-central/search?string=self.staging_directory
We do not experience those large fallouts on mozilla-beta, so I would only want to get this landed for mozilla-central and mozilla-aurora.
(In reply to Henrik Skupin (:whimboo) from comment #12)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #3)
> > You can find the directory with the following
> > const Cc = Components.classes; const Ci = Components.interfaces; var
> > fileLocator =
> > Cc["@mozilla.org/file/directory_service;1"].getService(Ci.nsIProperties);
> > var dir = fileLocator.get("UpdRootD", Ci.nsIFile); dir.reveal();
> 
> On Linux this actually opens the application directory so we would have to
> figure out ourselves where the downloaded updates are located. I will keep
> the code we currently have to determine this folder.
On all platforms this opens the update root directory and from there the download directory will be under updates/0.
The patch works great. No more failure for the last Nightly build on mozilla-central. Lets get it backported to mozilla-aurora too.
Whiteboard: [checkin-needed-aurora]
https://hg.mozilla.org/mozilla-central/rev/f37493706dd7
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla49
You need to log in before you can comment on or make changes to this bug.