244572 - md5sum files to speed up upload when files didn't change

Status: RESOLVED INCOMPLETE

(Core Graveyard :: Profile: Roaming, enhancement, P3)

Description

[Joe McCabe]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8a2) Gecko/20040524
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8a2) Gecko/20040524

When the browser quits it insists on pushing new copies of the configuration to
the remote store. It does not appear to compare the mod times of the files to
what is in the store (as reported by listing.xml).

It would be preferable to have Moz check the files and only push files really
needing refresh.

Unless I'm really mistaken the preferable behavior is what happens at startup
(regarding determining which files need to be pulled from the store).

Reproducible: Always
Steps to Reproduce:

Comment 1

[Joe McCabe]

Just a passing thought related to this as well -

It would probably be preferable for Moz to compare the files based on a checksum
instead of lastModifiedUnixTime.

Comment 2

[Ben Bucksch (:BenB)]

I do record filesizes and last modified times, at least I took great pains to do
so. See <http://bugzilla.mozilla.org/show_bug.cgi?id=4xroaming#c239>. If there's
any serious problem, please be very specific of symptoms and suspected causes.

INVALID

Comment 3

[Joe McCabe]

(In reply to comment #2)
> I do record filesizes and last modified times, at least I took great pains to do
> so. See <http://bugzilla.mozilla.org/show_bug.cgi?id=4xroaming#c239>. If there's
> any serious problem, please be very specific of symptoms and suspected causes.

I don't dispute that you go to great pains to log file size and modification
date. This is obvious by examining the contents of listing.xml:

  <file
    filename="bookmarks.html"
    mimetype="application/octet-stream"
    size="162366"
    lastModifiedUnixtime="1085441245000"
  />

The issue is that when pushing content to the store it appears as though this
information is ignored and all the files are pushed regardless of whether-or-not
they actually have changes that require a push.

Since dates on files can be misleading it may make more sense to store a
checksum (md5sum?) of the files rather than rely on size/date info.

Comment 4

[Ben Bucksch (:BenB)]

I was inprecise. I tried not only to record, but use the information. Again, see
the logic description. I'll need more than broad suspections.

I think I do upload everything, if the last mod time matches.

Using md5sums is a nice idea, might have worked, too. Not sure, if we have md5
generating tools in the moz tree, though (HTTP digest or so?). Last mod time
*should* work just as well, because I only compare client times with client
times, never use server times. I require exact matches, so even skewed clocks
should not be a problem.

Comment 5

[Joe McCabe]

(In reply to comment #4)
> I was inprecise. I tried not only to record, but use the information. Again, see
> the logic description. I'll need more than broad suspections.

I honestly wish I cold give you better than broad descriptions. At this moment
that's all I have though.

> I think I do upload everything, if the last mod time matches.

This is what I see. While this isn't a problem per-se, this causes a significant
shutdown lag requiring that the user either wait for a long period for all files
to be uploaded, or to hit "Cancel" to abort the operation (and not knowing if
this action may damage the remotely stored profile).

> Using md5sums is a nice idea, might have worked, too. Not sure, if we have md5
> generating tools in the moz tree, though (HTTP digest or so?). Last mod time
> *should* work just as well, because I only compare client times with client
> times, never use server times. I require exact matches, so even skewed clocks
> should not be a problem.

And generally even if the stamp does wander on a few files the overhead is
re-uploading them is small. I understand this.

Still - In testing the feature earlier each shutdown took around 30-60 seconds
to push the files to the remote FTP server. No harm came from this, but it
introduced a long lag in shutting down. Tighter checking of when to push files
would reduce this lag - particularly when several files want/need to be pushed.

I have no idea if MD5 functions exist inside Moz (outside of what NSS provides).

Comment 6

[Ben Bucksch (:BenB)]

Ah, so your problem is speed, not reliability. What you wrote initially
suggested that it would wrongly overwrite newer files on the server, causing
silent data loss.

Without md5summing, I can't detect that the files don't need update, at least
for many profile files, because they are always written on shutdown (even if the
actual data didn't change) and thus always have a very recent, changed last mod
time.

I was assuming that the roaming server is on a LAN, not accessed over slow
Internet. In the latter case, I realise that it may be slow.


I am REOPENing and turning this into a very low-priority enhancement bug.

Comment 7

[Ben Bucksch (:BenB)]

FYI, with an Internet server, I'd suggest that you only make those files roam
which you really need. BTW: Better use HTTPS :).

Comment 8

[Benjamin Smedberg]

This bug is filed in a bugzilla component related to pre-Firefox code which no longer exists. I believe it is no longer relevant and I am therefore closing it INCOMPLETE.

If you believe that this bug is still valid and needs to be fixed, please reopen it and move it to the Toolkit:Startup and Profile System product/component.