Closed Bug 816635 Opened 12 years ago Closed 12 years ago

partial mar verification failing for 17.0.1 esr

Categories

(Release Engineering :: Release Requests, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: bhearsum)

References

Details

Attachments

(2 files, 1 obsolete file)

Using  https://aus3.mozilla.org/update/1/Firefox/17.0/20121119074111/Linux_x86-gcc3/eo/esrtest/update.xml?force=1
retry: Calling <function run_with_timeout at 0xb7f3733c> with args: (['wget', '--no-check-certificate', '-S', '-O', 'update.xml', 'https://aus3.mozilla.org/update/1/Firefox/17.0/20121119074111/Linux_x86-gcc3/eo/esrtest/update.xml?force=1'], 300, None, None, False, True), kwargs: {}, attempt #1
Executing: ['wget', '--no-check-certificate', '-S', '-O', 'update.xml', 'https://aus3.mozilla.org/update/1/Firefox/17.0/20121119074111/Linux_x86-gcc3/eo/esrtest/update.xml?force=1']
--11:22:44--  https://aus3.mozilla.org/update/1/Firefox/17.0/20121119074111/Linux_x86-gcc3/eo/esrtest/update.xml?force=1
Resolving aus3.mozilla.org... 63.245.217.44
Connecting to aus3.mozilla.org|63.245.217.44|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: Apache
  X-Backend-Server: pp-app-dist06
  Cache-Control: no-store, must-revalidate, post-check=0, pre-check=0, private
  Content-Type: text/xml;
  Date: Thu, 29 Nov 2012 19:22:44 GMT
  Keep-Alive: timeout=5, max=997
  Connection: Keep-Alive
  Set-Cookie: aus2a=63.245.220.220.1354216964.2954; expires=Thu, 30-Nov-2017 00:26:34 GMT; path=/; domain=aus2.mozilla.org
  X-Powered-By: PHP/5.1.6
  X-Cache-Info: not cacheable; response specified "Cache-Control: no-store"
  Content-Length: 980
Cookie coming from aus3.mozilla.org attempted to set domain to aus2.mozilla.org
Length: 980 [text/xml]
Saving to: `update.xml'

     0K                                                       100% 46.7M=0s

11:22:44 (46.7 MB/s) - `update.xml' saved [980/980]

Got this response:
<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="17.0.1esr" appVersion="17.0.1" platformVersion="17.0.1" buildID="20121129050723" detailsURL="https://www.mozilla.com/eo/firefox/17.0.1/releasenotes/" actions="silent">
        <patch type="complete" URL="http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0.1esr-candidates/build1/update/linux-i686/eo/firefox-17.0.1esr.complete.mar" hashFunction="SHA512" hashValue="0913d745106729efd6cdeb9959f89e8f2a0929bb38d31468c15d7f41ab80e2bfd0aa2a9827b7882270f5629aadc7766164baaa51107ef43590ae9e0bcee0ecf7" size="21055106"/>
        <patch type="partial" URL="http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0.1esr-candidates/build1/update/linux-i686/eo/firefox-17.0esr-17.0.1esr.partial.mar" hashFunction="SHA512" hashValue="cb80b09f86d98b81b60b6d7ee2d21dac187d0840f0004f4902dcfe1b56268c6def746d7866de17120bf5ff9e61dbd75f1ce222319fe91e50285147db881088fc" size="1025659"/>
    </update>
</updates>

retry: Calling <function run_with_timeout at 0xb7eb433c> with args: (['wget', '--no-check-certificate', '-nv', '-O', 'update/partial.mar', 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0.1esr-candidates/build1/update/linux-i686/eo/firefox-17.0esr-17.0.1esr.partial.mar'], 300, None, None, False, True), kwargs: {}, attempt #1
Executing: ['wget', '--no-check-certificate', '-nv', '-O', 'update/partial.mar', 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0.1esr-candidates/build1/update/linux-i686/eo/firefox-17.0esr-17.0.1esr.partial.mar']
11:22:44 URL:http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0.1esr-candidates/build1/update/linux-i686/eo/firefox-17.0esr-17.0.1esr.partial.mar [1025659/1025659] -> "update/partial.mar" [1]
retry: Calling <function run_with_timeout at 0xb7e7333c> with args: (['wget', '--no-check-certificate', '-nv', 'http://stage.mozilla.org/pub/mozilla.org//firefox/releases/17.0esr/linux-i686/eo/firefox-17.0esr.tar.bz2'], 300, None, None, False, True), kwargs: {}, attempt #1
Executing: ['wget', '--no-check-certificate', '-nv', 'http://stage.mozilla.org/pub/mozilla.org//firefox/releases/17.0esr/linux-i686/eo/firefox-17.0esr.tar.bz2']
11:22:47 URL:http://stage.mozilla.org/pub/mozilla.org//firefox/releases/17.0esr/linux-i686/eo/firefox-17.0esr.tar.bz2 [21047064/21047064] -> "firefox-17.0esr.tar.bz2" [1]
retry: Calling <function run_with_timeout at 0xb7eca33c> with args: (['wget', '--no-check-certificate', '-nv', 'http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/17.0.1esr-candidates/build1/linux-i686/eo/firefox-17.0.1esr.tar.bz2'], 300, None, None, False, True), kwargs: {}, attempt #1
Executing: ['wget', '--no-check-certificate', '-nv', 'http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/17.0.1esr-candidates/build1/linux-i686/eo/firefox-17.0.1esr.tar.bz2']
11:22:49 URL:http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/17.0.1esr-candidates/build1/linux-i686/eo/firefox-17.0.1esr.tar.bz2 [21045591/21045591] -> "firefox-17.0.1esr.tar.bz2" [1]
SOURCE DIRECTORY ../../update
DESTINATION DIRECTORY .
UPDATE TYPE partial
PREPARE PATCH webapprt/webapprt.ini
PREPARE PATCH webapprt-stub
PREPARE PATCH update-settings.ini
PREPARE ADD precomplete
PREPARE PATCH platform.ini
PREPARE PATCH omni.ja
PREPARE PATCH libxul.so
PREPARE PATCH libssl3.so
PREPARE PATCH libsoftokn3.so
PREPARE ADD libsoftokn3.chk
PREPARE PATCH libsmime3.so
PREPARE PATCH libplds4.so
PREPARE PATCH libplc4.so
PREPARE PATCH libnssutil3.so
PREPARE PATCH libnssdbm3.so
PREPARE ADD libnssdbm3.chk
PREPARE PATCH libnssckbi.so
PREPARE PATCH libnss3.so
PREPARE PATCH libnspr4.so
PREPARE PATCH libmozsqlite3.so
PREPARE PATCH libfreebl3.so
PREPARE ADD libfreebl3.chk
PREPARE PATCH firefox-bin
PREPARE PATCH firefox
PREPARE PATCH extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/install.rdf
PREPARE PATCH application.ini
EXECUTE PATCH webapprt/webapprt.ini
LoadSourceFile: destination file crc 652474486 does not match expected crc -709355401
LoadSourceFile failed
### execution failed
FINISH PATCH webapprt/webapprt.ini
backup_restore: backup file doesn't exist: webapprt/webapprt.ini.moz-backup
FINISH PATCH webapprt-stub
backup_restore: backup file doesn't exist: webapprt-stub.moz-backup
FINISH PATCH update-settings.ini
backup_restore: backup file doesn't exist: update-settings.ini.moz-backup
FINISH ADD precomplete
backup_restore: backup file doesn't exist: precomplete.moz-backup
FINISH PATCH platform.ini
backup_restore: backup file doesn't exist: platform.ini.moz-backup
FINISH PATCH omni.ja
backup_restore: backup file doesn't exist: omni.ja.moz-backup
FINISH PATCH libxul.so
backup_restore: backup file doesn't exist: libxul.so.moz-backup
FINISH PATCH libssl3.so
backup_restore: backup file doesn't exist: libssl3.so.moz-backup
FINISH PATCH libsoftokn3.so
backup_restore: backup file doesn't exist: libsoftokn3.so.moz-backup
FINISH ADD libsoftokn3.chk
backup_restore: backup file doesn't exist: libsoftokn3.chk.moz-backup
FINISH PATCH libsmime3.so
backup_restore: backup file doesn't exist: libsmime3.so.moz-backup
FINISH PATCH libplds4.so
backup_restore: backup file doesn't exist: libplds4.so.moz-backup
FINISH PATCH libplc4.so
backup_restore: backup file doesn't exist: libplc4.so.moz-backup
FINISH PATCH libnssutil3.so
backup_restore: backup file doesn't exist: libnssutil3.so.moz-backup
FINISH PATCH libnssdbm3.so
backup_restore: backup file doesn't exist: libnssdbm3.so.moz-backup
FINISH ADD libnssdbm3.chk
backup_restore: backup file doesn't exist: libnssdbm3.chk.moz-backup
FINISH PATCH libnssckbi.so
backup_restore: backup file doesn't exist: libnssckbi.so.moz-backup
FINISH PATCH libnss3.so
backup_restore: backup file doesn't exist: libnss3.so.moz-backup
FINISH PATCH libnspr4.so
backup_restore: backup file doesn't exist: libnspr4.so.moz-backup
FINISH PATCH libmozsqlite3.so
backup_restore: backup file doesn't exist: libmozsqlite3.so.moz-backup
FINISH PATCH libfreebl3.so
backup_restore: backup file doesn't exist: libfreebl3.so.moz-backup
FINISH ADD libfreebl3.chk
backup_restore: backup file doesn't exist: libfreebl3.chk.moz-backup
FINISH PATCH firefox-bin
backup_restore: backup file doesn't exist: firefox-bin.moz-backup
FINISH PATCH firefox
backup_restore: backup file doesn't exist: firefox.moz-backup
FINISH PATCH extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/install.rdf
backup_restore: backup file doesn't exist: extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/install.rdf.moz-backup
FINISH PATCH application.ini
backup_restore: backup file doesn't exist: application.ini.moz-backup
failed: 4
calling QuitProgressUI
FAIL: update status was not succeeded: failed: 4
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-17.0esr.tar.bz2 vs. downloads/firefox-17.0.1esr.tar.bz2: 1
SUCCESS: partial updates are smaller than complete updates, all is well in the universe
command: END (855.67s elapsed)
It's pretty strange the binaries are what they're expected to be but webapprt.ini isn't. My gut says this is related to bug 813757, but that doesn't make sense since that bug was en-US only....
Well this is interesting. en-US partials succeeded, but l10n ones didn't:
Using  https://aus3.mozilla.org/update/1/Firefox/17.0/20121119074111/Linux_x86-gcc3/en-US/esrtest/update.xml?force=1
retry: Calling <function run_with_timeout at 0xb7ec533c> with args: (['wget', '--no-check-certificate', '-S', '-O', 'update.xml', 'https://aus3.mozilla.org/update/1/Firefox/17.0/20121119074111/Linux_x86-gcc3/en-US/esrtest/update.xml?force=1'], 300, None, None, False, True), kwargs: {}, attempt #1
Executing: ['wget', '--no-check-certificate', '-S', '-O', 'update.xml', 'https://aus3.mozilla.org/update/1/Firefox/17.0/20121119074111/Linux_x86-gcc3/en-US/esrtest/update.xml?force=1']
--11:21:34--  https://aus3.mozilla.org/update/1/Firefox/17.0/20121119074111/Linux_x86-gcc3/en-US/esrtest/update.xml?force=1
Resolving aus3.mozilla.org... 63.245.217.44
Connecting to aus3.mozilla.org|63.245.217.44|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: Apache
  X-Backend-Server: pp-app-dist05
  Cache-Control: no-store, must-revalidate, post-check=0, pre-check=0, private
  Content-Type: text/xml;
  Date: Thu, 29 Nov 2012 19:21:34 GMT
  Keep-Alive: timeout=5, max=750
  Connection: Keep-Alive
  Set-Cookie: aus2a=63.245.220.220.1354216894.4394; expires=Thu, 30-Nov-2017 00:25:24 GMT; path=/; domain=aus2.mozilla.org
  X-Powered-By: PHP/5.1.6
  X-Cache-Info: not cacheable; response specified "Cache-Control: no-store"
  Content-Length: 988
Cookie coming from aus3.mozilla.org attempted to set domain to aus2.mozilla.org
Length: 988 [text/xml]
Saving to: `update.xml'

     0K                                                       100% 47.2M=0s

11:21:34 (47.2 MB/s) - `update.xml' saved [988/988]

Got this response:
<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="17.0.1esr" appVersion="17.0.1" platformVersion="17.0.1" buildID="20121129050723" detailsURL="https://www.mozilla.com/en-US/firefox/17.0.1/releasenotes/" actions="silent">
        <patch type="complete" URL="http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0.1esr-candidates/build1/update/linux-i686/en-US/firefox-17.0.1esr.complete.mar" hashFunction="SHA512" hashValue="168980fe8a2e2256d4458c99cb57bb70a4226b3923b35d496db11869332dc9eb99baaa818e25e360007136ee6843483dd627fdd334a800a28b743364cbc39e82" size="21288338"/>
        <patch type="partial" URL="http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0.1esr-candidates/build1/update/linux-i686/en-US/firefox-17.0esr-17.0.1esr.partial.mar" hashFunction="SHA512" hashValue="2beccfcc902cdcc93c08285d7dcfc0ce6159f76e54d94eb4af1fe5bf213eb9ecaed73958cb4b24fe900329905164f309c57617dd5475e420d56402e88da6774d" size="974908"/>
    </update>
</updates>

retry: Calling <function run_with_timeout at 0xb7e8b33c> with args: (['wget', '--no-check-certificate', '-nv', '-O', 'update/partial.mar', 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0.1esr-candidates/build1/update/linux-i686/en-US/firefox-17.0esr-17.0.1esr.partial.mar'], 300, None, None, False, True), kwargs: {}, attempt #1
Executing: ['wget', '--no-check-certificate', '-nv', '-O', 'update/partial.mar', 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0.1esr-candidates/build1/update/linux-i686/en-US/firefox-17.0esr-17.0.1esr.partial.mar']
11:21:34 URL:http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0.1esr-candidates/build1/update/linux-i686/en-US/firefox-17.0esr-17.0.1esr.partial.mar [974908/974908] -> "update/partial.mar" [1]
retry: Calling <function run_with_timeout at 0xb7f4733c> with args: (['wget', '--no-check-certificate', '-nv', 'http://stage.mozilla.org/pub/mozilla.org//firefox/releases/17.0esr/linux-i686/en-US/firefox-17.0esr.tar.bz2'], 300, None, None, False, True), kwargs: {}, attempt #1
Executing: ['wget', '--no-check-certificate', '-nv', 'http://stage.mozilla.org/pub/mozilla.org//firefox/releases/17.0esr/linux-i686/en-US/firefox-17.0esr.tar.bz2']
11:21:38 URL:http://stage.mozilla.org/pub/mozilla.org//firefox/releases/17.0esr/linux-i686/en-US/firefox-17.0esr.tar.bz2 [21262365/21262365] -> "firefox-17.0esr.tar.bz2" [1]
retry: Calling <function run_with_timeout at 0xb7e9633c> with args: (['wget', '--no-check-certificate', '-nv', 'http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/17.0.1esr-candidates/build1/linux-i686/en-US/firefox-17.0.1esr.tar.bz2'], 300, None, None, False, True), kwargs: {}, attempt #1
Executing: ['wget', '--no-check-certificate', '-nv', 'http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/17.0.1esr-candidates/build1/linux-i686/en-US/firefox-17.0.1esr.tar.bz2']
11:21:41 URL:http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/17.0.1esr-candidates/build1/linux-i686/en-US/firefox-17.0.1esr.tar.bz2 [21260829/21260829] -> "firefox-17.0.1esr.tar.bz2" [1]
SOURCE DIRECTORY ../../update
DESTINATION DIRECTORY .
UPDATE TYPE partial
PREPARE PATCH webapprt/webapprt.ini
PREPARE PATCH webapprt-stub
PREPARE ADD precomplete
PREPARE PATCH platform.ini
PREPARE PATCH omni.ja
PREPARE PATCH libxul.so
PREPARE PATCH libssl3.so
PREPARE PATCH libsoftokn3.so
PREPARE ADD libsoftokn3.chk
PREPARE PATCH libsmime3.so
PREPARE PATCH libplds4.so
PREPARE PATCH libplc4.so
PREPARE PATCH libnssutil3.so
PREPARE PATCH libnssdbm3.so
PREPARE ADD libnssdbm3.chk
PREPARE PATCH libnssckbi.so
PREPARE PATCH libnss3.so
PREPARE PATCH libnspr4.so
PREPARE PATCH libmozsqlite3.so
PREPARE PATCH libfreebl3.so
PREPARE ADD libfreebl3.chk
PREPARE PATCH firefox-bin
PREPARE PATCH firefox
PREPARE PATCH extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/install.rdf
PREPARE PATCH application.ini
EXECUTE PATCH webapprt/webapprt.ini
EXECUTE PATCH webapprt-stub
EXECUTE ADD precomplete
EXECUTE PATCH platform.ini
EXECUTE PATCH omni.ja
EXECUTE PATCH libxul.so
EXECUTE PATCH libssl3.so
EXECUTE PATCH libsoftokn3.so
EXECUTE ADD libsoftokn3.chk
EXECUTE PATCH libsmime3.so
EXECUTE PATCH libplds4.so
EXECUTE PATCH libplc4.so
EXECUTE PATCH libnssutil3.so
EXECUTE PATCH libnssdbm3.so
EXECUTE ADD libnssdbm3.chk
EXECUTE PATCH libnssckbi.so
EXECUTE PATCH libnss3.so
EXECUTE PATCH libnspr4.so
EXECUTE PATCH libmozsqlite3.so
EXECUTE PATCH libfreebl3.so
EXECUTE ADD libfreebl3.chk
EXECUTE PATCH firefox-bin
EXECUTE PATCH firefox
EXECUTE PATCH extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/install.rdf
EXECUTE PATCH application.ini
FINISH PATCH webapprt/webapprt.ini
FINISH PATCH webapprt-stub
FINISH ADD precomplete
FINISH PATCH platform.ini
FINISH PATCH omni.ja
FINISH PATCH libxul.so
FINISH PATCH libssl3.so
FINISH PATCH libsoftokn3.so
FINISH ADD libsoftokn3.chk
FINISH PATCH libsmime3.so
FINISH PATCH libplds4.so
FINISH PATCH libplc4.so
FINISH PATCH libnssutil3.so
FINISH PATCH libnssdbm3.so
FINISH ADD libnssdbm3.chk
FINISH PATCH libnssckbi.so
FINISH PATCH libnss3.so
FINISH PATCH libnspr4.so
FINISH PATCH libmozsqlite3.so
FINISH PATCH libfreebl3.so
FINISH ADD libfreebl3.chk
FINISH PATCH firefox-bin
FINISH PATCH firefox
FINISH PATCH extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/install.rdf
FINISH PATCH application.ini
succeeded
calling QuitProgressUI
SUCCESS: partial updates are smaller than complete updates, all is well in the universe
Looks like we downloaded the wrong build # for l10n repacks:
Downloading http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0esr-candidates/build1/update/linux-i686/eo/firefox-17.0esr.complete.mar to firefox-17.0esr.eo.complete.mar

vs. this for en-US:
python /builds/slave/rel-m-esr17-lnx-bld/tools/buildfarm/utils/retry.py -s 1 -r 5 -t 1260 wget -O previous.mar --no-check-certificate http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/17.0esr-candidates/build2//update/linux-i686/en-US/firefox-17.0esr.complete.mar


and it's cause is me tagging https://hg.mozilla.org/build/buildbot-configs/rev/916e0d32fbee with FIREFOX_17_0_1_{RELEASE,BUILD1} instead of FIREFOX_17_0_1esr_{RELEASE,BUILD1}.
I moved the 17.0.1 tags back to their rightful place. This accidental movement didn't hurt that release at all because its release config was different between the two tag locations.
I was going to drop l10n from 17.0esr in the patcher config, but since we don't do multiple partials for ESR I don't think that's necessary. We won't be rerunning updates for this release, and when we do 17.0.2esr, the only partial will be from 17.0.1esr.

We do need to separate out en-US and l10n in the update verify configs in order to properly test them both. This patch should get us testing complete+partial for en-US, and only completes for l10n
Assignee: nobody → bhearsum
Status: NEW → ASSIGNED
Attachment #686745 - Flags: review?(nthomas)
(In reply to Ben Hearsum [:bhearsum] from comment #4)
> I moved the 17.0.1 tags back to their rightful place. This accidental
> movement didn't hurt that release at all because its release config was
> different between the two tag locations.

Sorry, this should read *WASN'T* different between the two locations.
Comment on attachment 686745 [details] [diff] [review]
drop l10n partials from update verify configs

Looks fine from low earth orbit, any closer and I'd have to spend some time in therapy.
Attachment #686745 - Flags: review?(nthomas) → review+
Comment on attachment 686745 [details] [diff] [review]
drop l10n partials from update verify configs

Landed this, but update verify still failed because the parser doesn't like this file now:

Traceback (most recent call last):
  File "/builds/slave/rel-m-esr17-lnx-update-verify-1/scripts/scripts/release/updates/chunked-verify.py", line 66, in <module>
    verifyConfig.read(path.join(UPDATE_VERIFY_DIR, verifyConfigFile))
  File "/builds/slave/rel-m-esr17-lnx-update-verify-1/scripts/scripts/release/updates/../../../lib/python/release/updates/verify.py", line 75, in read
    self._addEntry(self._parseLine(line), first)
  File "/builds/slave/rel-m-esr17-lnx-update-verify-1/scripts/scripts/release/updates/../../../lib/python/release/updates/verify.py", line 64, in _addEntry
    self.addRelease(**releaseKeys)
  File "/builds/slave/rel-m-esr17-lnx-update-verify-1/scripts/scripts/release/updates/../../../lib/python/release/updates/verify.py", line 107, in addRelease
    raise UpdateVerifyError("Couldn't add release identified by build_id '%s' and from_path '%s': already exists in config" % (build_id, from_path))
release.updates.verify.UpdateVerifyError: Couldn't add release identified by build_id '20121119074111' and from_path '/firefox/releases/17.0esr/linux-i686/%locale%/firefox-17.0esr.tar.bz2': already exists in config
Attachment #686745 - Flags: checked-in+
This a nasty nasty hack, but it works around the parser error.
Attachment #686760 - Flags: review?(nthomas)
Really sorry for all the churn.
Attachment #686760 - Attachment is obsolete: true
Attachment #686760 - Flags: review?(nthomas)
Attachment #686764 - Flags: review?(nthomas)
Comment on attachment 686764 [details] [diff] [review]
only switch the path, not the filename too

r+ for the combined effect of attachment 686760 [details] [diff] [review] and this.
Attachment #686764 - Flags: review?(nthomas) → review+
All fixed now. Huge thanks to Nick for catching potential problems along the way.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: