Closed Bug 373351 Opened 18 years ago Closed 15 years ago

Update tools don't error out/catch upon OOM/OOD errors

Categories

(Release Engineering :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 511967

People

(Reporter: cbook, Assigned: coop)

Details

Its not possible to run the latest nightly 20070309 when you update from 2007030804 via Aus. During the Update the firefox file(s) are missing and so firefox refuse to start. Steps to reproduce: Download a nightly from a day before like http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2007-03-08-04-mozilla1.8/firefox-2.0.0.3pre.en-US.linux-i686.tar.gz Use AUS - the update is found and choose Firefox restart -> Firefox fails to start because of missing firefox file
Problem was reproduced on Linux (sorry missed that in comment#0)
Summary: AUS Update to latest nightly is missing files → Firefox 1.8 Branch Linux AUS Update to latest nightly is missing files
It applies the patch and then exits. The files missing from the root directory are: firefox firefox-bin libfreebl3.chk libfreebl3.so libnspr4.so libnss3.so libnssckbi.so libsmime3.so libsoftokn3.chk libsoftokn3.so libssl3.so libxpcom_compat.so libxpistub.so mozilla-xremote-client removed-files run-mozilla.sh updater xpicleanup The files still there are several times larger than they are originally (debug info?)
Assignee: nobody → build
Component: Software Update → Build & Release
Product: Firefox → mozilla.org
QA Contact: software.update → preed
Version: 2.0 Branch → other
Make that prometheus-vm instead of prometheus.
Summary: Firefox 1.8 Branch Linux AUS Update to latest nightly is missing files → prometheus-vm disk full -- Firefox 1.8 Branch Linux update to latest nightly gives broken install
Assignee: build → rhelmer
I've cleared some space by removing old mozilla checkouts used to build the mar/mbsdiff tool in the past, but it's still pretty full. I am looking at the disk usage now to see if there are any other band-aids to apply. I've removed the bad update from both stage and AUS. It's been moved to a private area in case we want to examine it, but I'd guess that the cause is that tinderbox didn't check for errors when generating the full mar file and uploaded it anyway.
Status: NEW → ASSIGNED
Status: ASSIGNED → RESOLVED
Closed: 18 years ago
Resolution: --- → FIXED
I'm reopening this; we shouldn't publish nightly updates that are broken and/or failed. The patcher tools need to handle this more elegantly.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee: rhelmer → build
Status: REOPENED → NEW
Component: Build & Release → Software Update
OS: Linux → All
Product: mozilla.org → Firefox
Hardware: PC → All
Summary: prometheus-vm disk full -- Firefox 1.8 Branch Linux update to latest nightly gives broken install → Update tools don't error out/catch upon OOM/OOD errors
Version: other → 2.0 Branch
Severity: blocker → critical
Priority: -- → P3
Component: Software Update → Build & Release
Product: Firefox → mozilla.org
Version: 2.0 Branch → other
Assignee: build → nobody
QA Contact: mozpreed → build
Component: Release Engineering → Release Engineering: Future
QA Contact: build → release
We have nagios running on all the builders, so this hasnt happened in a while. However the root problem is still here, and still a valid concern: a problem with the update tools should never fail silently, allowing us to continue along and eventually send broken updates to users.
Just to clarify, the update tools run on prometheus-vm, not on the builders. But we also have nagios monitoring prometheus-vm.
Mass move of bugs from Release Engineering:Future -> Release Engineering. See http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Assignee: nobody → ccooper
Severity: critical → normal
This will be fixed when we start generating updates on the slaves rather than prometheus-vm (bug 511967). We already have nagios monitoring and required build space checks prior to every build on the slaves.
Status: NEW → RESOLVED
Closed: 18 years ago15 years ago
Resolution: --- → DUPLICATE
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.