573828 - "major update" failed to download file for 3.5.10 -> 3.6.4 MU ("Failed to remove patcher2.pid: No such file or directory")

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect, P5)

Description

[Armen [:armenzg]]

[104/447] firefox-3.5.10.gu-IN.mac.complete.mar... IN SHUTDOWN...
Failed to remove patcher2.pid: No such file or directory
IN SHUTDOWN...
Failed to remove patcher2.pid: No such file or directory
IN SHUTDOWN...
Failed to remove patcher2.pid: No such file or directory
DownloadFile(): FAILED: 0, output: --13:59:30--  http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1/update/mac/gu-IN/firefox-3.5.10.complete.mar
Resolving stage-old.mozilla.org... 10.2.74.116
Connecting to stage-old.mozilla.org|10.2.74.116|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17866723 (17M) [text/plain]
Saving to: `firefox-3.5.10.gu-IN.mac.complete.mar'


I have clobbered and re-triggered to continue.

Comment 1

[Armen [:armenzg]]

Related code:
http://mxr.mozilla.org/mozilla/source/tools/patcher/patcher2.pl#1526

Comment 2

[Armen [:armenzg]]

It seems that the actual problem was that stage-old.m.o was being used and that is not 100% reliable.

At the third attempt it worked so we never had to go and modofy the patcher-configs to use ftp.m.o instead of stage-old.m.o.

What should be the right solution so we don't hit this again?

Comment 3

[[e-mail] (:bhearsum)]

It surprises me that this is a problem specific to stage-old (aka surf). I've always assumed these sort of issues are a problem with the link overall. Armen, if you get a chance could you try running MajorUpdateFactory in staging, on a VM, and see if you can reproduce? If we can't reproduce on a VM, that points to the link being the problem.

If it is indeed stage-old that is the problem we could probably pass stageServer=ftpServer to MajorUpdateFactory, since it doesn't need to do any uploading. We couldn't do that for ReleaseUpdatesFactory, though. We might be able to modify the patcher bump script to use ftpServer for the completemarurl rather than stagingServer.

If we ever move to a two-tier stage/ftp setup (bug 394069) we'd have to go back to using the staging server.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Ben, my (hand waving) theory is that there is a fairly low rate of failures in requests to surf going on all the time. Sometimes that shows up as bogus data in the Castro proxy, but release jobs that do lots of downloads in one log make it more obvious to us. I don't have any hard data to back that up so I haven't filed it.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

This may not be an issue any more, as stage-old has the files mounted directly via NFS now. It could have been an issue with the way we were (transparently) proxying those requests back to ftp.m.o.

Comment 6

[Armen [:armenzg]]

That is good to know.

If we hit this issue again we could come back and opening it again.