Closed Bug 544587 Opened 16 years ago Closed 15 years ago

Broken addon after new version upload (-228 download error)

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jl, Assigned: chizu)

References

(
URL
)

Details

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.78 Safari/532.5 Build Identifier: I have uploaded a new version of my plugin. It failed the first time by just showing a white page. After fiddling around I finally got my addon uploaded after the third try. I've had these kind of problems before, as I'm not sure whether they are due to bugs or local network problems I ignored them. Now, one day later, I am surprised to see my addon cannot be downloaded anymore. Firefox gives a -228 error code (download failed), if I try to access the file directly(http://releases.mozilla.org/pub/mozilla.org/addons/46375/gutscheinrausch.de-2.3-fx.xpi) I get a Apache "Not found" page. Strangely, in the admin backend everything seems to be in order. There's a new version, a new file (which can't be found as in the frontend).. no errors or anything else to indicate some kind of problem. My addon code is fine btw, it works great when I install the addon from the local XPI file. I don't know if this is a general AMO problem or just an extraordinary problem, but any help to get this problem resolved would be greatly appreciated. - Johannes Reproducible: Always Steps to Reproduce: 1. Go to "https://addons.mozilla.org/de/firefox/addon/46375" 2. Click the green "add to firefox" button 3. Accept security warning & wait 3 seconds 4. Behold the -228 download error message Actual Results: Firefox konnte das Add-on von folgender Adresse nicht installieren: https://addons.mozilla.org/de/firefox/downloads/latest/46375/addon-46375-latest.xpi?src=addondetail Grund: Download-Fehler -228 Expected Results: just installing the addon would be great Uploaded a new file for the newly created version but now changes yet..
Severity: critical → normal
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P3
Target Milestone: --- → 5.7
Moving to critical, as there is huge pile of bug reports with this problem. Can you please fix this earlier than with the 3-weekly-AMO-update?
Severity: normal → critical
I've emailed with Jorge Villalobos about the issue. He advised me to release a new version. There was an error while uploading the file and some other strange behaviour. I'll paste the email I've sent yesterday, I think this can help to clear the issue: --- I've created a new version 2.4, only different by the version string. After the upload process, the upload page at https://addons.mozilla.org/de/developers/versions/add/46375 just kept spinning the AJAX wheel with the message "Add-on vervollständigen" (completing addon) After about a minute it said "Hoppla! Mit dieser Datei scheint es ein Problem zu geben..." (there seems to be a problem with the file" - please try again) I tried again, then it said the same error message and the notice "file already exists" In my backend addon overview I see about 7 new versions 2.4, about half of them without files. I have deleted the versions without files, I'm not sure about the other versions as they seem to point to the correct file (https://addons.mozilla.org/de/firefox/downloads/file/78888/gutscheinrausch.de-2.4-fx.xpi) ---
This happened a couple days ago and it was the cron job between AMO's PUBLIC_STAGING_PATH and ftp.mozilla.org. It had a stuck lock: rsync already running or /tmp/rsync-update.lock dangling -rw-r--r-- 1 root root 0 Feb 2 15:40 /tmp/rsync-update.lock Sending to IT to fix.
Assignee: nobody → server-ops
Severity: critical → major
Component: Public Pages → Server Operations
Product: addons.mozilla.org → mozilla.org
QA Contact: web-ui → mrz
Target Milestone: 5.7 → ---
Version: unspecified → other
Severity: major → critical
Assignee: server-ops → thardcastle
The problem was a missing nfs mount from the rebuild of mradm01. The rsync is running now.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
(In reply to comment #5) > The problem was a missing nfs mount from the rebuild of mradm01. The rsync is > running now. Can we add NFS mounts to nagios?
That was likely a one-time event (because of the upgrade from RHEL4 to RHEL5. Adding it to puppet is likely a better solution (that machine is on puppet now).
Thanks for helping out, the addon is downloadable again :) Nevertheless I think somebody should look into the upload issues (Upload failing after one minute, and especially having the same version 7 times after a second try).
Does (the browser or) the amo backend send some message somewhere when a 228 error happens? Perhaps it would be a good idea. Should the count suddenly jump to 10-fold, it would likely indicate a problem. Also, seems like 228 is some sort of catch all error, perhaps it could be more elaborate. I guess this entry also could be worked on to make it reflect the reality more. http://support.mozilla.com/en-US/kb/Unable+to+install+add-ons#Download_Error_228 Sorry for the bugspam, if that's the way you feel about this.
The -228 error just seems to be a reflection of the 404-HTTP error code. Strangely the download statistics still seemed to count installs, even when the version was broken.
My new uploaded addon file still not be synced: wget http://releases.mozilla.org/pub/mozilla.org/addons/4925/autopager-0.6.0.13-fx+mz+sm+fn.xpi --2010-02-07 16:43:25-- http://releases.mozilla.org/pub/mozilla.org/addons/4925/autopager-0.6.0.13-fx+mz+sm+fn.xpi Resolving releases.mozilla.org... 156.56.247.196, 149.20.20.5, 128.61.111.9, ... Connecting to releases.mozilla.org|156.56.247.196|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2010-02-07 16:43:26 ERROR 404: Not Found.
I experienced the -228 error today with an add-on upload. The file upload seemed to complete, but updating release notes failed with an error. Many users noticed. After a couple hours I uploaded a new version (leaving the old one in place). Shortly after, the error cleared up. This all started around 4AM PST IIRC while uploading version 0.6.20100206. https://addons.mozilla.org/en-US/firefox/addon/4908
My addon https://addons.mozilla.org/en-US/firefox/addon/4925 still has issue in update. Could you please have a look? Thanks Wind
Multiple comments concerning -228 errors, so reopening and marking as blocker.
Assignee: thardcastle → server-ops
Severity: critical → blocker
Status: RESOLVED → REOPENED
OS: Windows 7 → All
Priority: P3 → --
Resolution: FIXED → ---
The problem is permissions getting reverted, looking into why.
Assignee: server-ops → thardcastle
Status: REOPENED → ASSIGNED
Wind Li: something must have happened when you uploaded it; please upload the file again on https://addons.mozilla.org/en-US/developers/versions/edit/94674 . I removed the existing file entry since it pointed to a non-existing file.
(In reply to comment #12) > Shortly after, the error cleared up. Looks like the errors for my add-on have continued all day. I'm still getting reports coming from users. I just tried myself and it's -228 again. If I deleted both of my updates from earlier today (the next-most-recent was Jan 12) would that stop the error for now?
(In reply to comment #16) > Wind Li: something must have happened when you uploaded it; please upload the > file again on https://addons.mozilla.org/en-US/developers/versions/edit/94674 . > I removed the existing file entry since it pointed to a non-existing file. Done.
FWIW, the downloads will work immediately after it is uploaded because the addons site will serve the first directly for the first 30 minutes after you upload it, to allow time for the mirrors to pick it up, before it starts redirecting to the mirrors for downloads. It'll then stop working if the mirrors haven't actually picked it up. Deleting the files doesn't really solve anything. When the sync is fixed, they'll actually show up. This is among the reasons I always objected to the arbitrary "wait a set amount of time" thing and thought it would be better to check if the files were actually on the mirrors before starting to redirect. From the traffic I see on IRC from overnight, I think this did get fixed last night, can anyone verify? The machine running the rsync process was recently added to our puppet configuration management, and the ssh private key used for the rsync process has a different filename than the ones puppet knew about, so it was getting its permissions reset by puppet every time puppet scanned the machine and openssh didn't like it. The filename for that key file was added last night to the list of things puppet should ensure owner-only permissions on.
(In reply to comment #19) > From the traffic I see on IRC from overnight, I think this did get fixed last > night, can anyone verify? It's been working for me. For my add-on, install from AMO has worked for me every time I've checked over the last 24 hours. I've checked maybe 8 times. I've stopped getting complaints from users. (Thanks for saying something about what's going on behind the scenes. That makes it easier to know what actions as add-on author would make things better or worse.)
(In reply to comment #21) > It's still not working for me. > File: > https://addons.mozilla.org/en-US/firefox/downloads/file/73248/snip_it!_button_for_ebay-1.5.1-fx.xpi > Add-on: https://addons.mozilla.org/en-US/firefox/addon/55671/ > I've been directed here from > https://forums.addons.mozilla.org/viewtopic.php?f=20&t=490 This works for me. The install button installed v1.5.1
(In reply to comment #22) > (In reply to comment #21) > > It's still not working for me. > > File: > > https://addons.mozilla.org/en-US/firefox/downloads/file/73248/snip_it!_button_for_ebay-1.5.1-fx.xpi > > Add-on: https://addons.mozilla.org/en-US/firefox/addon/55671/ > > I've been directed here from > > https://forums.addons.mozilla.org/viewtopic.php?f=20&t=490 > > This works for me. The install button installed v1.5.1 That's why I provided a link to forum post. Installation works. Even XPI file is accessible from the browser. What's broken is updating from earlier version that leads to -228 error.
The rsync has been successfully running since the ssh key fix. Marking this fixed.
Status: ASSIGNED → RESOLVED
Closed: 16 years ago → 15 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.