537720 - Beta Channel add-on updates are missing from http://releases.mozilla.org servers

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Public Pages, defect, P2)

Description

[Michael Kraft [:morac]]

My add-on, Session Manager, has a beta channel version which I updated over the weekend.  I received notification of the new version today when I got into work, but installing it always fails with a download error of -228.

The download URL is listed as "http://releases.mozilla.org/pub/mozilla.org/addons/2324/session_manager-0.6.8pre20100103-fx+sm.xpi" which is a problem since none of the beta versions are listed at "http://releases.mozilla.org/pub/mozilla.org/addons/2324/".  Only the release versions are available there.  No beta versions (even the older one) are listed there.  I've tried reloading a number of times and it's the same on all the mirrors.

The update can be manually installed from https://addons.mozilla.org/en-US/firefox/downloads/file/74461/session_manager-0.6.8pre20100103-fx+sm.xpi so the updated file is available, just not at the location that the update manager is trying to retrieve it from.

Are the beta updates supposed to be on the http://releases.mozilla.org servers and are simply not there or are they hosted elsewhere?

Comment 1

[Wil Clouser [:clouserw]]

Attachment: fix?

This would make it copy the beta files as well.  There isn't a reason to keep those non-public, right?

Comment 2

[Dave Garrett]

Experimentals aren't there either, so I was under the impression that only release versions were put there to avoid having non-reviewed files there. Allowing betas would circumvent that, I think. (though, this isn't necessarily an argument against doing so)

Are files deleted off of there if removed from AMO? In other words, if a dev didn't want to leave their beta hosted publicly forever, could they remove it later?

Comment 3

[Michael Kraft [:morac]]

Can't the the updateLink in the beta channel update.rdf returned from versioncheck.addons.mozilla.org simply point to the file as hosted on https://addons.mozilla.org instead of https://releases.mozilla.org?  

If you are worried about server load, I wouldn't expect a large number of people to be using beta add-ons so there's shouldn't be a lot of extra load.  In other words, I don't think mirrors are required for beta updates.  I supposed you could set up a https://betas.mozilla.org or something if push came to shove.

To simply things, here's the URL for the beta channel update request on my machine.  Just plug it in to see the returned updateLink value:

https://versioncheck.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id={1280606b-2510-4fe0-97ef-9b5a22eafe30}&version=0.6.8pre20091226&maxAppVersion=3.7a1pre&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=3.5.6&appOS=WINNT&appABI=x86-msvc&locale=en-US&currentAppVersion=3.5.6

Comment 4

[Jeff Balogh (:jbalogh)]

Comment on attachment 419981 [details] [diff] [review]
fix?

r- on line 1, r+ on line 2.

>-            INNER JOIN addons ON versions.addon_id = addons.id AND addons.status = 4 AND addons.inactive = 0
>-            INNER JOIN files ON files.version_id = versions.id AND files.status = 4
>+            INNER JOIN addons ON versions.addon_id = addons.id AND addons.status IN (4, 7) AND addons.inactive = 0

Add-ons shouldn't have a status of 7, they should be public.  One of the prerequisites for us allowing betas is a public addon, so you can leave this line alone.

>+            INNER JOIN files ON files.version_id = versions.id AND files.status IN (4, 7)


At first glance, I also wondered why beta would go to releases when sandbox add-ons don't.  Then I remembered that we don't do updates for sandboxed add-ons at all, so it doesn't make sense to mirror them.

But we do want the mirror for beta, so that 1) there's no special casing of where to find add-ons, and 2) if beta traffic grows, we'll be ready to handle it.

Comment 5

[Jeff Balogh (:jbalogh)]

(In reply to comment #4) 
> >+            INNER JOIN files ON files.version_id = versions.id AND files.status IN (4, 7)

Why are you using numbers instead of constants?

Comment 6

[Wil Clouser [:clouserw]]

Thanks, r58926.  Numbers because it's consistent with the rest of the function/file and we'll push to releases.m.o because there isn't a reason not to.  If you delete your file on AMO it'll be removed the from mirrors once they sync (an hour or so).

Comment 7

[Dave Garrett]

Is there still a push needed for this? I'm using an extension in beta channel and am erroring trying to update it. (Update Scanner 3.1.0beta1 -> 3.1.0beta2)

Comment 8

[Wil Clouser [:clouserw]]

(In reply to comment #7)
> Is there still a push needed for this? I'm using an extension in beta channel
> and am erroring trying to update it. (Update Scanner 3.1.0beta1 -> 3.1.0beta2)

I'm not sure what the problem is.  My patch above should have fixed this, but the files still aren't on releases.m.o.  I'll ask IT tomorrow.

With the modified query:

mysql> SELECT DISTINCT addons.id as addon_id, files.filename as filename FROM versions INNER JOIN addons ON versions.addon_id = addons.id AND addons.status = 4 AND addons.inactive = 0 INNER JOIN files ON files.version_id = versions.id AND files.status IN (4, 7) AND addon_id = 2324 ORDER BY addons.id DESC;

  .....
|     2324 | session_manager-0.6.7.3-fx+sm.xpi          | 
|     2324 | session_manager-0.6.7.4-fx+sm.xpi          | 
|     2324 | session_manager-0.6.8pre20091226-fx+sm.xpi | 
|     2324 | session_manager-0.6.8pre20100103-fx+sm.xpi | 
+----------+--------------------------------------------+

Comment 9

[Wil Clouser [:clouserw]]

Attachment: fix the real problem

Well, this is not pretty.

So, my original thought was the publish_files maintenance job wasn't doing what it needed to do (the original patch).  It turns out, that maintenance job is no longer run (!?).  Instead, the copyFileToPublic() function was duplicated in the AMO Component and is now called on the fly each time it needs to be.  This patch fixes the behavior there so future uploads should work as expected.

Comment 10

[Wil Clouser [:clouserw]]

Attachment: temporary patch for the maintenance script

Even though the real fix is in place, we still should copy the files that were uploaded in the mean time but didn't make it to releases.m.o.  This is a temporary patch for the maintenance script which will change it to copy any beta files to releases.m.o.

The first change only looks at beta stuff, the second tells it not to overwrite by default, and the third is fixing some obvious fail.

In theory, we could just run the maintenance script as is and it would work fine, but it would also copy every version of every add-on over and we don't need that.  There are only a couple dozen files that need to go.

Comment 11

[Jeff Balogh (:jbalogh)]

Comment on attachment 422593 [details] [diff] [review]
fix the real problem

Could you make an RSYNCABLE_STATUS array in constants that controls this?

Even better would be to pass the status as a param of copyFileToPublic unconditionally, and let that function figure out if this addon should be copied.

With one or both of those changes, I'm r+.

Comment 12

[Wil Clouser [:clouserw]]

Alright, I added an rsyncable_status global in r60630.  I don't want to mess with copyFileToPublic; we can fix it in [z].

That should fix this bug.  In addition, I've filed bug 540933 to fix the currently uncopied add-ons and I'll write a note to run it again when we do the 5.6 push.

Comment 14

[Dave Garrett]

The beta files all seem to be up on releases.mozilla.org now. When will it be possible for one of us to upload a new beta file and have it copied over automatically? (I'd like to be able to post a Flagfox3.4beta2 in the not too distant future)

Comment 15

[Wil Clouser [:clouserw]]

Dave: betas that have been stuck in limbo are on releases.m.o, but new betas won't be on there until the next push; around Feb 2nd

Comment 16

[krupa raj[:krupa]]

QA is not clear on how to verify this bug on preview.Can anyone specify the steps to verify?

Comment 17

[Michael Kraft [:morac]]

It requires having an existing add-on on AMO and then submitting a beta version of that add-on.  See the following URL on how to upload a beta version:  https://addons.mozilla.org/en-US/developers/docs/policies/maintenance#beta-addons

The beta version should show up in the list of files at http://releases.mozilla.org/pub/mozilla.org/addons/####/ where #### is the add-on's number (the number that appears at the end of the URL on the add-on public AMO page).

Comment 18

[Stephen Donner [:stephend] Not actively reading bugmail]

I don't think we can test this on preview, since I don't think we have a staging server for release.mozilla.org; correct me if I'm wrong.

Comment 19

[Dave Garrett]

So, there was a push (bug 543514) and patches should be live now unless I misread something. I don't see Flagfox3.4beta2 (which I upload last week) on releases yet, so I guess bug 540933 comment 0 point #3 is up and a new run of the script is still needed to copy over beta files since the last run?

Comment 20

[Wil Clouser [:clouserw]]

It's on ftp now and will be rsync'd to releases.m.o shortly.  There was a stuck lock.

Comment 21

[Dave Garrett]

Thanks, works now. I successfully updated a profile from 3.4beta1 to 3.4beta2.

Comment 22

[Stephen Donner [:stephend] Not actively reading bugmail]

Thanks for commenting back, Dave; verified fixed based on comment 21.

Comment 23

[Dave Garrett]

Well, that just verifies that existing files were copied over and can be updated to. I'll be able to tell you for sure if this is fully fixed after I upload a 3rd (and probably final) beta.

Comment 24

[Michael Kraft [:morac]]

I'm still seeing the problem.

How long after an upload is done should it get pushed to r.m.o?  

I just uploaded a new beta version of Session Manager (0.6.8pre20100125) and
I'm getting an update notification when I check for a new version, but the file
is still missing off of r.m.o. so the update fails.

Comment 25

[Dave Garrett]

(In reply to comment #24)
> I just uploaded a new beta version of Session Manager (0.6.8pre20100125) and
> I'm getting an update notification when I check for a new version, but the file
> is still missing off of r.m.o. so the update fails.

That version is up there:
http://releases.mozilla.org/pub/mozilla.org/addons/2324/

Could've just taken a while to get copied over.

> How long after an upload is done should it get pushed to r.m.o?  

That's never been instantaneous; there's always a little bit of a wait. Don't know exactly how long, though.

Comment 26

[Dave Garrett]

I uploaded a 3rd Flagfox beta and after an hour or so was able to successfully update to it. So it's a bit slow, but it does all work now. :)

Comment 27

[Michael Kraft [:morac]]

I still don't think it's working 100%.  That or it's very, very slow to update.

I uploaded another Session Manager beta (0.6.8pre20100213) yesterday and 24 hours later, it's still not available as an update.

Comment 28

[Dave Garrett]

I've uploaded a couple other betas since the last time I commented here and they were up almost immediately. There may be some inconsistency here and I was just lucky. This initial bug, however, seems to be resolved (they weren't set to copy over at all before) so it may be a good idea to file a new bug for this new issue you're having. Maybe comment 20 again?

Comment 29

[Michael Kraft [:morac]]

Well I would expect it to copy over within a few hours.  It's been over a day (I actually uploaded on Saturday) so I'll file another bug I guess.   Before I do that though, maybe I'll try uploading it again.