Closed Bug 1074960 Opened 10 years ago Closed 10 years ago

ftp.mozilla.org/pub/addons/ hasn't updated since 08/2014

Categories

(Cloud Services :: Operations: Marketplace, task, P1)

Product:
Cloud Services
Component:
Operations: Marketplace
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: jorgev, Assigned: jason)

References

Details

This was brought up recently in the forum: https://forums.mozilla.org/viewtopic.php?f=20&t=20195. It looks like the FTP repository of add-on files at ftp://ftp.mozilla.org/pub/addons/ stopped updating about a month ago.
Assignee: server-ops → nobody
Component: FTP: Mirrors → Operations: Marketplace
Product: mozilla.org → Mozilla Services
QA Contact: operations-mkt
Assignee: nobody → jthomas
Priority: -- → P1
The addons.mozilla.org/public-staging directory, which is the directory we sync to ftp, contains no new files since Aug 14th.  It looks like this corresponds to 2014.08.13-2 prod push which contained changes to where addons are stored on the filesystem.

We have addons (including LWT) stored at addons.mozilla.org/shared_storage/uploads/addons. We could exclude LWT and sync this path to FTP instead. Are the addons in this path okay for public consumption?
Group: mozilla-employee-confidential
Mathieu might know if public-staging is used anymore in the code.  It might be just outdated code.
Flags: needinfo?(mathieu)
Is this something we actually care about?
I think that is the, important, question.

We'd have to sync the entire addons directory, with the changes in https://github.com/mozilla/olympia/commit/a35ab0834d69c1d8317941725b17afd99d109d2c and https://github.com/mozilla/olympia/commit/2638508cbc3d1bc5d3f380b43f689e2655454f25 .

Who can make the final decision about this?
I have no knowledge of public-staging or anything related to this bug I'm afraid. Please, if it's an information I can find, give me a few hints as two where I should look.
Flags: needinfo?(mathieu)
(In reply to Jeremy Orem [:oremj] from comment #4)
> Who can make the final decision about this?

Do we have stats for the ftp directory? I'd like to know if we are breaking stuff for a handful of people or if there's a significant audience.
I don't think this matters. We've never published this as a location to retrieve add-ons. Add-ons are organized in a cryptic and user-unfriendly way, using unpublished numeric add-on IDs. We've never linked to it directly, and have only ever pointed to the mirrors via 302s. We don't need it for distribution anymore, since we've moved from the mirror network to the CDNs.

If people are using it for anything, they should migrate to use the CDNs. I don't see this as something we have any obligation to support.
(In reply to Kris Maglione [:kmag] from comment #7)
> If people are using it for anything, they should migrate to use the CDNs. I
> don't see this as something we have any obligation to support.

Correct - no obligation.  The only reason it came up this time is because someone was spidering AMO to download all the add-ons and I suggested they just use FTP.  If that's no longer an option I don't mind.

Mathieu - I CC'd you just because if this isn't necessary anymore maybe there is some code we can strip out around the PUBLIC_STAGING URLs.  I'm not sure.

I'm fine with wontfixing this bug, just wanted everyone to be aware.
(I suppose we shouldn't just wontfix it - we should delete that dir off of FTP)
I'm not sure we really want to facilitate spidering all of the add-ons on AMO. I'm not really opposed to it, but they come with all sorts of licenses, and I suspect that a lot of developers would object to making it easier than it has to be.

In any case, I suspect that it might be easier to turn on directory indexes for the https://addons.cdn.mozilla.net/user-media/addons/ directory on the CDNs rather than attempting to keep ftp up-to-date.

Either way, I agree that if we're not going to go on syncing this, the directory should be removed.
For that matter, if we care about supporting this, it would probably be the work of about 5 minutes to just spit out a cashed JSON or CSV dump of all of the add-ons on the site with download URLs.
I know we said we'd wontfix this.  An interesting use case came up in bug 1085142 - AMO doesn't have a way for people to download add-ons for other operating systems.  The filer of that bug is a sysadmin who downloads the new versions of add-ons for other OS's from FTP.

That's the best case I've heard of for keeping it around -- as far as I know that functionality isn't anywhere on the site?
(In reply to Wil Clouser [:clouserw] from comment #12)
> That's the best case I've heard of for keeping it around -- as far as I know
> that functionality isn't anywhere on the site?

Right. The easiest way one could do that is to change the UA string so you see the button for the other platforms. It'd be good if we offered those links in the All Versions page, since that seems like the right place for them. It would also be easier to maintain and keep up to date than the FTP repo, no?
I don't think that's a good reason for keeping this. It may be useful to the 3 users who know how to find an add-on on the FTP site, but that's about it. If they need access to XPIs for other platforms, let's just fix that. They can file a bug and assign it to me.
Fair enough.  I was going to dupe the bug, but this one is confidential for some reason.  Any reason we can't open it up so the filer can see it?
I'm okay with opening up this bug.
Group: mozilla-employee-confidential
I've filed bug 1088088 with IT Webops to remove /pub/addons from product delivery.
Removed.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
(In reply to Kris Maglione [:kmag] from comment #14)
> I don't think that's a good reason for keeping this. It may be useful to the
> 3 users who know how to find an add-on on the FTP site, but that's about it.
> If they need access to XPIs for other platforms, let's just fix that. They
> can file a bug and assign it to me.

That would be bug 685721, which is currently in state "RESOLVED WONTFIX".
After resolving this bug, I was contacted by the Cuban community about it, since they were active users of the FTP repo.

Cuba has limited Internet access, so the local community has their own Mozilla site where Firefox and add-ons can be downloaded and installed. They are mirroring our data in a very ad-hoc way, and they used the FTP repo because it allowed them to sort by date to pick up on newer versions, as well as easily download the files for all platforms of an add-on version.

Can anyone think of a good way for them to set this up without the FTP repo?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Is like say Jorge, it is the Cuban site http://addons.firefoxmania.uci.cu/
Please tell us more about the flow here.  It looks like someone is posting new versions of add-ons in a wordpress install?  If that's the case, what would be the difference of going to the detail page and clicking download (assuming bug 685721 was fixed)?
They specifically mentioned sorting folders by date as a way to figure out which add-ons were recently updated. Do we have an AMO page or API call that returns a list of published add-ons sorted by the date of their last version?
(In reply to Wil Clouser [:clouserw] from comment #23)
> Please tell us more about the flow here.  It looks like someone is posting
> new versions of add-ons in a wordpress install?  If that's the case, what
> would be the difference of going to the detail page and clicking download
> (assuming bug 685721 was fixed)?

When I using Linux and visit AMO, it only allowed to my download the add-on version for that platform (if exists). In case that only have a version for all system, not need other links.
In our WP site, we detected the system of user and if the add-on is available for user platform, bring the installation or download (if is only for Thunderbird). 
For do that, we maintain the same name of ftp.mozilla.org/pub/addons/ (fx-tb-mac-windows-linux)
For name the add-ons correctly, we need a way where view the list of extensions and all versions.
Actually we have more than 450 usefully add-ons for Cuban community, including the featured add-ons selected by Members Board.
Hi people. Any update about it?
Why I can't access to bug #1088088?
Flags: needinfo?(clouserw)
Bug 1088088 is an infrastructure but. For security reasons, those bugs cannot be public.
I don't have another solution for sorting add-ons by last updated other than visiting each one individually.
Flags: needinfo?(clouserw)
Could we implement an API call that returns a list of add-ons that have been updated since a given date? Dealing with platform-specific files isn't that big of a problem, I think.
(In reply to Wil Clouser [:clouserw] from comment #28)
> I don't have another solution for sorting add-ons by last updated other than
> visiting each one individually.

I think that solution is publish again the addons folder on Mozilla FTP. Suppose that I use the default way [1] and [2], however may very hard and complicated update addons and themes.
For example, FXChrome is available for Windows, Mac and Linux, if I use Linux when visit AMO the URL for install not are the for all platforms, on Linux is [3] but if "platform:2" is replaced by "platform:1" or "platform:3", AMO answer not found.
So, for obtain the trusted URL for Mac and Windows, I should change my User Agent each time if addon is available for others platform.

Too think that solution of Jorge say sound good for us. After all, probably we (Mozilla Cuba) are the only who have a site like AMO. Many people miss the update of addons in our site.

I add to mail list of this bug to Amy Tsay, Guillermo Movia and Ruben Martin.


[1] https://addons.mozilla.org/en-US/firefox/extensions/?sort=updated
[2] https://addons.mozilla.org/en-US/firefox/complete-themes/?sort=updated
[3] https://addons.mozilla.org/firefox/downloads/latest/287749/platform:2/addon-287749-latest.xpi?src=cb-dl-users
I liked the previously way because allowed sort by date addons and saw all versions (stable, beta, etc). We always published the stable version and in case of add a new addon, first check in AMO if it was approved.
Like solution, we settled whit only show the last updated addons in a month.

Please help us. :*(
I don't think that making XPIs for other platforms only available by FTP, with an undocumented and arcane directory structure, which only a few people even know about, is a remotely desirable solution.
I don't have strong feelings on this other than not wanting to spend more time on it.  Jason: how much work is it to put the FTP structure back?
Putting the FTP structure back wouldn't be difficult. However we still need to fix the issue of what addons directory we will be syncing since the previous 'public-staging' directory is out dated due to changes (comment 1 and comment 4).
I'm currently maintaining a list of URLs here: https://people.mozilla.org/~kmaglione/latest_addons.tsv

Please sync to this rather than FTP.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → WONTFIX
(In reply to Kris Maglione [:kmag] from comment #35)
> I'm currently maintaining a list of URLs here:
> https://people.mozilla.org/~kmaglione/latest_addons.tsv
> 
> Please sync to this rather than FTP.

Thank you a lot!
> I'm currently maintaining a list of URLs here:
> https://people.mozilla.org/~kmaglione/latest_addons.tsv
> 
> Please sync to this rather than FTP.

This is a great help for now but it would be good to have a guaranteed, published, up-to-date source. In particular I have periodically done some analysis and metrics of the AMO ecosystem and I need a place to point my scripts at. Should I open a new bug for that? Thanks!
I'd like to see this data published as a JSON feed on AMO, so feel free to file a bug.

In the mean time, that list is being kept up-to-date with a nighty cron job. If you find it's not up-to-date when you need it, ping me and I'll fix it.
Filed bug 1140538, thanks.

Dave
You need to log in before you can comment on or make changes to this bug.