482160 - Replace Bouncer/Sentry with MirrorBrain

Status: RESOLVED WONTFIX

(Webtools :: Bouncer, defect)

Description

[Dave Miller [:justdave]]

A suggestion was posted at https://wiki.mozilla.org/Talk:MirrorNetwork to take a look at MirrorBrain...  I'd never heard of it before, but it looks like Novell/SuSE's independent attempt to pull off the same thing we did with Bouncer, and it looks like they've gotten a lot further along with it than we have, and they already have several of the features we've been complaining about Bouncer/Sentry lacking, and it is open source.

This might be worth looking into, to avoid a lot of unnecessary work if it turns out they've already done what we need, or if it would be less work to make what they had do what we need than to make Bouncer/Sentry do what we need.

Comment 1

[Peter Poeml]

Hi! I am the developer of MirrorBrain and I'd be very happy to help.

Comment 2

[Michael Morgan [:morgamic]]

Peter - Frédéric will be taking a look at MirrorBrain to see if it meets Mozilla's needs.  Thanks for offering to help, he will be in touch with you.  Are you on IRC?  Join us in #webdev sometime and we can discuss. :)  Thanks!

Comment 3

[Peter Poeml]

Hi! Cool, I visited #webdev shortly but for now I have to leave for two weeks of vacation. I'll be back on the 20th of April. Looking forward to meeting you and discuss your needs :) In general you can find me on the #mirrorbrain channel on freenode. See you then :-)

Comment 4

[Michael Morgan [:morgamic]]

Fred took at look at this, and posted his thoughts here:
https://wiki.mozilla.org/User:Wenzel/Bouncer:MirrorBrain

It looks like it supports mysql as well.  I think it's a viable replacement for Bouncer and we'd be willing to work on a front-end for it.  It could probably also make some use of Lars' work on Sentry back in the day (at least for logging).

Peter - would you be willing to chat for a bit with us?  Would like to go through some of desired features in the Bouncer bug list and figure out where our needs intersect with MB.  I think MB is a great project and accomplished a lot of the things Bouncer did not -- so if we can help with the web front end and improved monitoring/logging we're definitely interested.

Comment 5

[Michael Morgan [:morgamic]]

Making this a tracking bug.

Comment 6

[Michael Morgan [:morgamic]]

So, some things we're going to have to think about and deal with:
- comparative results between MB and Bouncer -- might be surprising; our file distribution is a bit different (more files, smaller, more mirrors)
- if we have to replace the redirect script (mod_mirrorbrain) with PHP or something easier to maintain, how do we use mod_asn?  replace with another script?
- can MB work with a slave DB?  we do this with bouncer right now, and have to.
- will MB stay up with real-time logging?  Bouncer died.
- admin interface - this is a must-have, and we'll pitch in for this
- mirror contacts and ACLs - allowing mirror admins to update their own info is necessary

More general stuff here:
https://wiki.mozilla.org/Bouncer/Sentry_Requirements

Comment 7

[[e-mail] (:bhearsum)]

Any idea of a timeline for rolling this out? bugs like 394498 are pretty strongly affected by this.

Comment 8

[Peter Poeml]

(In reply to comment #4)
> Fred took at look at this, and posted his thoughts here:
> https://wiki.mozilla.org/User:Wenzel/Bouncer:MirrorBrain

Yes, I saw that page, and Fred also sent mail, which I replied to in length. I ought to comment on some of the things listed on the wiki page.

> It looks like it supports mysql as well.  I think it's a viable replacement for
> Bouncer and we'd be willing to work on a front-end for it.  It could probably
> also make some use of Lars' work on Sentry back in the day (at least for
> logging).

That sounds great. A web frontend could be easily done in e.g. Django. Some of the features or Sentry would be a great fit into MirrorBrain I think, and it should be easy to integrate them. (Best in the form of Python modules that can be used both by commandline tools as well as by the web frontend, I'd think.)
 
> Peter - would you be willing to chat for a bit with us?  Would like to go
> through some of desired features in the Bouncer bug list and figure out where
> our needs intersect with MB.  I think MB is a great project and accomplished a
> lot of the things Bouncer did not -- so if we can help with the web front end
> and improved monitoring/logging we're definitely interested.

Thank you very much! Yes, I'd be very interested in discussing your needs in depth. And any contribution will be very welcome of course.

Comment 9

[Peter Poeml]

(In reply to comment #6)
> So, some things we're going to have to think about and deal with:
> - comparative results between MB and Bouncer -- might be surprising; our file
> distribution is a bit different (more files, smaller, more mirrors)

How many files, how many mirrors?

> - if we have to replace the redirect script (mod_mirrorbrain) with PHP or
> something easier to maintain, how do we use mod_asn?  replace with another
> script?

What would the PHP redirect script do that mod_mirrorbrain can't? I'd consider adding missing features or add flexibility. For example, it could make sense to use mod_mirrorbrain for lookup only, so it doesn't actually returns a redirect, but just pass the mirror selection to another module or script. It could already be done. Or what would you need? And what would it be that could be harder to maintain than a script?

> - can MB work with a slave DB?  we do this with bouncer right now, and have to.

It depends on what you mean - mod_mirrorbrain needs read-only access to the database only, thus it'll work with a read-only slave for instance. 

> - will MB stay up with real-time logging?  Bouncer died.

I am not exactly sure what you mean here. Which kind of logging? Apache logs - no problem that I'm aware of. 

I have written up a plan for logging download statistics (http://mirrorbrain.org/download-statistics/) which would also scale up very well. I wouldn't expect problems with that. However, it might make sense if you could give me some indication of how many requests per seconds you serve. The experience I have is in the range of 200-500req/s and there is a lot of headroom for more.

> - admin interface - this is a must-have, and we'll pitch in for this

That would be totally awesome!

I would also be interested in working on this, and I considered Django for this.

> - mirror contacts and ACLs - allowing mirror admins to update their own info is
> necessary

That would be a nice feature, and it would also allow mirror admins to take their mirror offline when maintaining it, or triggering a scan.

>  More general stuff here:
> https://wiki.mozilla.org/Bouncer/Sentry_Requirements

Yup. 

I am travelling right now, and don't have Internet access all the time, but I'll be happy to discuss later in more depth.

There are some things that I need to understand better, in regard of the needs that you have, and the means that you currently use to solve these challenges.

Thanks!

Comment 10

[matthew zeier [:mrz]]

> > - if we have to replace the redirect script (mod_mirrorbrain) with PHP or
> > something easier to maintain, how do we use mod_asn?  replace with another
> > script?
> 
> What would the PHP redirect script do that mod_mirrorbrain can't? 

Unbind me from Apache.

Comment 11

[Michael Morgan [:morgamic]]

Peter - do you have a VM that could help us get a head start on staging a parallel test server?

Comment 12

[Peter Poeml]

There is one here: http://mirrorbrain.org/news/mirrorbrain-eval-virtualbox-appliance/
The VM has actually been used once as starting point for a productive system.

Comment 13

[David Farning]

We have successfully been using mirror_brain as the Content Distribution Network for activities.sugarlabs.org at Sugar Labs for a month.

We rsync the Remora staging directory to http://download.sugarlabs.org/activities/ . Individual mirrors rsync against dl.sl.o .  Mirror_brain then scans the mirrors via a cron job to see what files are available at each mirror.  Download requests to sl.sl.o are redirected to the closest available  mirror.

From the Remora logparser POV mirrorbrain is transparent.  Parsing the Remora access logs gives correct download statics for individual activities.

Comment 14

[Fred Wenzel [:wenzel]]

We're not switching from Bouncer at this time.