831448 - [tracking] migrate to Etherpad Lite

Status: RESOLVED FIXED

(Websites Graveyard :: etherpad.mozilla.org, defect)

Description

[Robert Helmer [:rhelmer]]

We've had etherpad-lite with prototype teampad support for some time now, and have been working with/waiting on upstream to implement (or spec for us to implement) some changes to core that we need to upstream this (improving the auth api and the way sessions are implemented primarily)

I have the prototype teampad work in a repo here - https://github.com/rhelmer/etherpad-lite (technically it lives in ep_etherpad-lite repo, but this pulls that in as a plugin)
This is running in a dev instance at - http://etherpad-lite.paas.allizom.org/ (down for totally unrelated reasons at the moment but should be back soon)

I had been holding off proposing migration until the proposed core changes are done so we could avoid migration headaches later when teampad is upstreamed - but, the core changes are unlikely to happen soon, so I think we should forge ahead and resolve this later (we are continuing to work with etherpad developers - ongoing via irc/email and also meetups like http://beta.etherpad.org/p/usa2013 - so while it's a fork it'll be a temporary one.

I'd like to figure out how to get a staging-quality install that people can start testing ASAP. While we are figuring that out, I will do things like bring my repo up to date, follow up on the sec review we did for EL, etc.

Comment 1

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #0)
> This is running in a dev instance at -
> http://etherpad-lite.paas.allizom.org/ (down for totally unrelated reasons
> at the moment but should be back soon)

This is back up now.
 
> I'd like to figure out how to get a staging-quality install that people can
> start testing ASAP. While we are figuring that out

Spoke with jakem, we should be able to use the same stackato cluster for dev and staging (separate instances running different branches). Prod has it's own cluster.

> I will do things like bring my repo up to date, follow up on the sec review we did for EL, etc.

Repo(s) are up to date with upstream 'develop' branch:
https://github.com/rhelmer/etherpad-lite
https://github.com/rhelmer/ep_etherpad-lite

Blocking on bug 795501 for sec review.

Comment 2

[Robert Helmer [:rhelmer]]

See also the (alpha) VisualEditor project:

http://en.wikipedia.org/wiki/Wikipedia:VisualEditor

You can see it in action at http://www.mediawiki.org/w/index.php?title=VisualEditor:Test&useskin=vector ("VisualEditor" tab at the top next to Read/Edit)

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #2)
> See also the (alpha) VisualEditor project:
> 
> http://en.wikipedia.org/wiki/Wikipedia:VisualEditor
> 
> You can see it in action at
> http://www.mediawiki.org/w/index.php?title=VisualEditor:Test&useskin=vector
> ("VisualEditor" tab at the top next to Read/Edit)

Oops this was meant for bug 648589

Comment 4

[Robert Helmer [:rhelmer]]

After much discussion, we are going to drop the current teampad implementation and focus on shipping etherpad-lite as a replacement for etherpad.mozilla.org

Teampads live at subdomains *.etherpad.mozilla.org in the old server, so we can continue using the old server for those for the time being.

The current teampad impl was a prototype and I have a much better idea of what changes are needed in the core to do this right.

Comment 5

[Pomax]

keenly interested on being able to use etherpad-lite rather than the old etherpad; has there been any movement on this issue?

Comment 7

[Robert Helmer [:rhelmer]]

I have the latest Etherpad release up on stackato staging:

https://pad.paas.allizom.org/

We have a new production cluster, this bug is a little old so I am going to go over the dependencies and see if we need a re-review.

Comment 9

[Robert Helmer [:rhelmer]]

Since we're using Stackato for this, we came up with a decent way of doing team sites - just spinning up separate instances for separate teams.

Going to have discussions on remaining issues with some Etherpad folks in person next week, will have more updates in this bug soon.

Comment 10

[Aki Rose Braun [:akirose]]

Ping! Is there an update on this? I’d love to see us using a version of Etherpad with Markdown support.

Comment 11

[Robert Helmer [:rhelmer]]

(In reply to Aki Braun [:aki_] from comment #10)
> Ping! Is there an update on this? I’d love to see us using a version of
> Etherpad with Markdown support.

Yes! I just got a recent DB dump in bug 994705, going to try migrating it in stackato.

I have a staging instance I am keeping up to date with Etherpad Lite releases here if you'd like to test:
https://pad.paas.allizom.org/ 

It's backed by MySQL but I don't guarantee that data won't be wiped from stage for testing at some point.

Comment 12

[David Bolter [:davidb] (NeedInfo me for attention)]

FYI upstream a11y issue: https://github.com/ether/etherpad-lite/issues/1803

Comment 13

[Jean-Bernard Marcon]

(In reply to Robert Helmer [:rhelmer] from comment #11)
> (In reply to Aki Braun [:aki_] from comment #10)
> > Ping! Is there an update on this? I’d love to see us using a version of
> > Etherpad with Markdown support.
> 


> Yes! I just got a recent DB dump in bug 994705, going to try migrating it in
> stackato.
> 
> I have a staging instance I am keeping up to date with Etherpad Lite
> releases here if you'd like to test:
> https://pad.paas.allizom.org/ 
> 
> It's backed by MySQL but I don't guarantee that data won't be wiped from
> stage for testing at some point.

[simple info request] What is the current state and planned dev of this staging instance? Especially, is there any plan to add private and group pad management features? FYI there is a crowdfunding campaign here http://www.ulule.com/etherpad-framapad/ aiming at this range of features.

Comment 14

[Robert Helmer [:rhelmer]]

(In reply to Jean-Bernard Marcon from comment #13)
> (In reply to Robert Helmer [:rhelmer] from comment #11)
> > (In reply to Aki Braun [:aki_] from comment #10)
> > > Ping! Is there an update on this? I’d love to see us using a version of
> > > Etherpad with Markdown support.
> > 
> 
> 
> > Yes! I just got a recent DB dump in bug 994705, going to try migrating it in
> > stackato.
> > 
> > I have a staging instance I am keeping up to date with Etherpad Lite
> > releases here if you'd like to test:
> > https://pad.paas.allizom.org/ 
> > 
> > It's backed by MySQL but I don't guarantee that data won't be wiped from
> > stage for testing at some point.
> 
> [simple info request] What is the current state and planned dev of this
> staging instance? Especially, is there any plan to add private and group pad
> management features? FYI there is a crowdfunding campaign here
> http://www.ulule.com/etherpad-framapad/ aiming at this range of features.

We plan to support group pads by spinning up separate instances on different subdomains. I think having a crowdfunding campaign and getting it implemented in upstream Etherpad would be great!

Comment 15

[Dan Callahan [:callahad]]

For new teams, should we continue spinning up traditional Etherpad Teampads, or is Etherpad Lite imminent?

(Also, are any of the remaining tasks for the Etherpad Lite migration in Bugzilla? A naive search didn't surface anything, and I'd be happy to help out if anything needs to happen on the dev side.)

Comment 16

[Robert Helmer [:rhelmer]]

(In reply to Dan Callahan [:callahad] from comment #15)
> For new teams, should we continue spinning up traditional Etherpad Teampads,
> or is Etherpad Lite imminent?
> 
> (Also, are any of the remaining tasks for the Etherpad Lite migration in
> Bugzilla? A naive search didn't surface anything, and I'd be happy to help
> out if anything needs to happen on the dev side.)

We're still blocked on having a reasonable replacement for team pads - there's a strong desire not to upgrade e.g. public pads and leave team pads on old etherpad until this is figured out.

http://www.ulule.com/etherpad-framapad/ did get funded and I've been in touch with them - they are working on a teampad-like feature for Etherpad Lite. I'll cc: you on that conversation, and see if there's a way we can help out.

Comment 17

[Dan Mosedale (:dmosedale, :dmose)]

What is driving the desire to keep public pads and team pads stapled, given that it prevents incremental improvement for users?

Comment 18

[Jason Crowe [:jd]]

(In reply to Dan Mosedale (:dmose) - not reading bugmail; needinfo? for response from comment #17)
> What is driving the desire to keep public pads and team pads stapled, given
> that it prevents incremental improvement for users?

Primarily the additional IT overhead required to operate multiple systems over one single system. Additionally there is no way to manage the URL space if they are split which would require a bug and IT time every time a new team pad was needed. Further, there is no decent way to separate the public and private pads. The current schema does not make this a simple operation. In other words, there is a huge time commitment required to do that work. Finally, once we have team pads in etherpad light, there is no clear way to get the team pads back in and there are outstanding questions relating to authorship colors. We have actually had several working sessions and meetings in an attempt to figure out a better way to make progress, but we have not come up with anything reasonable. As long as the project mentioned to you in comment 17 continues to make progress (and so far they seem to be) that is going to be the best solution, especially as it looks like they might have something working before the end of the year.

Comment 19

[Dan Mosedale (:dmosedale, :dmose)]

Fair enough; thanks for the quick and detailed response!  :-)

Comment 20

[Wayne Mery (:wsmwk)]

I was seeing 10-15% cpu *per* etherpad.mozilla.org link in chrome as noted in https://code.google.com/p/chromium/issues/detail?id=465001, where John mentions this issue of old versions (ala bug 958834).   I have not yet determined yet whether the version of etherpad will make any difference. (but not sure I can)

Comment 21

[Robert Helmer [:rhelmer]]

I haven't been working on this for a while, we're still blocked by not having team pads.

At this point I'd suggest something like this: stand up a new service based on current Etherpad, and migrate everybody but team pad users over. Lock down the old site (it's not being maintained anymore) and move them if/when team support becomes available.

I know jp has set up and maintained various Etherpad instances in the past, and might have some interest/input here.

Comment 22

[Jean-Bernard Marcon]

(In reply to Robert Helmer [:rhelmer] from comment #21)
> I haven't been working on this for a while, we're still blocked by not
> having team pads.
> 
> At this point I'd suggest something like this: stand up a new service based
> on current Etherpad, and migrate everybody but team pad users over. Lock
> down the old site (it's not being maintained anymore) and move them if/when
> team support becomes available.
> 
> I know jp has set up and maintained various Etherpad instances in the past,
> and might have some interest/input here.

For information, there is an in progress plugin for private groups and team management with Etherpad https://git.framasoft.org/framasoft/ep_mypads/blob/master/README.md. Current state is second beta and a first stable release is scheduled within a couple of months

Comment 23

[JP Schneider [:jp]]

Check out etherpad.io...that's a version I worked with John Mclear on in the last few months...it supports exporting of documents, embedded media, etc...

Comment 24

[mstefanow]

+1

See: https://github.com/ether/etherpad-lite/wiki/Sites-that-run-Etherpad-Lite

Many random places, cannot tell one which one is reputable (as in trusted and having backups)

Mozilla is reputable however:

> Out of date instances (WARNING MAY CONTAIN SECURITY ISSUES)

Would be nice to run latest software and have backup policies / procedures / systems in place.

Comment 25

[Dietrich Ayala (:dietrich)]

https://discourse.mozilla-community.org/t/etherpad-transition-to-etherpad-lite/4529

I guess this can be closed now?

Comment 26

[Zibi Braniecki [:zbraniecki][:gandalf]]

Is there a bug for getting teampads/protected pads? Without them the transition is partial.

Comment 27

[JP Schneider [:jp]]

Does it make sense to replace protected pad functionality with Google apps now that we're using it as a domain?

The fact that there is very active development and market penetration for Google apps might make people more comfortable entering sensitive information there versus another Etherpad instance.

Comment 28

[Zibi Braniecki [:zbraniecki][:gandalf]]

I think you wouldn't have a problem to find people at Mozilla who would prefer to put sensitive information in our deployment of an open source technology than remote deployment of a closed-source technology by another provider.

Also, just from the functionality standpoint, ehterpad is serving text documents, google docs serve rich text. Etherpad lite is adding features like webrtc, commenting etc. that is much more suitable for most needs for which we've been using etherpad than gdocs are.
people who wanted to use gdocs already use gdocs. People who, for various reasons, did not want to use gdocs, used etherpad.

Comment 29

[:Atoll]

Now that we've announced the closure of the old Etherpad site and replaced it with a single public etherpad-lite instance, the terms of this bug have been met.

Great success!

Please direct any further questions about this migration to <etherpadtransition@m.c>.