804693 - Single releng etherpad giving Server Errors on load attempts

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Justin Wood (:Callek)]

So both myself and joduinn are getting Server Errors trying to load a *single* etherpad in the releng etherpad. It took John a long time to write, and was to be a proposal he sent to other teams this week, so anything we can do to restore the most-recent-possible-version, and/or prevent this from happening again is greatly appreciated.

Page at: https://releng.etherpad.mozilla.org/77

Error as reported:
---
Oops! A server error occured. It's been logged.

Please email <[e-mail]> if this persists.
---

On monday, ericz (as On-Call at the time) restarted the etherpad instance, causing me to be logged out, loading that specific page while logged out did prompt me to log back in, after doing so I once again got that error.

Other pages in the releng.etherpad.m.o seem unaffected.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

As callek notes, we can load other releng etherpad pages just fine - or at least spot testing a bunch have all worked so far. This specific pad, untitled 77, however, throws server error.

Comment 2

[Jake Maul [:jakem]]

I don't believe I have access to this team site, but bug 715061 is likely relevant.

One solution that sometimes helps is to try and open a read-only version of the pad (the gear next to the pad, in the pad list)... often that will let you see the data again. Then it's a matter of copying it out, sanitizing it with sed (or other tool), and pasting it back into a new pad.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Jake Maul [:jakem] from comment #2)
> I don't believe I have access to this team site, but bug 715061 is likely
> relevant.
> 
> One solution that sometimes helps is to try and open a read-only version of
> the pad (the gear next to the pad, in the pad list)... often that will let
> you see the data again. \

Tried that just now. clicking on gear gives me the same server error. :-(

Comment 4

[Justin Wood (:Callek)]

(In reply to Jake Maul [:jakem] from comment #2)
> I don't believe I have access to this team site, but bug 715061 is likely
> relevant.

Based on my skim of 715061 your suggestion seems to be to do the following (which I note below doesn't work) and then do a painful/hard delete/recreation. Can't webops check the stored Data directly to see if this is indeed an issue with our page, and/or find a fix to *PREVENT* these bad characters from inserting in the first place rather than coding a fix to make it easy to delete these pads?

> One solution that sometimes helps is to try and open a read-only version of
> the pad (the gear next to the pad, in the pad list)... often that will let
> you see the data again.

Did not work for me just now:

https://releng.etherpad.mozilla.org/ep/pad/view/77/latest

Comment 5

[Jake Maul [:jakem]]

(In reply to Justin Wood (:Callek) from comment #4)
> Based on my skim of 715061 your suggestion seems to be to do the following
> (which I note below doesn't work) and then do a painful/hard
> delete/recreation. Can't webops check the stored Data directly to see if
> this is indeed an issue with our page, and/or find a fix to *PREVENT* these
> bad characters from inserting in the first place rather than coding a fix to
> make it easy to delete these pads?

Finding the offending data in the database is difficult. The etherpad DB schema is extremely incomprehensible to mortals. I can't even guess what table to start looking in, it's that miserable.

As for preventing the error (or simply correcting it in the first place), that would be a task for webdev. However, their etherpad resources (aka: rhelmer) are aimed at getting Etherpad Lite up to snuff for use as a replacement to our current system altogether. It's a complete rewrite, so I suspect the same issue will not be present anymore.



I have been able to fetch a good portion of the data back from the pad, using the super-admin "Pad Inspector" functionality and walking through the list of revisions until failure. Certain parts of the database believe there should be 938 revisions... other parts stop at 859. I was able fetch up to revision 859 and send that data to Callek and John via email. Whatever revisions actually existed beyond that (if any) seem to be non-existent now.

Nothing more we can really do here. At least we got some of it back, though... that's a bigger win than I had hoped for.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Jake Maul [:jakem] from comment #5)
> (In reply to Justin Wood (:Callek) from comment #4)
> > Based on my skim of 715061 your suggestion seems to be to do the following
> > (which I note below doesn't work) and then do a painful/hard
> > delete/recreation. Can't webops check the stored Data directly to see if
> > this is indeed an issue with our page, and/or find a fix to *PREVENT* these
> > bad characters from inserting in the first place rather than coding a fix to
> > make it easy to delete these pads?
> 
> Finding the offending data in the database is difficult. The etherpad DB
> schema is extremely incomprehensible to mortals. I can't even guess what
> table to start looking in, it's that miserable.
> 
> As for preventing the error (or simply correcting it in the first place),
> that would be a task for webdev. However, their etherpad resources (aka:
> rhelmer) are aimed at getting Etherpad Lite up to snuff for use as a
> replacement to our current system altogether. It's a complete rewrite, so I
> suspect the same issue will not be present anymore.
> 
> 
> 
> I have been able to fetch a good portion of the data back from the pad,
> using the super-admin "Pad Inspector" functionality and walking through the
> list of revisions until failure. Certain parts of the database believe there
> should be 938 revisions... other parts stop at 859. I was able fetch up to
> revision 859 and send that data to Callek and John via email. Whatever
> revisions actually existed beyond that (if any) seem to be non-existent now.
> 
> Nothing more we can really do here. At least we got some of it back,
> though... that's a bigger win than I had hoped for.

Quick nod of thanks to Jake for doing this. And so quickly. Even this not-too-old revision was something I could use to recreate the work in progress and salvage something of my day yesterday. Thanks, Jake!

(Still find entire episode unsettling, as I'm not aware of adding any extended unicode text into the pad. I'll be more careful using etherpad from now on.)