Closed Bug 773505 Opened 12 years ago Closed 12 years ago

Clear or rebuild all documents before launch

Categories

(developer.mozilla.org Graveyard :: Wiki pages, defect, P2)

x86_64
Linux
defect

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: openjck, Unassigned)

References

Details

(Whiteboard: u= c= s= p=2)

16:40 <sheppy> Is there a way we can trigger a re-cache of everything?
16:40 <sheppy> I mean, to dump the whole cache and start over?
16:40 <sheppy> So many templates have been fixed in the last few days that a lot of pages are rendering really badly even though the templates they rely on have been updated.
16:43 <sheppy> We will want to kill the cache and let it start rebuilding before we launch.
Just to be pedantic: There isn't really an easily-cleared cache, with respect to kuma wiki documents.

In a nutshell, this is how it works now: On edits and shift-reload, pages are rendered through kumascript, and the result is saved in the DB. That way, most page views only touch the DB and skip kumascript altogether. We only hit kumascript when there's a clear event that page content has changed.

There's one additional way to trigger page rendering: In the admin pages, you can select groups of documents and force them to re-render. The problem with that, though, is it's only practical to do a few dozen at a time.

What would probably work best for this bug is: Write a small management command that loops through every document on the site and triggers rendering for each, one at a time. 

Running that would take time (I'm not sure how long), but it should be less demanding on the infrastructure than a total wipe. In fact, running this command should be lightweight enough that people can still use the site while it's going - it would basically just simulate someone visiting each page in turn and hitting shift-reload.

If we actually did *clear* the rendered content from documents with a MySQL UPDATE statement, pages *would* get rebuilt on demand. But, it would hammer kumascript like crazy in response to traffic. I suspect that would not turn out very well.
Summary: Clear cache before soft launch → Clear or rebuild all documents before soft launch
FWIW, it's also useful to note that, unlike caches, rendered content in the DB never expires. It *only* changes on edits, shift-reload, or admin tool actions.

I just filed bug 773522 to consider expiration or max-age fields on rendered content.
Blocks: 771763
No longer blocks: 770383
Depends on: 773795
Filed bug 773795. Worth noting that bug 772704 might make a management command unnecessary in the long term, unless I'm misunderstanding something.

In the medium term (after templates are complete but before launch), is there anything we could do to force a re-render of all pages, either through a management command or something else?
No longer depends on: 773795
Priority: -- → P3
(In reply to John Karahalis [:openjck] from comment #3)
> Filed bug 773795. Worth noting that bug 772704 might make a management
> command unnecessary in the long term, unless I'm misunderstanding something.

Very long term, maybe. As I commented on bug 772704, that will not be a quick-fix feature. It'll take some more IT infrastructure and some extra work to track templates dependencies in pages and other templates.

> In the medium term (after templates are complete but before launch), is
> there anything we could do to force a re-render of all pages, either through
> a management command or something else?

Yes, we need bug 773795, a management command that re-renders all pages. As I mentioned in comment #1, just clearing out the rendered content would trigger re-rendering, but that's likely to have a bad outcome
Depends on: 773795
Summary: Clear or rebuild all documents before soft launch → Clear or rebuild all documents before launch
If / when this is merged:
    https://github.com/mozilla/kuma/pull/417

We'll be able to run this command:
    ./manage.py render_document --all

Probably best to let that go over a weekend or something. It will go through documents one-by-one, in order of most-recently edited, and run a full re-rendering.
Whiteboard: u= c= s= p= → u= c= s=2012-07-17 p=
Priority: P3 → P2
Whiteboard: u= c= s=2012-07-17 p= → u= c= s=2012-07-25 p=
More retroactive estimates.
Whiteboard: u= c= s=2012-07-25 p= → u= c= s=2012-07-25 p=2
FWIW: This isn't really a thing that's appropriate for a sprint, in terms of timing. It's a command that needs to be run several days in advance of the Kuma launch switchover.

As far as I know, the command is ready to go as of last week. Could run it now, which will probably cover the bulk of template changes since we started developer-new. But, we should probably hold off running it as long as possible.
Whiteboard: u= c= s=2012-07-25 p=2 → u= c= s= p=2
Thanks Les. Adding this to the launch checklist (https://etherpad.mozilla.org/kuma-launch-checklist) and closing this bug.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → INVALID
Version: Kuma → unspecified
Component: Website → Landing pages
Product: developer.mozilla.org → developer.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.