1539170 - Run wiki pages re-render

Status: RESOLVED MOVED

(developer.mozilla.org Graveyard :: General, enhancement)

Description

[Florian Scholz (Open Web Docs)]

I'd like to request a re-render of wiki pages. Either all of them or just those using the {{compat}} macro.

We've had many BCD releases and we aren't displaying the latest data to users.

Also, the compat.ejs rendering has changed. Most notably, we've agreed to replace "Basic support" with the feature name. This hasn't seen complaints, so I'd like to see it rolled out across the site.

Comment 1

[John Whitlock [:jwhitlock]]

Is Thursday, after the next BCD release, soon enough? Ryan would like to learn the script.

Comment 2

[Florian Scholz (Open Web Docs)]

Yes, that is soon enough.

Comment 3

[Ryan Johnson (:rjohnson)]

This was started yesterday evening and successfully completed around 8pm PT last night. After some initial issues with Kuma getting timeout errors when making render calls to KumaScript, I temporarily increased the KUMASCRIPT_TIMEOUT setting from 25s to 120s and that eliminated the errors (I also scaled-up the number of api and kumascript pods, but that didn't help with the timeout errors, although it may have helped speed up the overall process).

Thanks to John Whitlock for the script and for showing me how he runs it.

Comment 4

[Florian Scholz (Open Web Docs)]

The last re-run was almost a month ago. I'm re-opening to request running re-rendering again. Let me know if you rather want a new bug. I will request this monthly very likely from now on as we make big data changes due to the BCD OKR this year.

fwiw, users already noticed discrepancies to what we have displayed on MDN vs what is in BCD, see this irc log: https://mozilla.logbot.info/mdn/20190425#c16243555

Ryan, can you run re-render next week?

Comment 5

[Ryan Johnson (:rjohnson)]

Will do Florian! It's fine with me that we re-use this bug. I'm leaving the needinfo open until I finish this request.

Comment 6

[Ryan Johnson (:rjohnson)]

This is done (it took about 4 hours).

Comment 7

[Florian Scholz (Open Web Docs)]

Re-opening to request a new monthly re-render after we've shipped BCD 0.0.82 tomorrow.

Comment 8

[Peter Bengtsson [:peterbe]]

I'm working on an exciting change as part of upgrading Celery to version 4, I've in that work, moved all the periodic tasks into code. They used to be configured in MySQL via Django Admin.

So, it would now be trivial to add a Celery task function that runs, say once a day. It's doesn't have to be a perfect function. It'd be a bit like a bash script except you write it in kuma Django ORM code.

It certainly would be nice to not have to do any manual bugzilla+kubernetes situps.

Comment 9

[Ryan Johnson (:rjohnson)]

Thanks Florian, I'll make sure that all of the documents using the compat macro are re-rendered tomorrow (Thursday) after the BCD 0.0.82 release.

Comment 10

[Ryan Johnson (:rjohnson)]

This was completed yesterday around 9:30pm PT.

Comment 11

[Florian Scholz (Open Web Docs)]

Re-opening to request a new re-render.
Due to all-hands and vacations this slipped through a bit. I'm setting up a reminder for myself, so we get back to monthly on this.

Comment 12

[Ryan Johnson (:rjohnson)]

I tried multiple times, but each time more tham 10% of the document re-renders resulted in errors (nearly all of them were connection errors between kuma and kumascript), which automatically shuts down the re-rendering process. I'm going to have to write some code that retries on connection errors and then try again. I probably won't get to this again until next week. Very frustrating.

Comment 13

[Peter Bengtsson [:peterbe]]

(In reply to Ryan Johnson (:rjohnson) from comment #12)

I tried multiple times, but each time more tham 10% of the document re-renders resulted in errors (nearly all of them were connection errors between kuma and kumascript), which automatically shuts down the re-rendering process. I'm going to have to write some code that retries on connection errors and then try again. I probably won't get to this again until next week. Very frustrating.

See https://github.com/mozilla/kuma/pull/5424#issuecomment-509780034

Comment 14

[Ryan Johnson (:rjohnson)]

I tried again late this morning. I found that I needed to scale up the api and kumascript deployments (both to 6 pods), and then the connection timeout errors effectively ceased. It ran for about 2 hours, finishing almost 60% of the re-renders (about 10k documents, with only 2-3 connection timeouts), but then the web pod it was running on was restarted.

I've just started another run, this time on a celery-worker pod (which I'm hoping are more immune from restarts). It usually takes about 4 hours, so we'll see how this one goes.

Once I have some time, I'm going to finish the work I started to make this more reliable and convenient (and also use Peter's https://github.com/mozilla/kuma/pull/5424 for the render calls to kumascript).

Comment 15

[Ryan Johnson (:rjohnson)]

This is done for this round. Running this gist within a ./manage.py shell_plus on a celery-worker pod, and using six (6) api pods and six (6) kumascript pods, 18,753 documents that used the compat macro were re-rendered in 2.8 hours without any errors.

Comment 16

[Florian Scholz (Open Web Docs)]

It's time again and we've updated our CSS compat data enormously to achieve the data quality goal there, see https://github.com/mdn/browser-compat-data/issues/3710#issuecomment-526254326

Please run a re-render, so that end users can benefit from the hard work :-)

Comment 17

[Peter Bengtsson [:peterbe]]

(In reply to Florian Scholz [:fscholz] (MDN) from comment #16)

It's time again and we've updated our CSS compat data enormously to achieve the data quality goal there, see https://github.com/mdn/browser-compat-data/issues/3710#issuecomment-526254326

Please run a re-render, so that end users can benefit from the hard work :-)

Am I right in assuming we first need to land something in KumaScript so we can build with v0.0.93 first?

Also, Ryan, what's stopping us from marking all documents that have BCD tables to get one of those render_max_age attribute or whatever it's called? Then our periodic worker would re-render stale documents more frequently.

Comment 18

[Ryan Johnson (:rjohnson)]

This is done. 19154 documents that use the compat macro were re-rendered in about 3 hours.

Comment 19

[Ryan Johnson (:rjohnson)]

(In reply to Peter Bengtsson [:peterbe] from comment #17)

(In reply to Florian Scholz [:fscholz] (MDN) from comment #16)

It's time again and we've updated our CSS compat data enormously to achieve the data quality goal there, see https://github.com/mdn/browser-compat-data/issues/3710#issuecomment-526254326

Please run a re-render, so that end users can benefit from the hard work :-)

Am I right in assuming we first need to land something in KumaScript so we can build with v0.0.93 first?

There are two cases to consider here, based on whether or not the latest Kumascript image tag has already been deployed within the Kubernetes cluster (if you have kubectl access, you can find the deployed Kumascript image tag by using kubectl get deploy kumascript -o yaml | grep image:). If it hasn't, and the last Kumascript build within Jenkins was done before the BCD release (if it was done after, it will contain the latest BCD release), you can simply restart the Jenkins build, and the Kumascript image will be re-built, automatically picking-up the latest BCD release. That's exactly what I did in this case.

In the other case, when the kumascript image tag that is currently deployed is already the latest, you will have to find something to merge, otherwise Kubernetes will simply ignore the Kumascript push thinking it's already done.

Also, Ryan, what's stopping us from marking all documents that have BCD tables to get one of those render_max_age attribute or whatever it's called? Then our periodic worker would re-render stale documents more frequently.

That's a great question! I don't think there is anything stopping us from doing that, but I'll have to dig into the details of that feature, as I don't know much about it.

Comment 20

[Peter Bengtsson [:peterbe]]

That's a great question! I don't think there is anything stopping us from doing that, but I'll have to dig into the details of that feature, as I don't know much about it.

Off the top of my head, if you set a render_max_age it means they get included in the render_stale_documents periodic task. But! If you set that number too high it takes too long for the stale documents to re-render. And if you set it too low it would trigger a crazy about of re-renders that would not only eat resources, it would also increase the risk that document renders fail and users have to stare at macros tags.

Comment 21

[Florian Scholz (Open Web Docs)]

I'd like to request we do this again now that there is BCD 1.0.0.
Would also be important so that we don't get signals/feedback on outdated compat tables.

Comment 22

[Ryan Johnson (:rjohnson)]

This is done for this round. Running this gist within a ./manage.py shell_plus on a celery-worker pod, and using six (6) api pods and six (6) kumascript pods, 19,860 documents that used the compat macro were re-rendered in about 2 hours without any errors.

Comment 23

[Florian Scholz (Open Web Docs)]

Hey Ryan, we've had again many changes to the data and I'd like to get it in front of our readers.
Very much appreciate if you would run this again. Thank you!

Comment 24

[Ryan Johnson (:rjohnson)]

Will do Florian! I've created a GitHub issue for this latest request (see https://github.com/mdn/kuma/issues/6325) and from now on we'll be using GitHub issues for making these requests, so closing this.