689645 - [webby] cron job db-intensive?

Status: RESOLVED WONTFIX

(Webtools Graveyard :: Elmo, defect, P3)

Description

[Axel Hecht [:Pike]]

Looking at https://ganglia.mozilla.org/sjc1/?c=Localization&h=l10n-db1.shared.sjc1.mozilla.com&m=load_one&r=hour&s=descending&hc=4&mc=2, there's spike on the db every 10 minutes, which looks like it could just be the webby cron job. Is there something expensive db-wise in the webby cron job?

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

yeah, we connect to Verbatim to pool for new locales/projects. Not sure if it's needed now or if Webby is used now....

Stas?

Comment 2

[Staś Małolepszy :stas]

It is used.  But why would it do any db operations?

Comment 3

[Peter Bengtsson [:peterbe]]

Hmm... the update_webby crontab is commented out. It **used to** run every 10 minutes but not any more. Unfortunately it's impossible to tell when it was commented out since it's just a crontab file. 

What *is* running every 10 minutes is a /home/dashboard/bin/stats which does this::

 tail -n 30 /home/dashboard/site/master-ball/twistd.log /home/dashboard/site/master-ball/l10n-master/twistd.log /home/dashboard/site/slave-ball/slave/twistd.log > /home/dashboard/site/elmo/collected/static/logs.txt

Doing that tail and writing it to logs.txt only writes a 14Kb file and I doubt it takes much more than a couple of milliseconds and I doubt it'd be CPU intensive. 

Is webby even used? Clicking around it appears to be out of touch with reality. For example, it lists 6 projects here https://l10n-stage-sj.mozilla.org/webby/ but here https://localize.mozilla.org/ I can see well over 10 projects. 

Gandalf? Axel? Shall we axe it?

Comment 4

[Peter Bengtsson [:peterbe]]

As a side-note, the update_webby management command uses 1,772 SQL queries to run one update. That definitely spikes up the CPU but like I said above, it appears that it's not being run.

Comment 5

[Peter Bengtsson [:peterbe]]

The cron job is commented out.  The update_webby script is CPU intensive and could do with some refactoring or optimization if anybody cared about webby. But they don't at the moment. 

From discussion we have agreed to not shut down webby until we have something better to replace it with.