514055 - PAGELIST data is not being updated on prod

Status: VERIFIED FIXED

(support.mozilla.org :: General, defect, P5)

Description

[Chris Ilias [:cilias]]

Go to <https://support.mozilla.com/en-US/kb/Article+list?bl=n>.

"Clear Recent History" is not listed. It should be.

Another article to look for, which isn't there, is "Backing up and restoring bookmarks".

The script seems to successfully export to <http://dm-sumotools01.mozilla.org/exports/export.txt>.

As I understand it, this also affects the l10n dashboard.

Comment 1

[Chris Ilias [:cilias]]

I don't know if the code that needs changing is in the sumo svn code or somewhere else, so I'm setting this to 1.3 just in case.

Comment 2

[E. Cooper [:ecooper]]

This works locally (once I connect to the VPN):

ecooper@sisyphus:~/projects/sumo/local/trunk/scripts$ lphp fetch_sumo_lists.php http://dm-sumotools01.mozilla.org/exports/export.txt
3 of 3 lists updated!

Most likely the cron on prod has died/been removed. I can't find the bug for setting this up on prod, but bug 484809 was for testing it on stage. We should have IT check for a weekly cron that runs "/scripts/fetch_sumo_lists.php http://dm-sumotools01.mozilla.org/exports/export.txt" or something similar.

Comment 3

[Chris Ilias [:cilias]]

Moving to IT, so someone from IT can check re comment 2.

"We should
have IT check for a weekly cron that runs "/scripts/fetch_sumo_lists.php
http://dm-sumotools01.mozilla.org/exports/export.txt" or something similar."

Comment 4

[Chris Ilias [:cilias]]

I noticed that <http://dm-sumotools01.mozilla.org/en-US/kb/Article+list> is not updated either, if that helps.

Comment 5

[Shyam Mani [:fox2mike]]

[root@mradm02 cron.d]# cat support.mozilla.com 
# vim: syntax=crontab

# cron jobs for support.mozilla.com

# regenerate the l10n dashboard every Monday at noon
0 12 * * 1 apache cd /data/php5/www/support.mozilla.com/scripts; php fetch_sumo_lists.php http://dm-sumotools01.mozilla.org/exports/export.txt

That's actually outdated. The location has changed to /data/generic/www 

I've updated the cronjob to point to the correct location now. Reopen if it's not updated by next Tuesday or so.

Comment 6

[Chris Ilias [:cilias]]

It's Tuesday, and it's still not updated, so reopening. :-(

Comment 7

[Shyam Mani [:fox2mike]]

From /var/log/cron :

Sep 14 12:00:01 mradm02 crond[4150]: (apache) CMD (cd /data/generic/www/support.mozilla.com/scripts; php fetch_sumo_lists.php http://dm-sumotools01.mozilla.org/exports/export.txt)

Checking to see if this runs by hand

Comment 8

[Chris Ilias [:cilias]]

Yup; https://support.mozilla.com/en-US/kb/Article+list?bl=n is now updated.

Comment 9

[Shyam Mani [:fox2mike]]

Chris,

So the issue is this : The scripts folder on mradm02 where the SUMO scripts reside, is an NFS partition...and so for the script to correctly work, the paths have to be absolute, not relative. I made the following changes for it to work :

[root@mradm02 scripts]# svn diff tiki_script_base.php 
Index: tiki_script_base.php
===================================================================
--- tiki_script_base.php        (revision 51077)
+++ tiki_script_base.php        (working copy)
@@ -7,15 +7,15 @@
  * libraries can be used if needed.
  *
  */  
-require_once('../webroot/lib/init/initlib.php');
-require_once('../webroot/lib/setup/tikisetup.class.php');
+require_once('/data/generic/www/support.mozilla.com/webroot/lib/init/initlib.php');
+require_once('/data/generic/www/support.mozilla.com/webroot/lib/setup/tikisetup.class.php');
 
-TikiSetup::appendIncludePath('../webroot');
-TikiSetup::appendIncludePath('../webroot/lib');
-TikiSetup::appendIncludePath('../webroot/lib/pear');
-TikiInit::appendIncludePath('../webroot/lib/adodb');
+TikiSetup::appendIncludePath('/data/generic/www/support.mozilla.com/webroot');
+TikiSetup::appendIncludePath('/data/generic/www/support.mozilla.com/webroot/lib');
+TikiSetup::appendIncludePath('/data/generic/www/support.mozilla.com/webroot/lib/pear');
+TikiInit::appendIncludePath('/data/generic/www/support.mozilla.com/webroot/lib/adodb');
 
 require_once('db/tiki-db.php');
 require_once('lib/tikilib.php');
 
-?>
\ No newline at end of file
+?>

So I guess you can make those changes and check in the script file (if this is the ONLY place this is being used) and the cron should work fine from next week. I can maybe change the frequency of the cron and we can check to see if it's running.

Comment 10

[Chris Ilias [:cilias]]

Thanks Shyam.
Moving this over to sumodev, because I don't have check-in privs.

Comment 11

[James Socol [:jsocol, :james]]

Shyam, do you know of any reason why PHP's realpath() function would not work, or could we use that to maintain generality?

Comment 12

[Shyam Mani [:fox2mike]]

James,

I've not looked up realpath() but I'm assuming it'd find out the path from the symlink. I'm open to giving it a try :) IANAD and so my "solutions" aren't exactly the right ones.

Comment 13

[Laura Thomson :laura]

Shyam, did you find out why scripts is a symlink to NFS?  If it could just be normal then that would make this problem go away and avoid future push problems like we had for 1.4.  I think you said you would look into it.

If we have to have it that way, then let's it get it set up that way on stage so we have some way of testing.

Comment 14

[Shyam Mani [:fox2mike]]

(In reply to comment #13)

> Shyam, did you find out why scripts is a symlink to NFS?  

Trevor explained over email : 

Supposedly the admin interface needed access to change these:
cfb183ba76f93da6471d39ea5b8d6f25  binary_weights.txt
cfb183ba76f93da6471d39ea5b8d6f25  binary_weights.txt.dist
98595e09c2a06ce1d1acba0d549d4380  stopwords.txt
98595e09c2a06ce1d1acba0d549d4380  stopwords.txt.dist
54c5f6c7ec1af678dcad18165fe7572d  weights_weights.txt
54c5f6c7ec1af678dcad18165fe7572d  weights_weights.txt.dist
a62d9b2cae4ecd73d7eb41a616beae4a  withc_weights.txt
a62d9b2cae4ecd73d7eb41a616beae4a  withc_weights.txt.dist

This was discovered during a production push and NFS was the quick fix. But since they don't actually change, we should move it off NFS.

-Trevor

So if these files need not be accessed by the admin interface I could just remove the symlink tomm and svn switch to get the folder and then push that out.

Comment 15

[Paul Craciunoiu [:paulc]]

I'm wondering if a good solution for the future is this:
If we want to allow scripts to change some files, we can write somewhere (shared place A) but _ask IT_ or _cron_ delivery of copies to all webheads. This would basically mean: have our admin pages write to a specific location, but the webheads read from their own.

Rest of team, what do you think about this?

This should work well for several of our bugs -- including, for example, the ability to have a hard-js file stored from the AAQ admin panel. It would also solve bug 518764, since we could have minify.conf be written somewhere but copied by scripts to the servers for use.

Maybe I'm missing something, but this seems to work well, even for stuff that we want to change relatively frequently?

Comment 16

[James Socol [:jsocol, :james]]

(In reply to comment #15)
> I'm wondering if a good solution for the future is this:
> If we want to allow scripts to change some files, we can write somewhere
> (shared place A) but _ask IT_ or _cron_ delivery of copies to all webheads.
> This would basically mean: have our admin pages write to a specific location,
> but the webheads read from their own.

How would we handle mid-airs?

Comment 17

[Paul Craciunoiu [:paulc]]

Mid-airs of?
In my mind, though perhaps not in my phrasing, the write-to and read-from were separate locations.

Comment 18

[James Socol [:jsocol, :james]]

I'd like to just implement Shyam's solution in comment 14. If the files don't practically change, then it seems like NFS is just unnecessary. This would solve our path issues, remove the potential for scripts/ to get out of sync during pushes, and couldn't hurt performance (Shyam: I'm assuming IO is faster from local drives than from NFS?).

Unless there are objections, maybe we can do this during the 1.4.1 push?

Comment 19

[Shyam Mani [:fox2mike]]

(In reply to comment #18)
> during pushes, and couldn't hurt performance (Shyam: I'm assuming IO is faster
> from local drives than from NFS?).

That is correct, local drives should be a decent bit faster compared to NFS.

Comment 20

[James Socol [:jsocol, :james]]

This was fixed during the push of 1.4.1 (bug 523264).

Comment 21

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #20)
> This was fixed during the push of 1.4.1 (bug 523264).

Chris, can you verify?  Looks good to me.

Comment 22

[Chris Ilias [:cilias]]

"Will Firefox work on Windows 7" was created on Friday and appears on the article list, so assuming the script wasn't run by hand, this if verified.

Comment 23

[Shyam Mani [:fox2mike]]

Nope, didn't run it by hand :)