774814 - Mozilla.org SVN update not showing on production

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P1)

Description

[Mike Alexis [:malexis]]

Pascal updated the text in an l10n block in SVN and pushed to production but it doesn't seem to be picked up (r107579)

It works locally and on dev-allizom but not on production

http://www.mozilla.org/ru/firefox/14.0.1/whatsnew/?oldversion=13.0.1
should be like:
http://www-dev.allizom.org/b/ru/firefox/14.0.1/whatsnew/

Need Web Ops to confirm that the repo is not locked and the cron still working.

Comment 1

[Pascal Chevrel:pascalc]

For testing, I also pushed a rewording to a french sentence in r107589 on a .lang file to see if this one is picked up

Comment 2

[Pascal Chevrel:pascalc]

The text change in r107589 was picked up on production but it may be that there is a partial file lock of the repo not affecting the /fr/ folder but affecting the /ru/ folder.

Could you please look into it? I am working in parallel with James from the webdev team to see if it could be a webdev bug.

Thanks

Comment 3

[Jake Maul [:jakem]]

I'm not showing any errors. This is in the Django side, correct?

[[e-mail]1 src]# cd /data/bedrock/src/www.mozilla.org-django/bedrock
[[e-mail]1 bedrock]# cd locale/
[[e-mail]1 locale]# svn up
Updating '.':
At revision 107591.
[[e-mail]1 locale]# cd ru/
[[e-mail]1 ru]# svn info
Path: .
Working Copy Root Path: /data/bedrock/src/www.mozilla.org-django/bedrock/locale
URL: https://svn.mozilla.org/projects/mozilla.com/tags/production/locales/ru
Repository Root: https://svn.mozilla.org
Repository UUID: 4eb1ac78-321c-0410-a911-ec516a8615a5
Revision: 107591
Node Kind: directory
Schedule: normal
Last Changed Author: [e-mail]
Last Changed Rev: 107579
Last Changed Date: 2012-07-17 11:03:55 -0700 (Tue, 17 Jul 2012)

Comment 4

[Pascal Chevrel:pascalc]

yes, it looks right, it seems that it looks like more a partial page caching issue :/

Do we have something related to caching on the production site that could cause that and that we don't have on dev-allizom?

Comment 5

[Jake Maul [:jakem]]

The Zeus cache settings for dev, stage, and prod all appear to be identical. I can flush cache entries if we think it will help.

Another quick check:

There are 6 cache points to think about: dev, stage, and prod, each in HTTP and HTTPS. Could you confirm which ones are right and which are wrong?

Also: for me the 2 pages in comment 0 are completely different in layout. That would seem to be not a locale issue, but perhaps it speaks to the underlying problem. Maybe we just need an update on the main site? Or maybe this is intended and not worth mentioning...

Comment 6

[Pascal Chevrel:pascalc]

(note that if one of the pages is blue, it means that you are not using Firefox 14 and are redirected)

yes, the page are different, we include an additional paragraph and an image for Russian only (that's called an l10n block and this is stored in ru/templates/firefox/whatsnew.html).

We discovered that partners builds and regular builds use that same whatsnew page so that means that we are sending an erroneous message to the users or Yandex Firefox builds, the last update to svn is just removing that paragraph until/if we find a way to differentiate these users at the webdev level and send them different messaging.

Comment 7

[Pascal Chevrel:pascalc]

(In reply to Jake Maul [:jakem] from comment #5)
> The Zeus cache settings for dev, stage, and prod all appear to be identical.
> I can flush cache entries if we think it will help.
> 
> Another quick check:
> 
> There are 6 cache points to think about: dev, stage, and prod, each in HTTP
> and HTTPS. Could you confirm which ones are right and which are wrong?

Right:
https://www-dev.allizom.org/ru/firefox/14.0.1/whatsnew/
http://www-dev.allizom.org/ru/firefox/14.0.1/whatsnew/

Wrong:
https://www.allizom.org/ru/firefox/14.0.1/whatsnew/
http://www.allizom.org/ru/firefox/14.0.1/whatsnew/
https://www.mozilla.org/ru/firefox/14.0.1/whatsnew/
http://www.mozilla.org/ru/firefox/14.0.1/whatsnew/

Comment 8

[Jake Maul [:jakem]]

(In reply to Pascal Chevrel:pascalc from comment #7)
> (In reply to Jake Maul [:jakem] from comment #5)
> > The Zeus cache settings for dev, stage, and prod all appear to be identical.
> > I can flush cache entries if we think it will help.
> > 
> > Another quick check:
> > 
> > There are 6 cache points to think about: dev, stage, and prod, each in HTTP
> > and HTTPS. Could you confirm which ones are right and which are wrong?
> 
> Right:
> https://www-dev.allizom.org/ru/firefox/14.0.1/whatsnew/
> http://www-dev.allizom.org/ru/firefox/14.0.1/whatsnew/

OK.

> Wrong:
> https://www.allizom.org/ru/firefox/14.0.1/whatsnew/
> http://www.allizom.org/ru/firefox/14.0.1/whatsnew/

These two have an additional problem... the entire page is in English! But that's a different issue we don't have to address in this bug.


> https://www.mozilla.org/ru/firefox/14.0.1/whatsnew/
> http://www.mozilla.org/ru/firefox/14.0.1/whatsnew/

These are fixed. I don't know what was wrong, but I restarted Apache on all of the prod nodes and now that block is gone. I suspect (but can not confirm) that Django's caching layer may have had old/bad data in it. www-dev gets restarted regularly, so it wouldn't be quite so affected by this kind of thing.

Comment 9

[Pascal Chevrel:pascalc]

awesome! I think that's enough for today, I'll talk with Anthony and James tomorrow to see if we can make sure it doesn't happen again on the Django side, thanks Jake!