660644 - mdn: fix staging errors

Status: VERIFIED FIXED

(developer.mozilla.org Graveyard :: Wiki pages, defect)

Description

[Luke Crouch [:groovecoder]]

[Django] Error (EXTERNAL IP): /en-US/mobile

Traceback (most recent call last):

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/apps/landing/views.py", line 47, in mobile
    return common_landing(request, section=SECTION_MOBILE)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/apps/landing/views.py", line 86, in common_landing
    return jingo.render(request, 'landing/%s.html' % section.short, data)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/jingo/jingo/__init__.py", line 78, in render
    rendered = render_to_string(request, template, context)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/jingo/jingo/__init__.py", line 96, in render_to_string
    return template.render(**get_context())

  File "/usr/lib/python2.6/site-packages/jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/apps/landing/templates/landing/mobile.html", line 1, in top-level template code
    {% extends "base.html" %}

  File "/data/www/django/kuma-stage.mozilla.org/kuma/templates/base.html", line 13, in top-level template code
    <link rel="home" href="{{ url('home') }}">

  File "/data/www/django/kuma-stage.mozilla.org/kuma/apps/sumo/helpers.py", line 39, in url
    return reverse(viewname, locale=locale, args=args, kwargs=kwargs)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/apps/sumo/urlresolvers.py", line 51, in reverse
    url = django_reverse(viewname, urlconf, args, kwargs, prefix)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/core/urlresolvers.py", line 351, in reverse
    *args, **kwargs)))

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/core/urlresolvers.py", line 272, in reverse
    possibilities = self.reverse_dict.getlist(lookup_view)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/core/urlresolvers.py", line 194, in _get_reverse_dict
    self._populate()

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/core/urlresolvers.py", line 173, in _populate
    for name in pattern.reverse_dict:

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/core/urlresolvers.py", line 194, in _get_reverse_dict
    self._populate()

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/core/urlresolvers.py", line 162, in _populate
    for pattern in reversed(self.url_patterns):

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/core/urlresolvers.py", line 244, in _get_url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/core/urlresolvers.py", line 239, in _get_urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/utils/importlib.py", line 35, in import_module
    __import__(name)

  File "/data/www/django/kuma-stage.mozilla.org/kuma/apps/wiki/urls.py", line 3, in <module>
    from kbforums.feeds import ThreadsFeed, PostsFeed

  File "/data/www/django/kuma-stage.mozilla.org/kuma/apps/kbforums/feeds.py", line 1, in <module>
    from django.contrib.syndication.views import Feed

  File "/data/www/django/kuma-stage.mozilla.org/kuma/vendor/src/django/django/contrib/syndication/views.py", line 3, in <module>
    from django.contrib.sites.models import get_current_site

ImportError: cannot import name get_current_site






Cron <root@mradm02> /data/devmostage/stage_update > /dev/null

warning: unable to rmdir src/smuggler: Directory not empty

Comment 1

[Luke Crouch [:groovecoder]]

I think stage_update script tries to delete src/smuggler? I don't see anything in update_site.py that would try to delete it.

Comment 2

[Jake Maul [:jakem]]

This is coming from update_site.py. Here it is with the full output, using -v. It's coming from 'git submodule update --init'.

--------------------
[root@mradm02 kuma]# python26 ./scripts/update_site.py -e stage -v
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma
git pull -q origin master
git submodule update --init
warning: unable to rmdir src/smuggler: Directory not empty
Submodule path 'vendor': checked out '955a916da5fc607933b9b9085ea52ab8109781ca'
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma/locale
svn update
At revision 89679.
./compile-mo.sh .
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma/vendor
git pull -q origin master
git submodule update --init
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma
python2.6 vendor/src/schematic/schematic migrations/
python2.6 manage.py compress_assets
[root@mradm02 kuma]# 
--------------------

Comment 3

[Luke Crouch [:groovecoder]]

Per ozten's comment on bug 655715 it looks like this happens if a git process crashes and/or a submodule is a deleted? We need to remove that dir manually.

Comment 4

[Jake Maul [:jakem]]

Okay, this has been removed manually and was recloned by git as expected.

Per jbalogh's comment in that bug (https://bugzilla.mozilla.org/show_bug.cgi?id=655715#c4), it would be a good idea if the script also does a 'git submodule sync'... it probably wouldn't have affected this particular bug, but we've seen it be a problem during pushes.

Closing this out though, since that's a separate issue.

Comment 5

[Luke Crouch [:groovecoder]]

yup, update_site.py does a sync in both the repos and the vendor repos.

Comment 6

[Jake Maul [:jakem]]

Doesn't appear to be doing a submodule sync:

[root@mradm02 kuma]# grep sync scripts/update_site.py 
[root@mradm02 kuma]# 


It doesn't appear in the verbose output (which claims to echo every command it runs):

[root@mradm02 kuma]# python26 ./scripts/update_site.py -e stage -v
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma
git pull -q origin master
git submodule update --init
Submodule path 'vendor': checked out '955a916da5fc607933b9b9085ea52ab8109781ca'
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma/locale
svn update
At revision 89728.
./compile-mo.sh .
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma/vendor
git pull -q origin master
git submodule update --init
Cloning into src/smuggler...
remote: Counting objects: 307, done.
remote: Compressing objects: 100% (114/114), done.
remote: Total 307 (delta 170), reused 274 (delta 156)
Receiving objects: 100% (307/307), 70.84 KiB, done.
Resolving deltas: 100% (170/170), done.
Submodule path 'src/smuggler': checked out 'f2ca1ca3e1e092b0fb426558b124a1f2b6eed23b'
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma
python2.6 vendor/src/schematic/schematic migrations/
python2.6 manage.py compress_assets
[root@mradm02 kuma]#

Comment 7

[Luke Crouch [:groovecoder]]

Ah, it's in our mdn branch, but not master nor mdn_prod. mdn will merge to both places in the next week so it will fix then.

Comment 8

[Luke Crouch [:groovecoder]]

Did we remove the dir from all locations? The errors haven't stopped.

Comment 9

[Jake Maul [:jakem]]

The error has come back, still on kuma-stage.

[root@mradm02 kuma]# python26 ./scripts/update_site.py -e stage -v
cd /data/dekiwiki-stage/src/kuma-stage.mozilla.org/kuma
git pull -q origin master
git submodule update --init
warning: unable to rmdir src/smuggler: Directory not empty
Submodule path 'vendor': checked out '955a916da5fc607933b9b9085ea52ab8109781ca'
(etc)

In fact, it always comes back. If I remove the dir, then run update_sity.py, it clones it as expected. If I then run it again, the error appears. This is not a transient "git was interrupted" type error.

I tested "git submodule --init --recursive", and it does not throw any errors. Maybe you just need to add that to the update_site.py script?

Comment 10

[Luke Crouch [:groovecoder]]

did we lose the mdn repos? Getting another new error:

/bin/sh: line 0: cd: /data/devmostage/src/developer.mozilla.org/mdn: No such file or directory
python26: can't open file 'manage.py': [Errno 2] No such file or directory

Comment 11

[Jake Maul [:jakem]]

See Bug 660046.... Where are you getting that error from?

Comment 12

[Jake Maul [:jakem]]

I found some cron jobs related to this and commented them out... lmk if you're still getting that error, and if so, where it's coming from.

Comment 13

[Luke Crouch [:groovecoder]]

Cron <root@mradm02> /data/devmostage/stage_update > /dev/null

warning: unable to rmdir src/smuggler: Directory not empty
Unknown command: 'migrate'
Type 'manage.py help' for usage.
There was an error while updating. Please try again later. Aborting.

Comment 14

[Luke Crouch [:groovecoder]]

Another confusing aspect of this is the stage_update script updates two staging servers via update_site.py: developer-stage9 and kuma-stage, but the output doesn't indicate which is causing errors.

Comment 15

[Jake Maul [:jakem]]

Comment 11 and comment 12 were directed at your comment 10, and is irrelevant to the stage_update issue.

(In reply to comment #14)
> Another confusing aspect of this is the stage_update script updates two
> staging servers via update_site.py: developer-stage9 and kuma-stage, but the
> output doesn't indicate which is causing errors.

It is kuma-stage. All I can tell you is there's something wrong with the git repo for vendor/src/smuggler on kuma-stage.

At the very end of this page (http://symphony-cms.com/discuss/thread/50623/), there's a note about this error:

"
for those that haven’t experienced it, if you add submodules to one branch and then checkout another branch that doesn’t have them, you will get this error:
"

Comment 16

[Luke Crouch [:groovecoder]]

Ah, that might be the key - I updated master's vendor subrepos to the same commit as mdn. <fingers-crossed/>

Comment 17

[Luke Crouch [:groovecoder]]

we think we're getting this fixed up. but, we broke the update_site.py script so we need a manual `git pull` in both kuma-stage and stage9 repos.

Comment 18

[Jake Maul [:jakem]]

just 'git pull'? 

kuma-stage normally does:
git pull origin master

developer-stage9 normally does:
git pull origin mdn

Don't want to wreck things up by pulling from the wrong place. :)

Comment 19

[Les Orchard [:lorchard]]

(In reply to comment #18)
> just 'git pull'? 
> 
> kuma-stage normally does:
> git pull origin master
> 
> developer-stage9 normally does:
> git pull origin mdn
> 
> Don't want to wreck things up by pulling from the wrong place. :)

Good call: I believe you're right.

Comment 20

[Luke Crouch [:groovecoder]]

> kuma-stage normally does:
> git pull origin master
> 
> developer-stage9 normally does:
> git pull origin mdn

yeah, do the specific repos refspec like that.

Comment 21

[Jake Maul [:jakem]]

[root@mradm02 kuma]# pwd
/data/devmostage/src/developer-stage9.mozilla.org/kuma
[root@mradm02 kuma]# git pull origin mdn
remote: Counting objects: 24, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 13 (delta 10), reused 5 (delta 2)
Unpacking objects: 100% (13/13), done.
From git://github.com/mozilla/kuma
 * branch            mdn        -> FETCH_HEAD
Updating 0586cc4..0c4c238
Fast-forward
 media/css/devderby.css   |   48 ++++++++++++------------
 media/css/mdn-screen.css |   88 ++++++++++++++++++++++++++--------------------
 media/css/screen.css     |   21 +++++------
 scripts/update_site.py   |   12 ++++---
 4 files changed, 91 insertions(+), 78 deletions(-)

[root@mradm02 kuma]# pwd
/data/devmostage/src/kuma-stage.mozilla.org/kuma
[root@mradm02 kuma]# git pull origin master
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 3), reused 1 (delta 0)
Unpacking objects: 100% (4/4), done.
From git://github.com/mozilla/kuma
 * branch            master     -> FETCH_HEAD
Updating 16d138c..f9a4883
Fast-forward
 scripts/update_site.py |   12 +++++++-----
 1 files changed, 7 insertions(+), 5 deletions(-)

---------------------------------------------------

I re-ran the stage-update portion that deals with kuma-stage, and the error is gone now. Similarly, the whole stage_update runs and exists with no output now. This looks fixed to me!

Comment 22

[Luke Crouch [:groovecoder]]

yup ..... for now. ;)

until we do something else stupid and/or crazy in our repos and/or update script. but hopefully the mods we've made in this case will help prevent future problems.