687106 - use .gitkeep instead of .gitignore as empty dir placeholder

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Webdev, task)

Description

[Jake Maul [:jakem]]

In some places we have historically used .gitignore as a placeholder file, to ensure that a particular directory is properly added to git (git will not add an empty directory). Sometimes this has been done on our side, but it seems more common for webdev to do.

However, our typical deployment process is to check out the upstream code into a src/ dir, merge local changes (settings_local.py, generate locales, etc), rsync to a www/ dir, and then commit the www/ dir to a local git repo from which the servers will check out. For this to work as expected, the rsync must exclude all .git content- the .git dir obviously, since we're committing to a different repo, as well as the .gitignore files, as at that point we no longer want to exclude anything from being checked in.

However, if we exclude .gitignore files in the rsync, we run the risk of git not checking in those empty directories where it was being used as a placeholder.

The solution is to use something else as a placeholder. It seems that webdev and IT have already collaborated on this and come up with ".gitkeep" as an acceptable substitute.

The purpose of this bug is to document all the places where .gitignore should be replaced by .gitkeep in the upstream repos. Thus after documenting everything, this bug will need to go to webdev for fixing.

Here's the first round, from Engagement.

Comment 1

[Jake Maul [:jakem]]

Here are the empty directories I'm seeing on various clusters. Note that most of these directories will already exist on the live servers, so there's not currently a problem with them. However, if a new server were to be rolled out, it would not have these directories. If that is a concern, then these directories need to have their upstream .gitignore placeholder replaced with a .gitkeep one (or really any file at all).

From Engagement:

./webifyme.org/webifyme/ff4/vendor/src/django-mozilla-product-details/product_details/json
./markup.mozilla.org/markup/ffdemo/vendor/src/django-mozilla-product-details/product_details/json
./affiliates.mozilla.org/affiliates-app/docs/_static
./affiliates.mozilla.org/affiliates-app/docs/_build
./affiliates.mozilla.org/affiliates-app/docs/_templates
./affiliates.mozilla.org/affiliates-app/vendor/src/django-mozilla-product-details/product_details/json
./demos.mozilla.org/webowonder/docs/_static
./demos.mozilla.org/webowonder/docs/_build
./demos.mozilla.org/webowonder/docs/_templates
./demos.mozilla.org/webowonder/vendor/src/django-mozilla-product-details/product_details/json

From Addons:

./addons.mozilla.org-remora/site/app/webroot/data
./builder.addons.mozilla.org/flightdeck/vendor/src/django-haystack/docs/_templates
./builder.addons.mozilla.org/flightdeck/vendor/src/django-haystack/docs/_build
./builder.addons.mozilla.org/flightdeck/vendor/src/django-haystack/docs/_static
./addons.mozilla.org/zamboni/tmp/addons
./addons.mozilla.org/zamboni/vendor/packages/billiard/billiard/utils
./addons.mozilla.org/zamboni/vendor/packages/html5lib/src/html5lib/treebuilders
./addons.mozilla.org/zamboni/vendor/packages/html5lib/src/html5lib/serializer
./addons.mozilla.org/zamboni/vendor/packages/html5lib/src/html5lib/treewalkers
./addons.mozilla.org/zamboni/vendor/packages/pytz/pytz
./addons.mozilla.org/zamboni/vendor/packages/anyjson/anyjson
./addons.mozilla.org/zamboni/vendor/packages/importlib/importlib
./addons.mozilla.org/zamboni/vendor/packages/python-memcached
./addons.mozilla.org/zamboni/vendor/packages/Babel/babel
./addons.mozilla.org/zamboni/vendor/packages/phpserialize
./addons.mozilla.org/zamboni/vendor/packages/suds/suds/sax
./addons.mozilla.org/zamboni/vendor/packages/suds/suds/mx
./addons.mozilla.org/zamboni/vendor/packages/suds/suds/transport
./addons.mozilla.org/zamboni/vendor/packages/suds/suds/bindings
./addons.mozilla.org/zamboni/vendor/packages/suds/suds/umx
./addons.mozilla.org/zamboni/vendor/packages/suds/suds/xsd
./addons.mozilla.org/zamboni/vendor/packages/django-picklefield/src/picklefield
./addons.mozilla.org/zamboni/vendor/packages/GitPython/lib/git
./addons.mozilla.org/zamboni/vendor/packages/celery/celery/task
./addons.mozilla.org/zamboni/vendor/packages/celery/celery/execute
./addons.mozilla.org/zamboni/vendor/packages/celery/celery/loaders
./addons.mozilla.org/zamboni/vendor/packages/celery/celery/utils
./addons.mozilla.org/zamboni/vendor/packages/celery/celery/backends
./addons.mozilla.org/zamboni/vendor/packages/setuptools
./addons.mozilla.org/zamboni/vendor/packages/recaptcha-client/recaptcha/client
./addons.mozilla.org/zamboni/vendor/packages/carrot/carrot/backends
./addons.mozilla.org/zamboni/vendor/packages/South/south/introspection_plugins
./addons.mozilla.org/zamboni/vendor/packages/amqplib/amqplib/client_0_8
./addons.mozilla.org/zamboni/vendor/src/django-mozilla-product-details/product_details/json
./addons.mozilla.org/zamboni/vendor/src/amo-validator/jetpack/addon-sdk
./addons.mozilla.org/zamboni/vendor/lib/python/carrot/backends
./addons.mozilla.org/zamboni/lib/multidb
./addons.mozilla.org/zamboni/lib/jingo
./addons.mozilla.org/zamboni/lib/caching/backends
./addons.mozilla.org/zamboni/lib/l10n
./addons.mozilla.org/zamboni/docs/_templates
./addons.mozilla.org/zamboni/docs/_build
./addons.mozilla.org/zamboni/docs/_static
./addons.mozilla.org/zamboni/build
./addons.mozilla.org/zamboni/apps/cronjobs/management/commands
./addons.mozilla.org/zamboni/apps/admin
./addons.mozilla.org/zamboni/apps/users/management/commands
./addons.mozilla.org/zamboni/apps/firefoxcup
./addons.mozilla.org/zamboni/apps/minify/management/commands
./addons.mozilla.org/zamboni/apps/nick

From SUMO:
./prod/locale

From MDN:
./django/developer.mozilla.org.old/mdn/vendor/src/django-mozilla-product-details/product_details/json
./django/developer.mozilla.org.old/mdn.backup/vendor/src/django-mozilla-product-details/product_details/json
./django/developer.mozilla.org/kuma/lib/product_details_json
./django/developer.mozilla.org/kuma/vendor/src/django-mozilla-product-details/product_details/json
./dekiwiki/ControlPanel
./dekiwiki/bin/cache/luceneindex/devmo.dekiwiki.mozilla.org
./devedge/viewsource/2002/xslt
./devedge/library/manuals/2002/ncadm/70pr1/relnotes

From Input:
./django/input.mozilla.com/reporter/media/data
./django/input.mozilla.com/reporter/vendor/src/django-mozilla-product-details/product_details/json

Comment 2

[Jake Maul [:jakem]]

Moving to webdev and CC'ing a couple folks.

Comment 3

[Fred Wenzel [:wenzel]]

Thanks, Jake, that's super helpful! I fixed the gitignore case for product-details and I can also fix the files in docs/ for playdoh.

Note that this will only apply going forward, not magically on existing projects, but better "soon" than never!

Comment 4

[Fred Wenzel [:wenzel]]

(In reply to Fred Wenzel [:wenzel] from comment #3)
> I fixed the gitignore case for
> product-details and I can also fix the files in docs/ for playdoh.

Done!

Comment 5

[Shyam Mani [:fox2mike]]

Thanks Jake & Wenzel! 

I spoke to Wenzel about this offline during the all-hands, glad to see it implemented :D

Comment 6

[Shyam Mani [:fox2mike]]

Ah, this bit me in the rear again, when setting up the sheriffs production site, CC'ing peterbe :)

Figured it out when I ran ./manage.py shell :

[[e-mail]1 sheriffs]# ./manage.py shell
Traceback (most recent call last):
  File "./manage.py", line 37, in <module>
    execute_manager(settings)
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django/django/core/management/base.py", line 351, in handle
    return self.handle_noargs(**options)
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django/django/core/management/commands/shell.py", line 47, in handle_noargs
    loaded_models = get_models()
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django/django/db/models/loading.py", line 167, in get_models
    self._populate()
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django/django/db/models/loading.py", line 61, in _populate
    self.load_app(app_name, True)
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django/django/db/models/loading.py", line 76, in load_app
    app_module = import_module(app_name)
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django-mozilla-product-details/product_details/__init__.py", line 104, in <module>
    product_details = ProductDetails()
  File "/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django-mozilla-product-details/product_details/__init__.py", line 49, in __init__
    for filename in os.listdir(json_dir):
OSError: [Errno 2] No such file or directory: '/data/www/sheriffs.mozilla.org/sheriffs/vendor/src/django-mozilla-product-details/product_details/json'

Will confirm this is it when the site is running

Comment 7

[Shyam Mani [:fox2mike]]

Yup, this was it. Adding a .gitkeep in json/ fixed this. The setup bug for sheriffs was Bug 669391

Comment 8

[Fred Wenzel [:wenzel]]

Fixed per comment 7