1042074 - bedrock.productdetails failed on release day

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Description

[Sylvestre Ledru [:Sylvestre]]

During the release day of 31 today, using this URL to update the product details from this URL:
http://bedrockadm.private.phx1.mozilla.com/chief/bedrock.productdetails

I had:
"
Internal Server Error

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
"

I started it twice today at 7:00 PST. Both failed.

I was trying to deploy r130496 & r130497

Comment 1

[Chris Turra [:cturra]]

it looks like there were two successful deploy attempts today:

  http://bedrockadm.private.phx1.mozilla.com/chief/bedrock.productdetails/logs/?C=M;O=D


i just ran the `update-www.mozilla.org-product-details.sh` script by hand and got the following stack trace. any idea what's going on here? 

++ python /data/bedrock/src/www.mozilla.org-django/bedrock/manage.py update_product_details
+ RUN='/data/bedrock/src/www.mozilla.org-django/bedrock/vendor/lib/python/django/core/management/__init__.py:465: DeprecationWarning: The '\''execute_manager'\'' function is deprecated, you likely need to update your '\''manage.py'\''; please see the Django 1.4 release notes (https://docs.djangoproject.com/en/dev/releases/1.4/).
  DeprecationWarning)
/data/bedrock/src/www.mozilla.org-django/bedrock/vendor/lib/python/django/core/management/__init__.py:409: DeprecationWarning: The '\''setup_environ'\'' function is deprecated, you likely need to update your '\''manage.py'\''; please see the Django 1.4 release notes (https://docs.djangoproject.com/en/dev/releases/1.4/).
  DeprecationWarning)
Product Details were up to date.
Product Details were up to date.'
+ grep -q 'Product Details were up to date.' /dev/fd/63
++ echo /data/bedrock/src/www.mozilla.org-django/bedrock/vendor/lib/python/django/core/management/__init__.py:465: DeprecationWarning: The ''\''execute_manager'\''' function is deprecated, you likely need to update your ''\''manage.py'\'';' please see the Django 1.4 release notes '(https://docs.djangoproject.com/en/dev/releases/1.4/).' 'DeprecationWarning)' /data/bedrock/src/www.mozilla.org-django/bedrock/vendor/lib/python/django/core/management/__init__.py:409: DeprecationWarning: The ''\''setup_environ'\''' function is deprecated, you likely need to update your ''\''manage.py'\'';' please see the Django 1.4 release notes '(https://docs.djangoproject.com/en/dev/releases/1.4/).' 'DeprecationWarning)' Product Details were up to date. Product Details were up to date.
+ echo 'No product details changes to deploy.'
No product details changes to deploy.
+ exit 1

Comment 2

[Paul [:pmac] McLanahan]

That's really odd. The `update_product_details` seems to be exiting non-zero when everything is up-2-date and fine... I'm looking into it.

Comment 3

[Jake Maul [:jakem]]

This is resolved.

This Chief job calls a simple shell script that in turn calls a python manage.py job to update product-details. That shell script was recently update to exit with a return code of "1" if there are no updates. This works fine on the command line, but fails from Chief. That's where the error was coming from.

The reason there were no updates and this error was triggerable is that there was also a cron job running very 15 minutes updating product details. This job would sync the new files, but not actually restart Apache, and so the app wouldn't see them. That's where the Chief button came in... pushing that not only syncs the new files, but also triggers a restart.

We've fixed both problems by 1) having the script return "0" even if there's nothing to do, and 2) disabling the cron.