983850 - NFS mount for crontabber to read and an NFS mount to the symbols netapp store

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)

Description

[Peter Bengtsson [:peterbe]]

Bug 977218 is about a crontabber app that unpacks zip files that have been uploaded through the webapp. 

The directory we need, for writing, in bug 980082 is a where we need to *read* from. This is basically an auxilliary mount where we don't store things long term. 

Now, we also need to do the unpacking to the NFS mount to the symbols netapp store. This is where we ultimately store the symbols. (Ted, can you help me fill in more details about this part?)

Comment 1

[Peter Bengtsson [:peterbe]]

The "destination" for this NFS mount to *write* to is going to have to be on symbols1.dmz.phx1 into a new directory called /mnt/netapp/breakpad/symbols_thirdparty

Comment 2

[Peter Bengtsson [:peterbe]]

Can phrawzty can help out here?

Comment 3

[Lonnen :lonnen]

From the dependent bugs this looks to be resolved. What's next for NFS mount stuff?

Comment 4

[Daniel Maher [:phrawzty]]

(In reply to Chris Lonnen :lonnen from comment #3)
> From the dependent bugs this looks to be resolved. What's next for NFS mount
> stuff?

The NFS mounts are in place:
--
$ for i in `cat nodes`; do; echo "\n${i}:"; ssh $i 'mount | grep symbols' | cut -f 1 -d '('; done

socorroadm.private.phx1.mozilla.com:
10.8.75.14:/symbols_upload/prod on /mnt/socorro/symbols_upload type nfs
10.8.75.14:/symbols on /mnt/socorro/symbols type nfs

socorroadm.stage.private.phx1.mozilla.com:
10.8.75.14:/symbols_upload/stage on /mnt/socorro/symbols_upload type nfs

socorro1.webapp.phx1.mozilla.com:
10.8.81.10:/symbols_upload/prod on /mnt/socorro/symbols_upload type nfs

socorro1.stage.webapp.phx1.mozilla.com:
10.8.81.10:/symbols_upload/stage on /mnt/socorro/symbols_upload type nfs
--


The production webheads have the symbols_upload symlink in place:
--
$ for i in `seq 1 4`; do; echo -n "${i}:"; ssh socorro$i.webapp.phx1.mozilla.com 'file /data/www/crash-stats.mozilla.org/socorro/webapp-django/media/symbols_upload | cut -d ':' -f 2'; done
1: symbolic link to `/mnt/socorro/symbols_upload'
2: symbolic link to `/mnt/socorro/symbols_upload'
3: symbolic link to `/mnt/socorro/symbols_upload'
4: symbolic link to `/mnt/socorro/symbols_upload'
--


The Stage webheads do *not* have the symlink in place.  This is because Stage is currently built by Jenkins[1] and I'm not sure of the best way to maintain the symlink in this instance (mostly because I've not yet examined the Stage build / deployment mechanism that closely, yet).

WIP.

[1] https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=5734601#crash-stats.mozilla.com%28Socorro%29-StageUpdateDetails

Comment 5

[Lonnen :lonnen]

My understanding is that stage and prod are built the same way. Stage is deployed by a cron that regularly kicks off the deployment script we manually kick off for prod.

Comment 6

[Daniel Maher [:phrawzty]]

(In reply to Chris Lonnen :lonnen from comment #5)
> My understanding is that stage and prod are built the same way. Stage is
> deployed by a cron that regularly kicks off the deployment script we
> manually kick off for prod.

Strictly speaking that's not true, as I'm in the process of discovering. Good times.

WIP.

Comment 7

[Daniel Maher [:phrawzty]]

Stage uses a bespoke deployment script[1] that, among other things, manipulates the contents of a given Socorro deployment before it goes out to the Stage webheads; in fact, it sets up at least one symlink already.  Given this precedent, it would seem that this script is the appropriate place to set up the symbols_upload symlink as well.

There are two different files named 'auto-update.sh' related to Socorro Stage in Puppet.  A quick md5sum reveals that the file currently deployed to socorroadm.stage.private.phx1 is the same as that of modules/socorro/files/stage/admin/scripts/auto-update.sh in the Puppet repo.


$ svn diff
Index: auto-update.sh
===================================================================
--- auto-update.sh	(revision 86708)
+++ auto-update.sh	(working copy)
@@ -19,7 +19,8 @@
 PYTHONPATH=/root/python_extras/lib:/data/socorro/application
 WEBAPP_PYTHON=/data/socorro/webapp-django/virtualenv/bin/python
 SOCORRO_CONFIG=/etc/socorro/common.conf
-export APPDIR PATH PYTHONPATH SOCORRO_CONFIG WEBAPP_PYTHON
+MEDIA_DIR="/data/socorro/webapp-django/media"
+export APPDIR PATH PYTHONPATH SOCORRO_CONFIG WEBAPP_PYTHON MEDIA_DIR


 function error {
@@ -91,6 +92,12 @@
 ln -s /etc/socorro/local.py /data/socorro/webapp-django/crashstats/settings/local.py
 error $? "unable to symlink webapp config"

+# recreate the symbols_upload symlink; bug 983850
+mkdir $MEDIA_DIR
+error $? "unable to mkdir ${MEDIA_DIR}"
+ln -s /mnt/socorro/symbols_upload ${MEDIA_DIR}/symbols_upload
+error $? "unable to symlink ${MEDIA_DIR}/symbols_upload"
+
 # Run database migrations
 $WEBAPP_PYTHON /data/socorro/webapp-django/manage.py syncdb
 error $? "unable to run database migrations"


$ svn commit -m 'create symbols_upload symlink; bug 983850'
Sending        auto-update.sh
Transmitting file data .
Committed revision 86719.


The symlink will become active on the Stage webhead once a) the updated script is distributed to the Admin node, and b) Stage is pushed.


[1] socorroadm.stage.private.phx1.mozilla.com:/root/bin/auto-update.sh

Comment 8

[Daniel Maher [:phrawzty]]

The symlink is active on the Stage webheads.

$ for i in `seq 1 2`; do; echo -n "${i}:"; ssh socorro$i.stage.webapp.phx1.mozilla.com 'file /data/socorro/webapp-django/media/symbols_upload | cut -d ':' -f 2'; done
1: symbolic link to `/mnt/socorro/symbols_upload'
2: symbolic link to `/mnt/socorro/symbols_upload'