1173085 - cloud-init should not auto-mount ephemeral storage

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Robert Helmer [:rhelmer]]

For some instance types, the local ephemeral storage (e.g. SSD instance storage) is already formatted, and cloud-init will automatically mount these on /mnt if detected.

However this breaks the puppet "role" manifest for processor, since it assumes that we need to format the device, and it can't since it's already mounted.

Additionally, we'd like to mount in e.g. /mnt/symbols and not take over the whole /mnt mountpoint - so let's turn have Packer turn this off auto-mounting in cloud-init.cfg when it makes the AMI.

Comment 1

[Robert Helmer [:rhelmer]]

Alternatively - we could make puppet handle this gracefully, and ensure that /mnt is unmounted before starting.

Comment 2

[Daniel Maher [:phrawzty]]

09:49:27 <rhelmer> "while you're in there", you could change the mountpoint to be e.g. 
                   /mnt/symbols if you like - that requires a corresponding consul change.
09:51:05 <rhelmer> in the socorro/processor prefix:
09:51:07 <rhelmer> companion_process.symbol_cache_path=/mnt/symbolcache
09:51:18 <rhelmer> and also
09:52:28 <rhelmer> timeout -s KILL 600 {command_pathname} --raw-json {raw_crash_pathname}
                   --symbols-url {public_symbols_url} --symbols-url {private_symbols_url}
                   --symbols-cache /mnt/symbolcache --symbols-tmp /mnt {dump_file_pathname} 2>
                   /dev/null
09:52:41 <rhelmer> {command_pathname} is stackwalker
09:54:18 <rhelmer> currently processor uses /mnt as tmp space and /mnt/symbolcache as the
                   cache, and puppet mounts the local ephemeral SSD on /mnt
09:53:10 <rhelmer> --symbols-cache and --symbols-tmp must be on the same block device (since
                   it does a rename() from tmp to cache)
09:53:39 <rhelmer> companion_process.symbol_cache_path is used by the cache cleaner, which
                   keeps the cache under a reasonable size
09:53:52 <rhelmer> anyway, my point is that these need to be changed if we change how this is
                   mounted
09:54:18 <rhelmer> currently processor uses /mnt as tmp space and /mnt/symbolcache as the
                   cache, and puppet mounts the local ephemeral SSD on /mnt
09:54:53 <rhelmer> the underlying point here is that stackwalker is responsible for populating
                   the cache, but processor is responsible for cleaning it up
09:55:37 <rhelmer> so I'd suggest e.g. "/mnt/symbols/", and perhaps creating "tmp" and "cache"
                   dirs under this (these must already exist, stackwalker will not create them
                   iirc)

Comment 3

[Daniel Maher [:phrawzty]]

https://github.com/mozilla/socorro-infra/pull/170