1687185 - release1 indexing error 2021-01-17 `umount: /index: target is busy.` on `sudo umount /index` at indexing completion likely due to `updatedb.mlocate`

Status: RESOLVED FIXED

(Webtools :: Searchfox, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

Log Excerpt

+ date
Sun Jan 17 06:25:47 UTC 2021
+ '[' -n /index ']'
+ mv /mnt/index-scratch/mozilla-mobile /index/mozilla-mobile
+ ln -s /index/mozilla-mobile /mnt/index-scratch/mozilla-mobile
+ date
Sun Jan 17 06:26:00 UTC 2021
+ echo 'Indexing complete'
Indexing complete
+ cp /home/ubuntu/index-log /index/index-log
+ sudo umount /index
umount: /index: target is busy.
++ handle_error
++ '[' -d /index ']'

Theories

The locate (via mlocate) database clearly knows about the contents of the /index so it seems possible it or some other daemon was being a busybody on the partition.

It's run from crontab from the daily group which has the crontab trigger of:

25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

From the timestamps above, we can see that we were attempting to unmount things at roughly 06:26 UTC and the cron job start is 06:25 UTC so this lines up pretty precisely.

Options

mlocate and cron specific:

• Uninstall or disable mlocate
• Add our key paths like /index to /etc/updatedb.conf's PRUNEPATHS to keep it out of there.
• Disable all cron jobs except for the failsafe 10hour shutdown created by https://github.com/mozsearch/mozsearch/blob/master/infrastructure/aws/make-crontab.py as invoked by https://github.com/mozsearch/mozsearch/blob/ee0c9dc7334dacc5ef4a4f9d730af972fd01c173/infrastructure/aws/main.sh#L74 which could perhaps be responsible for also disabling other cron jobs.
  • The rationale here is that these server instances are ephemeral and in general have never run their cron.daily/weekly/monthly tasks before and have no hourly tasks, so it's silly/wasteful/foot-gun-prone to run them.

Things we could do if we didn't strongly suspect it was updatedb at work here:

• For error handling, we could add a call to fuser -vm /index.
• We can use fuser's -k command (which can also take a signal like -9 to terminate any processes that are using the mount.
• We could add a sleep try loop as a grace period. There's also umount -l for lazy but I don't think that's sufficient for our purposes.
• We could use umount -f to try and force the unmount.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

I think disabling the other cron jobs seems reasonable.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Attachment: GitHub Pull Request

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

(Landed same day, Jan 17th, 2021.)