768603 - tbpl-dev.allizom.org updates are stuck

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[[e-mail] (:bhearsum)]

https://tbpl-dev.allizom.org/cache/revision_info.txt shows a last update on Saturday

Comment 1

[Chris Turra [:cturra]]

the local mercurial had a stuck lock file and as a result was hanging the updates with:

waiting for lock on repository /data/genericrhel6-dev/src/tbpl-dev.allizom.org/tbpl held by 'genericadm.private.phx1.mozilla.com:12114'

i removed the lock file (tbpl/.hg/store/lock) and manually kicked off an update. can you please validate everything looks as expected now?

Comment 2

[Ed Morley [:emorley]]

Looks good now, thank you :-)

Comment 3

[Ed Morley [:emorley]]

Sorry, seems to be stuck again:
https://tbpl-dev.allizom.org/cache/revision_info.txt says bc556afd4a02

tip of http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/ is 657b73898727

Comment 4

[Chris Turra [:cturra]]

updates to dev appear to all be running on the cron as expected. i can see the last run kicked off at 12:25 pacific.

Comment 5

[Ed Morley [:emorley]]

revision_info.txt now shows 657b73898727.

No idea why it took until now... :-)

Comment 6

[Chris Turra [:cturra]]

this is what the update script does with regards to mercurial. can you confirm that looks as expected?

  {
	set -x
	date
	echo -e "Updating code..."
	cd $CODE_DIR
	hg pull
	hg update
	checkretval
	cd $CODE_DIR
	hg parent
	checkretval
  } &> >(tee $CODE_DIR/cache/revision_info.txt)

Comment 7

[Ed Morley [:emorley]]

Looks ok to me

Comment 8

[[e-mail] (:bhearsum)]

Looks stuck again. It's at 657b73898727 but the repo is at d2d427bbd003.

Comment 9

[Chris Turra [:cturra]]

interesting. when i ran the cron manually, the revision number didn't update, but when i ran it manually it did. hit me up on irc next time you have an update which needs to be pushed so i can test some stuff out.

Comment 10

[Phil Ringnalda (:philor)]

Alas, I didn't know about "before," so I've already pushed one that hasn't been pulled.

Comment 11

[Chris Turra [:cturra]]

i spent some time this afternoon reviewing the update script that is run by cron and noticed the entire update section noted in comment 6 was being skipped due to ENV differences. i have resolved this by updating that section to look like the following:

  (
        set -x
	date
	echo -e "Updating code..."
        cd $CODE_DIR
        hg pull
        hg update
        checkretval
        cd $CODE_DIR
        hg parent
        checkretval
  ) | tee $CODE_DIR/cache/revision_info.txt