tbpl-dev.allizom.org updates are stuck

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
--
major
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: bhearsum, Assigned: cturra)

Tracking

Details

(Reporter)

Description

6 years ago
https://tbpl-dev.allizom.org/cache/revision_info.txt shows a last update on Saturday

Updated

6 years ago
Depends on: 758114
(Assignee)

Updated

6 years ago
Assignee: server-ops → cturra
(Assignee)

Comment 1

6 years ago
the local mercurial had a stuck lock file and as a result was hanging the updates with:

waiting for lock on repository /data/genericrhel6-dev/src/tbpl-dev.allizom.org/tbpl held by 'genericadm.private.phx1.mozilla.com:12114'

i removed the lock file (tbpl/.hg/store/lock) and manually kicked off an update. can you please validate everything looks as expected now?
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED

Comment 2

6 years ago
Looks good now, thank you :-)

Comment 3

6 years ago
Sorry, seems to be stuck again:
https://tbpl-dev.allizom.org/cache/revision_info.txt says bc556afd4a02

tip of http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/ is 657b73898727
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 4

6 years ago
updates to dev appear to all be running on the cron as expected. i can see the last run kicked off at 12:25 pacific.

Comment 5

6 years ago
revision_info.txt now shows 657b73898727.

No idea why it took until now... :-)
(Assignee)

Comment 6

6 years ago
this is what the update script does with regards to mercurial. can you confirm that looks as expected?

  {
	set -x
	date
	echo -e "Updating code..."
	cd $CODE_DIR
	hg pull
	hg update
	checkretval
	cd $CODE_DIR
	hg parent
	checkretval
  } &> >(tee $CODE_DIR/cache/revision_info.txt)

Comment 7

6 years ago
Looks ok to me
(Reporter)

Comment 8

6 years ago
Looks stuck again. It's at 657b73898727 but the repo is at d2d427bbd003.
(Assignee)

Comment 9

6 years ago
interesting. when i ran the cron manually, the revision number didn't update, but when i ran it manually it did. hit me up on irc next time you have an update which needs to be pushed so i can test some stuff out.
Alas, I didn't know about "before," so I've already pushed one that hasn't been pulled.
(Assignee)

Comment 11

6 years ago
i spent some time this afternoon reviewing the update script that is run by cron and noticed the entire update section noted in comment 6 was being skipped due to ENV differences. i have resolved this by updating that section to look like the following:

  (
        set -x
	date
	echo -e "Updating code..."
        cd $CODE_DIR
        hg pull
        hg update
        checkretval
        cd $CODE_DIR
        hg parent
        checkretval
  ) | tee $CODE_DIR/cache/revision_info.txt
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.