664994 - Blocked on Mercurial lock

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Armen [:armenzg]]

I am trying to do staging mobile release but I can't push to our staging repos.
I have tried deleting the repo and recreating it several times.

+ hg push -f -e 'ssh -i ~/.ssh/ffxbld_dsa -l stage-ffxbld' ssh://hg.mozilla.org/users/stage-ffxbld/mozilla-mobile-5.0
pushing to ssh://hg.mozilla.org/users/stage-ffxbld/mozilla-mobile-5.0
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/users/stage-ffxbld/mozilla-mobile-5.0 held by 'dm-svn02.mozilla.org:9431'
remote: abort: repository /repo/hg/mozilla/users/stage-ffxbld/mozilla-mobile-5.0: timed out waiting for lock held by dm-svn02.mozilla.org:9431
abort: unexpected response: empty string

This is very important to fix as I want to get my staging release done before aki and lsblakk head out for holidays which are the only ones that have done mobile releases.

On another side I am creating another alternative repo but that process never seems to finish:
>[cltbld@staging-master ~]$ ssh -l stage-ffxbld -i ~cltbld/.ssh/ffxbld_dsa hg.mozilla.org clone mozilla-mobile-beta releases/mozilla-mobile-5.0
> Please wait.  Cloning /releases/mozilla-mobile-5.0 to /users/stage-ffxbld/mozilla-mobile-beta

Comment 1

[Rail Aliiev [:rail]]

> On another side I am creating another alternative repo but that process
> never seems to finish:
> >[cltbld@staging-master ~]$ ssh -l stage-ffxbld -i ~cltbld/.ssh/ffxbld_dsa hg.mozilla.org clone mozilla-mobile-beta releases/mozilla-mobile-5.0
> > Please wait.  Cloning /releases/mozilla-mobile-5.0 to /users/stage-ffxbld/mozilla-mobile-beta

From my experience this step may take 8-24 hours. :/ See also bug 661828

Comment 2

[Armen [:armenzg]]

[10:57am] ashish: i see the lock file in /repo/hg/mozilla/users/stage-ffxbld/mozilla-mobile-5.0/.hg/store
[10:57am] ashish: it's a dead symlink:
[10:57am] ashish: lrwxrwxrwx  1 stage-ffxbld scm_level_1        25 Jun 16 14:12 lock -> dm-svn02.mozilla.org:9431
[10:57am] ashish: do you know if it's good to rm it?
[11:14am] dustin: ashish: btw that's not a dead symlink, it's just using symlniks as a way to store data (so it's a hostname:pid combination)

The lock has been removed and I tried to delete the repo and start from scratch.
I deleted the repo but it seems that is left behind since when I load http://hg.mozilla.org/users/stage-ffxbld/mozilla-mobile-5.0/ it doesn't return "The specified repository "mozilla-mobile-5.0" is unknown, sorry."
> ssh -l stage-ffxbld -i ~cltbld/.ssh/ffxbld_dsa hg.mozilla.org edit mozilla-mobile-5.0 delete YES;

It seems it might be cached/varnished.

[11:40am] jabba: hmm. I rm'd it
[11:40am] jabba: perhaps it is cached?
[11:40am] jabba: I don't see it here anymore http://hg.mozilla.org/users/stage-ffxbld/
[11:42am] dustin: jabba: this smells like varnish..

Comment 3

[Armen [:armenzg]]

[11:43am] jabba: any idea how to clear a single url from varnish?
[11:43am] jabba: ashish: dumitru ^
[11:44am] dumitru: jabba: from the varnish cli
[11:44am] dumitru: purge req.http.host == example.com && req.url ~ ^/somedirectory/.*$
[11:44am] dumitru: or
[11:45am] dumitru: purge
[11:45am] dumitru: jabba: but, if you restart varnish it will purge everything in its cache

Comment 4

[Armen [:armenzg]]

The lock is cleared.
There is web caching issue but it doesn't block me.
The issue that cloning takes *forever* is tracked in bug 661828.

Thanks ashish and jabba.