662760 - Can't delete repos using SSH commands on hg.mozilla.org

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Rail Aliiev [:rail]]

How to reproduce:

$ ssh hg.mozilla.org clone mozilla-beta releases/mozilla-beta
Sorry, you already have a repo called mozilla-beta
If you think this is wrong, please file an IT bug

# Let's delete it:
$ ssh hg.mozilla.org edit mozilla-beta delete YES

# recreate
$ ssh hg.mozilla.org clone mozilla-beta releases/mozilla-beta
Sorry, you already have a repo called mozilla-beta
If you think this is wrong, please file an IT bug

# Let's delete it 2 times
$ ssh hg.mozilla.org edit mozilla-beta delete YES
(no output)
$ ssh hg.mozilla.org edit mozilla-beta delete YES
(no output)

# try to delete not existent repo:
$ ssh hg.mozilla.org edit mozilla-beta-123 delete YES
Could not find the repository at /users/raliiev_mozilla.com/mozilla-beta-123.
Please check the list at https://hg.mozilla.org/users/raliiev_mozilla.com


Looks like SSH commands on hg.m.o don't allow to delete a repo.

Comment 1

[Rail Aliiev [:rail]]

The corresponding repo is https://hg.mozilla.org/users/raliiev_mozilla.com/mozilla-beta. I tried to clone the repo from /releases/mozilla-beta using "ssh hg.m.o clone mozilla-beta" yesterday and the cloning operation failed (my ssh connection was invalidated due to the dynamic IP change).

Comment 2

[Rail Aliiev [:rail]]

The same applies for http://hg.mozilla.org/users/prepr-ffxbld/tools

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

noahm: any ideas?

Comment 4

[Noah Meyerhans [:noahm]]

Deletion of repositories is supported by the following python functions in our hg wrapper.  It would seem that there's a bug here somewhere...

def do_delete(cname, repo_dir, repo_name):
    global doc_root
    run_command ('rm -rf %s/users/%s/%s' % (doc_root[cname], repo_dir, repo_name))
    purge_log = open ('/tmp/pushlog_purge.%s' % os.getpid(), "a")
    purge_log.write ('echo users/%s/%s\n' % (repo_dir, repo_name))
    purge_log.close()

def delete_repo (cname, repo_name, do_quick_delete):
  global doc_root
  user = os.getenv ('USER')
  user_repo_dir = user.replace ('@', '_')
  if os.path.exists ('%s/users/%s/%s' % (doc_root[cname], user_repo_dir, repo_name)):
    if do_quick_delete:
      do_delete (cname, user_repo_dir, repo_name)
    else:
      print '\nAre you sure you want to delete /users/%s/%s?' % (user_repo_dir, repo_name)
      print '\nThis action is IRREVERSIBLE.'
      selection = prompt_user ('Proceed?', ['yes', 'no'])
      if (selection == 'yes'):
        do_delete (cname, user_repo_dir, repo_name)
  else:
    sys.stderr.write ('Could not find the repository at /users/%s/%s.\n' % (user_repo_dir, repo_name))
    sys.stderr.write ('Please check the list at https://%s/users/%s\n' % (cname, user_repo_dir))
    sys.exit (1)
  sys.exit(0)

Comment 5

[Noah Meyerhans [:noahm]]

Hmm. Actually, I can't reproduce this:

khan:~$ ssh -l [e-mail] hg.mozilla.org clone hghooks hgcustom/hghooks
Please wait.  Cloning /hgcustom/hghooks to /users/nmeyerhans_mozilla.com/hghooks
Clone complete.
Fixing permissions, don't interrupt.

On the server, I see the newly cloned directory:
[root@dm-svn02 scripts]#  ls -l /repo/hg/mozilla/users/nmeyerhans_mozilla.com
total 4
drwxrwsr-x 3 [e-mail] users 4096 Jun  8 11:28 hghooks/

khan:~$ ssh -l [e-mail] hg.mozilla.org edit hghooks delete YES
khan:~$ echo $?
0

On the server, I see that the repository no longer exists:
[root@dm-svn02 scripts]#  ls -l /repo/hg/mozilla/users/nmeyerhans_mozilla.com
total 0

Comment 6

[Aki Sasaki (not active)]

I wonder if this was either a temporary thing, or if there are permission issues with prepr-ffxbld.

Comment 7

[Rail Aliiev [:rail]]

I think, the repos I listed above are "buggy", maybe there is a process which prevents to delete them?.. Do they really exist?

Comment 8

[Rail Aliiev [:rail]]

(In reply to comment #4)
>     run_command ('rm -rf %s/users/%s/%s' % (doc_root[cname], repo_dir,
> repo_name))


BTW, "rm -rf" should return zero in any case, even for "Text file busy" errors. Maybe using shutil.rmtree would be better here for catching such errors?

Comment 9

[Noah Meyerhans [:noahm]]

(In reply to comment #7)
> I think, the repos I listed above are "buggy", maybe there is a process
> which prevents to delete them?.. Do they really exist?

Is it ok for me to try to manually delete them on the server?

Comment 10

[Rail Aliiev [:rail]]

(In reply to comment #9)
> (In reply to comment #7)
> > I think, the repos I listed above are "buggy", maybe there is a process
> > which prevents to delete them?.. Do they really exist?
> 
> Is it ok for me to try to manually delete them on the server?

Sure, go ahead.

Comment 11

[Noah Meyerhans [:noahm]]

OK, I've nuked your mozilla-beta clone.  It does look like it was a rare issue caused by an NFS problem:

pwd; ls -RA mozilla-beta
/repo/hg/mozilla/users/raliiev_mozilla.com
mozilla-beta:
.hg/

mozilla-beta/.hg:
store/

mozilla-beta/.hg/store:
.nfs0000000000cd1e8a00000106

Comment 12

[Rail Aliiev [:rail]]

Thanks, both repos are gone!

Comment 13

[Rail Aliiev [:rail]]

http://hg.mozilla.org/users/prepr-ffxbld/tools is "undead" again... :/

Comment 14

[Noah Meyerhans [:noahm]]

It was in the same state as users/raliiev_mozilla.com/mozilla-beta.  I've deleted it.

Comment 15

[Armen [:armenzg]]

I have another problem on the same grounds.

When the delete command finishes I can execute the cloning one (even though it takes several hours as mentioned in bug 661828).

What I noticed that even after deletion I can load up an empty repo in
http://hg.mozilla.org/users/stage-ffxbld/mozilla-mobile-5.0/
when I should get "The specified repository "mozilla-mobile-abc" is unknown, sorry. "
http://hg.mozilla.org/users/stage-ffxbld/mozilla-mobile-abc

Comment 16

[Armen [:armenzg]]

(In reply to comment #15)
> I have another problem on the same grounds.
> 
> When the delete command finishes I can execute the cloning one (even though
> it takes several hours as mentioned in bug 661828).
> 
> What I noticed that even after deletion I can load up an empty repo in
> http://hg.mozilla.org/users/stage-ffxbld/mozilla-mobile-5.0/
> when I should get "The specified repository "mozilla-mobile-abc" is unknown,
> sorry. "
> http://hg.mozilla.org/users/stage-ffxbld/mozilla-mobile-abc

Filed bug 665013 to clear the varnish cache upon deletion.