407006 - Can't "ssh cp" on stage-new

Status: RESOLVED INVALID

(mozilla.org Graveyard :: Server Operations: Projects, task, P2)

Description

[Mark Mentovai]

I just pushed the files for our Camino 1.5.4 release, and did the process using stage-new.

Part of the release process involves copying a couple of files around on stage.  In the past, I did this by running "cp" on surf.  For the Camino 1.5.3 release, I did this on stage-new by using "ssh cp".  It worked properly then (a little more than a month ago), but does not work now.

For example, I tried this command:

caminobld@cb-xserve01 bash$ ssh stage-new.mozilla.org cp -p /home/ftp/pub/camino/releases/Camino-1.5.4.dmg /home/ftp/pub/camino/releases/en-US
caminobld@cb-xserve01 bash$ 

No output was produced (so it looked like the operation was successful), but ssh exited with 1 as its exit status (indicating failure).  I tried with and without the -p, with the same result.  In order to perform the copy, I had to scp the file back to the local machine, and then scp it to stage-new anew.

The other file operations I used during this session, including ls (with the -l flag) and rm (with the -r flag), functioned properly.  Only cp has stopped working.

Comment 1

[Dave Miller [:justdave]]

Got a timestamp for when you tried?  I can't find it in the log.

Comment 2

[Mark Mentovai]

Well, I added a file to this directory, so the mtime should tell us:

caminobld@cb-xserve01 bash$ ssh stage-new.mozilla.org ls -ld --full-time /home/ftp/pub/camino/releases/en-US
drwxrwxr-x 1 cltbld 7751 4096 2007-12-05 18:41:24.000000000 +0000 /home/ftp/pub/camino/releases/en-US

It was right around then - some of the attempts were a little bit before, and some were a little bit after.  That timestamp looks like UTC, so 10:41am Pacific sounds about right.

Comment 3

[Dave Miller [:justdave]]

OK, I see that one...

Dec  5 10:41:25 dm-stage02 scponly[11679]: running: /usr/bin/scp -p -t /home/ftp/pub/camino/releases/en-US (username: caminobld(2301), IP/port: 63.245.210.11 56449 22)

There's no record of you attempting a cp command.

When you try to do a command it doesn't like, it'll log something to this effect:

Dec  5 04:33:44 dm-stage02 scponly[32103]: bad request: cd /home/ftp/pub/thunderbird/nightly/experimental/stage-migration && if [ ! -h 2007-10-25-12-mozilla1.8 ]; then ln -s 2007/10/2007-10-25-12-mozilla1.8 .; fi [username: tbirdbld(2304), IP/port: 10.2.71.109 35769 22]

There are no such log entries for the caminobld user.

Comment 4

[Mark Mentovai]

OK, I just executed these commands.  Do they show up in your log?

11:57:03 ssh stage-new.mozilla.org ls -l /home/ftp/pub/camino/releases/
11:57:25 ssh stage-new.mozilla.org cp /home/ftp/pub/camino/releases/Camino-1.5.4.dmg /home/ftp/pub/camino/releases/Camino-1.5.4-Copy.dmg
11:57:33 ssh stage-new.mozilla.org ls -l /home/ftp/pub/camino/releases/
11:57:49 ssh stage-new.mozilla.org date
12:00:46 ssh stage-new.mozilla.org cd /home/ftp/pub/thunderbird/nightly/experimental/stage-migration \&\& if [ ! -h 2007-10-25-12-mozilla1.8 ]\; then ln -s 2007/10/2007-10-25-12-mozilla1.8 .\; fi
12:02:26 ssh stage-new.mozilla.org 'cp /home/ftp/pub/camino/releases/Camino-1.5.4.dmg /home/ftp/pub/camino/releases/Camino-1.5.4-Copy.dmg'
12:02:31 ssh stage-new.mozilla.org ls -l /home/ftp/pub/camino/releases/
12:04:07 ssh -v stage-new.mozilla.org cp /home/ftp/pub/camino/releases/Camino-1.5.4.dmg /home/ftp/pub/camino/releases/Camino-1.5.4-Copy.dmg

I added that long command in based on your previous comment.  In addition to failing, it produced this output:

invalid characters in scp command!
here:&& if [ ! -h 2007-10-25-12-mozilla1.8 ]; then ln -s 2007/10/2007-10-25-12-mozilla1.8 .; fi
try using a wildcard to match this file/directory

The cp commands cause ssh to exit with a status of 1, but don't produce any output.  During the last attempt, when I used ssh -v, I saw (snipped to contain only interesting bits):

debug1: Connecting to stage-new.mozilla.org [63.245.208.199] port 22.
debug1: Connection established.
[...]
debug1: Offering public key: /Users/caminobld/.ssh/id_dsa
debug1: Server accepts key: pkalg ssh-dss blen 817
debug1: read PEM private key done: type DSA
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: Sending command: cp /home/ftp/pub/camino/releases/Camino-1.5.4.dmg /home/ftp/pub/camino/releases/Camino-1.5.4-Copy.dmg
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.0 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 1

Comment 5

[Dave Miller [:justdave]]

(In reply to comment #4)
>> OK, I just executed these commands.  Do they show up in your log?

>> 11:57:03 ssh stage-new.mozilla.org ls -l /home/ftp/pub/camino/releases/

>Dec  5 11:57:04 dm-stage02 sshd[14153]: pam_unix(sshd:session): session opened for user caminobld by (uid=0)
>Dec  5 11:57:04 dm-stage02 scponly[14156]: running: /bin/ls -l /home/ftp/pub/camino/releases/ (username: caminobld(2301), IP/port: 63.245.210.11 56694 22)
>Dec  5 11:57:04 dm-stage02 sshd[14153]: pam_unix(sshd:session): session closed for user caminobld

>> 11:57:25 ssh stage-new.mozilla.org cp /home/ftp/pub/camino/releases/Camino-1.5.4.dmg /home/ftp/pub/camino/releases/Camino-1.5.4-Copy.dmg

>Dec  5 11:57:26 dm-stage02 sshd[14157]: pam_unix(sshd:session): session opened for user caminobld by (uid=0)
>Dec  5 11:57:26 dm-stage02 sshd[14157]: pam_unix(sshd:session): session closed for user caminobld

>> 11:57:33 ssh stage-new.mozilla.org ls -l /home/ftp/pub/camino/releases/

>Dec  5 11:57:34 dm-stage02 sshd[14162]: pam_unix(sshd:session): session opened for user caminobld by (uid=0)
>Dec  5 11:57:34 dm-stage02 scponly[14165]: running: /bin/ls -l /home/ftp/pub/camino/releases/ (username: caminobld(2301), IP/port: 63.245.210.11 56697 22)
>Dec  5 11:57:34 dm-stage02 sshd[14162]: pam_unix(sshd:session): session closed for user caminobld

>> 11:57:49 ssh stage-new.mozilla.org date

>Dec  5 11:57:49 dm-stage02 sshd[14166]: pam_unix(sshd:session): session opened for user caminobld by (uid=0)
>Dec  5 11:57:49 dm-stage02 sshd[14166]: pam_unix(sshd:session): session closed for user caminobld

>> 12:00:46 ssh stage-new.mozilla.org cd /home/ftp/pub/thunderbird/nightly/experimental/stage-migration \&\& if [ ! -h 2007-10-25-12-mozilla1.8 ]\; then ln -s 2007/10/2007-10-25-12-mozilla1.8 .\; fi

>Dec  5 12:00:39 dm-stage02 sshd[14324]: pam_unix(sshd:session): session opened for user caminobld by (uid=0)
>Dec  5 12:00:39 dm-stage02 scponly[14327]: bad request: cd /home/ftp/pub/thunderbird/nightly/experimental/stage-migration && if [ ! -h 2007-10-25-12-mozilla1.8 ]; then ln -s 2007/10/2007-10-25-12-mozilla1.8 .; fi [username: caminobld(2301), IP/port: 63.245.210.11 56702 22]
>Dec  5 12:00:39 dm-stage02 sshd[14324]: pam_unix(sshd:session): session closed for user caminobld

>> 12:02:26 ssh stage-new.mozilla.org 'cp /home/ftp/pub/camino/releases/Camino-1.5.4.dmg /home/ftp/pub/camino/releases/Camino-1.5.4-Copy.dmg'

>Dec  5 12:02:27 dm-stage02 sshd[14336]: pam_unix(sshd:session): session opened for user caminobld by (uid=0)
>Dec  5 12:02:27 dm-stage02 sshd[14336]: pam_unix(sshd:session): session closed for user caminobld

>> 12:02:31 ssh stage-new.mozilla.org ls -l /home/ftp/pub/camino/releases/

>Dec  5 12:02:31 dm-stage02 sshd[14341]: pam_unix(sshd:session): session opened for user caminobld by (uid=0)
>Dec  5 12:02:32 dm-stage02 scponly[14344]: running: /bin/ls -l /home/ftp/pub/camino/releases/ (username: caminobld(2301), IP/port: 63.245.210.11 56706 22)
>Dec  5 12:02:32 dm-stage02 sshd[14341]: pam_unix(sshd:session): session closed for user caminobld

>> 12:04:07 ssh -v stage-new.mozilla.org cp /home/ftp/pub/camino/releases/Camino-1.5.4.dmg /home/ftp/pub/camino/releases/Camino-1.5.4-Copy.dmg

>Dec  5 12:04:07 dm-stage02 sshd[14347]: pam_unix(sshd:session): session opened for user caminobld by (uid=0)
>Dec  5 12:04:07 dm-stage02 sshd[14347]: pam_unix(sshd:session): session closed for user caminobld

> The cp commands cause ssh to exit with a status of 1, but don't produce any
> output.  During the last attempt, when I used ssh -v, I saw (snipped to
> contain only interesting bits):

Hmm...  I suspect the scponly shell is crashing instead of properly dealing with the command...  cp should work, and I would suspect it would be trying to sanitize the arguments before running it.  Let me poke around a bit.

Comment 6

[Dave Miller [:justdave]]

I just upgraded to a new version of the scponly shell on stage-new which contains a bunch of changes in the way it deals with command-line arguments.  Can you give this a try again and see if it's working now?

Comment 7

[Dave Miller [:justdave]]

mento: any luck?

Comment 8

[Dave Miller [:justdave]]

This seems to be working now as far as I can tell.  Please reopen the bug if it isn't.

Comment 9

[Mark Mentovai]

Dave, this still isn't working.  I just tried this a few times:

caminobld@cb-xserve01 bash$ ssh stage-new cp -p /home/ftp/pub/camino/nightly/2008-03-24-20-1.6b3/Camino.dmg /home/ftp/pub/camino/releases/en-US/Camino-1.6b3.dmg
caminobld@cb-xserve01 bash$ echo $?1
caminobld@cb-xserve01 bash$ ssh stage-new ls -l /home/ftp/pub/camino/releases/en-US/Camino-1.6b3.dmg
/bin/ls: cannot access /home/ftp/pub/camino/releases/en-US/Camino-1.6b3.dmg: No such file or directory

Comment 10

[Dave Miller [:justdave]]

ok, scponly is at least admitting to blocking it now...

Mar 24 20:57:46 dm-stage02 scponly[2664]: denied request: cp -p /home/ftp/pub/camino/nightly/2008-03-24-20-1.6b3/Camino.dmg /home/ftp/pub/camino/releases/en-US/Camino-1.6b3.dmg [username: caminobld(2301), IP/port: 63.245.210.11 53357 22]

Not sure why it would do that, but I've got something to go on this time.

Comment 11

[Mark Mentovai]

Maybe it's got something to do with owners not matching?  caminobld != cltbld?  The underlying permissions system has no problem with that, but maybe scponly does.

caminobld@cb-xserve01 bash$ ssh stage-new ls -ld /home/ftp/pub/camino/nightly/2008-03-24-20-1.6b3 /home/ftp/pub/camino/releases/en-US
lrwxrwxrwx 1 caminobld camino   27 Mar 25 03:53 /home/ftp/pub/camino/nightly/2008-03-24-20-1.6b3 -> 2008/03/2008-03-24-20-1.6b3
drwxrwxr-x 1 cltbld    camino 4096 Mar 25 03:59 /home/ftp/pub/camino/releases/en-US
caminobld@cb-xserve01 bash$ ssh stage-new ls -ld /home/ftp/pub/camino/nightly/2008/03/2008-03-24-20-1.6b3
drwxrwxr-x 1 caminobld camino 4096 Mar 25 03:53 /home/ftp/pub/camino/nightly/2008/03/2008-03-24-20-1.6b3

Comment 12

[Jeremy Orem [:oremj]]

Changing QA Contact.

Comment 13

[matthew zeier [:mrz]]

Is this still a problem?

Comment 14

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #13)
> Is this still a problem?

We've (I've) been doing releases the old way, with regular stage and the old process.

I just tried to connect to stage-new (as caminobld on cb-xserve01), and I can't even get connected; stage-new prompts for a password and it doesn't seem to like the only one I know for caminobld.  (Also, stage-new appears to have a new RSA key from the last time anyone used cb-xserve01 to connect to 63.245.208.199.)

Comment 15

[matthew zeier [:mrz]]

stage-new is dm-stage02 and it's been reloaded since this was an issue.  To the best of my knowledge, this isn't in use for any release process.  If we ever migrate to a different system, this and any other problems should all be considered new.

Going to mark this INVALID and someone can reopen if I'm wrong.