This recipe is useful for transfers performed with scp and bbftp.
If not, you have to set them on your user configuration file or on the command line.
scp -i .ssh/identity.desy [user]@h1wgs.desy.de:[file] .
ssf -i .ssh/identity.desy h1wgs.desy.de
This method works only for afs directories. You need to have afs tokens to obtain complete access to the afs directories of the remote host. For example, for DESY you have to type:
klog -cell desy.de -principal [user]
and you will be asked for your desy password.
You can verify your tokens with the command
For the special case of DESY several alias and variables have been defined:
- klog.de = "klog -cell desy.de -principal [user]"
- cd.de = "cd /afs/desy.de/user/[u]/[user]"
- cd.de.h1 = "cd /afs/desy.de/group/h1"
- $DESYH = "/afs/desy.de/user/[u]/[user]"
The general way to use scp is:
scp [[user@]host1:]filename1... [[user@]host2:]filename2
and the transfer will be performed after that you have replied to the request for password by the remote host.
If you have set your RSA authentication in the remote host (see here), you will be not prompt for password and the transfer will be automatic. Pay attention that with the RSA authentication you don't get afs tokens. Therefore, in afs directories, you can only act on files for which you have sufficient afs rights.
BBFTP is a software designed to quickly transfer files across a wide area network. See bbftp web page for a more general description of this tool and the detailed list of the commands . The basic bbftp syntax is:
bbftp -e '[what_to_do]' -u [user] -I [identity.file] -E [remote.client] [remote.host]
means that bbftp is going to connect to remote host [remote.host] using [user] user name and the remote command to start the client will be "remote.client -s". It will then use the identity file [identity.file] to obtain RSA authentication. If the connection is successful then the command [what_to_do] will be executed. All transfer commands will use one stream.
bbftp -i [ctrlfile] -u [user] -I [identity.file] -E [remote.client] [remote.host]
As above, except that the commands in [ctrlfile] will be executed.
A client of bbftp has been installed in DESY (/shift/dice5/data45/h1/h1/bbftp/version-2.0.2/bbftpd/obj.Linux.intel/bbftpd) and in CPPM (/marcotte/bbftp/version-2.0.2/bbftpd/obj.Linux.intel/bbftpd). If you have properly set your your RSA authentication in the remote host (see here), you will be not prompt for password and the transfer will be automatic. Pay attention that with the RSA authentication you don't get afs tokens. Therefore, in afs directories, you can only act on files for which you have sufficient afs rights.
For the case of DESY and CPPM some alias have been defined:
- bbftp_desy = "bbftp -I ~/.ssh/identity.desy -u $USER -E /shift/dice5/data45/h1/h1/bbftp/version-2.0.2/bbftpd/obj.Linux.intel/bbftpd"
- bbftp_cppm = "bbftp -I ~/.ssh/identity.cppm -u $USER -E /marcotte/bbftp/version-2.0.2/bbftpd/obj.Linux.intel/bbftpd"
They permit to keep short the bbftp command: only the commands (or command file) and the remote host have to be specified. For example:
- bbftp_desy -e 'put file1 file2' h1mars01.desy.de
- bbftp_cppm -i [crtlfile] marhaba.in2p3.fr
Two scripts have been written to perform the automatic transfer of a list
of files from /acs in DESY to $H1HPSS in Lyon. One script,
has to run in DESY and the other,
exec_transf_lyon_desy , in Lyon.
The automatic transfer is based on RSA authentication (see
How to avoid to be prompt for password ) and it
assumes that the identity file is "/afs/in2p3.fr/home/[u]/[user]/.ssh/identity.desy".
The scripts communicate through a log file (that should be in a afs directory) and perform the following actions:
exec_transf_desy [listOfFiles] [outDirDesy] [logFile] [maxNumFiles] [debug]:
"exec_transf_desy -help"for a more detailed description of the input arguments.
exec_transf_lyon_desy [logFile] [outDirLyon] [outDirHPSS] [maxNumFiles] [debug]:
"exec_transf_lyon -help"for a more detailed description of the input arguments.
What to do:
"/shift/dice5/data45/h1/h1/scripts/exec_transf_desy"with the decided parameters.
"exec_transf_lyon_desy"with the decided parameters.
to transfer files from the disk in a remote institute to $H1HPSS. It assumes that
the RSA authentication is correctly set (see
How to avoid to be prompt for password ). The script performs the following actions:
exec_transf_lyon_labs [ListOfFiles] [outDirLyon] [outDirHPSS] [maxNumFiles] [remote_host] [identityFile] [debug]:
"exec_transf_lyon_labs -help"for a more detailed description of the input arguments.
What to do:
"exec_transf_lyon_labs"with the decided parameters