• How to avoid to be prompt for password
• Transfer through afs (little files)
• Transfer through scp (little files)
• Transfer through bbftp (big files)
• Automatic file transfer from /acs to $H1HPSS
• Automatic file transfer from one institute to $H1HPSS

• How to avoid to be prompt for password:

  This recipe is useful for transfers performed with scp and bbftp.

  1. In the remote host (desy, cppm, etc...):
     • check in /etc/ssh_config (the default ssh configuration file) that the following tags are set to "yes":
       

       RhostsRSAAuthentication yes
RSAAuthentication yes
PasswordAuthentication yes


       If not, you have to set them on your user configuration file or on the command line.

  2. In the local host (Lyon):
     • go in your directory .ssh .
     • execute the command ssh-keygen .
       Give a specific name to the identity file (for example: identity.[xxx] .where [xxx]=desy for the transfer from desy ) and no passphrase. As result, you should have two new files in .ssh : identity.[xxx] and identity.[xxx].pub
     • copy identity.[xxx].pub in the remote host ~/.ssh/authorized_keys file.
       Pay attention that this file has to be in a directory with afs reading right by anyuser. If your remote host .ssh directory doesn't have this right, you can copy authorized_keys in a directory with this right (for example your ~/public directory) and create a link to it in the .ssh directory.
     • specify the identity file when you perform the transfer or the connection. For example:

       scp -i .ssh/identity.desy [user]@h1wgs.desy.de:[file] .
ssf -i .ssh/identity.desy h1wgs.desy.de


  Pay attention that with the RSA authentication you don't get afs tokens. Therefore, in afs directories, you can only act on files for which you have sufficient afs rights.

  Up to Index

• Transfer through afs (little files):

  This method works only for afs directories. You need to have afs tokens to obtain complete access to the afs directories of the remote host. For example, for DESY you have to type:

  klog -cell desy.de -principal [user]


  and you will be asked for your desy password. You can verify your tokens with the command tokens .

  For the special case of DESY several alias and variables have been defined:

  • klog.de = "klog -cell desy.de -principal [user]"
    
  • cd.de = "cd /afs/desy.de/user/[u]/[user]"
    
  • cd.de.h1 = "cd /afs/desy.de/group/h1"
    
  • $DESYH = "/afs/desy.de/user/[u]/[user]"


  Up to Index

• Transfer through scp (little files):

  The general way to use scp is:

  scp [[user@]host1:]filename1... [[user@]host2:]filename2

  and the transfer will be performed after that you have replied to the request for password by the remote host.

  If you have set your RSA authentication in the remote host (see here), you will be not prompt for password and the transfer will be automatic. Pay attention that with the RSA authentication you don't get afs tokens. Therefore, in afs directories, you can only act on files for which you have sufficient afs rights.

  Up to Index

• Transfer through bbftp (big files):

  BBFTP is a software designed to quickly transfer files across a wide area network. See bbftp web page for a more general description of this tool and the detailed list of the commands . The basic bbftp syntax is:

  • bbftp -e '[what_to_do]' -u [user] -I [identity.file] -E [remote.client] [remote.host]


    means that bbftp is going to connect to remote host [remote.host] using [user] user name and the remote command to start the client will be "remote.client -s". It will then use the identity file [identity.file] to obtain RSA authentication. If the connection is successful then the command [what_to_do] will be executed. All transfer commands will use one stream.

  • bbftp -i [ctrlfile] -u [user] -I [identity.file] -E [remote.client] [remote.host]

    As above, except that the commands in [ctrlfile] will be executed.

  A client of bbftp has been installed in DESY (/shift/dice5/data45/h1/h1/bbftp/version-2.0.2/bbftpd/obj.Linux.intel/bbftpd) and in CPPM (/marcotte/bbftp/version-2.0.2/bbftpd/obj.Linux.intel/bbftpd). If you have properly set your your RSA authentication in the remote host (see here), you will be not prompt for password and the transfer will be automatic. Pay attention that with the RSA authentication you don't get afs tokens. Therefore, in afs directories, you can only act on files for which you have sufficient afs rights.

  For the case of DESY and CPPM some alias have been defined:

  • bbftp_desy = "bbftp -I ~/.ssh/identity.desy -u $USER -E /shift/dice5/data45/h1/h1/bbftp/version-2.0.2/bbftpd/obj.Linux.intel/bbftpd"

  • bbftp_cppm = "bbftp -I ~/.ssh/identity.cppm -u $USER -E /marcotte/bbftp/version-2.0.2/bbftpd/obj.Linux.intel/bbftpd"

  They permit to keep short the bbftp command: only the commands (or command file) and the remote host have to be specified. For example:

  • bbftp_desy -e 'put file1 file2' h1mars01.desy.de
  • bbftp_cppm -i [crtlfile] marhaba.in2p3.fr

  Up to Index

• Automatic file transfer from /acs to $H1HPSS:

  Two scripts have been written to perform the automatic transfer of a list of files from /acs in DESY to $H1HPSS in Lyon. One script, exec_transf_desy , has to run in DESY and the other, exec_transf_lyon_desy , in Lyon. The automatic transfer is based on RSA authentication (see How to avoid to be prompt for password ) and it assumes that the identity file is "/afs/in2p3.fr/home/[u]/[user]/.ssh/identity.desy". The scripts communicate through a log file (that should be in a afs directory) and perform the following actions:

  • exec_transf_desy [listOfFiles] [outDirDesy] [logFile] [maxNumFiles] [debug] :
    copy the files in the list [listOfFiles] from /acs to [outDirDesy] and update the log-file [logFile] each time that a file is ready for the transfer to Lyon. The copy to disk is performed only if the number of files is [outDirDesy] is less than [maxNumFiles].
    Type "exec_transf_desy -help" for a more detailed description of the input arguments.

  • exec_transf_lyon_desy [logFile] [outDirLyon] [outDirHPSS] [maxNumFiles] [debug] :
    read the log-file [logFile] and copy the available files from DESY to [outDirLyon] for a maximum of [maxNumFiles] files. Each file is then copied to [outDirHPSS].
    Type "exec_transf_lyon -help" for a more detailed description of the input arguments.

  What to do:

  1. check to have a correct RSA authentication (see How to avoid to be prompt for password ) and that h1mars01.desy.de is a "know host" for your account in Lyon (connect by ssf to h1mars01 and say "yes" to the question: "Host key not found from the list of known hosts. Are you sure you want to continue connecting (yes/no)?").

  2. in desy:
     1. create a file [listOfFiles] with the list of files to be copied from /acs (absolute path)
     2. decide your [outDirDesy] and check the space in it. On the base of the file size, decide your [maxNumFiles]
     3. decide your [logFile] in afs and be careful to have tokens to it (in the case that you place it in Lyon)
     4. start the execution of the script "/shift/dice5/data45/h1/h1/scripts/exec_transf_desy" with the decided parameters.

  3. in Lyon:
     1. check to have afs tokens to access the [logFile] (in that case you place it in desy)
     2. decide your [outDirLyon] and check the space in it. On the base of the file size, decide your [maxNumFiles]
     3. decide your [outDirHPSS] (giving "!" will copy the files in the same path as in /acs. For example: /acs/mc/pythia61/pippo.dat will be copied in $H1HPSS/mc/pythia61/pippo.dat)
     4. start the execution of the script "exec_transf_lyon_desy" with the decided parameters.

Up to Index