Data transfer

The archive offers two methods for data transfer

Before you begin

The research data archive offers large low cost storage by using tape as storage medium. While writing data (to the disk cache) will be quite fast, reading data may be delayed by the seek time on tape and can lead to very slow transfer rates in mean (tens of kB/s over many small files, ~100MB/s for a large single file). Getting a small file takes almost as long as a large file. Therefore we urge you

aggregate your data into large packages

with TAR, ZIP, RAR etc., before uploading. Do not use files < 100 MB. We recommend GB-sized files for best performance. If a download does not start immediately or you receive a client timeout when trying to read a file, this may be due to the fact that the file must be fetched from tape first and NOT that your data is damaged. In this case just try again after some minutes until the download starts.

Detailed information can be found in the WIKI.

 

SFTP

An easy way to transfer data is by SFTP. SFTP clients exist for all operating systems. Linux: e.g. sftp command, lftp (advanced), filezilla, ...; Windows: e.g. winscp, filezilla; MacOS: filezilla etc. Google for it and select the application you prefer - but read the manual for your client carefully.

  • A wrapper that allows easy bulk transfers with sftp on Linix is lftp . It allows you to mirror complete directories with the archive and supports restarting large operations from where it was left after a break. Note that this client is powerful but complex.

 

GridFTP (optional, only for power users)

For large volume and high speed transfers. You will need a x-509 certificate and a linux host. Detailed instructions for data transfers with GridFTP using the BWDAHub can be found in the  LSDF-Wiki. Please write us an email if you want to use gridFTP, since this requires manual action of the bwDataArchive personnel.