Archiving solexa data
rsync + Genome Center Kerberos
If you are using an UNIX (Linux/Apple/Solaris) system, the best way to transfer your data is via rsync transported with ssh. Our server is configured with the Genome Center Kerberos system, if you have kerberos enabled, it is quite easy. Let us know if you need assistance to configure your Kerberos configuration.
Please let us (zwlu at UCDAVIS dot EDU) know your Genome Center Kerberos user name so that we can enable data download for you on machine hofmann.genomecenter.ucdavis.edu
kinit (or kinit username@GENOMECENTER.UCDAVIS.EDU)
to obtain your kerberos ticket and you are ready to download files, please see section "Archiving Files" below.
Note: /etc/krb5.conf is called
in MAC OS X. Here is a sample Kerberos client configuration file for the Genome Center:
One also needs to enable kerberos (GSSAPI) client authentication for ssh, typically in /etc/ssh/ssh_config (most Linux distros) or /etc/ssh_config (in OS X)
GSSAPIAuthentication yes GSSAPIDelegateCredentials yes
rsync + ssh public key
Another way to download your files is rsync with ssh public key infrastructure. If you are familiar with the ssh public key scheme, simply send us (zwlu AT ucdavis.edu) your PUBLIC key (please DO NOT send us your PRIVATE key, it stays with you and only you).
If you are not familiar with ssh public key, here are a few simple steps to generate your public/private key pairs. Make sure that you have openssh utility installed (most Linux/Unix/OS X system do).
ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/your_username/.ssh/id_rsa): Return Enter passphrase (empty for no passphrase): Enter your own passphrase here Enter same passphrase again: Enter your own passphrase again Your identification has been saved in /home/your_username/.ssh/id_rsa. Your public key has been saved in /home/your_username/.ssh/id_rsa.pub. The key fingerprint is: a5:4c:8b:77:00:a4:3b:dd:84:2c:15:66:b5:f4:ff:0d your_username@yourmachine_name.genomecenter.ucdavis.edu
Send us your public key file (/home/your_username/.ssh/id_rsa.pub) to us and we can enable your access to fetch your files.
If your system have ssh-agent set up automatically for you, you can now authenticate to your ssh-agent by the following command:
give your pass phrase to authenticate. If your ssh-agent is not set up automatically, you need to run it manually,
eval `ssh-agent` ssh-add
Note that in the expression `ssh-agent`, we use backtick ` here rather than single quote '. You can follow the section #Archiving Files below to download your files.
You can read more about ssh public key authentication in the following references:
http://sial.org/howto/openssh/publickey-auth/ https://help.ubuntu.com/community/SSHHowto http://www.linuxtopia.org/online_books/rhel5/rhel5_administration/rhel5_ch-openssh.html
For Windows user, you need to install the freeware package basic cygwin plus the rsync and openssh packages. There are many online instructions on how to install cygwin, rsync, and openssh, here are a few links:
You can then download files using rsync similar to the other platforms.
You change to your archive directory on your local machine, for example,
rsync -av email@example.com:Data/random_string/ lane1
The random_string above can be found in your SLIMS (http://slims.bioinformatics.ucdavis.edu) web interface, for the run you want to download
at the "View Files" screen, the address bar contains the random string name similar to the following URL
where 79856d9a09 is the random string that you need to download your files.
"sol" is the user name with which you download your file, you are the guest of user "sol" and rsync is the only command available.
You may repeat the above command again to make sure that you have downloaded all your files, rsync will download missing files if the previous session was interrupted, it will report 0 download file if everything on the local machine is in sync with the remote server.
(Note compresion -z in rsync as in "rsync -avz" is not as effient as "rsync -av" for large files in solexa analysis pipeline)
Rsync 3.0 Note
We have patched the server affecting rsync 3.0.x previously and you can download your files using instructions above.
Client FAT32 File System Note
Many external USB hard drives sold on the market are pre-formatted to Microsoft FAT32 file system. Unfortunately, this FAT32 file system has a maximum file size of 4GB, which might prevent you from downloading your files (if there is a single file larger than 4 GB) from our server to your external USB drive. Your need to reformat your file system to NTFS (on Microsoft system), various Linux file systems, Apple OS X HFS+ or whatever other native file systems for your computer, please consult your friendly system administration staff to choose a best file system for you.