Managing data
Joachim Jacob 8 and 15 November 2013
Bioinformatics data
Historically, bioinformatics has al ays !sed te"t files to store data#
&'B file e"cer(t
$enban% record
HMM (rofile
N$) data
*he N$) machines s(it a lot of data, stored in plain text files# *hese files are m!lti(le gigabytes in si+e#
*i(s for managing N$) data
1# ,hen yo! move the data, do it in its smallest form# Compress the data# 2# ,hen yo! !n(ac% the data, leave it
here it is#
Symbolic links (oint to the data in different folders#
3# &rovide eno!gh storage for yo!r data#
choose yo!r file system type isely
-om(ression. tools in /in!"
2nd some more e"ist###
htt(.00
#lin!"lin%s#com0article0201102200111011310-om(ression*ools#html
*i(s
,idely !sed com(ression tools. $N3 +i( 4gzip5 Bloc% )orting com(ression 4 bzip25 *y(ically, com(ression tools or% on one file# Ho to com(ress directories and their contents6
*ar
itho!t com(ression
*ar 4*a(e 2rchive5 is a tool for bundling a set of files or directories into a single archive. *he res!lting file is called a tar ball# )ynta" to create a tarball. $ tar -cf archive.tar file1 file2 )ynta" to e"tract. $ tar -xvf /path/to/archive.tar
-om(ression. a ty(ical case
2rchiving and com(ression mostly occ!r together# *he most !sed formats are tar.gz or tar.bz. *hese files are the res!lt of two (rocesses#
Archiving 4tar5
Compressing 4g+i( or b+i(25
-om(ression. on yo!r des%to(
-om(ression. on yo!r des%to(
-om(ression. on the command line
Tar is the tool for creating #tar archives, b!t it can com(ress in one go, ith the + or 7 o(tion# Creating a com(ressed tar archive. $ tar cvfz mytararchive.tar.gz $ tar cvfj mytararchive.tar.bz
create -om(ression techni8!e
docs/ docs/
Decompressing a com(ressed tar archive $ tar xvfz mytararchive.tar.gz $ tar xvfj mytararchive.tar.bz
e"tract files verbose
'e90com(ression
*o com(ress one or more files. $ gzip [options] file $ bzip2 [options] file *o decom(ress one or more files. $ gunzip [options] file(s) $ bunzip2 [options] file(s)
*i(s
Many com(ression tools on the command line allo to read compressed files 4instead of first !n(ac%ing then reading5#
$ zcat file(s) $ bzcat file(s)
-om(ression is al ays a balance bet een time and com(ression ratio# $+i( is faster, b+i(2 com(resses harder# :f com(ression is im(ortant to yo!. benchmar%;
<"ercise
a little com(ression e"ercise#
)ymlin%s
&ay attention# )omething very convenient; 2 symbolic link 4or symlin%5 is a file hich (oints to the location of the lin%ed9to file# =o! can do anything ith the symlin% that yo! can do on the original file# 2s yo! move the original file from its location, the symlin% is >dead>#
'o nloads0
?
&ro7ects0
2nnotation0 @ice0 B!tterfly0 )e8!ences0 alignment.sam
)ymlin%s
*o create a symlin%, move to the folder in m!st be created, and e"ec!te ln # here the symlin%
~/Projects $ cd Butterfly ~/Butterfly $ ln -s ../Rice/ e!uences/alignment.sam "in#$to$alignment.sam
'o nloads0
?
&ro7ects0
2nnotation0 @ice0 B!tterfly0 )e8!ences0 alignment.sam
)ymlin%s
*he symlin% is created# =o! can chec% *o delete a symlin%, !se unlin#. ith ls.
~/Projects $ cd Butterfly ~/Butterfly $ ln -s ../Rice/ e!uences/alignment.sam "in#$to$alignment.sam ~/Butterfly $ ls -lh "in#$to$alignment.sam lr%xr%xr%x & joachim joachim '' (ct )) &'*'+ "in#$to$alignment.sam -, ../ e!uences/alignment.sam
'o nloads0
?
&ro7ects0
2nnotation0 @ice0 )e8!ences0 alignment.sam
B!tterfly0 ink!to!alignment.sam
<"ercise
a little symlin% e"ercise
'is%s and storage
:f yo! dive into bioinformatics, yo! manage dis%s and storage# * o ty(es of dis%s " solid state disks /o ca(acity, high s(eed, random " spinning hard disks High ca(acity, >normal> s(eed, se8!ential rites#
htt(.00en# i%i(edia#org0 i%i0)olid9stateAdrive htt(.00en# i%i(edia#org0 i%i0HardAdis%
ill have to
rites
2 dis% is a device
Bia the terminal, sho the dis%s !sing
$ sudo fdis# -l -sudo. /ass%ord for joachim* 0is# /dev/sda* &1.' 2B3 &1'45&'&1&) bytes ... 0is# /dev/sdb* 166+ 7B3 166+&+&+&) bytes ...
2 dis% is divided into (artitions
2 dis% can be divided in (arts, called (artitions# 2n internal disk hich r!ns an o(erating system is !s!ally divided in (artitions, one for each f!nctions# 2n external disk is !s!ally not divided in (artitions#
-hec% o!t the dis% !tility tool
*he system dis%
Name of the dis%
*he system dis%
Name c!rrently highlighted (artition
*he system dis%
&lace in the directory str!ct!re here the (artition can be accessed
2n e"am(le of an 3)B dis%
9
&lace in the directory str!ct!re here the (artition can be accessed
2n e"am(le of an 3)B dis%
*he 3)B dis% is >mo!nted> a!tomatically on the directory tree !nder #media#
2n e"am(le of an 3)B dis%
*his is the ty(e of file system on the (artition# *he (artition is said to be formatted in C2*32 4in this case5#
Cile system formats
By defa!lt, many 3)B flash dis%s are formatted in $AT%&# Dther ty(es are N*C), e"tE, FC)#
$AT%& G ma" E$B files 'T$S G ma"im!m (ortability 4also for !se !nder (xt) G defa!lt file system in /in!", indo s5
htt(.00en# i%i(edia#org0 i%i0CileAsystemHCileAsystemsAandAo(eratingAsystems
2n e"am(le of an 3)B dis%
Cirst !nmo!nt the device# Ne"t, choose format the device#
&
Cormat dis%s
ith dis% !tility
-hoose the ty(e of file system yo! ant to be on that device#
Cormat dis%s
ith dis% !tility
Cormat dis%s
ith dis% !tility
or%
=o! don>t ant to %no all the commands that behind the gnome9dis%9!tility for yo!# B!t if yo! do. 9 mo!nt 9 !mo!nt 9 fdis% 9 m%fs
=o! can read the man (ages and search for g!ides on the internet if yo! ant to get to %no these 4o!t of sco(e for this co!rse5#
-hec%ing storage s(ace
By defa!lt >dis% !sage analy+er>#
-hec%ing storage s(ace
Bon!s. IE'ir)tat# Not installed by defa!lt#
-hec%ing storage s(ace
Bon!s. IE'ir)tat# Not installed by defa!lt#
IE'irstat is a I'< (ac%age
@ehearsal. hat is I'<6
Bon!s. hat ha((ens hen yo! install this (ac%age on o!r system6
)(ace left on dis%s
ith df
*o chec% the storage that is !sed on the different dis%s#
~/ $ df -h
8ilesystem /dev/sda& udev tm/fs none none /dev/sdb&
~/ $ df -h .
ize &)2 '647 )447 <.47 '657 1.52
9sed :vail 9se; 7ounted on <.12 <.+2 '6; / '.4= '647 &; /dev 6)4= &667 &; /run 4 <.47 4; /run/loc# +>= '657 &; /run/shm )47 1.+2 &; /media/test
*he si+e of directories
*o chec% the si+e of files or directories#
~/ $ du -sh ? <)4= bin )5&7 @om/ression$exercise '.4= 0es#to/ '.4= 0ocuments <.47 0o%nloads '.4= 7usic '.4= Pictures '.4= Public 1+17 Rice Axam/le '.4= Bem/lates '.4= test &+7 test.img &&'7 ugene-&.&).) '.4= Cideos
,ildcards on the command line
,ildcards are !sed to describe the names of files#dirs# +, Dn that (osition, the character may be one of the characters bet een J K, e#g# saniti+sz,ation matches. sanitisation and sanitization Dn that (osition, any character is allo ed# e#g# saniti-ation matches. sanitisation, sanitiration, ### L Dn that (osition, any length of string is allo ed e#g# s. matches. san, sdd, sanitisation, sam#alignment,###
,ildcards on the command line
Many tools that re8!ire an argument to (oint to files or directories acce(t these ildcards#
~/ $ du -sh 0o?
,ildcards on the command line
Many tools that re8!ire an argument to (oint to files or directories acce(t these ildcards#
~/ $ du -sh 0o? '.4= 0ocuments )42 0o%nloads
,ildcards on the command line
Many tools that re8!ire an argument to (oint to files or directories acce(t these ildcards#
~/ $ ls ?.fast!
,ildcards on the command line
Many tools that re8!ire an argument to (oint to files or directories acce(t these ildcards#
~/ $ ls ?.fast! ARR&'5<<)$&.fast! testout.fast! ARR&'5<<)$&$/rinse!$good$zz%D.fast! ARR&'5<<)$).fast! test.fast!
Iey ords
-om(ression 2rchive )ymbolic lin% mo!nting Cile system format (artition @ec!rsively df d! !nlin% ,rite in yo!r o n ords hat the terms mean
Brea%
