This chapter shows how to create and maintain BLAST databases—one of the most neglected yet important aspects of using BLAST. We begin with a discussion of the proper use of the FASTA format, and then turns to BLAST database issues. We finish with a general exploration of sequence databases as well as the International Nucleotide Sequence Database.
11.1 FASTA Files
Regardless of where you get your sequences, you will eventually want them in FASTA format because it is the standard currency for sequence data. The FASTA format has a very simple specification consisting of two parts: the definition line and the sequence lines.
The definition line is a single line that begins with the mandatory > symbol immediately followed by an identifier and then a description. There are no spaces between the > and the identifier. The identifier itself must not contain any whitespace because it is the delimiter between the identifier and the description. The description is free-form text that may contain any characters except an end-of-line character. Figure 11-1 shows a simple definition line in which the identifier is "EcoRI" and the description reads "is a restriction enzyme."
Figure 11-1. The FASTA definition line
The sequence lines follow a very simple format: they may be any length and there may be any number of them. Usually, you'll see 50, 60, or 80 characters per line, but the choice is arbitrary. Some software relies on sequence lines not being too long, so it's generally a good idea to follow the convention of 50 to 80 characters per line.
The first and most important guideline for FASTA definition lines is that the identifier uniquely specifies the sequence in some database. The identifier and description are actually optional. The following is a valid, though confusing, FASTA file because there is no identifier (dumb is the description).
> dumb
GAATTC
The following definition lines are confusing because the identifier isn't unique:
>chromosome 1 sequence 1
>chromosome 1 sequence 2
This is easily remedied by replacing the whitespace with some other character:
>chromosome_1-sequence.1
>chromosome_1-sequence.2
On the surface, these look like good identifiers, but another researcher may have the same identifiers for completely different sequences from another organism. How can you prevent this? You can't, but you can minimize potential conflicts by including a unique tag, based on your name or institution. If your data will be made public, the best solution is to submit your sequences to the public databases and use the accession numbers they provide. If not, choose identifiers you think will be unique (and make sure you read about creating fake GI numbers in Section 11.2.3).
The sequencing world is usually very cooperative, and standards have been developed to minimize name conflicts. In particular, there is a tight collaboration between DDBJ, EMBL, and GenBank so that accession numbers among these databases are guaranteed to be unique. But this isn't true of all databases. Although the identifier "AAG39070" points to a specific DDBJ/EMBL/GenBank record, it may also point to a wholly different sequence in another database. A good way to avoid name conflicts is to make sure the identifier specifies a database in addition to some unique tag for the sequence. Let's look at how the NCBI solves this problem.
11.1.1 NCBI Identifier Format
The NCBI identifier format[1] indicates the name of the database in addition to an accession number. These tokens are separated by the "|" symbol, often called a bar or a pipe. This symbol can be confusing in some fonts, as it may look like a lowercase L or the number one. Try using a constant-width serif font such as Courier if you're having trouble seeing them.
[1] The NCBI identifier format is used by both NCBI-BLAST and WU-BLAST software and is necessary for proper indexing of BLAST databases. You should use it as your standard format.
In general terms, an NCBI definition line has the following specification:
>database|identifier
A little knowledge can be a dangerous thing, so don't stop reading now and assume that if you follow this syntax you will have valid identifiers. The names of databases are restricted and have particular syntaxes. If you don't follow the proper syntax, you will end up confusing formatdb or xdformat, which will prevent you performing certain operations such as retrieving sequences by accession number. And you may end up confusing people if you share your data. Table 11-1 shows the current database tokens and their syntax (for an up-to-date list, see the documentation distributed with NCBI-BLAST or WU-BLAST software). For example, if you use the "pat" database token, which corresponds to the patent database, you must supply the country and the patent number as well.
|
Table 11-1. NCBI identifier syntax |
|
|
Database name |
Syntax |
|
DDBJ |
dbj|accession|locus |
|
EMBL |
emb|accession|ID |
|
NCBI GenBank |
gb|accession|locus |
|
NCBI GenInfo |
gi|integer |
|
NCBI Reference Sequence |
ref|accession|locus |
|
NBRF Protein Information Resource |
pir||entry |
|
Protein Research Foundation |
prf||name |
|
SWISS-PROT |
sp|accesion|entry |
|
Brookhaven Protein Data Bank |
pdb|entry|chain |
|
Patents |
pat|country|number |
|
GenInfo Backbone ID |
bbs|number |
|
Local |
lcl|identifier |
|
General |
gnl|database|identifier |
Here are some real examples of NCBI identifiers:
>gi|21305377
>gb|AAM45611.1|AF384285_1
>ref|NP_104634.1|
In the final example, there is no locus, even though this is expected in the syntax. This isn't an error; the locus is just blank.
If you have a collection of your own sequences, with your own names, your best choice is to use the Local or General databases, which are designed specifically for that purpose. The advantage to using General is that you can specify your own sub-namespace in the database field. The following identifier strings are all different from one another (note that identifiers are case-sensitive):
>lcl|foo
>lcl|FOO
>gnl|mydatabase|foo
>gnl|yourdatabase|foo
If, for some reason, you don't want to use the Local or General databases, you can omit the database name and just use your own identifiers with the guidelines discussed earlier. If you're using NCBI-BLAST, your sequences are actually stored in the Local database, and the following identifiers are therefore identical:
>dna.001
>lcl|dna.001
If you retrieve sequences from the BLAST database with fastacmd, they will have the lcl| prepended (even if you didn't specify this), and they will have no definition line found if the definition line doesn't include a description.
If you use WU-BLAST, the previous two identifiers aren't considered identical because an additional unnamed database is separate from the Local database. Definition lines without descriptions are also reported unmodified.
11.1.1.1 Compound identifiers
It is typical for the same sequence to be known by various names. The NCBI identifier format supports this by using compound identifiers where individual identifiers are concatenated with a pipe symbol. The following identifiers are examples of such compound identifiers. In databases distributed by the NCBI, the GI number is the first identifier.
>gi|11611818|gb|AF287139.1|AF287139
>gi|1708198|sp|P80487|HHP_THICU
>gi|9910844|sp|Q9UWG2|RL3_METVA
>gi|7228451|dbj|BAA92411.1|
>gi|11277201|pir||T44712
11.1.1.2 Concatenated definition lines
If you download the nonredundant protein database from the NCBI or use one of the programs distributed with WU-BLAST that creates nonredundant databases, you will see concatenated definition lines. Each definition is separated with the Control-A character, which is a whitespace character that in text editors or word processors looks like a normal space. When it is forced to be visible, Control-A is often written as ^A (a white character on a black background is also common). You may wonder why you can't just create a larger compound identifier rather than a concatenated definition. The reason is that identical sequences may originate from different organisms or different loci and are therefore not identical in the biological sense; they may have different descriptions, which you may want to see. The following single definition line contains concatenated definitions as well as compound identifiers.
>gi|9845511|ref|NP_008839.2| ras-related C3 botulinum toxin substrate 1 isoform
Rac1; rho family, small GTP binding protein Rac1 [Homo sapiens]^Agi|131807|sp|P1
5154|RAC1_HUMAN Ras-related C3 botulinum toxin substrate 1 (p21-Rac1) (Ras-like
protein TC25)^Agi|68958|pir||TVHUC1 GTP-binding protein rac1 - human^Agi|108115|
pir||G36364 GTP-binding protein rac2 - dog^Agi|280956|pir||A60347 GTP-binding pr
otein rac1 - mouse^Agi|14277763|pdb|1I4D|D Chain D, Crystal Structure Analysis O
f Rac1-Gdp Complexed With Arfaptin (P21)^Agi|14277766|pdb|1I4L|D Chain D, Crysta
l Structure Analysis Of Rac1-Gdp In Complex With Arfaptin (P41)^Agi|922|emb|CAA3
9801.1| rac2 [Canis familiaris]^Agi|53886|emb|CAA40545.1| ras-related C3 botulin
ium toxin substrate [Mus musculus]^Agi|190824|gb|AAA36537.1| ras-related C3 botu
linum toxin substrate^Agi|249582|gb|AAB22206.1| rac1 p21=small GTP-binding prote
in [human, HL60, Peptide, 192 aa]^Agi|3184510|gb|AAC18960.1| GTPase cRac1A [Gall
us gallus]^Agi|6007014|gb|AAF00714.1|AF175262_1 GTPase [Bos taurus]^Agi|8574038|
emb|CAB53579.5| Rac1 protein [Homo sapiens]^Agi|12843555|dbj|BAB26027.1| RAS-rel
ated C3 botulinum substrate 1~data source:MGD, source key:MGI:97845, evidence:IS
S~putative [Mus musculus]^Agi|13277918|gb|AAH03828.1| ras-related C3 botulinum t
oxin substrate 1 (rho family, small GTP binding protein Rac1) [Mus musculus]^Agi
|15919905|dbj|BAB69451.1| RAS-related C3 botulinum substrate 1~data source:MGD,
source key:MGI:97845, evidence:ISS~putative [Mus musculus]^Agi|20379102|gb|AAM21
111.1|AF498964_1 small GTP binding protein RAC1 [Homo sapiens]
Most concatenated definitions aren't this long. This particular protein is highly conserved and is identical from human to chicken (Gallus gallus). You might take a moment to appreciate that in the eons during which continents have split apart and converged, this protein has remained completely unchanged.
11.1.2 Descriptions
While you should use the NCBI identifier format, there isn't a publicly recognized standard for descriptions. Some people choose to omit descriptions entirely, while others load up the definition line with the entire contents of a GenBank file. The best descriptions are both brief and informative. Descriptions from the NCBI include a short description and the species names in square brackets at the end of the line. This is a reasonably good practice, but you should be wary of trying to reliably parse descriptions that don't come from a controlled vocabulary. The following identifier is a real example of a difficult-to-parse description:
>gi|20820984|ref|XP_140836.1| similar to DiGeorge syndrome critical region gene DGSI
protein [Homo sapiens] [Mus musculus]
It's hard to tell if this is a human or mouse sequence. In reality, it's a mouse sequence similar to a human protein that originates from a region involved with the genetic disease called DiGeorge Syndrome. If you use a regular expression to find all Homo sapiens sequences and you don't bind the pattern match to the end of the line, this description can fool you. This kind of problem isn't limited to FASTA files; you'll also find fields in GenBank records that have embedded GenBank tags. It's both confusing and annoying. Unfortunately, automatically generating descriptions from transitive associations is a common practice. One way to cope with this problem is to rigorously construct your own definition lines from a controlled vocabulary. Another way is to trust only the identifiers, and when you need biological information, such as the species, retrieve it directly from the parent biological database.
11.2 BLAST Databases
The mechanics of creating BLAST databases is quite simple; just run formatdb or xdformat with the proper syntax. Chapter 10 discussed this topic, and you'll find the command summaries in Chapter 13 and Chapter 14. There are, however, subtleties that make this process more complicated than it may appear.
11.2.1 Large Databases
One of the most common database complications occurs with large files. Most computers today use 32-bit operating systems and 32-bit filesystems. This puts a physical limit of 4 GB on the amount of RAM and 4 GB on the size of any particular file. (You may find that you are actually limited to less than 4 GB in both cases, and a 2-GB limit is quite common.) Most computers these days don't have or need 4-GB RAM. However, most hard disks are quite a bit larger than 4 GB, and files can sometimes exceed these limits. Therefore many operating systems have the option of using 64-bit filesystems. Unfortunately you can't just change the filesystem and expect everything to work. Making software applications aware of large files often means recompiling them with special flags, and the process of migrating to a 64-bit filesystem can be painful because the applications don't tell you useful things like "I'm not large-file-aware." Instead, they just sit there quietly burning CPU time while they run in endless loops.
11.2.1.1 Large NCBI databases
The standard protocol for formatting a database is to run formatdb on a FASTA database:
formatdb -p F -i fasta_db -o
NCBI-BLAST databases are physically limited to 4 GB of sequence, which corresponds to about 4 billion amino acids or 16 billion nucleotides (nucleotides are compressed 4:1). On a 32-bit filesystem, the previous approach won't let you use all this space because the FASTA file can't contain more than 2 or 4 billion letters. Creating a database larger than 2 or 4 billion letters requires piping sequence to formatdb.
cat fasta1 fasta2 fasta3 | formatdb -p F -i stdin -n my_db -o
But what if you happen to have more than 16 billion letters? This isn't a problem because formatdb automatically segments individual BLAST databases to files containing 16 billion nucleotides and creates something called an alias database that stitches them all together. This is really convenient because it means that you can search enormous databases even on 32-bit filesystems. Alias databases are discussed in more detail later in this chapter.
It's still possible to run into file size issues by piping FASTA files to formatdb because the filesystem maximum may be 2 GB and the implicit BLAST maximum is 4 GB. Fortunately, formatdb lets you set the size of each database volume with the -v parameter. The following example sets this size this to 2 billion and includes a bit more realism by piping the FASTA files from a compressed format.
zcat file*.gz | formatdb -i stdin -p F -o -n my_db -v 2000000000
A word of caution: be precise. If you accidentally leave off one of the zeroes, you can create 10 times as many files. For large databases, this can be a problem because the maximum number of volumes is 100.
11.2.1.2 Large WU-BLAST databases
WU-BLAST doesn't use alias databases, so the only way to create a database larger than 4 GB is on a 64-bit filesystem. If you're accessing or distributing databases over a network, the network must also be 64-bit aware. To index a large number of compressed files, use a command such as the following:
zcat file* | xdformat -n -I -o ESTs -- -
In the typical Unix command line syntax, the double-dash indicates the end of the command line options, and the single-dash denotes standard input rather than a file.
If you're stuck with a 32-bit filesystem and need to search large BLAST databases, you can use virtual databases, which are explained next. If you use the free version of WU-BLAST, there is no large file support and no virtual database mechanism. Your best solution is to create several databases within the limits of your filesystem, search each independently and then merge the results.
11.2.2 Virtual Databases
Virtual databases let you combine multiple databases and use them as if they were one. It's as simple as including the various databases in quotes on the command line. Here's how it looks for NCBI-BLAST:
blastall -p blastp -d "db1 db2 db3" -i query
And here is the equivalent WU-BLAST command line:
blastp "db1 db2 db3" query
Virtual databases are useful for grouping related searches. Let's say you want to search individually against an EST and an mRNA database, as well as a transcripts database that combines the two. You can create each database individually, but there is some duplication in data. Alternatively, you can just create the EST and mRNA databases and use the virtual database EST mRNA for transcripts. Virtual databases behave just like normal databases. You can even retrieve sequences from virtual databases with fastacmd (this feature isn't yet available for xdget, but see the end of this chapter for a workaround).
One thing you probably don't want to do is to combine databases with redundant sequences. For example, you wouldn't want to group the NCBI nr database with SWISSPROT becausenr already includes SWISSPROT. Duplicated sequences decrease the statistical significance of matches and can be confusing in the output.
11.2.3 Alias Databases
Alias databases are a unique and powerful feature of NCBI-BLAST. You've already seen that formatdb creates alias databases when splitting large files, but alias databases also have other uses. Alias databases can be used as static virtual databases for any combination of databases; all you have to do is create a file with the proper name and syntax. Here's a simple alias file, transcripts.nal, that combines the previous ESTs and mRNAs example to create a transcripts databases:
TITLE transcripts
DBLIST ESTs mRNAs
The TITLE is the name of the database and the DBLIST is simply a list of the databases to merge. Using alias databases you can, for example, organize sequences by organism. All you have to do is create the individual databases and combine them in various ways with alias files to create more comprehensive sets.
Not only can you join databases, but you can also use alias files to restrict searches to particular sequences from a database. Let's say you create a comprehensive EST database and then want to create a human-only EST database. The alias file for such a database looks something like this:
TITLE humanESTs
DBLIST ESTs
GILIST human.gi
The GILIST specifies a list of files that contains the GI numbers of the sequences to search. There are a few complexities when working with GI lists. First, using a GI list assumes that your sequences have GI numbers. If your original FASTA identifiers don't include GI numbers, you can't use this feature. It is unfortunate that the file of GI numbers isn't a file of accession numbers, but that's the way it is. If you want to use GI lists for sequences without GI numbers, you have to add fake GI numbers to your identifiers. The following script counts backward from the maximum possible GI number and thus minimizes the potential conflict with real GI numbers:
#!/usr/bin/perl
$i = 2147483648;
while (<>) {
if (/^>/) {
$i--;
print ">gi|$i (fake-gi) ", substr($_, 1);
}
else {
print;
}
}
To use this script, fake-gi.pl, simply place it upstream of formatdb in your pipe:
zcat file*.gz | fake-gi.pl | formatdb -i stdin -p F -o -n my_db -v 2000000000
Creating an alias database restricted to GI numbers is a bit more complicated than just merging databases. First you need to get a list of GI numbers for your sequences of interest. There are a number of ways you can do this, but the easiest is to use NCBI Entrez. If you have a GI list called human.gi.list, the next step is to convert it to binary form using formatdb(this step isn't actually required, but it does improve performance).
formatdb -F human.gi.list -B human.gi
Finally, create the alias file. You can do this yourself, but you might as well let formatdb do it for you because it also adds the number of sequences and their length to the information in the alias file.
formatdb -p F -i ESTs -L humanESTs -F human.gi
-L identifies the name of the alias database to create, and -F is the name of the binary file of GI numbers.
As you can see, alias databases are a powerful way to join and split databases. As BLAST development continues, you should expect to see more structure in BLAST databases and greater control of sequences subsets.
11.2.4 Removing Redundancy
One way to improve the efficiency of your BLAST searches is to remove redundant sequences. Consider what happens when you search a redundant database. The statistical significance of a database hit depends on the size of the database. Each redundant sequence artificially increases the size of the database and therefore reduces the statistical significance of any hit. In addition, redundancy makes the search slower.
The nrdb program that comes with the licensed version of WU-BLAST concatenates the definition lines of all identical sequences. This program isn't available with the free version, and the NCBI distribution doesn't include such a program in the BLAST distribution. WU-BLAST also includes patdb, which is a bit more aggressive because it concatenates identical subsequences. Both programs are very efficient. You can find examples of their use in Chapter 10. You can also write your own database purifier using Bioperl tools. (Some entertaining discussions about the "best" way to do this may be found in the Bioperl mailing list archives at http://bioperl.org). Here's one way:
#!/usr/bin/perl
use Bio::SeqIO;
my %NR;
my $file = Bio::SeqIO->new(-fh => \*ARGV);
while (my $fasta = $file->next_seq){
my $def = $fasta->id ." " .$fasta->desc;
$NR{$fasta->seq}{$def} = 1;
}
for my $seq(keys %NR){
print ">",join(chr(1),keys
%{$NR{$seq}}),"\n",$seq,"\n";
}
It's more common to collapse redundant proteins than redundant nucleotide sequences. One reason is because nucleotide sequences are rarely identical to one another. Another is that nucleotide sequences can be very large, and the procedure becomes impractical with off-the-shelf hardware. Most importantly, it is better to assemble genomic fragments into chromosomes and ESTs into full-length transcripts. Because these tasks are complicated and compute-intensive, these feats of bioinformagic are best left to the experts.
11.2.5 Standard BLAST Databases
Every BLAST search is an experiment and should be planned as such. Just as you wouldn't want to use the same query for every BLAST search, you wouldn't want to use the same database for every BLAST search. However, a few databases are used so frequently that they have become standards. All databases described in this section are available from the NCBI at ftp://ftp.ncbi.nih.gov/blast/db/. The most important is the nonredundant protein database, nr. This database combines all translations from GenBank records (including RefSeq) with proteins from the SWISS-PROT, PIR, and PDB databases. If you want to do a comprehensive search against all known proteins, this is the database to use. Not all of the protein sequences have been verified experimentally, so you should expect some errors.
There are several essential nucleotide databases. The ecoli and vector databases may sound like uninteresting databases, but they're actually quite important. The procedures used to sequence DNA require various molecular biology techniques that rely on the E. coli bacterium and various vector sequences for carrying DNA. Because of this, many common sources of data contamination are from E. coli and vector sequences. Screening nucleotide sequences against these databases is a good way to detect these pollutants. Another database that is useful for detecting contaminants in genomic DNA is the mito database of mitochondrial sequences. The est database is also one of the most popular ones. It contains all the expressed sequence tags from DDBJ/EMBL/GenBank. Many undiscovered proteins lurk in the est database.
The NCBI FTP site includes several other databases. For those interested in the business side of bioinformatics, the pataa and patnt databases contain patented amino acid and nucleotide sequences. The sts database contains sequence tagged sites, which are mostly PCR amplimers that uniquely identify a region of a genome and are used in genome mapping. The gss sequences correspond to genome survey sequences. These are random-ish sequences from various organisms. Some sequences may not have been processed to remove sequencing vectors or low quality reads, so the quality of the sequences varies. Finally, the htg database contains all high-throughput genomic sequences that correspond to large genomic fragments from various organisms.
11.2.6 Custom BLAST Databases
In many cases, using a custom database rather than a standard database is more efficient. For example, if you're only interested in searching against human sequences, there's no point in including the rest of the public database. But the total cost of a BLAST search also includes creating the database, so it isn't always more efficient to use a custom database. Suppose you've just cloned a dog gene with mutations that bear some similarity to a human disease, and you want to know which human proteins correspond to the dog protein. You can build a human protein database and search it, or you could search against nr and ignore anything that isn't human. If this is the only experiment you're going to perform, it's probably more efficient to search nr than build a custom database.
If you occasionally want to make custom databases, it's worth getting to know one of the batch retrieval systems available on the Internet. You can make custom databases easily using these web-based systems. If you find yourself making custom databases frequently and find limitations with using web-based systems, you will probably want to have an in-house database. This can be a nontrivial task involving many hours of work and expensive computers, or it can be a relatively simple operation. It depends on the kind of performance and features you want. You will learn more about this topic in just a bit, but let's first discuss sequence databases in general.
11.3 Sequence Databases
The sequences in BLAST databases come from sequence databases. But what are sequence databases and where do you get them? The answers to these simple questions are surprisingly complex. Sequence databases come in many shapes and sizes. Some are just collections of raw sequence data from genome sequencing projects, while others contain comprehensive information about the origin and function of the sequences. Unfortunately, there isn't a one-stop shopping place to get all the information you may want, but there is one particular service worth mentioning above all others: the International Nucleotide Sequence Database.
11.3.1 International Nucleotide Sequence Database
Probably the most important molecular biology resource is the public sequence database maintained by the International Nucleotide Sequence Database (INSD). It is composed of three parties: the DNA Data Bank of Japan (DDBJ, http://www.ddbj.nig.ac.jp), the European Molecular Biology Laboratory, (EMBL, http://www.embl.org), and GenBank from the National Center for Biotechnology Information (NCBI, http://ncbi.nlm.nih.gov/GenBank). This consortium collaborates to form the largest public repository for DNA and protein sequences in the world. Because it is such an important resource, this chapter spends some time exploring it.
11.3.2 Database Growth
The amount of publicly available sequence has been growing geometrically, doubling approximately every 14 months (see Figure 11-2). Fortunately, computer technology has also kept pace. While it seems scary that GenBank is currently approaching 100 GB and will be half a terabyte in a few years, it's nice to know that this isn't going to be a problem. Not every database grows so fast, though. Organism-specific databases such as the Saccharomyces Genome Database, WormBase, and FlyBase are growing at a more moderate pace, principally because the sequence of their genomes is complete. But many new genome projects are just getting started, and they will probably grow very quickly.
Figure 11-2. Growth of DDBJ/EMBL/GenBank
11.3.3 Flat Files
Sequence databases usually offer their data in several different formats. The FASTA format is universally accepted for operating on sequences, but many sequence databases record a lot more data than just the sequence. Such extra information is commonly presented in a human-readable format called a flat file. The INSD uses two kinds of flat files. The DDBJ and GenBank flat file formats are identical, while the EMBL format is slightly different. The following DDBJ/GenBank record corresponds to a fragment of the Hoxa-11 gene from the coelacanth (the ancient fish on the cover of the book):
LOCUS AF287139 606 bp DNA linear VRT 10-DEC-2000
DEFINITION Latimeria chalumnae Hoxa-11 gene, partial cds.
ACCESSION AF287139
VERSION AF287139.1 GI:11611818
KEYWORDS .
SOURCE Latimeria chalumnae.
ORGANISM Latimeria chalumnae
Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi;
Coelacanthiformes; Coelacanthidae; Latimeria.
REFERENCE 1 (bases 1 to 606)
AUTHORS Chiu,C.H., Nonaka,D., Xue,L., Amemiya,C.T. and Wagner,G.P.
TITLE Evolution of Hoxa-11 in lineages phylogenetically positioned along
the fin-limb transition
JOURNAL Mol. Phylogenet. Evol. 17 (2), 305-316 (2000)
MEDLINE 20538275
PUBMED 11083943
REFERENCE 2 (bases 1 to 606)
AUTHORS Chiu,C.-H. and Wagner,G.P.
TITLE Direct Submission
JOURNAL Submitted (14-JUL-2000) Ecology and Evolutionary Biology, Yale
University, 165 Prospect St., New Haven, CT 06520-8106, USA
FEATURES Location/Qualifiers
source 1..606
/organism="Latimeria chalumnae"
/db_xref="taxon:7897"
CDS <1..>606
/codon_start=1
/product="Hoxa-11"
/protein_id="AAG39070.1"
/db_xref="GI:11611819"
/translation="YLPSCTYYVSGPDFSSLPSFLPQTPSSRPMTYSYSSNLPQVQPV
REVTFRDYAIDTSNKWHPRSNLPHCYSTEEILHRDCLATTTASSIGEIFGKGNANVYH
PGSSTSSNFYNTVGRNGVLPQAFDQFFETAYGTTENHSSDYSADKNSDKIPSAATSRS
ETCRETDEKERREESSSPESSSGNNEEKSSSSSGQRTRKKRC"
BASE COUNT 173 a 169 c 129 g 135 t
ORIGIN
1 tacttgccaa gttgcaccta ctacgtttcg ggtcccgatt tctccagcct cccttctttt
61 ttgccccaga ccccgtcttc tcgccccatg acatactcct attcgtctaa tctaccccaa
121 gttcaacctg tgagagaagt taccttcagg gactatgcca ttgatacatc caataaatgg
181 catcccagaa gcaatttacc ccattgctac tcaacagagg agattctgca cagggactgc
241 ctagcaacca ccaccgcttc aagcatagga gaaatctttg ggaaaggcaa cgctaacgtc
301 taccatcctg gctccagcac ctcttctaat ttctataaca cagtgggtag aaacggggtc
361 ctaccgcaag cctttgacca gtttttcgag acggcttatg gcacaacaga aaaccactct
421 tctgactact ctgcagacaa gaattccgac aaaatacctt cggcagcaac ttcaaggtcg
481 gagacttgca gggagacaga cgagaaggag agacgggaag aaagcagtag cccagagtct
541 tcttccggca acaatgagga gaaatcaagc agttccagtg gtcaacgtac aaggaagaag
601 aggtgc
//
The next example is the same record in the slightly different EMBL format. Most of the data is identical between the two formats, but there are a few important differences. The VERSION field of the DDBJ/GenBank record includes a GI number (discussed below) that isn't in the EMBL record. The EMBL record contains both a creation date and a modification date, while the DDBJ/GenBank record contains only a modification date.
ID AF287139 standard; DNA; VRT; 606 BP.
XX
AC AF287139;
XX
SV AF287139.1
XX
DT 11-DEC-2000 (Rel. 66, Created)
DT 11-DEC-2000 (Rel. 66, Last updated, Version 1)
XX
DE Latimeria chalumnae Hoxa-11 gene, partial cds.
XX
KW .
XX
OS Latimeria chalumnae (coelacanth)
OC Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi;
OC Coelacanthiformes; Coelacanthidae; Latimeria.
XX
RN [1]
RP 1-606
RX PUBMED; 11083943.
RA Chiu, Ch, Nonaka D., Xue L., Amemiya C.T., Wagner G.P.;
RT "Evolution of Hoxa-11 in Lineages Phylogenetically Positioned along the
RT Fin-Limb Transition";
RL Mol. Phylogenet. Evol. 17(2):305-316(2000).
XX
RN [2]
RP 1-606
RA Chiu C.-H., Wagner G.P.;
RT ;
RL Submitted (14-JUL-2000) to the RL Ecology and Evolutionary Biology, Yale
University, 165 Prospect St., New
RL Haven, CT 06520-8106, USA
XX
DR SPTREMBL; Q9DDT9; Q9DDT9.
XX
FH Key Location/Qualifiers
FH
FT source 1..606
FT /db_xref="taxon:7897"
FT /organism="Latimeria chalumnae"
FT CDS <1..>606
FT /codon_start=1
FT /db_xref="SPTREMBL:Q9DDT9"
FT /product="Hoxa-11"
FT /protein_id="AAG39070.1"
FT /translation="YLPSCTYYVSGPDFSSLPSFLPQTPSSRPMTYSYSSNLPQVQPVR
FT EVTFRDYAIDTSNKWHPRSNLPHCYSTEEILHRDCLATTTASSIGEIFGKGNANVYHPG
FT SSTSSNFYNTVGRNGVLPQAFDQFFETAYGTTENHSSDYSADKNSDKIPSAATSRSETC
FT RETDEKERREESSSPESSSGNNEEKSSSSSGQRTRKKRC"
XX
SQ Sequence 606 BP; 173 A; 169 C; 129 G; 135 T; 0 other;
tacttgccaa gttgcaccta ctacgtttcg ggtcccgatt tctccagcct cccttctttt 60
ttgccccaga ccccgtcttc tcgccccatg acatactcct attcgtctaa tctaccccaa 120
gttcaacctg tgagagaagt taccttcagg gactatgcca ttgatacatc caataaatgg 180
catcccagaa gcaatttacc ccattgctac tcaacagagg agattctgca cagggactgc 240
ctagcaacca ccaccgcttc aagcatagga gaaatctttg ggaaaggcaa cgctaacgtc 300
taccatcctg gctccagcac ctcttctaat ttctataaca cagtgggtag aaacggggtc 360
ctaccgcaag cctttgacca gtttttcgag acggcttatg gcacaacaga aaaccactct 420
tctgactact ctgcagacaa gaattccgac aaaatacctt cggcagcaac ttcaaggtcg 480
gagacttgca gggagacaga cgagaaggag agacgggaag aaagcagtag cccagagtct 540
tcttccggca acaatgagga gaaatcaagc agttccagtg gtcaacgtac aaggaagaag 600
aggtgc 606
//
Note that the sequence data is only one part of the record; there's a lot of other useful information in here including the organism, the taxonomic classification, the authors, a reference to the scientific literature, and a feature table indicating the translation of the DNA. This is great stuff, and INSD is full of these kinds of records. But there is a downside to using the public databases. They're a bit like public parks: huge, beautiful, inexpensive to use, and valuable, but there's always someone who doesn't pick up their trash. Some sequences are erroneous, and the ancillary information is sometimes wrong and misleading. But overall, the databases are high-quality resources, and you should take a moment to applaud the scientists who contribute their sequences to the INSD, as well as the administrators and curators at DDBJ/EMBL/GenBank who do an outstanding job. Now let's take a closer look at some parts of the sequence record.
11.3.3.1 ACCESSION, LOCUS, VERSION, and GI
One of the most important parts of any sequence record is its database identifier, which is often called its accession number. (Although it's called a number, it may be a mixture of letters, numbers, and other symbols, but not spaces.) This tag uniquely identifies the sequence in a database. There isn't necessarily a one-to-one correspondence between sequences and tags because sequences are sometimes known by multiple unique names. The DDBJ/GenBank ACCESSION (or AC in EMBL) is the primary name for a sequence record. Another unique name is the LOCUS (or ID in EMBL). The locus is supposed to be a "short mnemonic name for the entry, chosen to suggest the sequence's definition." For example, "HSMG01" is the locus name for the database entry containing Homo sapiens myoglobin exon 1. Over time, like the names of celestial objects, locus names have become less descriptive and are often just duplicates of the accession numbers.
Sequence records can also change over time. This often happens when the record is edited to correct a sequence error. The accession number and locus don't change, but the version number is increased (VERSION in DDBJ/GenBank and SV in EBML). In this way, an ACCESSION.VERSION points to a particular record at a particular time. It's a good idea to always refer to sequences in this way and not by ACCESSION alone or by LOCUS or ID.
DDBJ/GenBank records include an additional token called the GI number, which is a numeric identifier that points to a particular ACCESSION.VERSION. The GI number is especially important because NCBI-BLAST relies on it as an additional mechanism for indexing BLAST databases. This topic was covered in Section 11.2.3.
11.3.3.2 DEFINITION, KEYWORDS, and SOURCE
The DEFINITION is a concise description of the origin and function of a sequence, and is typically what you find a FASTA description. The text is structured, meaning that there are rules that define how it is produced. However, it doesn't use a controlled vocabulary, which means you can't be sure which words will or won't appear.
KEYWORDS are a historical relic like the locus name and aren't used in modern sequence records. Avoid the temptation to believe that keywords are meaningful.
The common name for an organism is often found in the SOURCE, or in parentheses after the OS in EMBL format. The scientific name is on the ORGANISM line (OS in EMBL) and the complete taxonomic classification is given on the following lines (OC in EMBL). The complete taxonomy may be abbreviated if it's especially long.
11.3.3.3 FEATURES
The FEATURES (FT in EMBL) list specific regions of importance on the sequence such as genes or repetitive elements. The general syntax of features is fairly simple; each has a key and location, and optional qualifiers. The key tells what kind of feature it is (e.g., a gene), the location (e.g., from nucleotide 100 to nucleotide 200), and the qualifiers include additional information, such as specific names, database cross references, and experimental notes. A detailed discussion of the feature table is beyond the scope of this book. See http://www.ncbi.nih.gov/projects/collab/FT for more information.
11.3.4 Other Common Databases
INSD is just one of many important databases. Some other favorites are listed in Table 11-2.
|
Table 11-2. Popular biological databases |
|||
|
Database |
Description |
||
|
RefSeq |
RefSeq provides reference sequences that represent the highest quality information about a particular sequence. Each record may be constructed from several INSD records, which makes the database nonredundant. All RefSeq accession numbers are preceded by two letters and an underscore, for example XP_102310. Some types of RefSeq records have been inspected manually by curators, and they are the highest quality records (indicated below). |
||
|
Prefix |
Molecule |
Description |
|
|
NC_ |
Genomic |
Curated complete genomic molecules including genomes, chromosomes, organelles, and plasmids. |
|
|
NG_ |
Genomic |
Curated incomplete genomic region; primarily supplied for Homo sapiens and Mus musculus to support the NCBI Genome Annotation pipeline. |
|
|
NM_ |
mRNA |
Curated mRNAs. |
|
|
NR_ |
RNA |
Curated noncoding transcripts including structural RNAs, transcribed pseudogenes, and others. |
|
|
NP_ |
Protein |
Curated proteins. |
|
|
NT_ |
Genomic |
Intermediate genomic assemblies of BAC sequence data. |
|
|
NW_ |
Genomic |
Intermediate genomic assemblies of Whole Genome Shotgun sequence data. |
|
|
XM_ |
mRNA |
Homo sapiens model mRNA provided by the Genome Annotation process; sequence corresponds to the genomic contig. |
|
|
XR_ |
RNA |
Homo sapiens model noncoding transcripts provided by the Genome Annotation process; sequence corresponds to the genomic contig. |
|
|
http://www.ncbi.nlm.nih.gov/LocusLink/refseq.html |
|||
|
Pfam |
Pfam is a collection of multiple sequence alignments and hidden Markov models (HMMs) for many common protein domains and families. If you are interested in a particular family, such as globins, or a particular domain, such as WD-40, this is a great resource. HMMs are probabilistic models that describe how whole domains evolve, which is quite different from a scoring matrix employed by BLAST that treats each amino acid of a protein independently. http://pfam.wustl.edu,http://www.sanger.ac.uk/pfam |
||
|
SWISS-PROT |
The SWISS-PROT Protein Knowledgebase is a curated protein sequence database that provides a high level of annotation (such as the description of protein function, domains structure, post-translational modifications, variants, etc.), a minimal level of redundancy, and high level of integration with other databases. http://www.ebi.ac.uk/swissprot |
||
|
TrEMBL |
The TrEMBL database contains translations of all coding sequences (CDS) present in the INSD, which aren't yet integrated into SWISS-PROT. TrEMBL is split into two main sections: SP-TrEMBL contains entries expected to be included in SWISS-PROT, and REM-TrEMBL contains those that aren't expected to be included. http://www.ebi.ac.uk/tremble |
||
|
UniGene |
UniGene is an experimental system for automatically partitioning GenBank sequences into a nonredundant set of gene-oriented clusters. Each UniGene cluster contains sequences that represent a unique gene, as well as related information such as the tissue types in which the gene has been expressed and the map location. UniGene sets are available for most genomes with a lot of EST sequences. http://www.ncbi.nlm.nih.gov/UniGene |
||
|
MGC |
The goal of the Mammalian Gene Collection (MGC) is to provide a complete set of full-length (open reading frame) sequences and cDNA clones of expressed mammalian genes. The current focus is limited to human and mouse. http://mgc.nci.nih.gov |
||
|
SGD |
SGD is a scientific database of the molecular biology and genetics of the yeast Saccharomyces cerevisiae, which is commonly known as baker's or budding yeast. S. cerevisiae was the first eukaryotic genome sequenced. http://genome-www.stanford.edu/Saccharomyces |
||
|
WormBase |
WormBase is a comprehensive database dedicated to the biology and genome of the nematode Caenorhabditis elegans. C. elegans was the first multicellular organism to have its genome sequenced. http://www.wormbase.org |
||
|
FlyBase |
FlyBase is a comprehensive database for information on the genetics and molecular biology of Drosophila. It includes data from the Drosophila Genome Projects and data curated from the literature. FlyBase is a joint project with the Berkeley Drosophila Genome Project. http://www.flybase.org |
||
|
TAIR |
The Arabidopsis Information Resource (TAIR) provides a comprehensive resource for the scientific community working with Arabidopsis thaliana, a widely used model plant. http://www.arabidopsis.org |
||
11.4 Sequence Database Management Strategies
There are many useful public sequence databases, and you may have access to some private ones as well. Because this is a book about BLAST, we assume you want to use these collections of sequences in BLAST searches. Some sequences may be used as queries, and others in databases. How are you going to manage them all in a rational way? Several possible strategies exist, and the correct one for you depends on your needs and resources. To demonstrate some of the issues, let's review a typical sequence analysis scenario.
Suppose a colleague of yours has just found the gene that makes cats go crazy for catnip. She wants to learn more about this gene and comes to you for help because you are a BLAST expert. The first thing she wants to do is a BLAST search to find out what vertebrate proteins are similar to this one. Where are you going to get such a database of proteins? Once you perform the BLAST search, you find several interesting similarities. Your colleague tells you that these are probably all part of a family of proteins, and she would like to build a phylogenetic tree to determine their relationships to one another. How are you going to get the individual sequences? Finally, she decides she wants more information about the human sequences, and to do that, she would like references to the scientific literature like the ones she would find in a DDBJ/EMBL/GenBank report. How are you going to retrieve such information? You could just refuse to help her because these aren't really BLAST problems, but these are the kinds of tasks many BLAST users must face. Let's take a look at how they can be solved.
This example has basically two solutions to each question: the first is to use tools available on the Internet. The second is to build the tools yourself. In general, it is much easier to use the Internet, but for high speed or high-throughput operations you'll want a local solution. After you read this chapter, you may decide that you want some services to be provided locally, while others are Internet-only operations. This section begins with a brief review of databases.
11.4.1 Queries, Indexes, and Reports
The most common database operation is a query. One person may want to retrieve a particular sequence. Another may want all human sequences. As you have seen, sequence records have quite a bit of useful information, and a user may request nonsequence information such as all the MEDLINE references for all sequences with the word disease in the description.
The efficiency with which a query is executed depends a lot on how the database is indexed. If there is no indexing, a query must operate on every record of the database. So, for example, if you want to find all the coelacanth sequences, you would have to look through millions of records to find the handful whose sequences originate from the coelacanth. Clearly, this isn't going to be efficient, so databases usually have indexes that, for example, keep lists of species and all the sequences for each species.
The most straightforward kind of indexing occurs when there is a unique relationship between a property and a sequence. This is called a one-to-one mapping, and an example would be an accession number. A more complex indexing occurs when a property points to many sequences. This is called a one-to-many mapping, and an example is a species name that is shared by millions of records.
Once a query is executed, the data must be reported in some format. For sequences, this is usually the FASTA format. For other kinds of data, there are other appropriate formats, such as lists, tables, and graphs.
11.4.2 Local Database Considerations
Having a local sequence database has some real advantages. First, local databases are faster and more reliable because they don't rely on an Internet connection. If you're involved in high-throughput research, these reasons are sufficient. Another compelling reason is that you can combine several databases, and even include your own sequences that aren't in the public databases. The downside to creating a local sequence database is the amount of work it takes. Depending on the scale of the operation, it can be a full-time job. Here are six important issues to address when building a local sequence database:
Downloading
Each database you support must be downloaded from time to time to keep the data current. For example, GenBank has five to six major releases each year, as well as daily updates. Other databases have their own update schedule. Managing updates can be a chore if you download a lot of databases, so automating the procedure is a good idea. In addition, you may want to take measures to ensure that during updates, which can take some time, the database that's presented to users isn't actually changing. This may require keeping a mirror of some data. Notice that having a local database doesn't mean you can completely insulate yourself from the Internet.
Processing records
Each database you support must have a parser to read the various fields of each record. This may be as simple as pulling out the accession number for a sequence, or it may be much more complicated, such when you record specific keywords. You can build your own parsers but it takes less time to use one already created, such as a parser from the Bioperl project.
Storing data
Your database schema will determine how each record is stored and what kinds of relationships exist between various pieces of data. Designing an appropriate schema is a difficult problem because it takes people who understand the data (biologists), the data models (software engineers), and the storage/backup of the data (systems administrators).
Indexing
The efficiency of queries will largely depend on what data is indexed. You may choose to index everything, but your indexes could grow much larger than your data. So you may have to make compromises. This is another place where users and engineers must interact to determine the appropriate solution.
Querying
Not all databases are queried in the same way. Relational databases usually employ SQL as the query language, but many popular databases have their own querying mechanisms. The details of how you interacts with the database may depend on what kind of database you use. Regardless of the underlying architecture, you may decide to present a different interface to users, such as a form in a web browser or a script/program interface that connects directly to the database.
Formatting
You'll definitely want to create FASTA files, but what other report formats will you want to support? The DDBJ/EMBL/GenBank flat file formats are sometimes used to exchange data, so this would be useful, as would tabular format and some kind of HTML that looks good in browsers. For each output format, you may need some specialized code to generate the report.
As you can see, building a local database isn't trivial. But it doesn't have to be a full-time job if you only want a subset of the information. For example, if all you want is to retrieve records by accession number, you don't need to invest more than a couple hours of work. The following section explores the common techniques for managing sequence data.
11.4.3 Retrieving FASTA Files by Accession
The task of retrieving FASTA files by accession number is so common and has such an easy solution that it should be a local resource. If you're using NCBI-BLAST, the fastacmdprogram retrieves sequences from BLAST databases singly or in batches. If you're using WU-BLAST, the xdget program does the same thing. To use these features, you must index the databases when you format them, which is as simple as including the -o or -I option (see the command-line tutorial in Chapter 10, the reference sections for formatdb and fastacmd inChapter 13, and xdformat and xdget in Chapter 14). One limitation of this approach is that the sequences are stored in a case-insensitive format in the database. If you use lowercase to denote regions containing repeats, for example, that information will be lost. If this is a serious problem for you, use one of the flat-file indexing schemes described later.
NCBI-BLAST users take note that unless you use the NCBI FASTA definition line format discussed earlier in this chapter, your definition lines may not look exactly the same when they come out of the database. For example, if you have a definition line such as this:
>FOO
When you retrieve it with fastacmd, it looks like:
>lcl|FOO no definition found
You can easily avoid such inconsistencies by using the recommended identifier format and by including descriptions on the definition line.
WU-BLAST users take note: xdget doesn't support virtual databases. You can work around this limitation with a simple script, such as this one:
#!/usr/bin/perl -w
use strict;
my (@DB, $i);
for ($i = 0; $i < @ARGV; $i++) {
if ($ARGV[$i] =~ /\s/) {
@DB = split(/\s+/, $ARGV[$i]);
last;
}
}
exec("xdget @ARGV") unless @DB;
my @pre = splice(@ARGV, 0, $i);
my @post = splice(@ARGV, 1);
foreach my $db (@DB) {
system("xdget @pre $db @post");
}
11.4.4 Flat File Indexing
One of the most common procedures used to manage sequence data is called flat file indexing. In this approach, you keep concatenated sequence reports in their native format and store the starting position of each record in a separate file. One advantage of this approach is that you don't have to do any work when you want to reproduce the data in flat file format. Another reason why flat file indexing is so common is that it is simple to implement, at least for one-to-one mappings. To illustrate the process, we'll show you how to index identifiers in FASTA files. Here is an example of a very short FASTA file:
>FOO
GAATTC
>BAR
ATAGCGAAT
This file has two records with identifiers FOO and BAR, and they begin at bytes 0 and 12, respectively (count the letters and don't forget to add one for the end of line—in Windows, the end of line is actually two characters, and this will change the positions to 0 and 14). You can now create an index file that tells where each record begins in the file:
BAR 12
FOO 0
To use this index file, simply find the identifier of interest in the index and seek to the appropriate position in the FASTA file. Note that you sorted the lookup file alphabetically by identifier. This makes it much more efficient to find the record because you can use a binary search to find the identifier. If you have an index file containing 1 million records, on average, a linear search looks through 500,000 records, but a binary search looks at only 20.
You can make a couple of improvements to this simplistic indexing scheme. The first is to allow the index file to support more than one FASTA file. This is a trivial modification because you can just add a filename to your index file:
BAR file-A 12
FOO file-A 0
XYZ file-B 0
Another easy improvement is to use a persistent indexed data structure such as a Perl tied-hash. The Bioperl project uses this strategy in its Bio::Index classes.
A slightly more complicated approach manages the indices with one of the many free or commercial database applications, such as MySQL, PostgreSQL, FileMaker, Microsoft Access, or whatever you happen to be familiar with. If you're going to do this, you might as well store a bit more data. For illustrative purposes, imagine you create a schema like that in Table 11-3. In addition to the accession number, file, and offset, this schema provides for a species and a molecule type (moltype). The actual sequence in the schema was not provided because some applications can't handle data this large. If you wish to store sequences as well, test the performance of the system with realistic data to see if the system scales well.
|
Table 11-3. Sequence database example |
||||
|
Accession |
Species |
Moltype |
File |
Offset |
|
A |
Homo sapiens |
AA |
file-1 |
12024 |
|
B |
Homo sapiens |
AA |
file-1 |
250 |
|
C |
Homo sapiens |
DNA |
file-2 |
28223 |
|
AF287139 |
Latimeria chalumnae |
cDNA |
file-3 |
0 |
Using such a database you can don't only the simple accession number retrievals, but also the one-to-many relationships such as all human sequences or all DNA sequences. All you have to do is query the database and seek to the appropriate place in the appropriate file for every record. Organizing the data this way has a number of advantages over just downloading DDBJ/EMBL/GenBank by division. For example, if you want to make a database of all human transcripts, you need to identify the human ESTs from the EST division, as well as all the mRNAs from the PRI (primate) division. But if you've designated all ESTs and mRNAs as the cDNA moltype, getting all human transcripts is as easy as retrieving all records in which the species is Homo sapiens and the moltype is cDNA. You can add several more fields to the database, like date created, division, keywords, etc., and get quite a bit of functionality without much more complexity.
Overall, flat file indexing is a very good strategy for sequence management because it is simple, fast, and retains the data in its original format. You don't even have to write any software, as both free and commercial software packages are designed specifically for managing flat file data. Check out the Bioperl project at http://bioperl.org, MyGenBank at http://sourceforge.net/projects/mgb, and SRS (see Table 11-4).
11.4.5 Commercial Sequence Management Software
Several commercial software packages are designed for managing biological sequence data. The database software is generally part of a much larger software suite that includes sequence analysis tools such as BLAST and visualization tools to make interpretation easier. The companies that develop these packages expend a great deal of effort to make the various sequence analysis tasks interoperable and user friendly. Table 11-4 gives a brief description of the software.
|
Table 11-4. Commercial sequence management software |
|
|
Company |
Product and description |
|
Accelrys |
The popular Wisconsin GCG package is now owned by Accelrys, which provides the SeqStore software for managing sequence data. The system uses an Oracle database and allows daily/weekly updates. To install and maintain the system, you must have personnel with experience in Unix systems administration and Oracle database administration. Accelrys recommends a computer with at least 4 CPUs, at 4-GB RAM, and 40- GB disk space. http://www.accelrys.com |
|
Informax |
The Genomax software suite provides sequence management along with a comprehensive set of interoperable tools. Informax recommends a project manager, a Unix systems administrator, and an Oracle database administrator to manage and maintain the system, as well as a life sciences expert to respond to users' questions. Informax uses a three-tiered architecture and recommends that the three computers be configured with 4 CPUs and 4-8 GB RAM, and the database server have 400-GB disk space. http://www.informaxinc.com |
|
LION Biosciences |
LION Biosciences offers the Sequence Retrieval System (SRS). SRS is probably the most popular sequence management software in use today and is used by both DDBJ and EMBL. SRS is free for academic users. LION produces a separate, related product PRISMA2, which is an automatic databank-updating and maintenance tool. SRS requires a person with competent Unix skills to install and maintain and a server with enough storage for the various databases and indexes. http://lionbioscience.com |
As you can see from the descriptions of the personnel and hardware requirements, using these comprehensive sequence analysis systems requires a serious commitment. For these reasons, these packages aren't recommended for small research groups. For larger groups, though, these products can save a lot of time and money. It's easy to underestimate the effort required to develop your own sequence management system, so take caution before embarking on such a task, and give the professionals a chance to show you their wares.
11.4.6 Tools on the Internet
There are good reasons to use web-based tools for sequence management rather than building a local database. First, you don't have to download more data than you need. Mirroring the entire public database isn't efficient if you need only a slice of it. Second, database providers take care of the most time-consuming and expensive tasks, namely processing, storing, and indexing the data. Third, the databases are self-updating, which means that you can always get the latest and most accurate information. Best of all, the service is completely free. Well, maybe not completely free since the databases are supported from taxes, but let's all thank the various governments and funding agencies for putting our hard-earned money toward a worthy cause, and let's especially recognize all the people that make it actually happen.
The downside to using web-based tools is that you have to spend time learning how to query the database efficiently and accurately, but that's going to be true of any sequence management system, even your own. A more serious issue is that you will depend on the computers and network between you and the database provider, but this will improve over time. Still, even if you have to put up with a few glitches here and there, the total cost in time and money is probably cheaper than building your own local mirror.