FSTpoolit - Flanking Sequence Tag POOLing and IdenTification ### release note The only difference between version 0.2 and 0.21 is that the example file "endogenous_insertions_test.txt" was replaced. ### Introduction FSTpoolit identifies tagged insertions in individuals pooled in a 2D scheme, barcoded, and sequenced using paired-end technology. FSTpoolit_config holds variable information used by all other programs FSTpoolit_index produces bowtie and bwa indices for reference files FSTpoolit_map annotates, trims and maps reads to the reference FSTpoolit_filter filters insertions and produces the final insertion list FSTpoolit_primer designs genotyping primers ### System requirements A unix-based system such as linux or MacOS with at least 8 GB of RAM. More for large reference genomes. You will also need plenty of disc space >100 GB for most applications. ### Installation untar the file package tar -xvzf FSTpoolit_v0.2_tarred.tgz Copy scripts and example files to your folders of choice. ## Make the shell scripts executable: chmod 755 [script_directory]FSTpoolit_config.sh chmod 755 [script_directory]FSTpoolit_index.sh chmod 755 [script_directory]FSTpoolit_map.sh chmod 755 [script_directory]FSTpoolit_filter.sh chmod 755 [script_directory]FSTpoolit_primer.sh ## Add the script directory path to your environment: export PATH=$PATH:[script_directory] We recommend to run FSTpoolit with the provided test data to validate correct installation. ## Dependencies FSTpoolit requires the following programs to be in your path environment: # perl (installed on most unix systems) # bwa (Li H, Durbin R. Bioinformatics. 2010 Mar 1;26(5):589-95; http://sourceforge.net/projects/bio-bwa/files/; version used: 0.5.9) # bowtie (Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol 10:R25. http://bowtie-bio.sourceforge.net/index.shtml; version used: 0.12.3) # primer3 (Rozen S, Skaletsky H, Primer3 on the WWW for general users and for biologist programmers. Methods Mol Biol. 2000;132:365-86. http://primer3.sourceforge.net/releases.php; version used: 2.2.3; primer3 is only required by FSTpoolit_primer) # when correctly installed, typing "bowtie", "bwa", and "primer3_core -v" (without the quotes) from your home directory should result in usage messages. You can add directories to your path using export PATH=$PATH:[directory] ### input file formats # FastQ files FSTpoolit requires fastq files with unique read headers formatted as: @HWI-ST132_0429:7:1:1249:1913#AAGGCC/1 where "HWI" is the machine name, which will be replaced by IA/LTR designations "#AAGGCC" is a multiplexing barcode, which will be replaced by 2D pooling coordinates. A dummy sequence can be inserted, but is has to be six characters long. "/1" indicates that this is read 1 of the pair. # IA and LTR sequence files has to be in FASTA format with >IA and >5_LTR headers, respectively. example files 'IA.fasta' and 'LTR.fasta'. # barcode sequence files multifasta with headers formatted as: > example header: >C_10 eample file: lore1_barcodes.fa # reference genome sequence multifasta format. Headers should be formatted as > example: . Example line: C_1 R_1 DK-01_30000001 Example file: 'plant_coordinates_DK01.txt' # list of original element positions Format: Exmpale line: chr4 40233527 Example file: 'endogenous_insertions_test.txt' ### example files # input files example input files can be found in the folders "ref_data" and "data". # output files example output files can be found in the folder "output". ### configuring FSTpoolit When all input files are correctly formatted, update the file FSTpoolit_config.sh' with your parameters: # directory variables: data_directory: Path to the fastq input files. All output files will also be written to this directory. Example: /u/sua/data/ script_directory: Path to all FSTpoolit scripts. ref_data_directory: Path to all input files except the fastq sequence files. # files names of files in the "data_directory": read1: Read 1 filename. read2: Read 2 filename. # file names of files in the "ref_data_directory" IA_seq: Fasta file containing the intermediate adapter sequence. LTR_seq: Fasta file containing the LTR or other insertion-specific sequence. barcode_seq: Multifasta file with barcode sequences and 2D pool coordinates. ref_genome_seq: Multifasta reference genome sequence. plant_coordinates: File specifying individual plant IDs within the 2D setup. endo_insertions: List of endogenus insertion positions # name variables BATCH: Sequencing batch ID. Serves as prefix for final output files. machine_name: Sequencing machine model. # read property variables read_length: Total read length IA_length: Total length of the intermediate adapter including random bases IA_random_bases: Number of random bases at the 5' end of IA LTR_length: LTR length excluding barcode barcode_length: Barcode length # computation variables cpu_number: Number of CPU cores to use for computation reference_indexing: Enter "yes" make to FSTpoolit_index index the reference genome. Only enter "no" if you already have a bwa-index of your reference genome sequence in the "ref_data_directory" folder. # cutoff variables coverage_cutoff: Minimum number of reads in each row and column to support insertion call. Try with 4. stdev_cutoff: Standard deviation cutoff. Read abundance has to be above this multiple of the standard deviation for all read abundances at a position. Entering '0' turns of standard deviation filtering. Try with 0. avg_cutoff='1': Average cutoff. Read abundance has to be above this multiple of the average for all read abundances at a position. Entering '0' turns of average filtering. Try with 1. ### Running FSTpoolit When configuration file is complete: type 'FSTpoolit_index.sh' (without the quotes) to produce bowtie indices for IA, LTR, and barcode sequences and a bwa index for the reference genome. You only need to run FSTpoolit_index once for one set of reference files. type 'FSTpoolit_map.sh' to annotate, trim and map your reads. type 'FSTpoolit_filter.sh' to do quality filtering and produce the final list of insertions. type 'FSTpoolit_primer.sh' to design genotyping primers. The bulk of the computation is done by 'FSTpoolit_map.sh', allowing quick reruns of 'FSTpoolit_filter.sh' to find the optimum quality cutoffs for a particular dataset. ### output file formats ## FSTpoolit_map output # *.log files holds the output from bowtie and bwa mapping read_1_1_dou.fq_bowtie_IA.log read_1_1_dou.fq_bowtie_LTR.log read_1_1_dou.fq_bwa_indexing.log read_1_1_dou.fq_bwa_mapping.log # _IA.sam.hit and _LTR.sam.hit are lists of read names matching IA and LTR sequences, respectively. Format: Example line: HWI-ST132_0429:7:1:1249:1913#AAGGCC/1 # *.ltr_ia.both.list is a list of read names of IA-LTR read pairs. Format: Example line: HWI-ST132_0429:7:1:14775:3142#AAGGCC # *.annot.fq contains fastq format files with added IA and LTR designations Example lines: @IA-ST132_0429:7:1:1249:1913#AAGGCC/1 TAGTGGCTGAATGAGACTGGTGTCGACACTAGTGGTAAGGGAAAATTTACATTTGGATCAGACTTCAGATNNNATCATGAAACAAGACAGTATTTTGTTA +IA-ST132_0429:7:1:1249:1913#AAGGCC/1 aefffffffdfffeeadd__addeefbff_dZ`ddUZaBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB # *.annot.ltr_only.fq same as .annot.fq files but exclusively with ltr matching sequences. # *.both.barcoded.sam.hit lists all LTR matching reads from IA-LTR pairs and their 2D pool coordinates. Format: <2D_pool_coordinate> Example line: 5_LTR-ST132_0429:7:1:1259:1957#AAGGCC/1 R_75 # *.annot.barcoded.fq are fastq format files with added IA, LTR and 2D pool coordinate designations Example lines: @5_LTR-ST132_0429:7:1:1259:1957#R_75/1 CGTTAAGCCAACTAACAACCCAGCTAACAGTAGACATCTGTAACATTTTCAGAATACAATTGAGACGGCGCCTTTGAAGAGATATAGAAAGAAGAAGGAT +5_LTR-ST132_0429:7:1:1259:1957#R_75/1 fffffeefffggggggggggggfggggefedfafffdeadc_d]Sc[bccedcdXb_aeab\]bbBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB # *.annot.barcoded_trimmed.fq are fastq format files with added with added IA, LTR and 2D pool coordinate designations and trimmed for all non-genomic sequences. Example lines: @5_LTR-ST132_0429:7:1:1259:1957#R_75/1 ATTTTCAGAATACAATTGAGACGGCGCCTTTGAAGAGATATAGAAAGAAGAAGGA +5_LTR-ST132_0429:7:1:1259:1957#R_75/1 Sc[bccedcdXb_aeab\]bbBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB # *_mapped.sam is a SAM format file containing information about all mapped LTR-IA reads. Example line: 5_LTR-ST132_0429:7:1:1259:1957#R_75 99 chr1 29429214 50 55M = 29429299 150 ATTTTCAGAATACAATTGAGACGGCGCCTTTGAAGAGATATAGAAAGAAGAAGGA 4DCC################################## XT:A:U NM:i:3 SM:i:25 AM:i:25 X0:i:1 X1:i:0 XM:i:3 XO:i:0 XG:i:0 MD:Z:22T2A10A18 # *_mapped.sam.hit is an excerpt from the _mapped.sam file listing insertion positions as the first base upstream of the 5'LTR Format: <2D_pool_coordinate> Example line: chr1 29429215 R R_75 # *_position_abundance_summary.txt shows read abundances for all coordinates detected at a genomic position Format: ... Example line: chr1 864862 R C_54 1 R_40 2 ## FSTpoolit_filter output # *_position_coordinate_summary.txt shows coordinate read abundances, which passed the abundance cutoff filter Format: <#_separated_coordinates> <#_separated_abundances> Example line: chr2 27103777 F C_1#C_49#R_1 45#11#73 129 # *_position_coordinate_summary_noendo.txt shows coordinate read abundances, which passed the abundance cutoff and original element filters Format: same as *_position_coordinate_summary.txt # *_position_coordinate_summary_noendo_nodupl.txt shows coordinate read abundances, which passed the abundance cutoff, original element, and mismapping filters Format: same as *_position_coordinate_summary.txt # *_insertion_table.txt shows passed filter insertion calls in individual planta Format: <#_separated_coordinates> <#_separated_abundances> Example line: C_1 R_1 chr2 27103777 F C_1#C_49#R_1 45#11#73 129 DK-01_30000001 ## FSTpoolit_primer output #*_primer3_input.txt holds insertion flanking region sequences for genotyping primer design Example: SEQUENCE_ID=chr0_6664661_F SEQUENCE_TEMPLATE=TAGCATATA[sequence_truncated]CTACAGCA SEQUENCE_TARGET=900,300 = # *_primers.txt Format: Example line: DK01 chr2 1105769 F GTAAGCATTACAATA[sequence_truncated]AGGATCAGAAAATGGATGGATCC GGTTGCGCCGGAAACCACTCTTCT TTCCCCAACCTTGCACTCATCGCT 520 669 ### Questions and comments Please contact Stig U. Andersen sua@mb.au.dk