FSTpoolit - 
Flanking Sequence Tag POOLing and IdenTification






### Introduction
FSTpoolit identifies tagged insertions in individuals pooled in a 2D scheme, barcoded, and sequenced using paired-end technology.

FSTpoolit_config	holds variable information used by all other programs
FSTpoolit_index		produces bowtie and bwa indices for reference files	
FSTpoolit_map		annotates, trims and maps reads to the reference
FSTpoolit_filter	filters insertions and produces the final insertion list
FSTpoolit_primer	designs genotyping primers





### System requirements
A unix-based system such as linux or MacOS with at least 8 GB of RAM. More for large reference genomes. You will also need plenty of disc space >100 GB for most applications.





### Installation
Copy scripts and example files to your folders of choice.

## Make the shell scripts executable:
chmod 755 [script_directory]FSTpoolit_config.sh
chmod 755 [script_directory]FSTpoolit_index.sh
chmod 755 [script_directory]FSTpoolit_map.sh
chmod 755 [script_directory]FSTpoolit_filter.sh
chmod 755 [script_directory]FSTpoolit_primer.sh

## Add the script directory path to your environment:
export PATH=$PATH:[script_directory]
We recommend to run FSTpoolit with the provided test data to validate correct installation.

## Dependencies
FSTpoolit requires the following programs to be in your path environment:
# perl 
(installed on most unix systems)
# bwa 
(Li H, Durbin R. Bioinformatics. 2010 Mar 1;26(5):589-95; http://sourceforge.net/projects/bio-bwa/files/; version used: 0.5.9)
# bowtie 
(Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol 10:R25. http://bowtie-bio.sourceforge.net/index.shtml; version used: 0.12.3)
# primer3 
(Rozen S, Skaletsky H, Primer3 on the WWW for general users and for biologist programmers. Methods Mol Biol. 2000;132:365-86.  http://primer3.sourceforge.net/releases.php; version used: 2.2.3; primer3 is only required by FSTpoolit_primer)
# when correctly installed, typing "bowtie", "bwa", and "primer3_core -v" (without the quotes) from your home directory should result in usage messages.
You can add directories to your path using export PATH=$PATH:[directory]






### input file formats 

# FastQ files
FSTpoolit requires fastq files with unique read headers formatted as:
@HWI-ST132_0429:7:1:1249:1913#AAGGCC/1
where 
"HWI" is the machine name, which will be replaced by IA/LTR designations
"#AAGGCC" is a multiplexing barcode, which will be replaced by 2D pooling coordinates.
	A dummy sequence can be inserted, but is has to be six characters long.
"/1" indicates that this is read 1 of the pair.

# IA and LTR sequence files
has to be in FASTA format with >IA and >5_LTR headers, respectively.
example files 'IA.fasta' and 'LTR.fasta'.

# barcode sequence files
multifasta with headers formatted as: ><coordinate>
example header: >C_10
eample file: lore1_barcodes.fa

# reference genome sequence
multifasta format. Headers should be formatted as ><chromosome> 
example: <chr1
Example file 'ljr25.fasta'

# plant 2D coordinate file
Format: <column_coordinate>	<row_coordinate>	<plant_id>. 
Example line:	C_1	R_1	DK-01_30000001
Example file: 'plant_coordinates_DK01.txt'

# list of original element positions
Format: <chromosome>	<position> 
Exmpale line: chr4	40233527
Example file: 'endogenous_insertions_test.txt'






### example files

# input files
example input files can be found in the folders "ref_data" and "data".

# output files
example output files can be found in the folder "output".






### configuring FSTpoolit

When all input files are correctly formatted, update the file FSTpoolit_config.sh' with your parameters:

# directory variables:
data_directory: Path to the fastq input files. All output files will also be written to this directory. Example: /u/sua/data/
script_directory: Path to all FSTpoolit scripts.
ref_data_directory: Path to all input files except the fastq sequence files.

# files names of files in the "data_directory":
read1: Read 1 filename.
read2: Read 2 filename.

# file names of files in the "ref_data_directory" 
IA_seq: Fasta file containing the intermediate adapter sequence.
LTR_seq: Fasta file containing the LTR or other insertion-specific sequence.
barcode_seq: Multifasta file with barcode sequences and 2D pool coordinates.
ref_genome_seq: Multifasta reference genome sequence.
plant_coordinates: File specifying individual plant IDs within the 2D setup.
endo_insertions: List of endogenus insertion positions

# name variables
BATCH: Sequencing batch ID. Serves as prefix for final output files.
machine_name: Sequencing machine model.

# read property variables
read_length: Total read length
IA_length: Total length of the intermediate adapter including random bases
IA_random_bases: Number of random bases at the 5' end of IA
LTR_length:	LTR length excluding barcode
barcode_length:	Barcode length

# computation variables
cpu_number:	Number of CPU cores to use for computation
reference_indexing: Enter "yes" make to FSTpoolit_index index the reference genome. Only enter "no" if you already have a bwa-index of your reference genome sequence in the "ref_data_directory" folder.

# cutoff variables
coverage_cutoff: Minimum number of reads in each row and column to support insertion call. Try with 4.
stdev_cutoff: Standard deviation cutoff. Read abundance has to be above this multiple of the standard deviation for all read abundances at a position. Entering '0' turns of standard deviation filtering. Try with 0.
avg_cutoff='1': Average cutoff. Read abundance has to be above this multiple of the average for all read abundances at a position. Entering '0' turns of average filtering. Try with 1.





### Running FSTpoolit

When configuration file is complete:
type 'FSTpoolit_index.sh' (without the quotes) to produce bowtie indices for IA, LTR, and barcode sequences and a bwa index for the reference genome. You only need to run FSTpoolit_index once for one set of reference files. 
type 'FSTpoolit_map.sh' to annotate, trim and map your reads.
type 'FSTpoolit_filter.sh' to do quality filtering and produce the final list of insertions.
type 'FSTpoolit_primer.sh' to design genotyping primers.

The bulk of the computation is done by 'FSTpoolit_map.sh', allowing quick reruns of 'FSTpoolit_filter.sh' to find the optimum quality cutoffs for a particular dataset.






### output file formats


## FSTpoolit_map output

# *.log files
holds the output from bowtie and bwa mapping

read_1_1_dou.fq_bowtie_IA.log
read_1_1_dou.fq_bowtie_LTR.log
read_1_1_dou.fq_bwa_indexing.log
read_1_1_dou.fq_bwa_mapping.log

# _IA.sam.hit and _LTR.sam.hit 
are lists of read names matching IA and LTR sequences, respectively.
Format: <read_name>
Example line: 
HWI-ST132_0429:7:1:1249:1913#AAGGCC/1

# *.ltr_ia.both.list 
is a list of read names of IA-LTR read pairs.
Format: <read_name>
Example line: 
HWI-ST132_0429:7:1:14775:3142#AAGGCC

# *.annot.fq 
contains fastq format files with added IA and LTR designations
Example lines:
@IA-ST132_0429:7:1:1249:1913#AAGGCC/1
TAGTGGCTGAATGAGACTGGTGTCGACACTAGTGGTAAGGGAAAATTTACATTTGGATCAGACTTCAGATNNNATCATGAAACAAGACAGTATTTTGTTA
+IA-ST132_0429:7:1:1249:1913#AAGGCC/1
aefffffffdfffeeadd__addeefbff_dZ`ddUZaBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

# *.annot.ltr_only.fq 
same as .annot.fq files but exclusively with ltr matching sequences.

# *.both.barcoded.sam.hit lists all LTR matching reads from IA-LTR pairs and their 2D pool coordinates.
Format: <read_name>	<2D_pool_coordinate>
Example line: 5_LTR-ST132_0429:7:1:1259:1957#AAGGCC/1	R_75

# *.annot.barcoded.fq 
are fastq format files with added IA, LTR and 2D pool coordinate designations
Example lines:
@5_LTR-ST132_0429:7:1:1259:1957#R_75/1
CGTTAAGCCAACTAACAACCCAGCTAACAGTAGACATCTGTAACATTTTCAGAATACAATTGAGACGGCGCCTTTGAAGAGATATAGAAAGAAGAAGGAT
+5_LTR-ST132_0429:7:1:1259:1957#R_75/1
fffffeefffggggggggggggfggggefedfafffdeadc_d]Sc[bccedcdXb_aeab\]bbBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

# *.annot.barcoded_trimmed.fq 
are fastq format files with added with added IA, LTR and 2D pool coordinate designations and trimmed for all non-genomic sequences.
Example lines:
@5_LTR-ST132_0429:7:1:1259:1957#R_75/1
ATTTTCAGAATACAATTGAGACGGCGCCTTTGAAGAGATATAGAAAGAAGAAGGA
+5_LTR-ST132_0429:7:1:1259:1957#R_75/1
Sc[bccedcdXb_aeab\]bbBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

# *_mapped.sam
is a SAM format file containing information about all mapped LTR-IA reads.
Example line:
5_LTR-ST132_0429:7:1:1259:1957#R_75	99	chr1	29429214	50	55M	=	29429299	150	ATTTTCAGAATACAATTGAGACGGCGCCTTTGAAGAGATATAGAAAGAAGAAGGA	4D<CDDFEDE9C@BFBC=>CC##################################	XT:A:U	NM:i:3	SM:i:25	AM:i:25	X0:i:1	X1:i:0	XM:i:3	XO:i:0	XG:i:0	MD:Z:22T2A10A18

# *_mapped.sam.hit
is an excerpt from the _mapped.sam file listing insertion positions as the first base upstream of the 5'LTR
Format: <chr>	<insert_position> <strand>	<2D_pool_coordinate>
Example line: chr1	29429215	R	R_75

# *_position_abundance_summary.txt
shows read abundances for all coordinates detected at a genomic position
Format: <chr> <position> <strand>	<coordinate_1> <read_abundance_coordinate_1> <coordinate_2> <read_abundance_coordinate_2> ... <coordinate_n> <read_abundance_coordinate_n>
Example line: chr1	864862	R	C_54	1	R_40	2


## FSTpoolit_filter output

# *_position_coordinate_summary.txt 
shows coordinate read abundances, which passed the abundance cutoff filter
Format: <chr> <position> <strand>	<#_separated_coordinates> <#_separated_abundances> <total_read_abundance>
Example line: chr2	27103777	F	C_1#C_49#R_1	45#11#73	129

# *_position_coordinate_summary_noendo.txt 
shows coordinate read abundances, which passed the abundance cutoff and original element filters
Format: same as *_position_coordinate_summary.txt

# *_position_coordinate_summary_noendo_nodupl.txt 
shows coordinate read abundances, which passed the abundance cutoff, original element, and mismapping filters
Format: same as *_position_coordinate_summary.txt

# *_insertion_table.txt
shows passed filter insertion calls in individual planta
Format: <column coordinate>	<row_coordinate>	<chr> <position> <strand> <#_separated_coordinates> <#_separated_abundances> <total_read_abundance> <plant_id>
Example line: C_1	R_1	chr2	27103777	F	C_1#C_49#R_1	45#11#73	129	DK-01_30000001


## FSTpoolit_primer output

#*_primer3_input.txt
holds insertion flanking region sequences for genotyping primer design
Example:
SEQUENCE_ID=chr0_6664661_F
SEQUENCE_TEMPLATE=TAGCATATA[sequence_truncated]CTACAGCA
SEQUENCE_TARGET=900,300
=

# *_primers.txt
Format: <batch>	<chr>	<positon>	<strand>	<sequence_+-1000bp_from_ insertion> <forward_primer_sequence>	<reverse_primer_sequence>	<positive_product_lenght>	<no_insertion_product_lenght>
Example line: DK01	chr2	1105769	F	GTAAGCATTACAATA[sequence_truncated]AGGATCAGAAAATGGATGGATCC	GGTTGCGCCGGAAACCACTCTTCT	TTCCCCAACCTTGCACTCATCGCT	520	669


