Lab 2: Public Data Retrieval

Connect and start an interactive session

# Connect to Ibex
ssh username@ilogin.ibex.kaust.edu.sa

# Start an interactive session (never run analyses on login nodes!)
srun --cpus-per-task=4 --mem=32G --time=04:00:00 --pty /bin/bash

Load required modules

# Load bioinformatics tools
module load sratoolkit
module load seqkit

# Verify tools are loaded
module list

Navigate to your workspace

# Navigate to your scratch directory
cd /ibex/user/$USER

# Create directories for this lab
mkdir workshop
cd workshop

# Create directories for this lab
mkdir -p raw_data references

# Verify
ls -la

Step 1: Navigate to NCBI GEO

NCBI GEO is a public repository for gene expression and genomics datasets.

1. Open your web browser and go to https://www.ncbi.nlm.nih.gov/geo/
2. You will see the GEO homepage with a search box at the top

Step 2: Search for a Dataset

As an example, search for accession GSE136366 (a TDP-43 RNA-seq study):

1. In the search box, type: GSE136366
2. Click "Search" or press Enter
3. Click on the accession number GSE136366 to view the full record

Step 3: Explore the GEO Series Page

The GEO Series page contains important information about the dataset:

• Title: Full study title and description
• Summary: Experimental design and objectives
• Overall design: Sample information and conditions
• Samples: List of all individual samples (GSM accessions)
• Citation: Link to the published paper

Step 4: Navigate to SRA from GEO

To access the raw FASTQ files, you need to go to the SRA:

1. Scroll down to the "Relations" section on the GEO page
2. Look for the link labeled "SRA" and click on it
3. Alternatively, click on any individual sample (GSM accession) to find its SRA run

Step 5: SRA Run Selector

The SRA Run Selector provides metadata about each sequencing run:

1. Click "Send results to Run selector" if not already there
2. The table shows important metadata columns:

   Column	Description
   Run	SRR accession number (used for download)
   Sample	Sample name from the study
   LibraryLayout	SINGLE or PAIRED (single-end vs paired-end)
   Bases	Total sequencing bases
   Spots	Number of reads (or read pairs)
   Platform	Sequencing platform (e.g., ILLUMINA)

Course Dataset: GSE136366

For this course, we use GSE136366 — an RNA-seq study investigating the role of TDP-43 (TARDBP) in gene regulation in human cells:

Navigate to your raw data directory

cd /ibex/user/$USER/workshop/raw_data

Option A: Copy from the shared course directory (recommended)

The full GSE136366 FASTQs are already available on Ibex at a shared course path. Files are already named with the KO_1_/WT_1_ prefix convention — no download or renaming needed just make a link to the files.

ln -s /biocorelab/BIX/resources/datasets/rnaseq/GSE136366/raw_fastq/*.fastq.gz /ibex/user/$USER/workshop/raw_data/

ls -lh /ibex/user/$USER/workshop/raw_data/

# Download all runs for GSE136366 in one command (takes too long)
prefetch GSE136366

# Convert each downloaded .sra file to paired FASTQ
for sra in ./*/*.sra; do
    echo "Converting $sra ..."
    fasterq-dump --split-files --threads 4 "$sra"
done

#!/bin/bash
#SBATCH --job-name=sra_download       # Name shown in squeue
#SBATCH --ntasks=1                    # Single task (not MPI)
#SBATCH --cpus-per-task=8             # 8 threads for fasterq-dump
#SBATCH --mem=16G                     # RAM — fasterq-dump needs ~4G per thread
#SBATCH --time=12:00:00               # Up to 12 hours for all samples
#SBATCH --partition=batch             # General compute queue on Ibex
#SBATCH --output=logs/sra_download_%j.out   # stdout log; %j = job ID
#SBATCH --error=logs/sra_download_%j.err    # stderr log
#SBATCH --mail-type=END,FAIL          # Email when done or if it fails
#SBATCH --mail-user=your.email@kaust.edu.sa # Replace with your email

# ---------- environment ----------
cd /ibex/user/$USER/workshop/raw_data

module load sratoolkit

mkdir -p logs

# ---------- step 1: prefetch (download all runs for GSE136366) ----------
echo "Starting prefetch ..."
prefetch GSE136366

# ---------- step 2: convert .sra to paired FASTQ ----------
echo "Starting fasterq-dump ..."
for sra in GSE136366/**/*.sra; do
    echo "  Converting $sra ..."
    fasterq-dump --split-files --threads 8 "$sra"
done
gzip *.fastq

# ---------- step 3: rename to add condition/replicate prefix ----------
echo "Renaming files to match shared folder convention ..."
mv SRR10045016_1.fastq.gz KO_1_SRR10045016_1.fastq.gz
mv SRR10045016_2.fastq.gz KO_1_SRR10045016_2.fastq.gz
mv SRR10045017_1.fastq.gz KO_2_SRR10045017_1.fastq.gz
mv SRR10045017_2.fastq.gz KO_2_SRR10045017_2.fastq.gz
mv SRR10045018_1.fastq.gz KO_3_SRR10045018_1.fastq.gz
mv SRR10045018_2.fastq.gz KO_3_SRR10045018_2.fastq.gz
mv SRR10045019_1.fastq.gz WT_1_SRR10045019_1.fastq.gz
mv SRR10045019_2.fastq.gz WT_1_SRR10045019_2.fastq.gz
mv SRR10045020_1.fastq.gz WT_2_SRR10045020_1.fastq.gz
mv SRR10045020_2.fastq.gz WT_2_SRR10045020_2.fastq.gz
mv SRR10045021_1.fastq.gz WT_3_SRR10045021_1.fastq.gz
mv SRR10045021_2.fastq.gz WT_3_SRR10045021_2.fastq.gz

echo "All downloads complete."
ls -lh *.fastq.gz

# Submit to SLURM
sbatch /ibex/user/$USER/workshop/raw_data/download_sra.sh

# Check job status (replace JOBID with the number printed by sbatch)
squeue -u $USER

# Watch live stdout while job runs
tail -f /ibex/user/$USER/workshop/raw_data/logs/sra_download_JOBID.out

# Cancel a job if needed
scancel JOBID

Navigate to your references directory

cd /ibex/user/$USER/workshop/references

Get the NCBI datasets tool on Ibex

#load NCBI datasets tool
module load ncbi_datasets_tools

# Check the help message
datasets --help

Search for a genome assembly

You can search for genomes by organism name or assembly accession:

# Search for human genome assemblies
datasets summary genome taxon "Homo sapiens" --reference | head -5

# Search by specific assembly accession (GRCh38 / hg38)
datasets summary genome accession GCF_000001405.40

Download genome and annotation

The datasets download command fetches the genome FASTA, annotation (GTF/GFF3), and metadata in a single zip archive:

# Download human GRCh38 reference genome with annotations
# Note: This is a large download (~3 GB)
datasets download genome accession GCF_000001405.40 \
    --include genome,gtf,seq-report

# This creates a file called ncbi_dataset.zip
ls -lh ncbi_dataset.zip

Extract and organize the downloaded files

# Extract the zip archive
unzip ncbi_dataset.zip

# See the directory structure
ls ncbi_dataset/data/GCF_000001405.40/

# The extracted files will include:
#   GCF_000001405.40_GRCh38.p14_genomic.fna  - genome FASTA (chromosomes)
#   genomic.gtf                               - gene annotation (GTF)
#   sequence_report.jsonl                     - sequence metadata

# Copy files to your references directory with clear names
cp ncbi_dataset/data/GCF_000001405.40/*.fna GRCh38_genome.fa
cp ncbi_dataset/data/GCF_000001405.40/genomic.gtf GRCh38_genes.gtf

# Verify the genome (should report ~3 GB sequence, 25 chromosomes)
seqkit stats GRCh38_genome.fa

# Clean up the zip and extracted directory
rm -r ncbi_dataset ncbi_dataset.zip

Understanding the Ensembl FTP Structure

The Ensembl FTP site is organized as follows:

ftp.ensemblgenomes.org/pub/bacteria/
├── release-57/
│   ├── fasta/                    # Sequence files (genome, cDNA, protein)
│   │   └── bacteria_0_collection/
│   │       └── escherichia_coli_.../
│   │           └── dna/          # Genomic DNA sequences
│   ├── gff3/                     # Gene annotations (GFF3 format)
│   │   └── bacteria_0_collection/
│   └── gtf/                      # Gene annotations (GTF format)
└── current/                      # Symlink to latest release

Download human GRCh38 reference genome from Ensembl

# Download human GRCh38 primary assembly (chromosome sequences only — smaller than full assembly)
wget https://ftp.ensembl.org/pub/release-110/fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna.primary_assembly.fa.gz

Download gene annotation

# Download GTF annotation (Ensembl release 110, GRCh38)
wget https://ftp.ensembl.org/pub/release-110/gtf/homo_sapiens/Homo_sapiens.GRCh38.110.gtf.gz

Decompress and verify

# Decompress all downloaded files
gunzip *.gz

# Check reference genome statistics (human GRCh38 primary assembly: ~3 GB, 25 sequences)
seqkit stats Homo_sapiens.GRCh38.dna.primary_assembly.fa

# Rename for convenience
mv Homo_sapiens.GRCh38.dna.primary_assembly.fa GRCh38_genome.fa
mv Homo_sapiens.GRCh38.110.gtf GRCh38_genes.gtf

# List files with sizes
ls -lh

Create the full directory structure

mkdir -p /ibex/user/$USER/workshop/{raw_data,trimmed,aligned,counts,results,references,logs}

ls /ibex/user/$USER/workshop/

Inspect raw reads

cd /ibex/user/$USER/workshop

# Quick look at the first few reads from one GSE136366 sample (12 lines = 3 reads)
zcat raw_data/$(head -1 raw_data/SRR_Acc_List.txt)_1.fastq.gz | head -12

Reference genome statistics

# Get stats for the human GRCh38 reference genome
seqkit stats references/GRCh38_genome.fa

Preview the annotation file

# View the first 30 lines of the GTF annotation (skip comment lines)
grep -v "^#" references/GRCh38_genes.gtf | head -10

# Count annotated genes
grep -P "\tgene\t" references/GRCh38_genes.gtf | wc -l