Practical Bioinformatics in 30 Days

From zero to bioinformatician — a structured journey through modern bioinformatics.

Who This Book Is For

This book is for anyone who wants to analyze biological data but does not know where to start. You might be:

• A biologist learning to code. You have lab experience, you understand PCR and gel electrophoresis, but when someone hands you a FASTQ file with 40 million reads, you freeze. You have tried Python tutorials, but they teach you web development when you need sequence analysis. This book teaches you programming through biology, not the other way around.

• A developer learning biology. You can write code in Python, R, or JavaScript, but you do not know a codon from a contig. You have heard that bioinformatics pays well and that genomics is the future, but the terminology is impenetrable. This book teaches you the biology alongside the code, so you understand why you are computing GC content, not just how.

• A student starting a bioinformatics program. Your coursework assumes you already know both biology and programming. You need a structured on-ramp that builds both skills simultaneously. This book gives you that foundation in 30 days.

• A researcher who needs to analyze their own data. You have been sending your sequencing data to a core facility and waiting weeks for results. You want to run your own analyses — quality control, variant calling, differential expression — without becoming a full-time software engineer. This book gets you there.

No matter which category you fall into, you share one thing: you want practical skills, not theory for its own sake. Every day in this book produces something you can use.

Your Path Through Week 1

Week 1 is designed so every reader gets the foundation they need, regardless of background. Here is which days to prioritize:

Complete beginner? That is completely fine. Day 3 teaches all the biology you need (no science background assumed), and Day 4 teaches all the coding you need (no programming experience assumed). By the end of Week 1, you will be on equal footing with everyone else.

What You Will Learn

Over 30 days, you will go from knowing nothing about bioinformatics to being able to:

• Read and write every major bioinformatics file format (FASTA, FASTQ, VCF, BED, GFF, SAM/BAM)
• Perform quality control on sequencing data
• Search biological databases programmatically (NCBI, Ensembl, UniProt, KEGG)
• Analyze gene expression data from RNA-seq experiments
• Call and interpret genetic variants
• Build publication-quality visualizations
• Write reproducible analysis pipelines
• Process datasets too large to fit in memory using streaming
• Use AI to assist your analysis
• Complete three capstone projects that mirror real research scenarios

You will learn all of this in BioLang, a language designed specifically for bioinformatics. But you will not be locked in. Every day includes comparison examples in Python and R, so you can see how the same task looks in all three languages and choose the right tool for your own work.

How This Book Is Structured

The book is organized into four weeks plus capstone projects:

Each day follows the same structure:

1. The Problem — a motivating scenario that shows why you need today’s skill
2. Core concepts — the biology and programming ideas, explained together
3. Hands-on examples — working code you type and run
4. Multi-language comparison — the same task in BioLang, Python, and R
5. Exercises — practice problems to cement understanding
6. Key Takeaways — the essential points to remember

Days are designed to take 1-3 hours each. Some days are shorter (Day 1 is mostly reading), while project days are longer. You do not have to finish a day in one sitting. Work at your own pace.

Prerequisites

You need:

• A computer running Windows, macOS, or Linux
• Basic computer literacy — you can open a terminal, navigate directories, and edit text files
• Curiosity — that is genuinely it

You do not need:

• Prior programming experience (Day 2 and Day 4 teach you from scratch)
• A biology degree (Day 1 and Day 3 cover the essential biology)
• Expensive software (everything in this book is free and open-source)
• A powerful machine (a laptop with 4 GB of RAM is sufficient for all exercises)

If you can open a terminal and type a command, you are ready.

The Companion Files

Every day in this book has a companion directory with runnable code. The structure looks like this:

practical-bioinformatics/
  days/
    day-01/
      init.bl           # Setup script — run this first
      scripts/
        exercise1.bl    # BioLang solutions
        exercise2.bl
        compare.py      # Python equivalent
        compare.R       # R equivalent
      expected/
        output1.txt     # Expected output for verification
        output2.txt
      compare.md        # Side-by-side language comparison
    day-02/
      ...

To use the companion files:

1. Run init.bl first. Each day’s init script downloads sample data, creates test files, or sets up whatever that day’s exercises need. Run it with bl run init.bl.

2. Work through the exercises. Try to solve them yourself before looking at the solutions in scripts/.

3. Check your output. Compare your results against the files in expected/ to verify correctness.

4. Read compare.md. After completing a day in BioLang, read the comparison document to see how the same tasks look in Python and R. This is especially valuable if you already know one of those languages.

To get the companion files:

git clone https://github.com/bioras/practical-bioinformatics.git
cd practical-bioinformatics

Or download the ZIP from the book’s website and extract it.

Setting Up Your Environment

Full installation instructions are in Appendix A, but here is the short version:

# Install BioLang
curl -sSf https://biolang.org/install.sh | sh

# Verify it works
bl --version

# Launch the REPL
bl repl

On Windows, use the PowerShell installer:

irm https://biolang.org/install.ps1 | iex

If you want to run the Python and R comparison scripts (optional but recommended), you will also need Python 3.8+ and R 4.0+. See Appendix A for details.

The BioLang Philosophy

BioLang was designed around three principles that make it different from general-purpose languages:

1. Biology is first-class. DNA, RNA, and protein sequences are native types, not strings you have to wrap in objects. When you write dna"ATGCGATCG", BioLang knows it is DNA and gives you biological operations — complement, reverse complement, translation, GC content — without importing anything.

2. Pipes make data flow visible. In BioLang, you chain operations with the pipe operator |>. Data flows left to right, just like reading English. No nested function calls, no temporary variables, no losing track of what feeds into what.

3. Conciseness without crypticness. BioLang aims for the shortest correct code, but never at the expense of readability. Function names say what they do: gc_content, reverse_complement, find_motif. You should be able to read BioLang code aloud and have it make sense.

A Quick Taste

Here is what BioLang looks like in practice. This script reads a FASTQ file, filters for high-quality reads, and reports basic statistics:

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

let reads = read_fastq("data/reads.fastq")

reads
  |> filter(|r| r.quality >= 30)
  |> map(|r| gc_content(r.sequence))
  |> mean()
  |> println("Mean GC content of high-quality reads: {}")

Five lines. No imports. No boilerplate. The pipe operator makes it clear what happens at each step: read the file, filter by quality, extract GC content, compute the mean, print the result.

Here is another example — searching NCBI for a gene and analyzing its sequence:

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

let gene = ncbi_gene("BRCA1", "human")
let seq = ncbi_sequence(gene.id)

seq
  |> kmers(21)
  |> filter(|k| gc_content(k) > 0.6)
  |> len()
  |> println("High-GC 21-mers in BRCA1: {}")

You will understand every line of this by Day 9. For now, just notice how naturally the code reads: get the gene, get its sequence, break it into 21-mers, keep the GC-rich ones, count them, print.

Week-by-Week Overview

Week 1: Foundations (Days 1-5)

You start with the big picture. What is bioinformatics? Why does it matter? Then you learn BioLang itself — variables, types, pipes, functions. Day 3 is a biology crash course for developers. Day 4 is a coding crash course for biologists. Day 5 covers data structures: lists, records, and tables. By Friday, everyone is on the same page regardless of background.

Week 2: Core Skills (Days 6-12)

Now the real work begins. You learn to read FASTA and FASTQ files, understand quality scores, and process data too large for memory. You explore biological databases, master tables (the workhorse of bioinformatics), compare sequences, and find variants in genomes. These are the skills you will use every day as a bioinformatician.

Week 3: Applied Analysis (Days 13-20)

You apply your skills to real research problems. Gene expression analysis with RNA-seq. Statistical testing. Publication-quality plots. Pathway enrichment. Protein structure. Genomic intervals and coordinate systems. Biological visualization. Multi-species comparative analysis. Each day tackles a different domain of bioinformatics.

Week 4: Professional Skills (Days 21-27)

You learn to work like a professional. Parallel processing for speed. Reproducible pipelines. Batch processing at scale. Programmatic database queries. Robust error handling. AI-assisted analysis. Building your own tools and plugins. These are the skills that separate a script-writer from a bioinformatician.

Capstone Projects (Days 28-30)

Three full projects that integrate everything you have learned. Day 28: build a clinical variant interpretation report from whole-exome sequencing data. Day 29: conduct a complete RNA-seq differential expression study. Day 30: perform a multi-species gene family analysis with phylogenetics. Each project mirrors real research workflows.

Learning Path

The following diagram shows how the days build on each other. Each week’s skills feed into the next, culminating in the capstone projects.

Conventions Used in This Book

Throughout this book, you will see several recurring elements:

Code Blocks

BioLang code appears in fenced code blocks:

let seq = dna"ATGCGATCG"
gc_content(seq)

When a code block shows REPL interaction, lines starting with bl> are what you type, and the lines below are the output:

bl> gc_content(dna"ATGCGATCG")
0.5556

Shell commands use bash syntax:

bl run my_script.bl

Python and R Comparisons

Multi-language comparisons appear with labeled blocks:

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

BioLang:

read_fasta("data/sequences.fasta") |> filter(|s| len(s.sequence) > 1000)

Python:

from Bio import SeqIO
[r for r in SeqIO.parse("genes.fa", "fasta") if len(r.seq) > 1000]

R:

library(Biostrings)
seqs <- readDNAStringSet("genes.fa")
seqs[width(seqs) > 1000]

Exercises

Each day ends with exercises labeled by difficulty:

Exercise 1: Sequence Length — Write a script that reads a FASTA file and prints the length of each sequence.

Key Takeaways

Each day concludes with a bulleted list of the most important points:

• Takeaway in bold. Explanation follows in regular text.

Callout Boxes

Important notes, warnings, and tips appear as blockquotes:

Note: NCBI rate-limits unauthenticated requests to 3 per second. Set NCBI_API_KEY to increase this to 10 per second.

Warning: Streaming operations consume the stream. Once you iterate through a stream, it is exhausted and cannot be reused.

A Note on the Multi-Language Approach

This book uses BioLang as its primary language, but it is not a BioLang advocacy book. It is a bioinformatics book. The concepts — GC content, quality filtering, differential expression, variant calling — are universal. They do not change because you switch languages.

We include Python and R comparisons for two reasons:

1. Translation. If you already know Python or R, seeing the BioLang equivalent helps you learn faster. If you learn BioLang first, seeing the Python and R equivalents prepares you for the real world where those languages dominate.

2. Perspective. Different languages make different tradeoffs. BioLang is concise for biology but young. Python has the largest ecosystem. R has the best statistics libraries. Seeing all three helps you appreciate what each brings to the table.

The compare.md file in each day’s companion directory provides a detailed side-by-side comparison. The compare.py and compare.R scripts are runnable equivalents you can execute and compare output.

Let’s Begin

You have everything you need. The next 30 days will transform how you think about biological data. Day 1 starts with the fundamental question: what is bioinformatics, and why does it matter?

Turn the page. Your journey starts now.

Day 1: What Is Bioinformatics?

The Problem

A patient walks into a clinic. Their tumor is sequenced. Three billion base pairs of data arrive on a hard drive. Somewhere in there is the mutation driving their cancer. How do you find it?

You cannot read three billion letters by hand. You cannot compare them against a reference genome by eye. You cannot search for patterns across thousands of patients using a spreadsheet. Biology has become a data science, and the data is enormous.

This is why bioinformatics exists.

What Is Bioinformatics?

Bioinformatics sits at the intersection of three fields: biology, computer science, and statistics. But it is more than just “biology plus computers.” It is the discipline of asking biological questions and answering them with data. When a researcher wants to know which genes are active in a tumor, when a clinician needs to identify a drug-resistant mutation, when an ecologist traces the evolutionary history of a species — that is bioinformatics.

The field was born out of necessity. In 1977, Frederick Sanger published the first complete DNA genome sequence — a bacteriophage with 5,386 base pairs. That was manageable by hand. By 2003, the Human Genome Project had sequenced 3.2 billion base pairs at a cost of $2.7 billion. Today, a single Illumina NovaSeq run produces over 6 terabytes of raw data in less than two days. The cost of sequencing a human genome has dropped below $200. The bottleneck is no longer generating data — it is making sense of it.

Every year, the gap between data generation and data analysis widens. Modern sequencing machines produce data faster than biologists can analyze it. This is where you come in. Whether you are a developer learning biology or a biologist learning to code, bioinformatics needs both perspectives. The biology tells you what questions to ask. The code tells you how to answer them.

The Central Dogma of Molecular Biology

Before you can analyze biological data, you need to understand what that data represents. The central dogma describes how genetic information flows in living cells:

Let’s break this down:

DNA — The Double Helix

DNA is the blueprint. It is a long molecule made of four chemical bases: Adenine, Thymine, Cytosine, and Guanine. Your entire genome — all the instructions to build and run your body — is written in these four letters. The human genome is about 3.2 billion base pairs long, organized into 23 pairs of chromosomes.

DNA has a unique structure: two strands wound around each other in a double helix, connected by base pairs. A always pairs with T (2 hydrogen bonds), and C always pairs with G (3 hydrogen bonds — making CG pairs stronger):

Each DNA strand has a direction, like a one-way street. Every nucleotide has a sugar with numbered carbon atoms. The 5’ (five-prime) carbon connects to the next nucleotide’s 3’ (three-prime) carbon via a phosphate bond — so the strand has a built-in direction: 5’→3’. Both strands are built the same way, but they run in opposite directions (called antiparallel):

5'──A──T──G──C──G──3'   ← coding strand (read left to right)
    |  |  |  |  |
3'──T──A──C──G──C──5'   ← template strand (runs the other way)

The base pairing (A-T, C-G) holds the two strands together, but notice the 5’ and 3’ ends are flipped. This antiparallel arrangement is why enzymes like RNA polymerase can only read in one direction (3’→5’ on the template, producing mRNA in 5’→3’).

When we write a DNA sequence like ATGCGATCG, we mean the coding strand read 5’→3’ — this is the universal convention in biology and bioinformatics. The other strand is implied — you can always reconstruct it using the base pairing rules.

RNA — The Single-Stranded Messenger

RNA is the working copy. When a cell needs to use a gene, it copies that region of DNA into RNA through a process called transcription. Remember that DNA has two strands. The cell’s RNA polymerase reads the template strand (also called the antisense strand) and builds a complementary RNA. The resulting mRNA sequence ends up matching the coding strand (the other strand, also called the sense strand) — except RNA uses Uracil (U) instead of Thymine (T). So in practice, every T in the coding strand becomes U in the mRNA: ATGCG in DNA becomes AUGCG in RNA.

Why bioinformatics uses the coding strand: When databases like NCBI store a gene sequence, they store the coding strand (5’→3’). To get the mRNA, just replace T with U. You rarely need to think about the template strand directly.

Unlike DNA’s stable double helix, RNA is single-stranded — it is a temporary copy meant to be read and then degraded:

There are several types of RNA, but the one most relevant to the central dogma is mRNA (messenger RNA) — the copy that carries gene instructions to the ribosome for protein synthesis.

Protein — The Folded Machine

Protein is the machine. Proteins do most of the work in cells — they catalyze reactions, transport molecules, provide structure, and signal between cells. The RNA sequence is read three letters at a time (called codons), and each codon maps to one of 20 amino acids. This process is called translation. For example, the codon AUG always codes for Methionine (abbreviated M) and also serves as the “start” signal.

A protein starts as a linear chain of amino acids, but it immediately folds into a specific 3D shape. This shape determines its function — and is why mutations can be so devastating:

The key insight: sequence determines structure determines function. Change one amino acid (via a DNA mutation) and the entire fold can collapse. This is why the TP53 R175H mutation causes cancer — swapping Arginine for Histidine at position 175 disrupts the DNA-binding domain, and p53 can no longer activate tumor suppression genes.

Why Proteins Are Essential

Proteins are not optional extras — they are what makes life work. Every function your body performs depends on specific proteins doing their jobs correctly:

With working proteins — your body functions:

Without working proteins — disease happens:

This is why a single mutation in a gene can cause devastating disease. The mutation changes the DNA, which changes the RNA, which changes the protein’s amino acid sequence, which can alter its 3D shape, which can destroy its function. One wrong letter out of billions — and the protein misfolds, or never gets made, or loses its ability to do its job.

“But I Eat Protein Every Day — Why Can’t I Just Use That?”

You have heard it your whole life: “Eat protein — eggs, chicken, lentils, fish.” So a natural question is: if proteins are so essential, why does the body need to manufacture them from DNA instructions? Why not just use the protein from food directly?

The answer is that dietary protein and your body’s proteins are completely different things. When you eat a chicken breast, you are eating chicken muscle proteins — myosin, actin, troponin — proteins designed to make a chicken’s wing move. Your body cannot use chicken myosin as-is. It is the wrong shape, the wrong size, the wrong function.

Here is what actually happens:

Think of it like this: eating a wooden chair does not give you furniture. But if you break that chair down into individual planks and nails, you can use those raw materials to build something completely different — a bookshelf, a table, whatever your blueprint calls for.

Food protein = raw materials (amino acids). Your DNA = the blueprints. Your ribosomes = the factory. The 20 amino acids are like 20 types of LEGO bricks — the same bricks can build completely different structures depending on the instructions. (You will find the complete table of all 20 amino acids with their single-letter codes and properties in Day 3.)

This is why the central dogma matters so profoundly:

Your body contains roughly 20,000 different proteins, each encoded by its own gene, each with a unique amino acid sequence and 3D structure. You cannot get these from food. You can only get the raw building blocks (amino acids) from food, and then your cells assemble them according to the instructions in your DNA.

This is also why protein deficiency is so dangerous — without enough amino acids from food, your cells cannot build the proteins your DNA encodes. And it is why genetic mutations are so consequential — even with perfect nutrition, a mutated gene produces a misfolded or missing protein that no amount of food can fix.

The central dogma is not an abstract concept — it is the reason your body works, and the reason disease happens when it goes wrong. Understanding this chain (DNA encodes RNA, RNA builds protein, protein does the work) is essential for everything in bioinformatics. When we analyze variants, we are asking: “Does this DNA change affect the protein?” When we measure gene expression, we are asking: “How much of this protein is the cell making?” Every analysis connects back to this fundamental flow.

Genes, Genomes, and Chromosomes

Now that you understand the molecules (DNA, RNA, Protein), let’s define the structures that organize them.

Genome — The Complete Instruction Manual

A genome is the complete set of DNA in an organism — every instruction needed to build and run that organism. Think of it as the entire hard drive, not a single file.

Notice something surprising: genome size does not correlate well with organism complexity. Wheat has 5x more DNA than humans. The difference lies not in how much DNA you have, but in how it is organized and regulated.

Chromosomes — The Volumes

Chromosomes are the physical units that DNA is packaged into. If the genome is an encyclopedia, chromosomes are the individual volumes. Humans have 23 pairs (46 total) — one set from each parent. Each chromosome is a single, very long DNA molecule wrapped tightly around proteins called histones.

Human Genome (3.2 billion base pairs)
├── Chromosome 1   (249 million bp)   ← largest
├── Chromosome 2   (242 million bp)
├── ...
├── Chromosome 17  (83 million bp)    ← home of TP53 and BRCA1
├── ...
├── Chromosome 22  (51 million bp)    ← smallest autosome
├── Chromosome X   (156 million bp)
└── Chromosome Y   (57 million bp)

When we say a gene is “on chromosome 17”, we mean its DNA sequence is part of that specific chromosome’s molecule.

Genes — The Individual Instructions

A gene is a specific region of DNA that contains the instructions for building one protein (or sometimes a functional RNA molecule). If the genome is the encyclopedia and chromosomes are volumes, genes are individual articles.

Key facts about genes:

• The human genome has roughly 20,000 protein-coding genes
• Genes make up only about 1.5% of total human DNA
• The rest includes regulatory sequences (promoters, enhancers), structural elements, and regions still being characterized
• A gene is not just one continuous stretch — it contains exons (coding parts) interrupted by introns (non-coding parts that get spliced out)
• The same gene can produce multiple different proteins through alternative splicing

A gene is like a recipe in a cookbook:
- The cookbook = genome
- The chapter = chromosome
- The recipe = gene
- The ingredients list = exons (the parts that matter)
- The chef's notes = introns (removed before cooking)
- The finished dish = protein

Some landmark genes you will encounter throughout this book:

Why Data?

Here is the scale problem that makes bioinformatics necessary:

• A single human genome: ~3 GB of text (just the bases, no metadata)
• A typical whole-genome sequencing run: 100-500 GB of raw data (because each position is read multiple times for accuracy)
• NCBI GenBank (the world’s public sequence archive): over 10 trillion nucleotide bases
• The Sequence Read Archive: over 80 petabytes of raw sequencing data

You cannot do this by hand. You need code.

This is not an exaggeration. Before computational tools existed, identifying a single disease gene could take a decade of work by large teams. Today, clinical sequencing pipelines identify candidate variants in hours. The biology has not changed. The tools have.

Your First Bioinformatics

Try it right now — no installation needed! You can run all the code examples in this chapter directly in your browser at lang.bio/playground. The online playground is perfect for the exercises in Days 1 through 5. For later chapters that work with files (FASTQ, VCF, CSV), you will need the local bl installation — see Appendix A for setup instructions.

Let’s write some code. BioLang treats DNA, RNA, and protein sequences as first-class types — not strings, but biological objects that understand what they are.

Creating a DNA sequence

# Your first DNA sequence
let seq = dna"ATGCGATCGATCGATCG"
println(f"Sequence: {seq}")
println(f"Length: {len(seq)} bases")
println(f"Type: {type(seq)}")

# Output:
# Sequence: DNA(ATGCGATCGATCGATCG)
# Length: 17
# Type: DNA

That dna"..." is a sequence literal. BioLang knows this is DNA, not a random string. It will enforce that only valid bases appear. Try putting a Z in there — you will get an error, because Z is not a nucleotide.

The central dogma in code

# Walk through the central dogma
let gene = dna"ATGAAACCCGGGTTTTAA"
println(f"DNA:     {gene}")

let mrna = transcribe(gene)
println(f"RNA:     {mrna}")

let protein = translate(gene)
println(f"Protein: {protein}")

# Output:
# DNA:     DNA(ATGAAACCCGGGTTTTAA)
# RNA:     RNA(AUGAAACCCGGGUUUUAA)
# Protein: Protein(MKPGF)

Six codons in that DNA sequence: ATG (Met/M), AAA (Lys/K), CCC (Pro/P), GGG (Gly/G), TTT (Phe/F), and TAA (Stop). The translate function reads until the stop codon and returns the protein sequence MKPGF. That is the central dogma — DNA to RNA to Protein — in three lines of code.

Analyzing sequence composition

# What's in this sequence?
let genome_fragment = dna"ATGCGATCGATCGAATTCGATCG"
let counts = base_counts(genome_fragment)
println(f"Base composition: {counts}")
println(f"GC content: {gc_content(genome_fragment)}")

# Output:
# Base composition: {A: 6, T: 6, G: 6, C: 5, N: 0, GC: 0.4782608695652174}
# GC content: 0.4782608695652174

Why GC content and not AT content? Since GC% + AT% = 100%, knowing one tells you the other. The convention is to report GC because it’s the biologically interesting number:

• Thermal stability — G-C base pairs form three hydrogen bonds (versus two for A-T), so GC-rich regions are harder to melt apart. This directly affects PCR primer design — you need primers with the right melting temperature.
• Gene density — GC-rich regions in the human genome tend to be gene-dense, and CpG islands (clusters of CG dinucleotides) mark promoter regions where genes start.
• Sequencing quality — Illumina sequencers have lower coverage in regions with very high or very low GC content, so checking GC distribution is a standard quality control step.
• Species fingerprint — Organisms have characteristic GC content. Plasmodium falciparum (malaria parasite) has about 19% GC, while Streptomyces bacteria can exceed 70%. If you sequence a sample and see unexpected GC content, it might indicate contamination or a novel organism.

Finding patterns in DNA

# Finding a restriction enzyme site
let seq = dna"ATCGATCGAATTCGATCGATCG"
let sites = find_motif(seq, "GAATTC")
println(f"EcoRI cuts at positions: {sites}")

# Output:
# EcoRI cuts at positions: [7]

EcoRI is a restriction enzyme — a molecular scissor that cuts DNA at a specific recognition sequence (GAATTC). These enzymes are fundamental tools in molecular biology. Before sequencing was cheap, scientists used restriction enzymes to cut genomes into fragments for analysis. Even today, they are essential for cloning, genotyping, and quality control.

Using the pipe operator

BioLang’s pipe operator |> lets you chain operations naturally — data flows left to right, just like a bench protocol:

# Chain operations with pipes
let result = dna"ATGCGATCGATCG"
    |> complement()
    |> reverse_complement()
    |> transcribe()
println(f"Result: {result}")

# Output:
# Result: RNA(AUGCGAUCGAUCG)

If you are coming from biology, think of pipes as steps in a lab protocol. If you are coming from programming, think of them as method chaining or Unix pipes. Either way, they make multi-step analyses readable.

The Bioinformatics Workflow

Every bioinformatics project — from a student homework to a clinical sequencing pipeline — follows the same general pattern:

The Eight Steps

Steps 4 through 7 are where bioinformatics lives. That is what you will learn in this book.

What You’ll Build in 30 Days

This book is structured as four weeks, each building on the last:

Week 1: Foundations (Days 1-5) — You are here. By the end of this week, you will understand the biology behind the data, be comfortable with BioLang’s syntax, and know the core data structures used in bioinformatics.

Week 2: Core Skills (Days 6-12) — Reading real sequencing data (FASTQ, BAM, VCF), working with biological databases, processing large files efficiently, and finding variants in genomes. This is the bread and butter of bioinformatics.

Week 3: Applied Analysis (Days 13-20) — Gene expression analysis, statistics, publication-quality visualization, pathway analysis, protein structure, and multi-species comparison. This is where you start doing real science.

Week 4: Professional Skills (Days 21-30) — Performance optimization, reproducible pipelines, batch processing, error handling, and three capstone projects that tie everything together: a clinical variant report, an RNA-seq study, and a multi-species gene family analysis.

By Day 30, you will be able to take raw sequencing data from a public database, process it through a quality control pipeline, identify biologically meaningful results, and produce publication-quality figures. That is not a promise about what you might achieve — it is the actual content of the capstone projects.

Exercises

Exercise 1: Sequence Composition

Create a DNA sequence of at least 20 bases and analyze its composition:

let my_seq = dna"ATGCCCAAAGGGTTTATGCCC"
let counts = base_counts(my_seq)
println(f"Counts: {counts}")
println(f"GC content: {gc_content(my_seq)}")

Is your sequence GC-rich (>50% GC) or AT-rich (<50% GC)?

Exercise 2: Central Dogma

Translate this DNA sequence and determine what protein it encodes:

let gene = dna"ATGGATCCCTAA"
println(f"DNA:     {gene}")
println(f"RNA:     {transcribe(gene)}")
println(f"Protein: {translate(gene)}")

# What amino acids are M, D, and P?
# Hint: M = Methionine, D = Aspartic acid, P = Proline

Exercise 3: Base Counting

Count the bases in this perfectly balanced sequence:

let balanced = dna"AAAAATTTTTCCCCCGGGGG"
println(f"Counts: {base_counts(balanced)}")
println(f"GC content: {gc_content(balanced)}")

# Is it GC-rich, AT-rich, or perfectly balanced?

Exercise 4: Motif Search

Find all start codons (ATG) in this sequence:

let seq = dna"ATGATGATGATG"
let starts = find_motif(seq, "ATG")
println(f"Start codons at positions: {starts}")

# How many start codons are there?
# What positions are they at?

Key Takeaways

• Bioinformatics exists because biology generates data at computational scale. Modern sequencing produces terabytes daily — no human can process that by hand.
• DNA to RNA to Protein is the central dogma — the foundation of molecular biology. DNA stores the information, RNA carries it, and proteins do the work.
• BioLang treats sequences as first-class types, not just strings. dna"ATGC" is a DNA value with biological semantics, not four arbitrary characters.
• Every bioinformatics project follows the same workflow: Question, Data, QC, Analysis, Insight. The tools change, but the pattern does not.
• Scale is the defining challenge. A single genome is 3 GB. A research project can involve thousands of genomes. Code is the only way to work at this scale.

Setting Up for the Comparison Scripts

Each day in this book includes equivalent scripts in Python and R alongside the BioLang version, so you can compare approaches. Before starting the exercises, install the required packages once:

Python (run in a terminal):

pip install biopython scipy pandas matplotlib requests openai

R (run in an R console):

install.packages(c("dplyr", "jsonlite", "httr2", "digest", "logging", "ggplot2"))
# For Bioconductor packages (optional, used in later chapters):
# if (!require("BiocManager")) install.packages("BiocManager")
# BiocManager::install(c("Biostrings", "GenomicRanges"))

Note: You do not need Python or R to follow this book — all examples work in BioLang alone. The comparison scripts are provided so you can see how the same analysis looks across languages. See Appendix A for detailed setup instructions.

What’s Next

Tomorrow, we go hands-on with BioLang itself — variables, types, pipes, functions, and the interactive REPL. You will learn the language that powers every example in this book. If today was about why bioinformatics exists, tomorrow is about how you do it.

Day 2: Your First Language — BioLang

The Problem

You have seen what bioinformatics can do. You know that DNA becomes RNA becomes protein, that genomes are billions of letters long, and that computation is the only way to make sense of this data. Now you need a tool to do it.

Every programming language makes tradeoffs. Python is general-purpose but verbose for biology — you need imports, object wrappers, and ten lines to do what should take two. R is excellent for statistics but awkward for building pipelines. Perl was the original bioinformatics language but has fallen out of favor for good reason. Each of these languages was designed for something else and then adapted for biology.

BioLang was designed for one thing: making biological data analysis as natural as describing it in English. DNA sequences are not strings you have to convert. Pipes are not a library you have to import. The language thinks about biology the way you do.

Today you will learn BioLang from scratch. By the end, you will be writing real analysis code — filtering sequences, computing statistics, and chaining operations together with a fluency that would take weeks in other languages.

Getting Started: The REPL

A REPL (Read-Eval-Print Loop) is an interactive environment where you type code, it runs immediately, and you see the result. It is the best way to learn a language because you get instant feedback.

No installation yet? You can try all the examples in this chapter at lang.bio/playground — it runs BioLang directly in your browser. Perfect for learning the basics before committing to a local install.

Launch it:

bl repl

Or simply:

bl

You will see a prompt:

bl>

Try some arithmetic:

bl> 2 + 3
5
bl> 10 * 7
70
bl> 2 ** 10
1024
bl> 17 % 5
2

Try strings:

bl> "Hello, bioinformatics!"
Hello, bioinformatics!
bl> len("ATCGATCG")
8
bl> upper("atcgatcg")
ATCGATCG

To exit the REPL, type Ctrl+D or Ctrl+C.

The REPL is your laboratory bench. Throughout this book, any time you see a new concept, try it there first. Get a feel for it. Break it. Fix it. That is how you learn.

Variables and Types

BioLang has a clean type system designed for biology. Here is how it is organized:

Declaring Variables

Use let to create a variable. BioLang infers the type automatically — you never need type annotations.

let name = "BRCA1"           # Str
let length = 81189            # Int
let gc = 0.423                # Float
let is_oncogene = false       # Bool
let seq = dna"ATGCGATCG"     # DNA

Use type() to check what type a value is:

println(type(name))     # Str
println(type(length))   # Int
println(type(gc))       # Float
println(type(seq))      # DNA

Reassignment

Once a variable exists, you can update it without let:

let count = 0
count = count + 1
println(count)          # 1

Why Bio Types Matter

In Python, DNA is just a string: "ATCG". You can accidentally concatenate it with a name, reverse it incorrectly, or pass it to a function that expects a protein. Nothing stops you.

In BioLang, dna"ATCG" is a DNA value. The language knows it is DNA. Functions like transcribe() accept DNA and return RNA. Functions like gc_content() accept DNA or RNA and return a float. If you try to transcribe a protein, you get an error — immediately, not three hours into a pipeline run.

let d = dna"ATGCGATCG"
let r = transcribe(d)         # Works: DNA -> RNA
let p = translate(r)          # Works: RNA -> Protein

# This would fail:
# let bad = transcribe(p)     # Error: transcribe requires DNA

The Pipe Operator

This is the most important concept in BioLang. If you learn one thing today, learn this.

The pipe operator |> takes the result of one expression and feeds it as the first argument to the next function. It turns nested, inside-out code into left-to-right, top-to-bottom code that reads like English.

  data  ──|>──  transform1()  ──|>──  transform2()  ──|>──  result

Without Pipes vs. With Pipes

# Without pipes (nested calls — read inside-out)
println(round(gc_content(dna"ATCGATCGATCG"), 3))

# With pipes (left to right — natural reading order)
dna"ATCGATCGATCG"
    |> gc_content()
    |> round(3)
    |> println()

Both lines produce the same result: 0.5. But the pipe version reads like a recipe: take this sequence, compute its GC content, round it, print it.

How Pipes Work

The rule is simple: a |> f(b) becomes f(a, b). The pipe inserts the left side as the first argument to the function on the right.

# These two are identical:
round(gc_content(dna"ATCG"), 3)

dna"ATCG" |> gc_content() |> round(3)

Pipes with Biology

Pipes follow the fundamental bioinformatics pattern: read, transform, summarize.

# Transcribe and translate in one pipeline
dna"ATGAAACCCGGG"
    |> transcribe()
    |> translate()
    |> println()
# Output: Protein(MKPG)

# Find start codons in a sequence
let positions = find_motif(dna"ATGATGCCGATG", "ATG")
println(f"Start codon positions: {positions}")
# Output: Start codon positions: [0, 3, 9]

println(f"Found {len(positions)} start codons")
# Output: Found 3 start codons

You will use pipes constantly. Every chapter in this book builds pipe chains. They are the backbone of BioLang.

Lists and Records

Lists — Ordered Collections

A list holds values in order. Create one with square brackets:

# Lists — ordered collections
let genes = ["BRCA1", "TP53", "EGFR", "KRAS"]
println(len(genes))           # 4
println(genes[0])             # BRCA1
println(genes[3])             # KRAS
println(genes[-1])            # KRAS (negative indices count from the end)
println(genes[-2])            # EGFR

# Lists can hold any type
let lengths = [81189, 19149, 188307, 45806]
let mixed = ["BRCA1", 81189, true, dna"ATCG"]

Useful list operations:

let nums = [3, 1, 4, 1, 5, 9]
println(first(nums))          # 3
println(last(nums))           # 9
println(sort(nums))           # [1, 1, 3, 4, 5, 9]
println(reverse(nums))        # [9, 5, 1, 4, 1, 3]
println(contains(nums, 5))    # true

Records — Key-Value Pairs

Records are collections of named fields, like a dictionary or a struct:

# Records — key-value pairs
let gene = {
    name: "TP53",
    chromosome: "17",
    length: 19149,
    is_tumor_suppressor: true
}
println(gene.name)            # TP53
println(gene.chromosome)      # 17
println(gene.length)          # 19149

Records are everywhere in bioinformatics. Every gene has a name, a location, a function. Every experiment has samples, conditions, results. Records let you group related data together naturally.

Functions

Defining Functions

Use fn to define a function:

fn gc_rich(seq) {
    gc_content(seq) > 0.6
}

let s = dna"GCGCGCGCATGC"
println(gc_rich(s))            # true

let t = dna"AAAATTTT"
println(gc_rich(t))            # false

Functions can take multiple parameters and use any logic:

fn classify_gc(seq) {
    let gc = gc_content(seq)
    if gc > 0.6 {
        "GC-rich"
    } else if gc < 0.4 {
        "AT-rich"
    } else {
        "balanced"
    }
}

println(classify_gc(dna"GCGCGCGC"))    # GC-rich
println(classify_gc(dna"ATATATATAT"))  # AT-rich
println(classify_gc(dna"ATCGATCG"))    # balanced

Lambdas (Anonymous Functions)

A lambda is a small function without a name. The syntax is |params| expression:

let double = |x| x * 2
println(double(5))             # 10

let add = |a, b| a + b
println(add(3, 7))             # 10

Lambdas are used constantly with higher-order functions (coming up next). They let you define behavior inline, right where you need it.

Control Flow

If / Else

let gc = 0.65
if gc > 0.6 {
    println("GC-rich region")
} else if gc < 0.4 {
    println("AT-rich region")
} else {
    println("Balanced composition")
}
# Output: GC-rich region

if in BioLang is also an expression — it returns a value:

let label = if gc > 0.6 { "high" } else { "normal" }
println(label)                 # high

For Loops

let codons = ["ATG", "GCT", "TAA"]
for codon in codons {
    println(f"Codon: {codon}")
}
# Output:
# Codon: ATG
# Codon: GCT
# Codon: TAA

Pattern Matching

match is like a more powerful if/else chain:

let base = "A"
match base {
    "A" | "G" => println("Purine"),
    "C" | "T" => println("Pyrimidine"),
    _ => println("Unknown"),
}
# Output: Purine

The _ is a wildcard — it matches anything. Pattern matching is especially useful for handling different cases cleanly.

Higher-Order Functions

Higher-order functions (HOFs) take a function as an argument. They are the power tools of BioLang. Once you learn map, filter, and reduce, you will rarely need explicit loops.

map — Transform Each Element

map applies a function to every element and returns a new list:

let sequences = [dna"ATCG", dna"GCGCGC", dna"ATATAT"]
let gc_values = sequences |> map(|s| gc_content(s))
println(gc_values)
# Output: [0.5, 1.0, 0.0]

filter — Keep Elements Matching a Condition

filter keeps only elements where the function returns true:

let sequences = [dna"ATCG", dna"GCGCGC", dna"ATATAT"]
let gc_rich = sequences
    |> filter(|s| gc_content(s) > 0.4)
println(gc_rich)
# Output: [DNA(ATCG), DNA(GCGCGC)]
println(len(gc_rich))
# Output: 2

each — Do Something with Each Element

each runs a function on every element for its side effects (like printing). It does not collect results:

["BRCA1", "TP53", "EGFR"]
    |> each(|g| println(f"Gene: {g}"))
# Output:
# Gene: BRCA1
# Gene: TP53
# Gene: EGFR

reduce — Combine into a Single Value

reduce combines all elements into one value by applying a function pairwise:

let sequences = [dna"ATCG", dna"GCGCGC", dna"ATATAT"]
let total_length = sequences
    |> map(|s| len(s))
    |> reduce(|a, b| a + b)
println(f"Total bases: {total_length}")
# Output: Total bases: 16

Combining HOFs with Pipes

The real power comes from chaining these together:

# Find names of all GC-rich sequences
[dna"ATCG", dna"GCGCGCGC", dna"AAAA", dna"CCGG"]
    |> filter(|s| gc_content(s) > 0.5)
    |> map(|s| f"GC={round(gc_content(s), 2)}: {s}")
    |> each(|line| println(line))
# Output:
# GC=1.0: DNA(GCGCGCGC)
# GC=0.75: DNA(CCGG)

Putting It All Together

Here is a mini-analysis that uses everything you have learned today — variables, records, pipes, functions, and HOFs:

# Analyze a set of gene fragments
let fragments = [
    {name: "exon1", seq: dna"ATGCGATCGATCG"},
    {name: "exon2", seq: dna"GCGCGCATATAT"},
    {name: "exon3", seq: dna"TTTTAAAACCCC"},
]

# Find GC-rich exons using pipes + HOFs
let gc_rich_exons = fragments
    |> filter(|f| gc_content(f.seq) > 0.5)
    |> map(|f| f.name)

println(f"GC-rich exons: {gc_rich_exons}")
# Output: GC-rich exons: [exon1]

# Summary statistics
let gc_values = fragments |> map(|f| round(gc_content(f.seq), 3))
println(f"GC contents: {gc_values}")
# Output: GC contents: [0.538, 0.5, 0.333]

println(f"Mean GC: {round(mean(gc_values), 3)}")
# Output: Mean GC: 0.457

# Classify each fragment
fn classify_gc(gc) {
    if gc > 0.6 { "GC-rich" }
    else if gc < 0.4 { "AT-rich" }
    else { "balanced" }
}

fragments |> each(|f| {
    let gc = round(gc_content(f.seq), 3)
    println(f"{f.name}: GC={gc} ({classify_gc(gc)})")
})
# Output:
# exon1: GC=0.538 (balanced)
# exon2: GC=0.5 (balanced)
# exon3: GC=0.333 (AT-rich)

This is the pattern you will use for the rest of this book: load data, transform it with pipes and HOFs, summarize the results. The data gets more complex — FASTQ files, VCF variants, gene expression tables — but the pattern stays the same.

BioLang vs Python vs R

Let’s see the same task in all three languages: given a list of DNA sequences, find the GC-rich ones and display them with their GC content.

BioLang (6 lines, 0 imports)

let seqs = [dna"ATCGATCG", dna"GCGCGCGC", dna"ATATATAT"]
seqs
    |> filter(|s| gc_content(s) > 0.5)
    |> map(|s| {seq: s, gc: round(gc_content(s), 3)})
    |> each(|r| println(f"{r.seq}: {r.gc}"))

Python (15 lines)

from Bio.Seq import Seq
from Bio.SeqUtils import gc_fraction

sequences = [Seq("ATCGATCG"), Seq("GCGCGCGC"), Seq("ATATATAT")]

gc_rich = []
for seq in sequences:
    gc = gc_fraction(seq)
    if gc > 0.5:
        gc_rich.append({"seq": str(seq), "gc": round(gc, 3)})

for item in gc_rich:
    print(f"{item['seq']}: {item['gc']}")

# Or with list comprehension (more compact but harder to read):
# [print(f"{s}: {round(gc_fraction(s),3)}") for s in sequences if gc_fraction(s)>0.5]

R (12 lines)

library(Biostrings)

sequences <- DNAStringSet(c("ATCGATCG", "GCGCGCGC", "ATATATAT"))

gc_values <- letterFrequency(sequences, letters="GC", as.prob=TRUE)
gc_rich_idx <- which(gc_values > 0.5)

gc_rich_seqs <- sequences[gc_rich_idx]
gc_rich_vals <- round(gc_values[gc_rich_idx], 3)

for (i in seq_along(gc_rich_seqs)) {
  cat(sprintf("%s: %s\n", as.character(gc_rich_seqs[i]), gc_rich_vals[i]))
}

Why the Difference Matters

BioLang is not shorter because it is a toy. It is shorter because:

• No imports: DNA, GC content, and pipes are built in
• Bio types: dna"..." is a type, not a string you convert
• Pipes: chaining reads top-to-bottom, not inside-out
• HOFs: filter, map, each replace loops

When your script is 6 lines instead of 15, you spend less time writing boilerplate and more time thinking about biology. That advantage compounds — a 200-line pipeline in BioLang would be 500 lines in Python.

Exercises

Try these in the REPL or in a .bl script file.

Exercise 1: Longest Sequence

Create a list of 5 DNA sequences of different lengths. Find the longest one using sort_by and last:

let seqs = [dna"ATG", dna"ATCGATCG", dna"ATCG", dna"AT", dna"ATCGATCGATCG"]
# Hint: sort_by takes a lambda that returns the sort key
# seqs |> sort_by(|s| len(s)) |> last() |> print()

Exercise 2: Classify Bases

Write a function classify_base(base) that uses match to return "purine" for A or G, "pyrimidine" for C or T, and "unknown" for anything else:

# fn classify_base(base) { ... }
# Test: classify_base("A") should return "purine"

Exercise 3: Central Dogma Pipeline

Use pipes to: create a DNA sequence, transcribe it to RNA, translate it to protein, and get its length — all in one pipeline:

# dna"ATGAAACCCGGGTTTTAA" |> transcribe() |> translate() |> len() |> print()

Exercise 4: Filter Records

Given a list of gene expression records, keep only those with expression above 3.0:

let genes = [
    {gene: "BRCA1", expr: 5.2},
    {gene: "TP53", expr: 1.8},
    {gene: "EGFR", expr: 7.1},
    {gene: "KRAS", expr: 2.3},
    {gene: "MYC", expr: 4.0},
]
# Hint: genes |> filter(|g| g.expr > 3.0) |> each(|g| print(f"{g.gene}: {g.expr}"))

Exercise 5: Join vs Reduce

Use reduce to concatenate a list of strings with " | " as separator. Then discover that join does it more simply:

let items = ["DNA", "RNA", "Protein"]
# Hard way: items |> reduce(|a, b| a + " | " + b) |> print()
# Easy way: join(items, " | ") |> print()

Key Takeaways

Here is what you learned today, distilled:

The pipe |> is the core of BioLang. It makes data flow visible. When you read data |> transform() |> summarize() |> print(), you know exactly what happens at each step. No nesting, no temporary variables, no ambiguity.

Bio types (DNA, RNA, Protein) are not strings. They carry meaning, and the language enforces it. You cannot accidentally transcribe a protein or translate a string.

map, filter, and reduce replace most loops. They are cleaner, less error-prone, and they compose with pipes beautifully.

What’s Next

You now have a working language. You can write variables, functions, pipes, and HOFs. But so far, all our sequences have been short strings we typed by hand.

Tomorrow, we step back from code and into biology: genomes, genes, mutations, and why they matter. You need this foundation before you can analyze real data. Understanding what a VCF file represents matters as much as knowing how to parse it.

Day 3: The Biology You Need — genomes, chromosomes, variants, and the questions bioinformatics answers.

Day 3: Biology Crash Course for Developers

The Problem

You can write code, but you do not know what a gene actually is, why mutations matter, or what “expression” means. Without this foundation, bioinformatics code is just meaningless data shuffling. Every variable name, every file format, every analysis pipeline assumes you understand the biology underneath. Today we build the biological intuition you need.

If you already have a biology background, skim this chapter or use it as a refresher. For everyone else: this is the day that makes everything else click.

The Cell: Biology’s Computer

If you understand computers, you already have the mental framework for molecular biology. A living cell is an information-processing system, and the analogy is surprisingly precise.

Your DNA is the master copy of every instruction your body needs. It never leaves the nucleus, just like critical data stays in a server room. When the cell needs to build something, it copies the relevant section of DNA into RNA — a temporary, disposable working copy. That RNA travels to a ribosome, which reads it and assembles a protein, amino acid by amino acid.

This flow — DNA to RNA to Protein — is called the central dogma of molecular biology. Nearly everything in bioinformatics relates to measuring, comparing, or interpreting data at one of these three levels.

The analogy breaks down at scale, of course. Your cells are not running one program at a time. A single human cell has about 20,000 genes, thousands of which are active simultaneously, producing millions of protein molecules. It is less like a laptop and more like a data center running 20,000 microservices.

DNA: The Source Code

DNA is built from four chemical bases, each represented by a single letter:

These bases pair up in a strict pattern called Watson-Crick base pairing: A always pairs with T, and C always pairs with G. This gives DNA its famous double-helix structure — two complementary strands wound around each other.

    5'─A─T─G─C─G─A─T─C─G─3'    (coding strand)
        |  |  |  |  |  |  |  |  |
    3'─T─A─C─G─C─T─A─G─C─5'    (template strand)

Direction matters. DNA strands have a chemical directionality called 5’ (five-prime) to 3’ (three-prime). By convention, sequences are always written 5’ to 3’, just like we read text left to right. When bioinformatics tools say “the sequence is ATGCGATCG,” they mean reading the coding strand from 5’ to 3’.

The complement of a sequence flips each base according to the pairing rules: A becomes T, T becomes A, C becomes G, G becomes C. The reverse complement also reverses the order, giving you the other strand read in its own 5’-to-3’ direction.

let coding = dna"ATGCGATCG"
let comp = complement(coding)
let rc = reverse_complement(coding)
println(f"Coding:     5'-{coding}-3'")
println(f"Complement: 3'-{comp}-5'")
println(f"RevComp:    5'-{rc}-3'")
# Output:
# Coding:     5'-ATGCGATCG-3'
# Complement: 3'-TACGCTAGC-5'
# RevComp:    5'-CGATCGCAT-3'

Why does the reverse complement matter? Because sequencing machines can read either strand. If a read comes from the opposite strand, you need the reverse complement to map it back to the reference. This is one of the most common operations in bioinformatics.

Genes: Functions in the Genome

If DNA is the source code, a gene is a function — a defined region with a specific purpose. A gene contains the instructions for building one protein (a simplification, but a useful one; some genes produce functional RNA instead).

The numbers are humbling:

• The human genome has about 3.2 billion base pairs
• Only about 1.5% of that codes for proteins
• We have roughly 20,000 protein-coding genes
• The rest used to be called “junk DNA,” but much of it has regulatory roles

A gene is not a simple, contiguous stretch of code. It has structure:

• Exons are the coding sections — the parts that actually encode protein
• Introns are non-coding sections between exons — they get removed
• Splicing is the process of cutting out introns and joining exons together
• The result is mRNA (messenger RNA), the template used to build the protein

Think of it this way: a gene in DNA is like a source file full of commented-out blocks. Splicing is the preprocessor that strips the comments and produces clean, executable code.

# Simulating exon splicing
let exon1 = dna"ATGCGA"
let exon2 = dna"TCGATC"
let exon3 = dna"GCGTAA"

# In reality, splicing is done by the cell's machinery
# In BioLang, we can transcribe individual exons
let mrna = transcribe(exon1)
println(f"Exon 1 transcribed: {mrna}")
# Output:
# Exon 1 transcribed: AUGCGA

One of the most surprising facts in biology: the same gene can produce different proteins depending on which exons are included. This is called alternative splicing, and it is one reason humans can get by with only 20,000 genes — each one can produce multiple protein variants.

Proteins: The Machines

Proteins are the workhorses of the cell. They are built from 20 amino acids, and the sequence of amino acids determines what the protein does. The mapping from DNA to amino acid uses a three-letter code: every group of three bases (a codon) specifies one amino acid.

The math works out neatly: 4 bases taken 3 at a time gives 4^3 = 64 possible codons. Those 64 codons map to just 20 amino acids plus 3 stop signals. This redundancy is important — it means some mutations are harmless because different codons can encode the same amino acid.

Key codons to remember:

Let’s trace through the central dogma in code:

# Exploring the genetic code
let seq = dna"ATGGCTAACTGA"
let rna = transcribe(seq)
let protein = translate(seq)
println(f"DNA:     {seq}")
println(f"RNA:     {rna}")
println(f"Protein: {protein}")
# Output:
# DNA:     ATGGCTAACTGA
# RNA:     AUGGCUAACUGA
# Protein: MAN
# M = Methionine (start), A = Alanine, N = Asparagine
# (TGA = stop codon, translation halts before it)

Notice that translate() accepts DNA directly — BioLang handles the T-to-U conversion internally. The function stops at the first stop codon, which is the biologically correct behavior.

# Codon usage in a sequence
let gene = dna"ATGGCTGCTTCTGATTGA"
let usage = codon_usage(gene)
println(usage)
# Output:
# {ATG: 1, GCT: 2, TCT: 1, GAT: 1, TGA: 1}
# Notice GCT appears twice — both encode Alanine

Protein function depends on how the amino acid chain folds into a 3D structure. A single change in the sequence can alter the fold and destroy the protein’s function. This is why mutations matter.

Mutations: Bugs in the Code

A mutation is any change in the DNA sequence. Like a bug in software, the consequences depend entirely on where it happens and what changes. Some mutations are invisible; others are catastrophic.

Types of mutations

Normal:      ATG GCT AAC TGA  -->  M-A-N  (stop)
                                    |||
Missense:    ATG GCT GAC TGA  -->  M-A-D  (stop)   one amino acid changed (N->D)
                                    ||
Nonsense:    ATG TAA AAC TGA  -->  M  (premature stop!)   protein truncated
                                    |
Frameshift:  ATG -CT AAC TGA  -->  reading frame destroyed — total chaos

• SNP (Single Nucleotide Polymorphism): one base swapped for another. The most common type of variation.
• Synonymous (silent): the codon changes but still encodes the same amino acid, thanks to redundancy in the genetic code. No effect on the protein.
• Missense: the codon changes to encode a different amino acid. May or may not affect protein function, depending on how different the new amino acid is.
• Nonsense: the codon changes to a stop codon, truncating the protein. Almost always damaging.
• Frameshift: an insertion or deletion that is not a multiple of 3 shifts the entire reading frame. Every codon downstream is wrong. This is the biological equivalent of an off-by-one error that corrupts everything after it.

# Comparing normal vs mutant
let normal = dna"ATGGCTAACTGA"
let mutant = dna"ATGGCTGACTGA"  # A->G at position 7 (0-indexed)

let normal_protein = normal |> translate()
let mutant_protein = mutant |> translate()

println(f"Normal:  {normal_protein}")
println(f"Mutant:  {mutant_protein}")
println(f"Changed: {normal_protein != mutant_protein}")
# Output:
# Normal:  MAN
# Mutant:  MAD
# Changed: true
# One base change (A->G) changed Asparagine (N) to Aspartate (D)

The position within a codon matters enormously. The third position (called the “wobble position”) is the most tolerant of mutations because of codon redundancy. Mutations at the first or second position almost always change the amino acid.

The 20 Amino Acids

Every protein in every living organism is built from the same 20 amino acids. Each has a three-letter abbreviation and a single-letter code — the one-letter codes are what you will see constantly in bioinformatics data:

Why this table matters: When you see a protein sequence like MEEPQSDP in bioinformatics, each letter is one of these 20 amino acids. When a mutation report says “R175H”, it means Arginine (R) at position 175 was changed to Histidine (H). The single-letter codes are the language of protein bioinformatics.

Notice the properties column. Amino acids are not interchangeable:

• Hydrophobic amino acids (A, V, I, L, F, W, M) cluster in the protein’s interior, away from water
• Charged amino acids (R, K, D, E) sit on the surface and interact with other molecules
• Polar amino acids (S, T, N, Q) form hydrogen bonds and participate in catalysis

This is why a mutation that swaps a hydrophobic amino acid for a charged one (like V600E in BRAF — Valine to Glutamate) can be catastrophic: it puts a charged residue where a hydrophobic one should be, disrupting the protein’s entire 3D fold.

Gene Expression: Which Programs Are Running?

Every cell in your body contains the same DNA — the same complete set of ~20,000 genes. But a liver cell looks and behaves nothing like a neuron. The difference is gene expression: which genes are turned on and how strongly.

Gene expression is measured by how much RNA is being produced from a gene. A highly expressed gene produces thousands of RNA copies; a silenced gene produces none. Different cell types have dramatically different expression profiles:

• Housekeeping genes are always on — they handle basic cell maintenance (like system services that always run)
• Tissue-specific genes are only active in certain cell types (like applications that only launch on specific servers)
• Stress-response genes activate only under certain conditions — heat, DNA damage, infection (like error handlers)

The developer analogy is precise: gene expression is like running ps aux on a server. You see which processes are active, how much CPU they are using, and which ones just started or stopped. In biology, the equivalent tool is RNA-seq — a sequencing technology that counts RNA molecules, telling you exactly which genes are active and at what level.

Differential expression analysis compares expression between conditions. Which genes are more active in tumor tissue versus normal tissue? Which genes turn on when a cell is infected by a virus? These comparisons are one of the most common tasks in bioinformatics.

Reference Genomes and Coordinates

Just as every address system needs a map, genomics needs a reference genome — a canonical, consensus sequence for a species. The current human reference genome is called GRCh38 (Genome Reference Consortium Human Build 38), released in 2013 and continually patched.

Genomic coordinates use a simple system: chromosome + position. The location chr17:7,687,490 means chromosome 17, position 7,687,490. This is the universal addressing system in genomics — every variant, every gene, every regulatory element has coordinates on the reference.

Two coordinate conventions matter:

If you have ever been bitten by off-by-one errors in code, genomic coordinates will give you sympathy pain. The BED-vs-VCF coordinate difference is responsible for more bioinformatics bugs than any other single issue.

# Genomic intervals
let brca1_location = interval("chr17", 43044295, 43125483)
let tp53_location = interval("chr17", 7668402, 7687550)

println(f"BRCA1: {brca1_location}")
println(f"TP53:  {tp53_location}")
println(f"Same chromosome: {brca1_location.chrom == tp53_location.chrom}")
# Output:
# BRCA1: chr17:43044295-43125483
# TP53:  chr17:7668402-7687550
# Same chromosome: true

Both BRCA1 and TP53 are on chromosome 17, but they are millions of base pairs apart. BRCA1 is a breast/ovarian cancer gene; TP53 is the most commonly mutated gene across all cancers. We will meet both again throughout this book.

The “-omics” Landscape

Modern biology is organized into layers, each with its own data types and analysis methods:

• Genomics: the study of complete DNA sequences — finding genes, identifying variants, comparing species
• Transcriptomics: measuring which genes are expressed and at what level, usually via RNA-seq
• Proteomics: identifying and quantifying proteins in a sample using mass spectrometry
• Metabolomics: profiling small molecules (metabolites) that result from cellular processes
• Epigenomics: studying chemical modifications to DNA that affect gene expression without changing the sequence
• Variant analysis: cataloging mutations and polymorphisms, assessing their clinical significance
• Single-cell -omics: any of the above, but measured in individual cells rather than bulk tissue

Each -omics field has its own file formats, databases, and analytical pipelines. This book will focus on genomics, transcriptomics, and variant analysis — the areas where most bioinformatics work happens.

Putting It All Together: A Gene Story

Let’s make this concrete with TP53, the most studied gene in cancer biology. TP53 encodes the protein p53, sometimes called the “guardian of the genome.” When DNA gets damaged, p53 activates to either repair the damage or trigger cell death. When TP53 is mutated, this safety mechanism fails — damaged cells keep dividing, leading to cancer.

TP53 is mutated in more than 50% of all human cancers. It is the single most commonly mutated gene across cancer types. Understanding why requires everything we have covered today.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# The story of TP53 — the most mutated gene in cancer
# Requires internet connection for NCBI lookup
# Optional: set NCBI_API_KEY for higher rate limits

let tp53 = ncbi_gene("TP53")
println(f"Gene:        {tp53.symbol}")
println(f"Description: {tp53.description}")
println(f"Chromosome:  {tp53.chromosome}")
println(f"Location:    {tp53.location}")
# Output (approximate — NCBI data updates):
# Gene:        TP53
# Description: tumor protein p53
# Chromosome:  17
# Location:    17p13.1

# A normal TP53 fragment — the start of the coding sequence
let normal = dna"ATGGAGGAGCCGCAGTCAGATCCTAGC"
let protein = normal |> translate()
println(f"Normal protein starts: {protein}")
# Output:
# Normal protein starts: MEEPQSDPS
# M=Met, E=Glu, E=Glu, P=Pro, Q=Gln, S=Ser, D=Asp, P=Pro, S=Ser

# GC content of this region
let gc = gc_content(normal)
println(f"GC content: {gc}")
# Output:
# GC content: 0.5555555555555556

The most common TP53 mutation in cancer is R248W: a single base change that swaps Arginine (R, coded by CGG) for Tryptophan (W, coded by TGG) at position 248 of the protein. One letter changes. The protein misfolds. The guardian is disabled. Cells lose their brake pedal.

This is why we study mutations with such care. A single base out of 3.2 billion can be the difference between a cell that functions normally and one that becomes cancerous.

Exercises

Exercise 1: Hand-translate a sequence

Given dna"ATGAAAGCTTGA", what protein does it encode? Work it out by hand first:

• Split into codons: ATG | AAA | GCT | TGA
• Look up each codon: ATG=M, AAA=K, GCT=A, TGA=Stop
• Expected protein: MKA

Then verify with BioLang:

let seq = dna"ATGAAAGCTTGA"
let protein = translate(seq)
println(f"Protein: {protein}")
# Output:
# Protein: MKA

Exercise 2: Wobble position experiment

Create two DNA sequences that differ by one base. Translate both. Does the amino acid change? Try mutating position 1, 2, and 3 of the second codon to see which position tolerates mutations best:

# Original: GCT = Alanine (A)
let original = dna"ATGGCTTGA"
let mut_pos1 = dna"ATGTCTTGA"   # G->T at codon position 1
let mut_pos2 = dna"ATGGATTGA"   # C->A at codon position 2
let mut_pos3 = dna"ATGGCATGA"   # T->A at codon position 3

println(f"Original (GCT): {translate(original)}")
println(f"Pos1 mut (TCT): {translate(mut_pos1)}")
println(f"Pos2 mut (GAT): {translate(mut_pos2)}")
println(f"Pos3 mut (GCA): {translate(mut_pos3)}")
# Output:
# Original (GCT): MA
# Pos1 mut (TCT): MS   (Alanine -> Serine — changed!)
# Pos2 mut (GAT): MD   (Alanine -> Aspartate — changed!)
# Pos3 mut (GCA): MA   (Alanine -> Alanine — silent! same amino acid)
# The third position is most tolerant of mutations (wobble position)

Exercise 3: Look up a gene

Look up what chromosome EGFR is on using ncbi_gene("EGFR"). EGFR (Epidermal Growth Factor Receptor) is a major drug target in lung cancer.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Requires internet connection
let egfr = ncbi_gene("EGFR")
println(f"EGFR chromosome: {egfr.chromosome}")
println(f"EGFR description: {egfr.description}")
# Expected: chromosome 7

Exercise 4: Interval overlap check

Create intervals for two genes on chromosome 7 and check whether they overlap:

let egfr = interval("chr7", 55019017, 55211628)
let braf = interval("chr7", 140719327, 140924929)

# Manual overlap check: two intervals overlap if
# they are on the same chrom AND start < other.end AND other.start < end
let same_chrom = egfr.chrom == braf.chrom
let overlaps = same_chrom and egfr.start < braf.end and braf.start < egfr.end

println(f"EGFR: {egfr}")
println(f"BRAF: {braf}")
println(f"Same chromosome: {same_chrom}")
println(f"Overlap: {overlaps}")
# Output:
# EGFR: chr7:55019017-55211628
# BRAF: chr7:140719327-140924929
# Same chromosome: true
# Overlap: false
# (They're ~85 million bases apart — same chromosome, but far away)

Key Takeaways

• DNA -> RNA -> Protein: the central dogma governs how genetic information becomes function. DNA is transcribed into RNA; RNA is translated into protein.
• Genes are regions of DNA that encode proteins. Humans have approximately 20,000 protein-coding genes in a 3.2-billion-base genome.
• Mutations are changes in DNA. They can be silent (synonymous), damaging (missense/nonsense), or catastrophic (frameshift). The wobble position (third base of a codon) is the most tolerant.
• Gene expression tells us which genes are active. It varies by cell type, condition, and time. RNA-seq measures it by counting RNA molecules.
• Genomic coordinates (chromosome + position) are the universal addressing system. Watch out for 0-based (BED) vs 1-based (VCF) conventions.
• The reference genome (GRCh38) is the baseline. Variants are always described relative to it.

What’s Next

Tomorrow: Day 4 — Coding Crash Course for Biologists. The complementary perspective — thinking in data structures, debugging strategies, and building confidence with code. Biologists learn the computational thinking they need; developers can skip or skim.

Day 4: Coding Crash Course for Biologists

The Problem

You understand biology deeply. You can design a CRISPR experiment, interpret a Western blot, and explain the Krebs cycle from memory. But your data analysis is stuck in Excel. You copy-paste between spreadsheets, manually rename files, and spend hours on repetitive tasks that a script could do in seconds.

Today you learn to think like a programmer — not to become one, but to become a more effective biologist. By the end of this chapter, you will be able to read and write short programs that automate the tedious parts of your research.

If you already know how to code, skim this chapter or use it to understand how biologists think about data. The lab analogies here will help you communicate with your biology collaborators.

Why Code Beats Spreadsheets

Every biologist has been there: a spreadsheet with 500 gene names in column A, sequences in column B, and a formula in column C that took 20 minutes to get right. Then someone asks you to do the same thing with a different dataset. Or worse, asks you to prove your analysis is reproducible.

Code solves four problems that spreadsheets cannot:

Reproducibility. A script runs the same way every time. No forgotten steps, no accidental edits, no “I think I sorted column B before filtering.” You can hand your script to a colleague and they get the exact same results.

Scale. Processing 1,000 samples is exactly as hard as processing 1. You do not manually drag formulas down 1,000 rows or open 1,000 files by hand.

Automation. Chain steps together. Run overnight. Schedule weekly analyses. Code does not get tired, does not skip a step, and does not introduce random errors at 3 AM.

Sharing. Send a colleague a script, not a 47-step protocol with screenshots. They run it, it works. Done.

Here is a concrete example. Suppose you need to count how many genes in a list of 500 sequences have GC content above 60%.

In Excel: create a column with a LEN formula, another column to count G and C characters, a third column for the ratio, then a COUNTIF on that column. Manually set up. Fifteen minutes if nothing goes wrong.

In BioLang:

# What takes 15 minutes in Excel takes 2 lines in BioLang
let sequences = [dna"GCGCGCATGC", dna"ATATATATAT", dna"GCGCGCGCGC", dna"ATCGATCGAT", dna"GCGCTAGCGC"]
let count = sequences |> filter(|s| gc_content(s) > 0.6) |> len()
println(f"{count} sequences are GC-rich")

Two lines. Instant. And when your collaborator gives you a new list of 5,000 sequences, you change nothing — the same two lines handle it.

Thinking in Steps

You already know how to think in steps. Every wet lab protocol is a sequence of instructions, executed in order, with decisions along the way. Programming is exactly that, except you write the protocol in a language the computer understands.

Lab Protocol:                Code Equivalent:
─────────────────────────    ─────────────────────────
1. Get sample                1. Read input file
2. Extract DNA               2. Parse sequences
3. Run PCR                   3. Filter / transform
4. Gel electrophoresis       4. Analyze results
5. Photograph gel            5. Visualize / save output

The point: you already think in recipes. Code just writes them down so a computer can follow them. Every program you will ever write follows this pattern — get data in, do something to it, get results out.

Throughout this chapter, we will use lab analogies to make each concept click. If you can run a protocol, you can write a program.

Variables: Labeling Your Tubes

In the lab, you label every tube. Without labels, you have mystery liquids and ruined experiments. Variables work the same way — they are named labels attached to data.

# Variables are like labeled tubes in your rack
let sample_name = "Patient_042"
let concentration = 23.5       # ng/uL
let is_contaminated = false
let bases_sequenced = 3200000

println(f"Sample: {sample_name}")
println(f"Concentration: {concentration} ng/uL")
println(f"Clean: {not is_contaminated}")

Every variable has a type — the kind of data it holds. You already know these types from your lab notebook:

Notice that BioLang has types specifically for biology. You do not store DNA as plain text and hope nobody passes it to a function expecting a gene name. The type system catches mistakes before they become wrong results.

# Types prevent mistakes — like labeling tubes correctly
let gene = dna"ATGGCTAACTGA"
let name = "BRCA1"

# These work:
let gc = gc_content(gene)       # gc_content expects DNA
println(f"GC content: {gc}")

# This would be an error:
# let gc = gc_content(name)     # "BRCA1" is a string, not DNA!

The let keyword creates a new variable. Think of it as reaching for a fresh tube and writing a label on it. Without let, you get an error — the computer does not know what you are referring to.

Lists: Your Sample Rack

A list is an ordered collection of items — like a rack of labeled tubes. Each tube has a position (starting from 0), and you can add, remove, or check what is in the rack.

# A list is like a rack of tubes
let samples = ["Control_1", "Control_2", "Treated_1", "Treated_2"]
println(f"Number of samples: {len(samples)}")
println(f"First sample: {first(samples)}")

# Add a new sample
let updated = push(samples, "Treated_3")
println(f"Now have {len(updated)} samples")

# Check if a sample exists
println(contains(samples, "Control_1"))  # true
println(contains(samples, "Control_9"))  # false

Lists can hold any type of data — strings, numbers, even DNA sequences:

# A rack of DNA samples
let primers = [
    dna"ATCGATCGATCG",
    dna"GCGCGCGCGCGC",
    dna"AAATTTAAATTT"
]

println(f"Number of primers: {len(primers)}")
println(f"First primer: {first(primers)}")

Records: Your Lab Notebook Entry

A record groups related information together — like one entry in your lab notebook. Instead of five separate variables for one experiment, you have one record with named fields.

# A record is like one entry in your lab notebook
let experiment = {
    date: "2024-03-15",
    investigator: "Dr. Chen",
    cell_line: "HeLa",
    treatment: "Doxorubicin",
    concentration_uM: 0.5,
    viability_percent: 72.3
}

println(f"Cell line: {experiment.cell_line}")
println(f"Viability: {experiment.viability_percent}%")

You access fields with a dot — experiment.cell_line pulls out the cell line, just like flipping to the right page in your notebook. Records keep related data together, which prevents the spreadsheet problem of accidentally sorting one column without the others.

# A list of records — like a table in your notebook
let qc_results = [
    {sample: "S001", reads: 25000000, quality: 35.2},
    {sample: "S002", reads: 18000000, quality: 33.1},
    {sample: "S003", reads: 500000,   quality: 28.7}
]

println(f"First sample: {first(qc_results).sample}")
println(f"Its quality: {first(qc_results).quality}")

Loops: Processing Every Sample the Same Way

In the lab, you rarely process one sample. You process twenty, or a hundred, or a thousand — all with the same protocol. A loop does exactly that: repeat a set of instructions for every item in a list.

Without a loop, you would write:

# Without loops — painful and error-prone
# print("Analyzing BRCA1...")
# print("Analyzing TP53...")
# print("Analyzing EGFR...")
# ... what if you have 500 genes?

With a loop:

# With a loop — works for 5 genes or 5,000
let genes = ["BRCA1", "TP53", "EGFR", "KRAS", "MYC"]
for gene in genes {
    println(f"Analyzing {gene}...")
}

The for loop takes each item from the list, one at a time, assigns it to the variable gene, and runs the code inside the curly braces. When the list is done, the loop stops.

Think of it as a protocol where you say “do this for every tube in the rack”:

Here is a more practical example — processing actual sequences:

# Calculate GC content for each sequence
let sequences = [dna"ATCGATCG", dna"GCGCGCGC", dna"AATTAATT"]

for seq in sequences {
    let gc = gc_content(seq)
    let gc_pct = round(gc * 100.0, 1)
    println(f"{seq} -> GC: {gc_pct}%")
}

Conditions: Quality Control Decisions

Every lab has QC checkpoints. Is the concentration high enough? Is the sample contaminated? Did we get enough reads? In code, you make these decisions with if, else if, and else.

# Making QC decisions in code — just like at the bench
let read_count = 15000000
let gc_bias = 0.52
let duplication_rate = 0.15

if read_count < 1000000 {
    println("FAIL: Too few reads — resequence")
} else if duplication_rate > 0.3 {
    println("WARNING: High duplication — check library prep")
} else {
    println("PASS: Sample meets QC thresholds")
}

You can combine conditions with and and or:

# Multiple criteria
let reads = 20000000
let quality = 32.5

if reads > 10000000 and quality > 30.0 {
    println("High-quality sample — proceed to analysis")
} else {
    println("Sample needs review")
}

Conditions are especially powerful inside loops. Here is QC on a whole batch:

let samples = [
    {name: "S001", reads: 25000000, quality: 35.2},
    {name: "S002", reads: 500000,   quality: 28.7},
    {name: "S003", reads: 18000000, quality: 33.1},
    {name: "S004", reads: 12000000, quality: 22.0}
]

for s in samples {
    if s.reads < 1000000 {
        println(f"  {s.name}: FAIL (too few reads: {s.reads})")
    } else if s.quality < 25.0 {
        println(f"  {s.name}: FAIL (low quality: {s.quality})")
    } else {
        println(f"  {s.name}: PASS")
    }
}

Functions: Reusable Protocols

A function is a reusable protocol. You write it once, name it, and use it whenever you need it — just like an SOP in your lab manual.

# A function is a reusable protocol
fn qc_check(reads, min_reads) {
    if reads < min_reads {
        "FAIL"
    } else {
        "PASS"
    }
}

# Use it on any sample
println(qc_check(25000000, 1000000))   # PASS
println(qc_check(500000, 1000000))     # FAIL
println(qc_check(12000000, 5000000))   # PASS

The beauty of functions is that when you change your QC threshold, you change it in one place — not in 50 spreadsheet cells.

Functions can take any number of inputs and return a result. The last expression in the function is the result (no need to write “return”):

# Calculate fold change between conditions
fn fold_change(control, treated) {
    round(treated / control, 2)
}

println(f"FC: {fold_change(5.2, 12.8)}")   # 2.46
println(f"FC: {fold_change(8.1, 7.9)}")    # 0.98
println(f"FC: {fold_change(3.4, 15.2)}")   # 4.47

You can use functions together with loops for powerful batch processing:

# Apply QC to every sample
let results = [12000000, 500000, 8000000, 25000000]
    |> map(|r| {reads: r, status: qc_check(r, 1000000)})

for r in results {
    println(f"Reads: {r.reads} -> {r.status}")
}

The |r| ... syntax is a shorthand function — a quick, unnamed protocol you use once and throw away. Think of it as a sticky note with one instruction, versus a full SOP in the lab manual.

Pipes: Connecting Steps Together

In the lab, you chain steps: take sample, extract DNA, measure concentration, decide if you have enough, proceed. Each step feeds into the next. Pipes (|>) work the same way in BioLang — the result of one step flows into the next.

# Lab protocol as pipes:
# Take sample -> extract DNA -> measure concentration -> decide -> proceed

# In BioLang, pipes connect processing steps:
let result = dna"ATGCGATCGATCGATCGATCGATCG"
    |> gc_content()
    |> round(3)

println(f"GC content: {result}")

Read it left to right: start with a DNA sequence, calculate its GC content, round to 3 decimal places. The |> operator takes the result from the left side and feeds it as the first input to the right side.

Without pipes, you would nest functions inside each other, which gets hard to read:

# Without pipes — hard to follow
let result_nested = round(gc_content(dna"ATGCGATCGATCGATCGATCGATCG"), 3)
println(f"Same result: {result_nested}")

Both produce the same answer, but the pipe version reads like English: “take this sequence, get its GC content, round it.”

Here is a more realistic pipeline:

# Multi-step analysis pipeline
let sequences = [
    dna"ATCGATCGATCG",
    dna"GCGCGCGCGCGC",
    dna"ATATATATATATAT",
    dna"GCGCATATAGCGC",
    dna"TTTTTAAAAACCCCC"
]

let gc_rich_count = sequences
    |> map(|s| {seq: s, gc: gc_content(s)})
    |> filter(|r| r.gc > 0.5)
    |> len()

println(f"{gc_rich_count} out of {len(sequences)} sequences are GC-rich")

Read the pipeline step by step:

1. Start with a list of sequences
2. map: for each sequence, create a record with the sequence and its GC content
3. filter: keep only records where GC content is above 50%
4. len: count how many passed the filter

This is the power of pipes — complex multi-step analyses that read like a protocol.

Errors: When Things Go Wrong

Code errors are like failed experiments — they give you information. A PCR that does not work tells you the primers are wrong, the temperature is off, or the DNA is degraded. Code errors tell you exactly what went wrong and where.

# Errors are informative, not catastrophic
try {
    let x = int("not_a_number")
    println(f"This won't print: {x}")
} catch e {
    println(f"Error: {e}")
    # Just like a failed PCR tells you something useful
}

The try/catch pattern says: “try this, and if it fails, do this instead.” It prevents your whole analysis from crashing when one step goes wrong — like having a backup plan in your protocol.

Common errors you will see:

Do not fear errors. Read them, understand them, fix them. Every error makes you a better programmer, just like every failed experiment makes you a better scientist.

Your First Complete Analysis

Let us combine everything into a realistic mini-project. You have gene expression data from a control and treated condition, and you want to find which genes are upregulated.

# Experiment: Analyze gene expression across treatments
let samples = [
    {gene: "BRCA1", control: 5.2,  treated: 12.8},
    {gene: "TP53",  control: 8.1,  treated: 7.9},
    {gene: "EGFR",  control: 3.4,  treated: 15.2},
    {gene: "MYC",   control: 6.7,  treated: 6.5},
    {gene: "KRAS",  control: 4.1,  treated: 11.3}
]

# Calculate fold changes
fn fold_change(control, treated) {
    round(treated / control, 2)
}

let results = samples |> map(|s| {
    gene: s.gene,
    fold_change: fold_change(s.control, s.treated),
    direction: if s.treated > s.control { "UP" } else { "DOWN" }
})

# Find significantly upregulated genes (fold change > 2)
let upregulated = results
    |> filter(|r| r.fold_change > 2.0)
    |> sort_by(|r| r.fold_change)
    |> reverse()

println("=== Upregulated Genes (FC > 2.0) ===")
for gene in upregulated {
    println(f"  {gene.gene}: {gene.fold_change}x {gene.direction}")
}
println(f"\nTotal: {len(upregulated)} of {len(samples)} genes upregulated")

Let us trace through what this does:

1. Data: five genes, each with a control and treated expression value
2. Function: fold_change calculates the ratio and rounds it
3. Map: transforms each sample into a result with fold change and direction
4. Filter: keeps only genes with fold change above 2.0
5. Sort and reverse: orders by fold change, highest first
6. Print: displays the results

This is a complete analysis pipeline. It is reproducible (run it again, get the same answer), scalable (add 500 more genes to the list, nothing else changes), and readable (anyone can follow the logic).

Common Mistakes and How to Fix Them

Every programmer makes these mistakes. They are not signs that you are doing it wrong — they are a normal part of learning.

Forgetting let

# Wrong — x is not defined
# x = 42
# print(x)

# Right — use let to create a variable
let x = 42
println(x)

Wrong type

# Wrong — gc_content needs DNA, not a string
# let gc = gc_content("ATGCGA")

# Right — use a DNA literal
let gc = gc_content(dna"ATGCGA")
println(f"GC: {gc}")

Positions start at 0, not 1

In biology, we count from 1 (base 1, exon 1). In most programming, counting starts at 0. This trips up everyone at first. Just remember: the first item is at position 0.

Using = when you mean ==

# Single = assigns a value
let x = 5

# Double == compares values
if x == 5 {
    println("x is five")
}

Exercises

Exercise 1: QC Filter

Create a list of 5 samples, each with name, read_count, and quality_score fields. Use filter to keep only high-quality samples (quality above 30). Print how many passed.

let samples = [
    {name: "S1", read_count: 20000000, quality_score: 35.0},
    # ... add 4 more
]
let passed = samples |> filter(|s| s.quality_score > 30.0)

Exercise 2: Fold Change Function

Write a function calc_fc(control, treated) that returns the fold change (treated divided by control, rounded to 2 decimal places). Test it with at least 3 pairs of values.

fn calc_fc(control, treated) {
    round(treated / control, 2)
}

Exercise 3: GC Content Pipeline

Build a pipeline that takes a list of DNA sequences, calculates GC content for each, finds the average GC content using mean, and prints a summary.

let seqs = [dna"ATCGATCG", dna"GCGCGCGC", dna"AATTAATT"]
let gc_values = seqs |> map(|s| gc_content(s))
let avg_gc = mean(gc_values)

Exercise 4: Dilution Table

Use nested for loops to print a dilution table. Starting concentrations: [0.1, 0.5, 1.0, 5.0, 10.0]. Dilution factors: [1, 2, 4]. Print each combination.

let concentrations = [0.1, 0.5, 1.0, 5.0, 10.0]
let dilutions = [1, 2, 4]
for conc in concentrations {
    for dil in dilutions {
        let final_conc = round(conc / dil, 3)
        println(f"  {conc} / {dil} = {final_conc}")
    }
}

Key Takeaways

• Code is a lab protocol written in a language computers understand
• Variables are labeled tubes — let name = value creates one
• Lists are sample racks — ordered collections you can loop through
• Records are notebook entries — groups of related fields accessed with .
• Loops process every sample the same way — write the protocol once
• Conditions make QC decisions — if/else branches based on thresholds
• Functions are reusable SOPs — write once, use everywhere
• Pipes (|>) connect processing steps — like a lab workflow, left to right
• Errors are informative — they tell you what went wrong, just like a failed experiment

You do not need to memorize all of this. You will look things up, copy patterns from previous scripts, and gradually build fluency — exactly like learning any other lab technique.

What’s Next

Tomorrow: data structures designed specifically for biology — how to work with collections of sequences, genomic intervals, and tables of results. You will see how BioLang’s built-in types make common bioinformatics tasks concise and safe.

Day 5: Data Structures for Biology

The Problem

You have got 500 gene expression values, 20,000 variants, and 3 reference databases to cross-check. How do you organize this data so your analysis does not drown in complexity?

The difference between a messy script and a clean one is rarely the algorithm. It is the data structure. Pick the right container for your data, and filtering, comparing, and summarizing become one-liners. Pick the wrong one, and you spend hours writing code to work around it.

Today you learn five structures that cover virtually every bioinformatics task: lists, records, tables, sets, and genomic intervals. By the end of this chapter you will know which one to reach for and why.

Lists: Ordered Collections

A list holds items in a specific order. Use lists when sequence matters: time-series measurements, ordered coordinates, ranked gene lists, sample queues.

# Gene expression values in order
let expression = [2.1, 5.4, 3.2, 8.7, 1.1, 6.3]

# Statistics on lists
println(f"Mean: {round(mean(expression), 2)}")
println(f"Median: {round(median(expression), 2)}")
println(f"Stdev: {round(stdev(expression), 2)}")
println(f"Min: {min(expression)}, Max: {max(expression)}")

# Sorting and slicing
let sorted_expr = sort(expression) |> reverse()
let top3 = sorted_expr |> take(3)
println(f"Top 3 values: {top3}")

Expected output:

Mean: 4.47
Median: 4.3
Stdev: 2.65
Min: 1.1, Max: 8.7
Top 3 values: [8.7, 6.3, 5.4]

Lists hold any type. You can filter, transform, and reduce them with pipes:

# Sample names
let samples = ["control_1", "control_2", "treated_1", "treated_2", "treated_3"]

# Filter to treated samples
let treated = samples |> filter(|s| contains(s, "treated"))
println(f"Treated: {treated}")

# Count elements
println(f"Total: {len(samples)}, Treated: {len(treated)}")

Nested lists model matrix-like data when you need something quick:

# Matrix-like data: samples x genes
let data = [
    [2.1, 3.4, 5.6],
    [1.8, 4.2, 6.1],
    [3.0, 2.9, 4.8],
]
# Access: data[1][2] = 6.1 (Sample 2, Gene 3)
println(f"Sample 2, Gene 3: {data[1][2]}")

Records: Structured Metadata

A record groups named fields together. Use records when you have heterogeneous data about a single entity: a gene, a sample, a variant, an experiment.

# A gene record
let gene = {
    symbol: "BRCA1",
    name: "BRCA1 DNA repair associated",
    chromosome: "17",
    start: 43044295,
    end: 43125483,
    strand: "+",
    biotype: "protein_coding"
}

# Access fields
println(f"{gene.symbol} on chr{gene.chromosome}")
println(f"Length: {gene.end - gene.start} bp")
println(f"Keys: {keys(gene)}")

# Check if field exists
println(f"Has strand: {has_key(gene, "strand")}")
println(f"Has expression: {has_key(gene, "expression")}")

Expected output:

BRCA1 on chr17
Length: 81188 bp
Keys: [symbol, name, chromosome, start, end, strand, biotype]
Has strand: true
Has expression: false

The most common pattern in bioinformatics is a list of records. Each record describes one item (a variant, a sample, a gene), and the list collects them:

let variants = [
    {chrom: "chr17", pos: 43091434, ref_allele: "A", alt_allele: "G", gene: "BRCA1"},
    {chrom: "chr17", pos: 7674220,  ref_allele: "C", alt_allele: "T", gene: "TP53"},
    {chrom: "chr7",  pos: 55249071, ref_allele: "C", alt_allele: "T", gene: "EGFR"},
]

# Filter to chromosome 17
let chr17_vars = variants |> filter(|v| v.chrom == "chr17")
println(f"Chr17 variants: {len(chr17_vars)}")

# Extract just gene names
let genes = variants |> map(|v| v.gene)
println(f"Affected genes: {genes}")

Tables: The Bioinformatician’s Workhorse

Tables are the primary structure for analysis results. If you have named columns and multiple rows, you want a table. Differential expression results, sample sheets, variant annotations, QC metrics – all tables.

┌──────┬──────────┬──────────┐
│ gene │ log2fc   │ pval     │
├──────┼──────────┼──────────┤
│ BRCA1│  2.4     │ 0.001    │
│ TP53 │ -1.1     │ 0.23     │
│ EGFR │  3.8     │ 0.000001 │
│ MYC  │  1.9     │ 0.04     │
│ KRAS │ -0.3     │ 0.67     │
└──────┴──────────┴──────────┘

Create a table from a list of records with to_table():

# Creating tables from records
let results = [
    {gene: "BRCA1", log2fc: 2.4,  pval: 0.001},
    {gene: "TP53",  log2fc: -1.1, pval: 0.23},
    {gene: "EGFR",  log2fc: 3.8,  pval: 0.000001},
    {gene: "MYC",   log2fc: 1.9,  pval: 0.04},
    {gene: "KRAS",  log2fc: -0.3, pval: 0.67},
] |> to_table()

println(f"Rows: {nrow(results)}, Columns: {ncol(results)}")
println(f"Columns: {colnames(results)}")

# Filter and sort
let significant = results
    |> filter(|r| r.pval < 0.05)
    |> arrange("log2fc")

println(significant |> head(5))

Expected output:

Rows: 5, Columns: 3
Columns: [gene, log2fc, pval]

Tables support the operations you know from dplyr or pandas, all connected with pipes:

# select -- choose columns
let gene_pvals = results |> select("gene", "pval")
println(gene_pvals |> head(3))

# mutate -- add or transform columns
let annotated = results |> mutate("significant", |r| r.pval < 0.05)
println(annotated |> head(3))

# group_by + summarize
let direction_table = results
    |> mutate("direction", |r| if r.log2fc > 0 { "up" } else { "down" })
    |> group_by("direction")
    |> summarize(|key, rows| {direction: key, count: len(rows)})
println(direction_table)

Here is what each operation does:

Sets: Unique Membership and Comparisons

A set holds unique items with no duplicates and no particular order. Use sets when you care about membership: Which genes appear in both experiments? Which samples are unique to one cohort? Sets give you Venn diagram logic in code.

# Genes from two experiments
let experiment_a = set(["BRCA1", "TP53", "EGFR", "MYC", "KRAS"])
let experiment_b = set(["TP53", "EGFR", "PTEN", "RB1", "MYC"])

# Set operations
let shared = intersection(experiment_a, experiment_b)
let only_a = difference(experiment_a, experiment_b)
let only_b = difference(experiment_b, experiment_a)
let all_genes = union(experiment_a, experiment_b)

println(f"Shared genes: {shared}")
println(f"Only in A: {only_a}")
println(f"Only in B: {only_b}")
println(f"Total unique: {len(all_genes)}")

Expected output:

Shared genes: {TP53, EGFR, MYC}
Only in A: {BRCA1, KRAS}
Only in B: {PTEN, RB1}
Total unique: 7

Sets are the natural fit whenever you ask “which items overlap?” – a question that appears constantly in bioinformatics. Gene panels, GO term lists, differentially expressed gene sets, sample cohorts.

Genomic Intervals: Coordinates and Overlaps

Genomic data lives on coordinates. A promoter spans chr17:43125283-43125483. An exon runs from chr17:43124017 to chr17:43124115. You need to ask: do these regions overlap? What falls within this window?

BioLang has built-in interval types and an interval tree for fast overlap queries:

# Working with genomic regions
let promoter = interval("chr17", 43125283, 43125483)
let exon1 = interval("chr17", 43124017, 43124115)
let enhancer = interval("chr17", 43125000, 43125600)

println(f"Promoter: {promoter}")
println(f"Exon 1: {exon1}")
println(f"Enhancer: {enhancer}")

# Build an interval tree for fast overlap queries
let regions = [
    {chrom: "chr17", start: 43125283, end: 43125483, name: "promoter"},
    {chrom: "chr17", start: 43124017, end: 43124115, name: "exon1"},
    {chrom: "chr17", start: 43125000, end: 43125600, name: "enhancer"},
] |> to_table()

let tree = interval_tree(regions)

# Query: what overlaps this 100bp window?
let hits = query_overlaps(tree, "chr17", 43125300, 43125400)
println(f"Overlapping regions: {nrow(hits)}")
println(hits)

The interval_tree function builds a searchable index from a table containing chrom, start, and end columns. The query_overlaps function takes the tree, a chromosome name, a start position, and an end position, and returns a table of matching rows. This is the same algorithm that powers tools like bedtools – but built into the language.

Choosing the Right Structure

When you sit down with a new dataset, ask yourself three questions:

1. Does my data have named fields? (record or table)
2. Do I have one item or many? (record vs table)
3. Do I need order, or just membership? (list vs set)

Here is the summary:

Real-World Pattern: Combining Structures

Real analyses combine multiple structures. Here is a pattern you will see repeatedly: samples described by records, gene sets for comparison, and results collected into a table.

# Combining data structures in a real analysis

# Each sample is a record with a set of detected genes
let samples = [
    {id: "S1", condition: "control", genes: set(["BRCA1", "TP53", "EGFR"])},
    {id: "S2", condition: "treated", genes: set(["TP53", "MYC", "KRAS", "EGFR"])},
    {id: "S3", condition: "treated", genes: set(["BRCA1", "TP53", "PTEN"])},
]

# Find genes detected in ALL samples
let all_genes = samples |> map(|s| s.genes)
let common = all_genes |> reduce(|a, b| intersection(a, b))
println(f"Core genes (in all samples): {common}")

# Find genes unique to treated samples
let treated_genes = samples
    |> filter(|s| s.condition == "treated")
    |> map(|s| s.genes)
    |> reduce(|a, b| union(a, b))

let control_genes = samples
    |> filter(|s| s.condition == "control")
    |> map(|s| s.genes)
    |> reduce(|a, b| union(a, b))

let treatment_specific = difference(treated_genes, control_genes)
println(f"Treatment-specific genes: {treatment_specific}")

# Build a summary table
let summary = [
    {category: "Core (all samples)", count: len(common)},
    {category: "Treatment-specific", count: len(treatment_specific)},
    {category: "Control genes", count: len(control_genes)},
    {category: "Treated genes", count: len(treated_genes)},
] |> to_table()

println(summary)

Expected output:

Core genes (in all samples): {TP53}
Treatment-specific genes: {MYC, KRAS, PTEN}

Notice how naturally the structures compose. Records hold per-sample metadata. Sets enable Venn-diagram logic. Lists let you iterate over samples with map and filter. Tables collect the final summary. Each structure does what it is best at.

Exercises

1. List statistics. Create a list of 10 expression values (make up realistic numbers between 0 and 50). Compute the mean, median, min, max, and standard deviation. Sort the list in descending order and print the top 5 values.

2. Variant record. Build a record representing a VCF variant with fields: chrom, pos, ref_allele, alt_allele, qual, filter_status, gene. Print each field. Use keys() to list all field names and has_key() to check for a field called annotation.

3. Table filtering. Create a table from 5 gene records, each with gene, chromosome, and expression fields. Filter to genes on chromosome 17. Use select to show only the gene and expression columns.

4. Set overlap. Define two gene panels as sets: a cancer panel (10 genes) and a cardiac panel (10 genes) with 3 genes in common. Use intersection, difference, and union to find shared genes, genes unique to each panel, and the total gene count.

5. Interval queries. Create a table with 3 genomic regions (give them names like “promoter”, “exon1”, “enhancer” with realistic coordinates on the same chromosome). Build an interval_tree and use query_overlaps to find which regions overlap a given 200bp window.

Key Takeaways

• Lists hold ordered data. Use them for expression values, sample queues, ranked results. Key operations: sort, filter, map, reduce, take.

• Records group named fields. Use them for metadata about a single entity – one gene, one sample, one experiment. Access fields with dot notation. Check fields with has_key.

• Tables are the workhorse. Named columns, many rows. Use to_table() to create them from lists of records. Manipulate with select, filter, mutate, arrange, group_by, summarize.

• Sets eliminate duplicates and enable Venn-diagram logic. intersection finds shared items, difference finds unique items, union combines everything.

• Intervals represent genomic coordinates. Build an interval_tree for fast overlap queries with query_overlaps. This is the same approach that powers bedtools.

• Choose the right structure upfront. It makes everything downstream easier. When in doubt: if it has named fields and multiple rows, it is a table. If you need unique membership, it is a set. If order matters, it is a list.

What’s Next

Week 1 is complete. You now have the foundations: biological sequences, sequence analysis, coding skills, and data structures. Starting in Week 2, you will work with real sequencing data. Day 6 opens with FASTA and FASTQ files – the raw material of genomics.

Day 6: Reading Sequencing Data

The Problem

Your sequencing facility sends you a 50 GB FASTQ file. It contains millions of short DNA reads, each with a quality score for every base. Some reads are garbage — adapter contamination, low quality, too short. Before any analysis, you must separate the good reads from the bad. This is quality control, and it is the first step of every sequencing project.

Today is the first day we work with real bioinformatics data formats. Everything before this was foundations. From here on, the biology gets real.

What Is a FASTQ File?

FASTQ is the universal format for sequencing data. Every sequencing platform — Illumina, PacBio, Oxford Nanopore — outputs FASTQ files. Each record in a FASTQ file has exactly four lines:

@SRR123456.1 length=150        <- Read name (starts with @)
ATCGATCGATCGATCGATCG...         <- DNA sequence
+                               <- Separator (always a single +)
IIIIIIIHHHHHGGGFFF...           <- Quality scores (ASCII-encoded)

The first line is the read identifier — it starts with @ and usually contains a unique ID, sometimes with metadata like instrument name, flowcell, and tile coordinates.

The second line is the DNA sequence itself — the bases called by the sequencer.

The third line is a separator. It is always +, sometimes followed by the read name again.

The fourth line is the quality string. Every character encodes the confidence the sequencer has in the corresponding base call. This is where the real information lives.

Phred Quality Scores

Quality scores use the Phred scale, named after the original base-calling program from the Human Genome Project. The formula is:

Q = -10 * log10(P_error)

A higher Q means lower error probability. The score is encoded as an ASCII character by adding 33 to the numeric value (this is the Sanger/Illumina 1.8+ encoding used by all modern sequencers):

Most Illumina sequencers produce reads with average quality between Q28 and Q35. A Q30 average is generally considered good. Reads below Q20 are usually discarded.

To decode: take the ASCII value of the character and subtract 33. The character I has ASCII value 73, so its Phred score is 73 - 33 = 40.

Reading FASTQ Files in BioLang

BioLang provides two ways to read FASTQ files: eager loading and streaming.

Eager Loading

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Read a FASTQ file (eager --- loads all into memory)
let reads = read_fastq("data/reads.fastq")
println(f"Total reads: {len(reads)}")
println(f"First read: {first(reads)}")

Total reads: 100
First read: {name: "read_001", seq: "ATCGATCG...", qual: "IIIIIIII..."}

Each read is a record with three fields:

• name — the read identifier (without the @)
• seq — the DNA sequence
• qual — the quality string (same length as seq)

Streaming

For large files, loading everything into memory is impractical. A 50 GB FASTQ file might contain 300 million reads. Use streaming instead:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Streaming --- process one at a time, constant memory
let stream = fastq("data/large_sample.fastq")
let count = stream |> count()
println(f"Read count: {count}")

Read count: 300000000

The rule of thumb: if the file fits comfortably in RAM, use read_fastq(). Otherwise, use fastq(). For this chapter, we use read_fastq() because our sample data is small.

Exploring Read Quality

Before filtering, you need to know what you are working with. BioLang’s read_stats() gives you a summary in one call:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Quality statistics for a FASTQ file
let stats = read_stats("examples/sample.fastq")
println(f"Total reads: {stats.total_reads}")
println(f"Total bases: {stats.total_bases}")
println(f"Mean length: {round(stats.mean_length, 1)}")
println(f"Mean quality: {round(stats.mean_quality, 1)}")
println(f"GC content: {round(stats.gc_content * 100, 1)}%")

Total reads: 100
Total bases: 15000
Mean length: 150.0
Mean quality: 28.4
GC content: 48.2%

For deeper analysis, you can compute per-read quality scores using pipes:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Per-read quality analysis
let reads = read_fastq("data/reads.fastq")
let qualities = reads |> map(|r| mean_phred(r.qual))
println(f"Quality range: {round(min(qualities), 1)} - {round(max(qualities), 1)}")
println(f"Mean quality: {round(mean(qualities), 1)}")

Quality range: 12.3 - 38.7
Mean quality: 28.4

The mean_phred() function takes a quality string and returns the average Phred score across all bases. This is the single most useful number for judging a read.

Quality Visualization

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Quality distribution as ASCII plot
let reads = read_fastq("data/reads.fastq")
reads
    |> map(|r| mean_phred(r.qual))
    |> quality_plot()

Quality Distribution
  Q10-15: ####          (8)
  Q15-20: ########      (15)
  Q20-25: ##########    (22)
  Q25-30: ############  (28)
  Q30-35: ##########    (19)
  Q35-40: ######        (8)

This immediately tells you the shape of your quality distribution. A good library will be skewed toward the right (higher quality). If most reads pile up below Q20, something went wrong with sequencing.

Filtering Reads

Not every read deserves to continue to analysis. Filtering removes reads that would introduce noise or artifacts. A typical filtering pipeline applies three checks:

Built-in Filtering

BioLang provides filter_reads() for the most common quality filters:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Filter reads by quality and length
let reads = read_fastq("data/reads.fastq")

let clean = reads |> filter_reads(min_length: 50, min_quality: 20)
println(f"Before: {len(reads)} reads")
println(f"After:  {len(clean)} reads")
println(f"Kept:   {round(len(clean) / len(reads) * 100, 1)}%")

Before: 100 reads
After:  82 reads
Kept:   82.0%

Custom Filtering with Pipes

For more specific criteria, compose your own filters using filter():

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Custom filtering with pipes
let reads = read_fastq("data/reads.fastq")

let clean = reads
    |> filter(|r| len(r.seq) >= 50)
    |> filter(|r| mean_phred(r.qual) >= 20)
    |> filter(|r| gc_content(r.seq) > 0.2 and gc_content(r.seq) < 0.8)
    |> collect()

println(f"Clean reads: {len(clean)}")

Clean reads: 78

Each filter() call removes reads that fail the predicate. The pipe chain reads like a checklist: keep reads that are long enough, high enough quality, and have reasonable GC content.

Why filter on GC content? Extreme GC values (below 20% or above 80%) often indicate contamination — adapter dimers, primer artifacts, or DNA from a different organism. A typical mammalian genome has ~40% GC content.

Trimming Low-Quality Bases

Sometimes a read has good bases at the start but degrades toward the end. This is normal — Illumina quality drops along the read. Rather than throwing away the entire read, you can trim off the bad bases:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Quality trimming --- remove low-quality bases from ends
let reads = read_fastq("data/reads.fastq")
let trimmed = trim_quality(reads, min_quality: 20)

# Check how trimming affected lengths
let original_lengths = reads |> map(|r| len(r.seq))
let trimmed_lengths = trimmed |> map(|r| len(r.seq))

println(f"Mean length before: {round(mean(original_lengths), 1)}")
println(f"Mean length after:  {round(mean(trimmed_lengths), 1)}")

Mean length before: 150.0
Mean length after:  138.6

trim_quality() uses a sliding window from the 3’ end of the read. It removes bases until the average quality in the window meets the threshold. This is the same algorithm used by Trimmomatic’s SLIDINGWINDOW mode.

After trimming, you typically filter again to remove reads that became too short:

let trimmed_and_filtered = trimmed
    |> filter(|r| len(r.seq) >= 50)
    |> collect()
println(f"Reads after trim + length filter: {len(trimmed_and_filtered)}")

Adapter Detection and Removal

Sequencing adapters are synthetic DNA sequences ligated to your library fragments. If the insert is shorter than the read length, the sequencer reads through into the adapter. These adapter sequences must be removed because they are not part of the genome.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Detect adapters in reads
let adapters = detect_adapters("examples/sample.fastq")
println(f"Detected adapters: {adapters}")

Detected adapters: [AGATCGGAAGAGC, CTGTCTCTTATACACATCT]

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Trim adapters
let reads = read_fastq("data/reads.fastq")
let trimmed = trim_adapters(reads)
println(f"Adapter-trimmed reads: {len(trimmed)}")

Adapter-trimmed reads: 100

detect_adapters() scans the reads and identifies overrepresented sequences at read ends — these are almost always adapters. trim_adapters() removes any adapter contamination it finds.

Common adapters include:

• Illumina TruSeq: AGATCGGAAGAGC
• Nextera: CTGTCTCTTATACACATCT
• Small RNA: TGGAATTCTCGG

K-mer Analysis for Quality Assessment

K-mers are subsequences of length k. Counting k-mer frequencies across your reads can reveal contamination, library bias, or technical artifacts. In a clean library, k-mer frequencies should follow a roughly normal distribution. Spikes at specific k-mers suggest adapter contamination or PCR duplicates.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# K-mer frequency analysis
let reads = read_fastq("data/reads.fastq")

# Count k-mers in the first read
let first_seq = first(reads).seq
let kmer_freq = kmer_count(first_seq, 5)
println(f"5-mers found: {nrow(kmer_freq)}")
println(kmer_freq |> head(10))

5-mers found: 146
 kmer  | count
 ATCGA | 3
 TCGAT | 3
 CGATC | 2
 GATCG | 2
 GCTAG | 2
 TAGCA | 2
 ACGTA | 1
 CGTAC | 1
 GTACG | 1
 TACGT | 1

If you see a single 5-mer appearing hundreds of times, that is a red flag — it likely corresponds to adapter sequence or a PCR artifact.

Writing Clean Reads

After filtering and trimming, save the clean reads to a new FASTQ file:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Save filtered reads to a new file
let reads = read_fastq("data/reads.fastq")
let clean = reads |> filter_reads(min_length: 50, min_quality: 20)
write_fastq(clean, "results/clean_reads.fastq")
println(f"Wrote {len(clean)} clean reads")

Wrote 82 clean reads

The output FASTQ preserves the original read names, sequences (potentially trimmed), and quality scores. Downstream tools like aligners (BWA, Bowtie2) expect standard FASTQ input, so this step ensures compatibility.

Complete QC Pipeline

Here is a complete quality control pipeline that combines everything from this chapter. This is the kind of script you would run on every new sequencing dataset:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Complete FASTQ QC Pipeline

println("=== FASTQ Quality Control Pipeline ===")

# Step 1: Read stats
let stats = read_stats("examples/sample.fastq")
println(f"\n1. Raw data summary:")
println(f"   Reads: {stats.total_reads}")
println(f"   Bases: {stats.total_bases}")
println(f"   Mean quality: {round(stats.mean_quality, 1)}")

# Step 2: Load and filter
let reads = read_fastq("data/reads.fastq")
let clean = reads
    |> filter(|r| len(r.seq) >= 50)
    |> filter(|r| mean_phred(r.qual) >= 20)
    |> collect()

let pass_rate = round(len(clean) / len(reads) * 100, 1)
println(f"\n2. Filtering results:")
println(f"   Input:  {len(reads)} reads")
println(f"   Passed: {len(clean)} reads ({pass_rate}%)")

# Step 3: Quality summary of clean reads
let clean_quals = clean |> map(|r| mean_phred(r.qual))
println(f"\n3. Clean read quality:")
println(f"   Mean: {round(mean(clean_quals), 1)}")
println(f"   Min:  {round(min(clean_quals), 1)}")

# Step 4: GC content check
let gc_values = clean |> map(|r| gc_content(r.seq))
println(f"\n4. GC content:")
println(f"   Mean GC: {round(mean(gc_values) * 100, 1)}%")

# Step 5: Write output
write_fastq(clean, "results/qc_passed.fastq")
println(f"\n5. Output written to results/qc_passed.fastq")
println("=== Pipeline complete ===")

=== FASTQ Quality Control Pipeline ===

1. Raw data summary:
   Reads: 100
   Bases: 15000
   Mean quality: 28.4

2. Filtering results:
   Input:  100 reads
   Passed: 82 reads (82.0%)

3. Clean read quality:
   Mean: 30.6
   Min:  20.3

4. GC content:
   Mean GC: 49.1%

5. Output written to results/qc_passed.fastq
=== Pipeline complete ===

This pipeline takes about 5 seconds on a 100-read sample file. On a real 50 GB FASTQ with 300 million reads, you would switch to streaming with fastq() and it would take a few minutes.

Exercises

1. Top 10 longest reads. Write a script that reads a FASTQ file and prints the 10 longest reads by sequence length. Hint: use sort() on a mapped list of lengths, or arrange() on a table.

2. Q30 percentage. Calculate what percentage of reads have a mean quality score >= Q30. This is a standard QC metric reported by sequencing facilities.

3. Strict base filter. Build a custom filter that keeps only reads where every base has quality >= Q15. Use min_phred() instead of mean_phred(). How many reads survive compared to the mean-based filter?

4. GC shift analysis. Compare GC content distributions before and after quality filtering. Does removing low-quality reads change the GC distribution? Calculate mean GC for raw reads and for filtered reads.

Key Takeaways

• FASTQ = sequence + quality for every base. Four lines per record, always.
• Phred scores: Q20 = 99% accurate, Q30 = 99.9%, Q40 = 99.99%. Higher is better.
• Always QC before analysis — garbage in, garbage out. This is not optional.
• Use fastq() streaming for large files, read_fastq() for small ones.
• filter_reads() handles standard filtering; custom filter() chains handle special cases.
• trim_quality() removes low-quality bases from read ends — better than discarding entire reads.
• K-mer analysis can reveal contamination and artifacts before they corrupt your results.

What’s Next

Tomorrow we tackle the rest of the bioinformatics file format zoo: FASTA for reference genomes, VCF for variants, BED for genomic regions, GFF for gene annotations, and BAM for alignments. You will learn when to use each format and how to convert between them.

Day 7: Bioinformatics File Formats

The Problem

Bioinformatics has accumulated dozens of file formats over 30 years. Each stores different information in a different way. FASTA for sequences, VCF for variants, BED for regions, GFF for annotations, BAM for alignments. Knowing which format holds what — and how to read each — is essential.

Every analysis you will ever do starts by reading one of these files and ends by writing another. Get the formats wrong and your pipeline silently produces garbage. Get the coordinate systems confused and every interval is off by one. Today we build the mental map that prevents those mistakes.

The Format Landscape

Where does each format appear in a typical genomics workflow?

The sequencer produces raw reads in FASTQ (Day 6). Those reads get aligned to a reference genome (FASTA), producing alignments (SAM/BAM). Variant callers compare the alignments to the reference and output differences (VCF). Annotators overlay gene models (GFF/GTF) and region lists (BED) onto the variants.

Every arrow in that diagram is a file format conversion. Today you learn to read and write each one.

FASTA — Reference Sequences

FASTA is the oldest and simplest bioinformatics format. It stores named sequences — DNA, RNA, or protein. Every reference genome, every transcript database, every protein collection uses FASTA.

Anatomy

>chr1 Homo sapiens chromosome 1     <- Header line (starts with >)
ATCGATCGATCGATCGATCGATCGATCG        <- Sequence (can span multiple lines)
ATCGATCGATCGATCG
>chr2 Homo sapiens chromosome 2     <- Next sequence
GCGCGCATATATATGCGCGCGCGC
>BRCA1_mRNA NM_007294.4             <- Can be any named sequence
ATGGATTTATCTGCTCTTCGCGTTGAAG

The header line starts with > followed by an identifier and optional description. The sequence follows on one or more lines. There is no quality information — FASTA is for known sequences, not raw reads.

Reading FASTA

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let seqs = read_fasta("data/sequences.fasta")
println(f"Sequences: {len(seqs)}")

for s in seqs {
    println(f"  {s.id}: {len(s.seq)} bp, GC={round(gc_content(s.seq) * 100, 1)}%")
}

Sequences: 5
  chr1_fragment: 200 bp, GC=49.0%
  chr17_brca1: 150 bp, GC=52.0%
  chrX_region: 180 bp, GC=41.1%
  ecoli_16s: 120 bp, GC=54.2%
  insulin_mrna: 100 bp, GC=47.0%

Each sequence is a record with two fields:

• id — the identifier from the header line (text after > up to the first space)
• seq — the full sequence as a string

Streaming for Large Genomes

A human reference genome is 3.1 billion bases across 24 chromosomes. Loading it all into memory uses ~3 GB. For large FASTA files, stream instead:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let total_bases = fasta("data/sequences.fasta")
    |> map(|s| len(s.seq))
    |> reduce(|a, b| a + b)
println(f"Total bases: {total_bases}")

Total bases: 750

FASTA Statistics

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let stats = fasta_stats("data/sequences.fasta")
println(f"Sequences: {stats.count}")
println(f"Total bases: {stats.total_bases}")
println(f"Mean length: {round(stats.mean_length, 1)}")

Sequences: 5
Total bases: 750
Mean length: 150.0

VCF — Variant Calls

VCF (Variant Call Format) stores genetic variants — positions where a sample’s DNA differs from the reference genome. It is the standard output of every variant caller (GATK, bcftools, DeepVariant, etc.).

Anatomy

##fileformat=VCFv4.3                                <- Meta-information lines
##INFO=<ID=DP,Number=1,Type=Integer,Description="Read Depth">
##FILTER=<ID=LowQual,Description="Low quality">
#CHROM  POS     ID      REF  ALT  QUAL FILTER INFO  <- Column header
chr1    100     .       A    G    30   PASS   DP=45  <- SNP (A -> G)
chr1    200     rs123   CT   C    45   PASS   DP=62  <- Deletion (T deleted)
chr17   43091   .       G    A    99   PASS   DP=88  <- High-quality SNP
chr17   43200   .       C    T    12   LowQual DP=5  <- Low-quality, filtered

The file has three sections:

1. Meta-information lines (start with ##) — describe the file structure, INFO fields, FILTER definitions, and sample metadata.
2. Column header (starts with #CHROM) — names the eight mandatory columns plus any sample columns.
3. Data lines — one variant per line.

The key columns:

• CHROM and POS — where the variant is (1-based coordinate)
• REF and ALT — what the reference has vs what the sample has
• QUAL — confidence score (Phred-scaled)
• FILTER — PASS if the variant passed all filters, otherwise a filter name
• INFO — semicolon-delimited key=value annotations

Reading VCF

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let variants = read_vcf("data/variants.vcf")
println(f"Total variants: {len(variants)}")

# Examine first variant
let v = first(variants)
println(f"Chrom: {v.chrom}, Pos: {v.pos}, Ref: {v.ref}, Alt: {v.alt}")

# Filter to passing variants
let passed = variants |> filter(|v| v.filter == "PASS")
println(f"PASS variants: {len(passed)}")

# Count by chromosome
let by_chrom = passed
    |> to_table()
    |> group_by("chrom")
    |> summarize(|chrom, rows| {chrom: chrom, count: len(rows)})
println(by_chrom)

Total variants: 10
Chrom: chr1, Pos: 100, Ref: A, Alt: G
PASS variants: 8

 chrom | count
 chr1  | 3
 chr17 | 3
 chrX  | 2

Variant Types

Not all variants are the same. The REF and ALT lengths tell you what kind of variant you have:

# Classify variants by type
let snps = variants |> filter(|v| len(v.ref) == 1 and len(v.alt) == 1)
let indels = variants |> filter(|v| len(v.ref) != len(v.alt))
println(f"SNPs: {len(snps)}")
println(f"Indels: {len(indels)}")

SNPs: 7
Indels: 3

Streaming Large VCF Files

Whole-genome VCF files can contain millions of variants. Stream them:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let snp_count = vcf("data/variants.vcf")
    |> filter(|v| len(v.ref) == 1 and len(v.alt) == 1)
    |> count()
println(f"SNPs (streaming): {snp_count}")

SNPs (streaming): 7

BED — Genomic Regions

BED (Browser Extensible Data) stores genomic intervals — regions of a chromosome with a start and end position. It is used for gene coordinates, exon boundaries, peaks from ChIP-seq, target capture regions, blacklisted regions, and anything else that can be described as “chromosome X from position A to position B.”

Anatomy

chr1    1000    2000    gene_A    100    +      <- 6-column BED
chr1    3000    4000    gene_B    200    -
chr17   43044295 43125483 BRCA1   0      +

The columns are tab-separated:

1. chrom — chromosome name
2. start — start position (0-based)
3. end — end position (exclusive, half-open)
4. name — feature name (optional, columns 4+)
5. score — numeric score (optional)
6. strand — + or - (optional)

The Critical Coordinate Convention

BED uses 0-based, half-open coordinates. This is the single most important thing to remember about BED files.

Position:  0  1  2  3  4  5  6  7  8  9
Bases:     A  T  C  G  A  T  C  G  A  T

BED:       chr1  2  5     <- Covers bases at positions 2, 3, 4 (= C, G, A)
                          <- Start is inclusive, end is exclusive
                          <- Length = end - start = 5 - 2 = 3

VCF/GFF:   chr1  3        <- Position 3 refers to the base at 1-based position 3
                          <- Which is the same base C at 0-based position 2

This means:

• BED chr1 100 200 covers 100 bases (positions 100 through 199)
• The length of a BED interval is always end - start
• To convert VCF position (1-based) to BED: subtract 1 from the start

Reading BED

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let regions = read_bed("data/regions.bed")
println(f"Regions: {len(regions)}")

# Calculate total covered bases
let total = regions
    |> map(|r| r.end - r.start)
    |> reduce(|a, b| a + b)
println(f"Total bases covered: {total}")

# Filter to a specific chromosome
let chr17 = regions |> filter(|r| r.chrom == "chr17")
println(f"Chr17 regions: {len(chr17)}")

Regions: 10
Total bases covered: 92500
Chr17 regions: 3

Region Statistics

let sizes = regions |> map(|r| r.end - r.start)
println(f"Region sizes:")
println(f"  Min: {min(sizes)}")
println(f"  Max: {max(sizes)}")
println(f"  Mean: {round(mean(sizes), 1)}")

Region sizes:
  Min: 500
  Max: 81189
  Mean: 9250.0

GFF/GTF — Gene Annotations

GFF (General Feature Format) and GTF (Gene Transfer Format) store gene structure annotations — where genes are, where their exons are, where the coding regions start and stop. GFF3 is the current standard; GTF is an older Ensembl-specific variant that is still widely used.

Anatomy

chr1  ensembl  gene   11869  14409  .  +  .  gene_id "ENSG00000223972"; gene_name "DDX11L1"
chr1  ensembl  exon   11869  12227  .  +  .  gene_id "ENSG00000223972"; exon_number "1"
chr1  ensembl  exon   12613  12721  .  +  .  gene_id "ENSG00000223972"; exon_number "2"
chr1  ensembl  exon   13221  14409  .  +  .  gene_id "ENSG00000223972"; exon_number "3"

The nine tab-separated columns:

1. seqid — chromosome or contig name
2. source — who produced the annotation (ensembl, refseq, etc.)
3. type — feature type (gene, exon, mRNA, CDS, etc.)
4. start — start position (1-based, inclusive)
5. end — end position (1-based, inclusive)
6. score — numeric score or . if not applicable
7. strand — +, -, or .
8. phase — reading frame for CDS features (0, 1, or 2) or .
9. attributes — semicolon-delimited key-value pairs

Coordinates: 1-Based, Inclusive

GFF uses 1-based, fully inclusive coordinates. A feature at 11869..14409 covers all 2541 bases from position 11869 through position 14409 inclusive.

To convert GFF to BED:
  BED_start = GFF_start - 1
  BED_end   = GFF_end          (already exclusive in the half-open sense)

Example:
  GFF:  chr1  11869  14409     (1-based inclusive, covers 14409 - 11869 + 1 = 2541 bases)
  BED:  chr1  11868  14409     (0-based half-open, covers 14409 - 11868 = 2541 bases)

Reading GFF

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let features = read_gff("data/annotations.gff")
println(f"Features: {len(features)}")

# Count feature types
let genes = features |> filter(|f| f.type == "gene")
let exons = features |> filter(|f| f.type == "exon")
let cds = features |> filter(|f| f.type == "CDS")
println(f"Genes: {len(genes)}")
println(f"Exons: {len(exons)}")
println(f"CDS:   {len(cds)}")

Features: 15
Genes: 3
Exons: 8
CDS:   4

Extracting Gene Information

# List all gene names
let gene_names = features
    |> filter(|f| f.type == "gene")
    |> map(|f| f.attributes.gene_name)
println(f"Genes: {gene_names}")

Genes: [DDX11L1, BRCA1, TP53]

Streaming

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let exon_count = gff("data/annotations.gff")
    |> filter(|f| f.type == "exon")
    |> count()
println(f"Exons (streaming): {exon_count}")

Exons (streaming): 8

SAM/BAM — Alignments

SAM (Sequence Alignment/Map) stores read alignments — which reads mapped where on the reference genome, and how. BAM is the binary compressed version of SAM. You almost always work with BAM files because they are smaller and indexed for fast random access.

Anatomy

@HD  VN:1.6  SO:coordinate                           <- Header: format version, sort order
@SQ  SN:chr1  LN:248956422                           <- Header: reference sequence lengths
@SQ  SN:chr17 LN:83257441
read_001  99   chr1   100   60   150M        *  0  0  ATCG...  IIII...  <- Alignment
read_002  83   chr1   250   42   75M2I73M    *  0  0  ATCG...  IIII...  <- Alignment with insertion
read_003  4    *      0     0    *           *  0  0  ATCG...  IIII...  <- Unmapped read

The key fields in each alignment record:

• QNAME — read name
• FLAG — bitwise flags encoding paired-end status, strand, mapping status
• RNAME — reference chromosome
• POS — leftmost mapping position (1-based)
• MAPQ — mapping quality (0-60, higher is better)
• CIGAR — alignment description string (e.g., 150M = 150 matches, 75M2I73M = 75 matches + 2 inserted bases + 73 matches)

SAM Flags

The FLAG field is a bitwise integer. Common values:

Reading BAM

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let alignments = read_bam("data/alignments.bam")
println(f"Total alignments: {len(alignments)}")

# Basic alignment statistics
let mapped = alignments |> filter(|r| r.is_mapped)
let unmapped = alignments |> filter(|r| not r.is_mapped)
println(f"Mapped: {len(mapped)}")
println(f"Unmapped: {len(unmapped)}")

# Mapping quality distribution
let mapqs = mapped |> map(|r| r.mapq)
println(f"Mean MAPQ: {round(mean(mapqs), 1)}")
println(f"High quality (MAPQ >= 30): {len(mapqs |> filter(|q| q >= 30))}")

Total alignments: 20
Mapped: 17
Unmapped: 3
Mean MAPQ: 48.2
High quality (MAPQ >= 30): 14

Streaming BAM

BAM files from a whole-genome sequencing run can be 50-100 GB. Always stream:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let mapped_count = bam("data/alignments.bam")
    |> filter(|r| r.is_mapped)
    |> count()
println(f"Mapped reads (streaming): {mapped_count}")

Mapped reads (streaming): 17

BAM vs SAM

Rule: always store BAM, never SAM. Convert to SAM only when you need to visually inspect a few records.

The Coordinate System Trap

The single biggest source of bugs in bioinformatics is mixing up coordinate systems. Here is the definitive comparison:

Genome:    A  T  C  G  A  T  C  G
0-based:   0  1  2  3  4  5  6  7      <- BED, BAM (internal)
1-based:   1  2  3  4  5  6  7  8      <- VCF, GFF/GTF, SAM (POS)

The region covering "CGAT" (4 bases):
  BED:     chr1  2  6     (0-based, half-open: positions 2,3,4,5)
  GFF:     chr1  3  6     (1-based, inclusive: positions 3,4,5,6)
  VCF:     POS=3          (1-based: position 3 for a single variant)

Conversion Rules

# VCF (1-based) to BED (0-based, half-open)
bed_start = vcf_pos - 1
bed_end   = vcf_pos - 1 + len(ref)

# GFF (1-based, inclusive) to BED (0-based, half-open)
bed_start = gff_start - 1
bed_end   = gff_end              # already correct for half-open

# BED (0-based) to GFF (1-based, inclusive)
gff_start = bed_start + 1
gff_end   = bed_end              # already correct for inclusive

Format Conversion Patterns

Converting between formats is a daily task. Here are the most common conversions:

VCF to BED — Variant Positions as Intervals

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let variants = read_vcf("data/variants.vcf")
let beds = variants |> map(|v| {
    chrom: v.chrom,
    start: v.pos - 1,
    end: v.pos - 1 + len(v.ref)
})
println(f"Converted {len(beds)} variants to BED intervals")
println(f"First: {first(beds).chrom}:{first(beds).start}-{first(beds).end}")

Converted 10 variants to BED intervals
First: chr1:99-100

GFF Genes to BED

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let features = read_gff("data/annotations.gff")
let gene_beds = features
    |> filter(|f| f.type == "gene")
    |> map(|f| {
        chrom: f.seqid,
        start: f.start - 1,
        end: f.end,
        name: f.attributes.gene_name
    })
println(f"Gene BED regions: {len(gene_beds)}")

Gene BED regions: 3

Writing Files

BioLang can write all the formats it reads.

Writing FASTA

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let seqs = [
    {id: "seq1", seq: dna"ATCGATCGATCG"},
    {id: "seq2", seq: dna"GCGCGCATATGC"},
]
write_fasta(seqs, "results/output.fasta")
println("Wrote 2 sequences to FASTA")

Wrote 2 sequences to FASTA

Writing BED

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let regions = [
    {chrom: "chr1", start: 100, end: 200},
    {chrom: "chr1", start: 300, end: 400},
    {chrom: "chr17", start: 43044295, end: 43125483},
]
write_bed(regions, "results/output.bed")
println(f"Wrote {len(regions)} regions to BED")

Wrote 3 regions to BED

Tables to CSV

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let results = [
    {gene: "BRCA1", pval: 0.001, chrom: "chr17"},
    {gene: "TP53", pval: 0.05, chrom: "chr17"},
    {gene: "EGFR", pval: 0.003, chrom: "chr7"},
] |> to_table()

write_csv(results, "results/output.csv")
println(f"Wrote {nrow(results)} rows to CSV")
println(f"Columns: {colnames(results)}")

Wrote 3 rows to CSV
Columns: [gene, pval, chrom]

Putting It All Together

Here is a realistic mini-pipeline that reads multiple formats and produces a summary:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Multi-format analysis pipeline
println("=== Multi-Format Analysis ===")

# 1. Read reference sequences
let ref_seqs = read_fasta("data/sequences.fasta")
println(f"Reference: {len(ref_seqs)} sequences, {fasta_stats('data/sequences.fasta').total_bases} bp")

# 2. Read variants
let variants = read_vcf("data/variants.vcf")
let passed = variants |> filter(|v| v.filter == "PASS")
println(f"Variants: {len(variants)} total, {len(passed)} PASS")

# 3. Read target regions
let targets = read_bed("data/regions.bed")
let target_bp = targets |> map(|r| r.end - r.start) |> reduce(|a, b| a + b)
println(f"Target regions: {len(targets)}, covering {target_bp} bp")

# 4. Read gene annotations
let features = read_gff("data/annotations.gff")
let genes = features |> filter(|f| f.type == "gene")
println(f"Annotations: {len(features)} features, {len(genes)} genes")

# 5. Summary table
let snps = passed |> filter(|v| len(v.ref) == 1 and len(v.alt) == 1)
let indels = passed |> filter(|v| len(v.ref) != len(v.alt))
let summary = [
    {metric: "Reference sequences", value: len(ref_seqs)},
    {metric: "Total variants", value: len(variants)},
    {metric: "PASS variants", value: len(passed)},
    {metric: "SNPs", value: len(snps)},
    {metric: "Indels", value: len(indels)},
    {metric: "Target regions", value: len(targets)},
    {metric: "Target bases", value: target_bp},
    {metric: "Genes", value: len(genes)},
] |> to_table()
println(summary)

=== Multi-Format Analysis ===
Reference: 5 sequences, 750 bp
Variants: 10 total, 8 PASS
Target regions: 10, covering 92500 bp
Annotations: 15 features, 3 genes

 metric              | value
 Reference sequences | 5
 Total variants      | 10
 PASS variants       | 8
 SNPs                | 6
 Indels              | 2
 Target regions      | 10
 Target bases        | 92500
 Genes               | 3

Format Cheat Sheet

Keep this table handy. You will refer to it constantly.

When to use eager vs stream:

Exercises

Exercise 1: FASTA GC Champion. Read data/sequences.fasta and find the sequence with the highest GC content. Print its ID and GC percentage.

let seqs = read_fasta("data/sequences.fasta")
let best = seqs
    |> sort(|a, b| gc_content(b.seq) - gc_content(a.seq))
    |> first()
println(f"Highest GC: {best.id} at {round(gc_content(best.seq) * 100, 1)}%")

Exercise 2: SNP Census. Read data/variants.vcf, filter to SNPs only (single-base REF and ALT), and count them by chromosome.

let snps = read_vcf("data/variants.vcf")
    |> filter(|v| len(v.ref) == 1 and len(v.alt) == 1)
let by_chrom = snps
    |> to_table()
    |> group_by("chrom")
    |> summarize(|chrom, rows| {chrom: chrom, count: len(rows)})
println(by_chrom)

Exercise 3: Mean Region Size. Read data/regions.bed and calculate the mean region size in base pairs.

let regions = read_bed("data/regions.bed")
let sizes = regions |> map(|r| r.end - r.start)
println(f"Mean region size: {round(mean(sizes), 1)} bp")

Exercise 4: VCF to BED. Convert all variants in data/variants.vcf to BED format, properly adjusting the coordinate system (1-based to 0-based).

let variants = read_vcf("data/variants.vcf")
let bed_regions = variants |> map(|v| {
    chrom: v.chrom,
    start: v.pos - 1,
    end: v.pos - 1 + len(v.ref)
})
for b in bed_regions {
    println(f"{b.chrom}\t{b.start}\t{b.end}")
}

Exercise 5: Feature Types. Read data/annotations.gff and list all unique feature types with their counts.

let features = read_gff("data/annotations.gff")
let type_counts = features
    |> to_table()
    |> group_by("type")
    |> summarize(|feat_type, rows| {type: feat_type, count: len(rows)})
println(type_counts)

Key Takeaways

• FASTA = sequences, FASTQ = sequences + quality, VCF = variants, BED = regions, GFF = annotations, BAM = alignments.
• BED is 0-based half-open, VCF and GFF are 1-based — coordinate conversion is a constant source of bugs. Always check which system you are in.
• Use streaming readers (fasta(), vcf(), bam()) for large files — they process one record at a time in constant memory.
• Use eager readers (read_fasta(), read_vcf()) for small files you need to access multiple times or sort.
• Every format has a BioLang reader — you never need to parse tab-separated text manually.
• When converting between formats, always account for the coordinate system difference. VCF position 100 becomes BED start 99.

What’s Next

Tomorrow: when files are too big to fit in memory. Day 8 covers streaming, lazy evaluation, and constant-memory processing — the techniques that let you handle whole-genome data on a laptop.

Day 8: Processing Large Files

The Problem

Your laptop has 16 GB of RAM. Your FASTQ file is 50 GB. Your BAM file is 200 GB. Loading everything into memory crashes your machine. You need to process data one piece at a time — like reading a book page by page instead of memorizing the whole thing at once.

This is not a theoretical problem. A single Illumina NovaSeq run produces 1–3 TB of FASTQ data. Whole-genome sequencing at 30x coverage yields ~100 GB of compressed FASTQ per sample. If your analysis script starts with “load the entire file,” it will never finish.

The solution is streaming: reading and processing one record at a time, keeping only what you need in memory. BioLang makes this the default for large-file operations.

Eager vs Streaming

There are two fundamentally different approaches to processing a file:

Eager — load everything, then process:

[File: 50 GB] --> [RAM: Load all 50 GB] --> [Process] --> [Result]
                       Out of memory!

The eager approach reads every record into a list in memory. This is simple and works fine for small files, but fails catastrophically on large ones.

Streaming — process one record at a time:

[File: 50 GB] --> [RAM: 1 record] --> [Process] --> [Next record] --> ... --> [Result]
                       ~10 MB constant

The streaming approach reads one record, processes it, discards it, then reads the next. Memory usage stays constant regardless of file size. A 1 GB file and a 100 GB file use the same amount of RAM.

BioLang streaming functions return a StreamValue — a lazy iterator that is consumed once. No data is loaded until you ask for it.

Every green box is lazy — no data moves until the blue terminal operation runs.

Stream Basics

BioLang provides two ways to read every file format: an eager function that loads everything into a list, and a streaming function that returns a lazy iterator.

The eager versions are the ones you used in Days 6 and 7. They are convenient for small files. The streaming versions are what you use for anything large.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Eager: loads everything into a list
let all_reads = read_fastq("data/reads.fastq")
println(type(all_reads))  # List

# Streaming: nothing loaded yet
let stream = fastq("data/reads.fastq")
println(type(stream))  # Stream

# Streams are lazy — nothing happens until you consume
let count = stream |> count()
println(f"Reads: {count}")

List
Stream
Reads: 500

The key rule: streams can only be consumed once. Once you have iterated through a stream, the data is gone. You cannot rewind. If you need multiple passes over the same file, create a new stream each time.

let s = fastq("data/reads.fastq")
let n = s |> count()    # consumes the stream
# let m = s |> count()  # ERROR: stream already exhausted

This is not a limitation — it is the reason streaming works. If you could rewind, the system would need to keep all the data in memory or re-read the file from scratch. The one-pass constraint is what guarantees constant memory.

Stream Operations

Stream operations are lazy: they build up a processing pipeline without moving any data. Data only flows when you call a terminal operation like count(), collect(), or reduce().

Orange is the source. Green boxes are lazy transformations — each returns a new stream. Blue boxes are terminal operations that trigger data flow.

Lazy operations (return streams)

Terminal operations (consume the stream)

Here is a complete lazy pipeline:

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# This builds a pipeline — no data moves yet
let pipeline = fastq("data/reads.fastq")
    |> filter(|r| mean_phred(r.qual) >= 30)
    |> map(|r| {id: r.id, gc: gc_content(r.seq), length: len(r.seq)})
    |> take(1000)

# NOW data flows — only when you collect
let results = pipeline |> collect()
println(f"Got {len(results)} high-quality reads")

Got 170 high-quality reads

The fastq() call opens the file but reads nothing. The filter() call attaches a predicate but reads nothing. The map() call attaches a transformation but reads nothing. The take(1000) call sets a limit but reads nothing. Only when collect() runs does data actually flow through the pipeline, one record at a time.

Constant-Memory Patterns

These five patterns cover the vast majority of large-file processing tasks in bioinformatics. Each uses constant memory regardless of input size.

Pattern 1: Count without loading

The simplest streaming operation. Count the records in a file without loading any of them.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Count reads in a large file using ~10 MB of RAM
let total = fastq("data/reads.fastq") |> count()
println(f"Total reads: {total}")

Total reads: 500

Pattern 2: Filter and count

Apply a quality filter and count how many records pass, without storing any of them.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# How many reads pass quality filter?
let passed = fastq("data/reads.fastq")
    |> filter(|r| mean_phred(r.qual) >= 20)
    |> count()
println(f"Passed Q20: {passed}")

Passed Q20: 392

Pattern 3: Reduce to a single value

Combine all records into a single summary value. The reduce() function maintains a running accumulator, so only two values are ever in memory at once.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Calculate mean GC content without loading all reads
let result = fastq("data/reads.fastq")
    |> map(|r| {gc: gc_content(r.seq), n: 1})
    |> reduce(|a, b| {gc: a.gc + b.gc, n: a.n + b.n})
let mean_gc = result.gc / result.n
println(f"Mean GC: {round(mean_gc * 100, 1)}%")

Mean GC: 49.8%

Pattern 4: Take a sample

Peek at the first few records to verify file contents without reading the entire file. The stream stops after take(n) records.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Peek at the first 5 reads
let sample = fastq("data/reads.fastq") |> take(5) |> collect()
for r in sample {
    println(f"{r.id}: {len(r.seq)} bp, Q={round(mean_phred(r.qual), 1)}")
}

read_0001: 150 bp, Q=33.2
read_0002: 148 bp, Q=27.1
read_0003: 150 bp, Q=35.0
read_0004: 145 bp, Q=22.8
read_0005: 150 bp, Q=31.4

Pattern 5: Stream, filter, write

Read from one file, filter, and write to another. The entire operation uses constant memory — the input and output files can be any size.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Filter a FASTQ to keep only high-quality, long reads
# Memory: constant regardless of input size
let filtered = fastq("data/reads.fastq")
    |> filter(|r| len(r.seq) >= 100 and mean_phred(r.qual) >= 25)
    |> collect()
write_fastq(filtered, "results/filtered.fastq")
println(f"Wrote {len(filtered)} filtered reads")

Wrote 264 filtered reads

Chunked Processing

Some operations need groups of records rather than individual ones — for example, computing statistics on batches. The stream_chunks() function groups a stream into fixed-size chunks.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Process reads in chunks of 100
let stream = fastq("data/reads.fastq")
let chunks = stream_chunks(stream, 100)

let batch_num = 0
for chunk in chunks {
    batch_num = batch_num + 1
    let gc_vals = chunk |> map(|r| gc_content(r.seq))
    let mean_gc = mean(gc_vals)
    println(f"Batch {batch_num}: {len(chunk)} reads, mean GC: {round(mean_gc * 100, 1)}%")
}

Batch 1: 100 reads, mean GC: 50.2%
Batch 2: 100 reads, mean GC: 49.5%
Batch 3: 100 reads, mean GC: 49.9%
Batch 4: 100 reads, mean GC: 50.1%
Batch 5: 100 reads, mean GC: 49.3%

Each chunk is a list of records small enough to fit in memory. The stream reads only one chunk at a time, so memory usage stays bounded by the chunk size rather than the file size.

Streaming All Formats

Every BioLang file reader has a streaming counterpart. Here are examples for each format.

FASTA streaming

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Find the sequence with the highest GC content
let gc_stats = fasta("data/sequences.fasta")
    |> map(|s| {id: s.id, gc: gc_content(s.seq)})
    |> collect()
let gc_sorted = gc_stats |> sort_by(|s| s.gc)
let highest = gc_sorted |> last()
println(f"Highest GC: {highest.id} at {round(highest.gc * 100, 1)}%")

Highest GC: ecoli_16s at 54.2%

VCF streaming

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Count variants per chromosome, PASS only
let chr_counts = vcf("data/variants.vcf")
    |> filter(|v| v.filter == "PASS")
    |> map(|v| v.chrom)
    |> frequencies()
println(chr_counts)

{chr1: 3, chr17: 3, chrX: 2}

BED streaming

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Total bases covered by all regions
let total_bp = bed("data/regions.bed")
    |> map(|r| r.end - r.start)
    |> reduce(|a, b| a + b)
println(f"Total covered: {total_bp} bp")

Total covered: 92500 bp

BAM streaming

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Count mapped reads
let mapped = bam("data/alignments.bam")
    |> filter(|r| r.is_mapped)
    |> count()
println(f"Mapped reads: {mapped}")

Mapped reads: 17

GFF streaming

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Count exons
let exon_count = gff("data/annotations.gff")
    |> filter(|f| f.type == "exon")
    |> count()
println(f"Exons: {exon_count}")

Exons: 8

Every format follows the same pattern: open a stream, chain lazy operations, terminate with a consumer. Once you learn the pattern for one format, you know it for all of them.

The tee Pattern: Inspect Without Consuming

Sometimes you want to see what is flowing through a pipeline without changing it. The tee() function calls a function on each record for its side effect (typically printing) and passes the record through unchanged.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# tee lets you peek at data as it flows through
let high_q = fastq("data/reads.fastq")
    |> tee(|r| println(f"Checking: {r.id}"))
    |> filter(|r| mean_phred(r.qual) >= 30)
    |> take(3)
    |> collect()
println(f"\nKept {len(high_q)} reads")

Checking: read_0001
Checking: read_0002
Checking: read_0003
Checking: read_0004
Checking: read_0005
Checking: read_0006

Kept 3 reads

Notice that tee() printed six read IDs but only three passed the filter. The stream stopped early because take(3) was satisfied — the file was not read to the end.

This is extremely useful for debugging pipelines. If your filter is producing zero results, add a tee() before the filter to see what records actually look like.

Memory Comparison

Here is why streaming matters, with concrete numbers:

The eager approach scales linearly with file size. The streaming approach stays constant.

Streaming is not just about memory. It is also faster because there is no allocation overhead for storing millions of records in a list. The records are processed and discarded immediately.

Rule of thumb: use eager (read_fastq()) for files under 100 MB. Use streaming (fastq()) for anything larger. When in doubt, stream.

Complete Example: Streaming QC Report

This script generates a quality report for a FASTQ file using streaming. Each pass through the file creates a new stream. Memory usage stays constant at ~20 MB regardless of file size.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Generate QC report for a FASTQ file
# Memory usage: constant ~20 MB regardless of file size
# requires: data/reads.fastq in working directory

println("=== Streaming QC Report ===")
println("")

# Pass 1: Basic counts using read_stats
let stats = read_stats("data/reads.fastq")
println(f"Total reads: {stats.total_reads}")
println(f"Total bases: {stats.total_bases}")

# Pass 2: Quality distribution (stream again — each pass is a new stream)
let quality_bins = fastq("data/reads.fastq")
    |> map(|r| {
        q: mean_phred(r.qual),
        category: if mean_phred(r.qual) >= 30 { "excellent" }
                  else if mean_phred(r.qual) >= 20 { "good" }
                  else { "poor" }
    })
    |> map(|r| r.category)
    |> frequencies()

println("")
println("Quality distribution:")
for category in keys(quality_bins) {
    println(f"  {category}: {quality_bins[category]}")
}

# Pass 3: Length distribution
let lengths = fastq("data/reads.fastq")
    |> map(|r| len(r.seq))
    |> collect()

println("")
println("Length stats:")
println(f"  Mean: {round(mean(lengths), 1)}")
println(f"  Min: {min(lengths)}")
println(f"  Max: {max(lengths)}")

# Pass 4: Filtered output
let filtered = fastq("data/reads.fastq")
    |> filter(|r| len(r.seq) >= 100 and mean_phred(r.qual) >= 25)
    |> collect()
write_fastq(filtered, "results/filtered.fastq")

println("")
println(f"Filtered reads written: {len(filtered)}")
println("")
println("=== Report complete ===")

=== Streaming QC Report ===

Total reads: 500
Total bases: 73750

Quality distribution:
  excellent: 170
  good: 222
  poor: 108

Length stats:
  Mean: 147.5
  Min: 100
  Max: 150

Filtered reads written: 264

=== Report complete ===

Each of the four passes creates a fresh stream from the file. The file is read four times, but each pass uses only ~10 MB of memory. For a 100 GB file, this script would use 20 MB of RAM and take about 40 minutes — but it would finish, while an eager approach would crash.

Exercises

1. Count total bases in a FASTQ file using streaming. Hint: map each read to its sequence length, then reduce by summing.

2. Find the read with the highest mean quality using streaming. Hint: use reduce() with a comparator that keeps the better record.

3. Batch statistics — use stream_chunks() to process a FASTQ in batches of 50 and print per-batch mean read length and quality.

4. Stream a VCF and count how many variants are SNPs (same length ref and alt) vs indels (different length).

5. FASTA length filter — write a streaming pipeline that reads a FASTA file, filters to sequences longer than 100 bp, and writes the results to a new file.

Key Takeaways

• Streams process data one record at a time — constant memory regardless of file size.
• fastq(), fasta(), vcf(), bed(), gff(), bam() all return streams.
• Streams are lazy — nothing happens until you consume with count(), collect(), or reduce().
• Streams can only be consumed once — create a new stream for each pass over the file.
• Use collect() only when you need all data in memory; prefer count(), reduce(), or stream-to-file.
• stream_chunks() groups records for batch processing when you need per-group statistics.
• Rule of thumb: eager for files under 100 MB, streaming for anything larger.

What’s Next

Tomorrow we connect to the outside world. Day 9: Biological Databases and APIs — looking up what the world already knows about your genes, proteins, and variants.

Day 9: Biological Databases and APIs

The Problem

You found a mutation in gene BRCA1. What does this gene do? Is this mutation known? What pathway is it in? What protein does it encode? What other proteins does it interact with? What 3D structures are available?

This information exists — scattered across a dozen databases maintained by organizations around the world. NCBI in Bethesda, EBI in Cambridge, KEGG in Kyoto, RCSB in New Jersey. Manually searching each one, copying identifiers between browser tabs, cross-referencing results — it takes hours for a single gene. For a list of 50 candidate genes from a screen, it takes days.

With API calls, it takes seconds.

BioLang has built-in clients for 12+ biological databases. No packages to install. No authentication boilerplate. No JSON parsing. You call a function, you get structured data back.

The Database Landscape

Biological knowledge is distributed across specialized databases. Each one is the authoritative source for a particular kind of information:

No single database has the complete picture. NCBI has the sequences but not the pathways. KEGG has the pathways but not the 3D structures. PDB has the structures but not the interaction networks. The real power comes from querying multiple databases and combining the results.

NCBI — The Central Repository

The National Center for Biotechnology Information (NCBI) is the largest repository of biological data. It hosts GenBank (sequences), PubMed (literature), Gene (gene records), and dozens of other databases. Nearly every bioinformatician interacts with NCBI daily.

BioLang’s NCBI functions wrap the E-utilities API, handling the XML parsing, rate limiting, and error recovery for you.

Looking Up a Gene

The simplest operation: look up a gene by symbol.

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection
# optional: NCBI_API_KEY for higher rate limits

let gene = ncbi_gene("BRCA1")
println(f"Symbol: {gene.symbol}")
println(f"Name: {gene.name}")
println(f"Description: {gene.description}")
println(f"Chromosome: {gene.chromosome}")
println(f"Location: {gene.location}")
println(f"Organism: {gene.organism}")

Expected output (approximate — NCBI data is updated regularly):

Symbol: BRCA1
Name: BRCA1 DNA repair associated
Description: BRCA1 DNA repair associated
Chromosome: 17
Location: 17q21.31
Organism: Homo sapiens

ncbi_gene() returns a record with fields: id, symbol, name, description, organism, chromosome, location, summary. When the search matches a single gene, you get the full record directly. When it matches multiple genes, you get a list of NCBI Gene IDs.

Searching NCBI Databases

NCBI hosts over 40 databases. You can search any of them with ncbi_search():

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection
# optional: NCBI_API_KEY for higher rate limits

# Search PubMed for articles about BRCA1 and breast cancer
let pubmed_ids = ncbi_search("pubmed", "BRCA1 breast cancer", 5)
println(f"PubMed hits: {len(pubmed_ids)}")
for id in pubmed_ids {
    println(f"  PMID: {id}")
}

# Search the Gene database
let gene_ids = ncbi_search("gene", "TP53 homo sapiens", 5)
println(f"Gene IDs: {len(gene_ids)}")

Note the argument order: ncbi_search(database, query, max_results). The max_results parameter is optional (defaults to 20).

Fetching Sequences

Retrieve a sequence by its accession number:

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection
# optional: NCBI_API_KEY for higher rate limits

# Fetch BRCA1 mRNA sequence (RefSeq accession)
let fasta = ncbi_sequence("NM_007294")
println(f"Sequence (first 100 chars):")
println(fasta |> take(200))

ncbi_sequence() returns the raw FASTA text. You can parse it further or write it to a file.

Ensembl — Gene Models and Variants

Ensembl, maintained by the European Bioinformatics Institute (EBI), provides gene annotations, comparative genomics, and variant effect prediction. Its REST API is particularly well-designed and fast.

Looking Up a Gene by Symbol

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

let gene = ensembl_symbol("homo_sapiens", "BRCA1")
println(f"Ensembl ID: {gene.id}")
println(f"Symbol: {gene.symbol}")
println(f"Biotype: {gene.biotype}")
println(f"Chromosome: {gene.chromosome}")
println(f"Start: {gene.start}")
println(f"End: {gene.end}")
println(f"Strand: {gene.strand}")

Expected output (approximate):

Ensembl ID: ENSG00000012048
Symbol: BRCA1
Biotype: protein_coding
Chromosome: 17
Start: 43044295
End: 43170245
Strand: -1

Note the argument order: ensembl_symbol(species, symbol). Species uses Ensembl’s underscore-separated format: "homo_sapiens", "mus_musculus", "danio_rerio".

Getting Protein Sequences

Once you have an Ensembl gene ID, you can retrieve its sequence in different forms:

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

let gene = ensembl_symbol("homo_sapiens", "BRCA1")

# Get the protein sequence
let protein = ensembl_sequence(gene.id, "protein")
println(f"Protein length: {len(protein.seq)} amino acids")
println(f"First 50 aa: {protein.seq |> take(50)}")

# Get the coding sequence (CDS)
let cds = ensembl_sequence(gene.id, "cds")
println(f"CDS length: {len(cds.seq)} bases")

ensembl_sequence() takes an Ensembl ID and an optional sequence type: "genomic" (default), "cds", "cdna", or "protein". It returns a record with id, seq, and molecule fields.

Variant Effect Prediction (VEP)

One of Ensembl’s most powerful features is VEP — the Variant Effect Predictor. Given a variant, it tells you the predicted biological consequence:

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Predict the effect of a BRCA1 variant (HGVS notation)
let results = ensembl_vep("17:g.43091434G>A")
for r in results {
    println(f"Alleles: {r.allele_string}")
    println(f"Most severe: {r.most_severe_consequence}")
    for tc in r.transcript_consequences {
        println(f"  Transcript: {tc.transcript_id}")
        println(f"  Impact: {tc.impact}")
        println(f"  Consequences: {tc.consequences}")
    }
}

VEP accepts HGVS notation (e.g., "17:g.43091434G>A") and returns a list of result records, each containing transcript-level consequence predictions with impact severity (HIGH, MODERATE, LOW, MODIFIER).

UniProt — Protein Knowledge

UniProt is the definitive resource for protein function, domains, post-translational modifications, and literature. Every well-characterized protein has a UniProt entry curated by expert biologists.

Looking Up a Protein

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Look up BRCA1 by its UniProt accession
let entry = uniprot_entry("P38398")
println(f"Name: {entry.name}")
println(f"Organism: {entry.organism}")
println(f"Length: {entry.sequence_length} aa")
println(f"Gene names: {entry.gene_names}")
println(f"Function: {entry.function}")

Expected output (approximate):

Name: BRCA1_HUMAN
Organism: Homo sapiens (Human)
Length: 1863 aa
Gene names: ["BRCA1", "RNF53"]
Function: E3 ubiquitin-protein ligase that...

uniprot_entry() returns a record with accession, name, organism, sequence_length, gene_names (a list), and function.

Searching UniProt

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Search for human BRCA1 proteins
let results = uniprot_search("BRCA1 AND organism_name:human", 5)
println(f"Results: {len(results)}")
for entry in results {
    println(f"  {entry.accession}: {entry.name} ({entry.sequence_length} aa)")
}

uniprot_search() takes a query string (using UniProt’s query syntax) and an optional limit (defaults to 10). It returns a list of protein entry records.

Protein Features and Domains

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Get structural and functional features of BRCA1
let features = uniprot_features("P38398")
println(f"Total features: {len(features)}")

# Find just the domains
let domains = features |> filter(|f| f.type == "Domain")
println(f"Domains: {len(domains)}")
for d in domains {
    println(f"  {d.description} ({d.location})")
}

# Find binding sites
let sites = features |> filter(|f| f.type == "Binding site")
println(f"Binding sites: {len(sites)}")

Each feature record has type, location, and description fields. Common types include "Domain", "Region", "Binding site", "Modified residue", "Disulfide bond", and "Chain".

Gene Ontology Terms from UniProt

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Get GO terms associated with BRCA1
let go_terms = uniprot_go("P38398")
println(f"GO terms: {len(go_terms)}")
for t in go_terms |> take(5) {
    println(f"  {t.id}: {t.term} ({t.aspect})")
}

KEGG — Pathways and Metabolism

The Kyoto Encyclopedia of Genes and Genomes links genes to metabolic and signaling pathways. It is especially valuable for understanding how individual genes fit into larger biological systems.

Finding Genes in KEGG

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Find BRCA1 in the KEGG database
let results = kegg_find("genes", "BRCA1")
println(f"KEGG hits: {len(results)}")
for r in results |> take(5) {
    println(f"  {r.id}: {r.description}")
}

kegg_find() takes a database name and a query string. The database can be "genes", "pathway", "compound", "disease", "drug", and more. It returns a list of records with id and description.

Getting Detailed Entries

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Get detailed entry for human BRCA1
let entry = kegg_get("hsa:672")
println(f"KEGG entry (first 500 chars):")
println(entry |> take(500))

kegg_get() returns the raw KEGG flat-file text for any KEGG identifier. KEGG IDs use an organism prefix: hsa for Homo sapiens, mmu for Mus musculus, etc.

Linking to Pathways

The real power of KEGG is connecting genes to pathways:

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Find pathways that BRCA1 participates in
let links = kegg_link("pathway", "hsa:672")
println(f"Pathways involving BRCA1: {len(links)}")
for link in links {
    println(f"  {link.source} -> {link.target}")
}

kegg_link() takes two arguments: target database and source identifier. It returns a list of records with source and target fields.

PDB — 3D Protein Structures

The Protein Data Bank (PDB) contains experimentally determined 3D structures of proteins, nucleic acids, and their complexes. If you want to see what a protein actually looks like, this is where you go.

Looking Up a Structure

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Get information about BRCA1 BRCT domain structure
let structure = pdb_entry("1JM7")
println(f"Title: {structure.title}")
println(f"Method: {structure.method}")
println(f"Resolution: {structure.resolution}")
println(f"Release date: {structure.release_date}")
println(f"Organism: {structure.organism}")

Expected output (approximate):

Title: Crystal structure of the BRCT repeat region from...
Method: X-RAY DIFFRACTION
Resolution: 2.5
Release date: 2001-07-06
Organism: Homo sapiens

pdb_entry() returns a record with id, title, method, resolution (may be nil for NMR structures), release_date, and organism.

Searching for Structures

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Find all PDB structures related to BRCA1
let pdb_ids = pdb_search("BRCA1")
println(f"PDB structures for BRCA1: {len(pdb_ids)}")
for id in pdb_ids |> take(10) {
    println(f"  {id}")
}

pdb_search() returns a list of PDB ID strings.

Getting Entity and Sequence Information

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Get entity details for a specific chain
let entity = pdb_entity("1JM7", 1)
println(f"Entity type: {entity.entity_type}")
println(f"Description: {entity.description}")

# Get the protein sequence from the structure
let seq = pdb_sequence("1JM7", 1)
println(f"Sequence: {seq}")
println(f"Length: {len(seq)} aa")

STRING — Protein Interactions

STRING (Search Tool for Recurring Instances of Neighbouring Genes) maps known and predicted protein-protein interactions. Understanding which proteins interact is crucial for interpreting experimental results.

Getting an Interaction Network

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Get interaction partners for BRCA1
# string_network takes a list of protein identifiers and a species taxonomy ID
let network = string_network(["BRCA1"], 9606)
println(f"Interaction partners: {len(network)}")

# Show top interactors by score
let top = network
    |> sort_by(|n| n.score)
    |> reverse()
    |> take(5)

for partner in top {
    println(f"  {partner.protein_a} <-> {partner.protein_b}: score={partner.score}")
}

Note that string_network() takes a list of protein identifiers (not a single string) and a species taxonomy ID. Common taxonomy IDs: 9606 (human), 10090 (mouse), 7955 (zebrafish), 6239 (C. elegans), 7227 (D. melanogaster).

Each interaction record has protein_a, protein_b, and score fields. The score ranges from 0 to 1, where higher scores indicate more confident interactions.

Functional Enrichment

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Check if a set of genes is enriched for specific functions
let enrichment = string_enrichment(["BRCA1", "BRCA2", "RAD51", "TP53", "ATM"], 9606)
println(f"Enriched terms: {len(enrichment)}")
for e in enrichment |> take(5) {
    println(f"  [{e.category}] {e.description}: p={e.p_value}, FDR={e.fdr}")
}

string_enrichment() takes a list of gene symbols and a species taxonomy ID. It returns a list of enrichment records with category, term, description, gene_count, p_value, and fdr.

Gene Ontology and Reactome

Gene Ontology (GO)

The Gene Ontology provides a standardized vocabulary for describing gene function across all organisms. Every GO term belongs to one of three namespaces:

• Molecular Function — what the protein does (e.g., “kinase activity”)
• Biological Process — what pathway it participates in (e.g., “DNA repair”)
• Cellular Component — where in the cell it acts (e.g., “nucleus”)

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Look up a specific GO term
let term = go_term("GO:0006281")
println(f"ID: {term.id}")
println(f"Name: {term.name}")
println(f"Aspect: {term.aspect}")
println(f"Definition: {term.definition}")

Expected output:

ID: GO:0006281
Name: DNA repair
Aspect: biological_process
Definition: The process of restoring DNA after damage...

GO Annotations for a Gene

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Get GO annotations for BRCA1 (by UniProt accession)
let annotations = go_annotations("P38398")
println(f"GO annotations: {len(annotations)}")
for a in annotations |> take(5) {
    println(f"  {a.go_id}: {a.go_name} ({a.aspect})")
    println(f"    Evidence: {a.evidence}")
}

go_annotations() takes a gene/protein identifier and an optional limit (defaults to 25). Each annotation has go_id, go_name, aspect, evidence, and gene_product_id fields.

Navigating the GO Hierarchy

GO terms form a directed acyclic graph (DAG). You can traverse it:

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Find child terms of "DNA repair"
let children = go_children("GO:0006281")
println(f"Child terms of DNA repair: {len(children)}")
for c in children |> take(5) {
    println(f"  {c.id}: {c.name}")
}

# Find parent terms
let parents = go_parents("GO:0006281")
println(f"Parent terms: {len(parents)}")
for p in parents {
    println(f"  {p.id}: {p.name}")
}

Reactome — Biological Pathways

Reactome is a curated database of biological pathways and reactions, maintained by EBI and the Ontario Institute for Cancer Research.

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

# Find pathways involving BRCA1
let pathways = reactome_pathways("BRCA1")
println(f"Reactome pathways: {len(pathways)}")
for p in pathways |> take(5) {
    println(f"  {p.id}: {p.name} ({p.species})")
}

reactome_pathways() takes a gene symbol and an optional species (defaults to "Homo sapiens"). It returns a list of pathway records with id, name, and species.

You can also search Reactome by keyword:

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection

let results = reactome_search("DNA damage response")
println(f"Search results: {len(results)}")

Combining Multiple Databases

The real power of programmatic database access is cross-referencing. A single gene symbol unlocks information across every database simultaneously. What would take 30 minutes of browser-tab switching takes 10 lines of code.

A Complete Gene Profile

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection
# optional: NCBI_API_KEY for higher rate limits

fn gene_profile(symbol) {
    println(f"\n{'=' * 50}")
    println(f"  Gene Profile: {symbol}")
    println(f"{'=' * 50}")

    # NCBI: basic gene info
    let gene = ncbi_gene(symbol)
    println(f"\n[NCBI Gene]")
    println(f"  Description: {gene.description}")
    println(f"  Chromosome: {gene.chromosome}")
    println(f"  Location: {gene.location}")

    # Ensembl: genomic coordinates
    let ens = ensembl_symbol("homo_sapiens", symbol)
    println(f"\n[Ensembl]")
    println(f"  ID: {ens.id}")
    println(f"  Biotype: {ens.biotype}")
    println(f"  Position: chr{ens.chromosome}:{ens.start}-{ens.end}")

    # Ensembl: protein sequence
    let protein = ensembl_sequence(ens.id, "protein")
    println(f"  Protein: {len(protein.seq)} amino acids")

    # UniProt: function
    let results = uniprot_search(f"{symbol} AND organism_name:human", 1)
    if len(results) > 0 {
        let entry = results |> first()
        println(f"\n[UniProt]")
        println(f"  Accession: {entry.accession}")
        println(f"  Name: {entry.name}")
        println(f"  Function: {entry.function}")
    }

    # STRING: interactions
    let network = string_network([symbol], 9606)
    println(f"\n[STRING]")
    println(f"  Interaction partners: {len(network)}")
    let top3 = network
        |> sort_by(|n| n.score)
        |> reverse()
        |> take(3)
    for partner in top3 {
        println(f"    {partner.protein_b}: {partner.score}")
    }

    # PDB: structures
    let structures = pdb_search(symbol)
    println(f"\n[PDB]")
    println(f"  Available structures: {len(structures)}")

    # Reactome: pathways
    let pathways = reactome_pathways(symbol)
    println(f"\n[Reactome]")
    println(f"  Pathways: {len(pathways)}")
    for p in pathways |> take(3) {
        println(f"    {p.name}")
    }

    sleep(1)  # respect rate limits between genes
}

Profiling Multiple Genes

# requires: internet connection
# optional: NCBI_API_KEY for higher rate limits

# Profile a set of cancer-related genes
let cancer_genes = ["BRCA1", "TP53", "EGFR"]
for gene in cancer_genes {
    gene_profile(gene)
}

This is the kind of analysis that is impractical to do manually but trivial with API calls. Three genes, six databases each, complete profiles in under a minute.

Building a Comparison Table

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection
# optional: NCBI_API_KEY for higher rate limits

# Collect structured data for comparison
let genes = ["BRCA1", "TP53", "EGFR", "KRAS", "MYC"]
let rows = []
for symbol in genes {
    let gene = ncbi_gene(symbol)
    let ens = ensembl_symbol("homo_sapiens", symbol)
    let protein = ensembl_sequence(ens.id, "protein")
    let network = string_network([symbol], 9606)
    let pathways = reactome_pathways(symbol)

    rows = push(rows, {
        gene: symbol,
        chromosome: gene.chromosome,
        protein_length: len(protein.seq),
        interactions: len(network),
        pathways: len(pathways)
    })

    sleep(0.5)  # be respectful
}

let results = rows |> to_table()
println(results)

Expected output (approximate):

gene   | chromosome | protein_length | interactions | pathways
-------|------------|----------------|--------------|--------
BRCA1  | 17         | 1863           | 10           | 25
TP53   | 17         | 393            | 10           | 18
EGFR   | 7          | 1210           | 10           | 30
KRAS   | 12         | 189            | 10           | 22
MYC    | 8          | 439            | 10           | 15

Rate Limiting and Best Practices

Biological databases are shared public resources. Hammering them with thousands of requests per second will get your IP temporarily blocked — and slow down the service for everyone.

Rate Limits by Database

Setting Up API Keys

NCBI strongly recommends registering for an API key. It is free and takes 30 seconds:

1. Go to ncbi.nlm.nih.gov/account/settings
2. Click “Create an API Key”
3. Set the environment variable:

export NCBI_API_KEY="your_key_here"

BioLang automatically detects and uses the NCBI_API_KEY environment variable for all NCBI calls.

Batch Queries with Rate Limiting

When querying multiple genes, add delays between requests:

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection
# optional: NCBI_API_KEY for higher rate limits

let genes = ["BRCA1", "TP53", "EGFR", "KRAS", "MYC",
             "PIK3CA", "BRAF", "APC", "RB1", "PTEN"]

let results = []
for gene in genes {
    let info = ncbi_gene(gene)
    results = push(results, {gene: gene, chrom: info.chromosome, desc: info.description})
    sleep(0.5)  # be respectful
}

let results_table = results |> to_table()
println(results_table)

Best Practices

1. Cache results — if you are going to query the same gene repeatedly during development, save the result to a variable or file instead of calling the API each time.

2. Use sleep() in loops — add at least 0.3–0.5 seconds between requests when iterating over a list of genes.

3. Handle errors gracefully — API calls can fail due to network issues, maintenance windows, or invalid identifiers. Use try/catch for production scripts.

4. Start small — test your query with 2–3 genes before running it on 500.

5. Set NCBI_API_KEY — it is free and triples your rate limit.

Requires CLI: This example uses network APIs not available in the browser. Run with bl run.

# requires: internet connection
# optional: NCBI_API_KEY for higher rate limits

# Robust batch query with error handling
let genes = ["BRCA1", "TP53", "INVALID_GENE", "EGFR"]
let results = []
let errors = []

for gene in genes {
    let result = try {
        let info = ncbi_gene(gene)
        push(results, {gene: gene, chrom: info.chromosome})
    } catch e {
        push(errors, {gene: gene, error: e})
    }
    sleep(0.5)
}

println(f"Successful: {len(results)}")
println(f"Failed: {len(errors)}")
for err in errors {
    println(f"  {err.gene}: {err.error}")
}

Exercises

1. Gene Lookup: Look up your favorite gene in NCBI using ncbi_gene() and print its chromosome location, description, and summary. Try at least two different genes.

2. Protein Size Estimation: Use ensembl_symbol() and ensembl_sequence() to get the protein sequence of TP53. Calculate its length and estimate its molecular weight (average amino acid weight is approximately 110 daltons).

3. UniProt Search: Search UniProt for "insulin AND organism_name:human" and list the accession numbers and names of the results.

4. Interaction Network: Use string_network() to find interaction partners for MYC (species 9606). Sort by score and print the top 5.

5. Multi-Database Report: Write a gene_report(symbol) function that queries at least 3 databases (NCBI, Ensembl, and one other) and returns a summary record with fields like chromosome, protein_length, num_interactions, and num_pathways. Test it on EGFR and KRAS.

Key Takeaways

• BioLang has built-in clients for 12+ biological databases — no packages to install, no JSON to parse.

• NCBI is the central repository for sequences, genes, and literature. ncbi_gene() is often your starting point.

• Ensembl provides gene models, coordinates, and the invaluable Variant Effect Predictor (ensembl_vep()).

• UniProt is the authoritative source for protein function, domains, and curated annotations.

• KEGG connects genes to metabolic and signaling pathways. Use kegg_link() to find pathway memberships.

• PDB gives you 3D protein structures. STRING maps protein-protein interaction networks.

• GO and Reactome provide functional annotations and biological pathway context.

• Combining databases gives a complete picture no single source provides. A 10-line function can profile a gene across six databases.

• Respect rate limits: use sleep() in batch queries, set NCBI_API_KEY for NCBI, and cache results when possible.

• All API functions require internet access. Some need API keys: NCBI (optional, recommended), COSMIC (required).

What’s Next

Tomorrow we move from fetching data to organizing it. Day 10: Tables — The Bioinformatician’s Workbench covers selecting, filtering, joining, and reshaping tabular data — the format that most bioinformatics analysis ultimately lives in.

Day 10: Tables — The Bioinformatician’s Workbench

What You’ll Learn

• How to create tables from CSV files, records, and column vectors
• How to select, drop, and rename columns
• How to filter rows with predicates
• How to add and transform columns with mutate
• How to sort, slice, and deduplicate rows
• How to group rows and compute summaries (split-apply-combine)
• How to join tables by key columns (inner, left, right, outer, anti, semi)
• How to reshape between wide and long formats (pivot)
• How to use window functions for running totals and ranks
• How to chain all of these into a complete analysis pipeline

The Problem

Every analysis produces tabular data — gene expression matrices, variant call results, sample metadata, statistical summaries. A differential expression tool gives you thousands of rows with gene names, fold changes, and p-values. A variant caller gives you chromosomes, positions, and quality scores. A clinical database gives you patient IDs, phenotypes, and treatment groups.

Knowing how to slice, dice, join, reshape, and summarize tables is the single most valuable data skill in bioinformatics. It is the skill that turns raw output into biological insight.

In R, this is dplyr and tidyr. In Python, this is pandas. In BioLang, tables are built in — no imports, no package managers, no configuration. You load a CSV and start working.

Creating Tables

There are three ways to get data into a table.

From CSV/TSV Files

The most common case: you have a file from another tool.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

let expr = csv("data/expression.csv")
println(f"Rows: {nrow(expr)}, Cols: {ncol(expr)}")
println(f"Columns: {colnames(expr)}")
println(expr |> head(3))

Expected output:

Rows: 20, Cols: 6
Columns: [gene, log2fc, pval, padj, chr, biotype]
gene   | log2fc | pval     | padj     | chr | biotype
EGFR   | 3.8    | 0.000001 | 0.00001  | 7   | protein_coding
BRCA1  | 2.4    | 0.001    | 0.005    | 17  | protein_coding
VEGFA  | 2.1    | 0.002    | 0.008    | 6   | protein_coding

csv() reads comma-separated files. For tab-separated files, use tsv(). Both auto-detect headers and infer column types (integers, floats, strings).

From a List of Records

When you construct data programmatically, build a list of records and convert it.

let data = [
    {gene: "BRCA1", log2fc: 2.4, pval: 0.001, chr: "17"},
    {gene: "TP53", log2fc: -1.1, pval: 0.23, chr: "17"},
    {gene: "EGFR", log2fc: 3.8, pval: 0.000001, chr: "7"},
    {gene: "MYC", log2fc: 1.9, pval: 0.04, chr: "8"},
    {gene: "KRAS", log2fc: -0.3, pval: 0.67, chr: "12"},
] |> to_table()

println(data)

Expected output:

gene  | log2fc | pval     | chr
BRCA1 | 2.4    | 0.001    | 17
TP53  | -1.1   | 0.23     | 17
EGFR  | 3.8    | 0.000001 | 7
MYC   | 1.9    | 0.04     | 8
KRAS  | -0.3   | 0.67     | 12

From Column Vectors

When you already have parallel arrays, pass a record of lists.

let t = table({
    gene: ["BRCA1", "TP53", "EGFR"],
    value: [1.0, 2.0, 3.0]
})
println(t)

Expected output:

gene  | value
BRCA1 | 1.0
TP53  | 2.0
EGFR  | 3.0

This is the Polars/R column-oriented style. Each key is a column name, each value is a list of that column’s data. All lists must have the same length.

Selecting Columns

Tables often have more columns than you need. select() keeps only the ones you name. drop_cols() removes the ones you don’t want.

let data = [
    {gene: "BRCA1", log2fc: 2.4, pval: 0.001, chr: "17"},
    {gene: "TP53", log2fc: -1.1, pval: 0.23, chr: "17"},
    {gene: "EGFR", log2fc: 3.8, pval: 0.000001, chr: "7"},
] |> to_table()

# Keep specific columns
let slim = data |> select("gene", "pval")
println(slim)

Expected output:

gene  | pval
BRCA1 | 0.001
TP53  | 0.23
EGFR  | 0.000001

# Drop a column
let no_chr = data |> drop_cols("chr")
println(no_chr)

Expected output:

gene  | log2fc | pval
BRCA1 | 2.4    | 0.001
TP53  | -1.1   | 0.23
EGFR  | 3.8    | 0.000001

# Rename a column
let renamed = data |> rename("log2fc", "fold_change")
println(renamed)

Expected output:

gene  | fold_change | pval     | chr
BRCA1 | 2.4         | 0.001    | 17
TP53  | -1.1        | 0.23     | 17
EGFR  | 3.8         | 0.000001 | 7

select() takes the table as the first argument (piped) and column names as the remaining arguments. rename() takes the old name and the new name.

Filtering Rows

filter() keeps only the rows where a predicate returns true. The predicate receives each row as a record.

let data = [
    {gene: "BRCA1", log2fc: 2.4, pval: 0.001, chr: "17"},
    {gene: "TP53", log2fc: -1.1, pval: 0.23, chr: "17"},
    {gene: "EGFR", log2fc: 3.8, pval: 0.000001, chr: "7"},
    {gene: "MYC", log2fc: 1.9, pval: 0.04, chr: "8"},
    {gene: "KRAS", log2fc: -0.3, pval: 0.67, chr: "12"},
] |> to_table()

# Single condition: significant genes
let sig = data |> filter(|r| r.pval < 0.05)
println(sig)

Expected output:

gene  | log2fc | pval     | chr
BRCA1 | 2.4    | 0.001    | 17
EGFR  | 3.8    | 0.000001 | 7
MYC   | 1.9    | 0.04     | 8

# Multiple conditions: significant AND upregulated
let sig_up = data |> filter(|r| r.pval < 0.05 and r.log2fc > 1.0)
println(sig_up)

Expected output:

gene  | log2fc | pval     | chr
BRCA1 | 2.4    | 0.001    | 17
EGFR  | 3.8    | 0.000001 | 7
MYC   | 1.9    | 0.04     | 8

# Filter by category
let chr17 = data |> filter(|r| r.chr == "17")
println(chr17)

Expected output:

gene  | log2fc | pval  | chr
BRCA1 | 2.4    | 0.001 | 17
TP53  | -1.1   | 0.23  | 17

You can combine conditions with and and or. Parentheses clarify precedence when mixing them:

# Chromosome 17 OR very significant
let subset = data |> filter(|r| r.chr == "17" or r.pval < 0.001)
println(subset)

Expected output:

gene  | log2fc | pval     | chr
BRCA1 | 2.4    | 0.001    | 17
TP53  | -1.1   | 0.23     | 17
EGFR  | 3.8    | 0.000001 | 7

Mutating: Adding and Transforming Columns

mutate() adds a new column (or replaces an existing one) by applying a function to each row. It takes three arguments: the table, the new column name, and a closure that receives each row as a record.

let data = [
    {gene: "BRCA1", log2fc: 2.4, pval: 0.001},
    {gene: "TP53", log2fc: -1.1, pval: 0.23},
    {gene: "EGFR", log2fc: 3.8, pval: 0.000001},
    {gene: "MYC", log2fc: 1.9, pval: 0.04},
    {gene: "KRAS", log2fc: -0.3, pval: 0.67},
] |> to_table()

# Add a significance flag
let with_sig = data |> mutate("significant", |r| r.pval < 0.05)
println(with_sig)

Expected output:

gene  | log2fc | pval     | significant
BRCA1 | 2.4    | 0.001    | true
TP53  | -1.1   | 0.23     | false
EGFR  | 3.8    | 0.000001 | true
MYC   | 1.9    | 0.04     | true
KRAS  | -0.3   | 0.67     | false

# Add a direction column
let with_dir = data |> mutate("direction", |r| if r.log2fc > 0 { "up" } else { "down" })
println(with_dir)

Expected output:

gene  | log2fc | pval     | direction
BRCA1 | 2.4    | 0.001    | up
TP53  | -1.1   | 0.23     | down
EGFR  | 3.8    | 0.000001 | up
MYC   | 1.9    | 0.04     | up
KRAS  | -0.3   | 0.67     | down

# Add a negative log10 p-value (common for volcano plots)
let with_nlp = data |> mutate("neg_log_p", |r| -1.0 * log10(r.pval))
println(with_nlp)

Expected output:

gene  | log2fc | pval     | neg_log_p
BRCA1 | 2.4    | 0.001    | 3.0
TP53  | -1.1   | 0.23     | 0.638...
EGFR  | 3.8    | 0.000001 | 6.0
MYC   | 1.9    | 0.04     | 1.397...
KRAS  | -0.3   | 0.67     | 0.173...

To add multiple columns, chain mutate() calls:

let enriched = data
    |> mutate("significant", |r| r.pval < 0.05)
    |> mutate("direction", |r| if r.log2fc > 0 { "up" } else { "down" })
    |> mutate("neg_log_p", |r| -1.0 * log10(r.pval))
println(enriched)

Each mutate() adds one column. The pipe chains them together so the result flows naturally.

Sorting

arrange() sorts a table by a column in ascending order. For descending order, pipe through reverse().

let data = [
    {gene: "BRCA1", log2fc: 2.4, pval: 0.001},
    {gene: "TP53", log2fc: -1.1, pval: 0.23},
    {gene: "EGFR", log2fc: 3.8, pval: 0.000001},
    {gene: "MYC", log2fc: 1.9, pval: 0.04},
] |> to_table()

# Sort by p-value (ascending --- most significant first)
let by_pval = data |> arrange("pval")
println(by_pval)

Expected output:

gene  | log2fc | pval
EGFR  | 3.8    | 0.000001
BRCA1 | 2.4    | 0.001
MYC   | 1.9    | 0.04
TP53  | -1.1   | 0.23

# Sort by fold change descending (largest first)
let by_fc_desc = data |> arrange("log2fc") |> reverse()
println(by_fc_desc)

Expected output:

gene  | log2fc | pval
EGFR  | 3.8    | 0.000001
BRCA1 | 2.4    | 0.001
MYC   | 1.9    | 0.04
TP53  | -1.1   | 0.23

Combine with head() to get top-N results:

# Top 2 most significant genes
let top2 = data |> arrange("pval") |> head(2)
println(top2)

Expected output:

gene  | log2fc | pval
EGFR  | 3.8    | 0.000001
BRCA1 | 2.4    | 0.001

Grouping and Summarizing

The most powerful table operation is split-apply-combine: split the data into groups, apply an aggregation to each group, and combine the results into a new table.

The Pattern

group_by() splits a table into a map of subtables, keyed by the distinct values in the grouping column. summarize() then takes that map and a function that receives each key and subtable, and must return a record. The records are assembled into a new table.

let data = [
    {gene: "BRCA1", log2fc: 2.4, pval: 0.001, chr: "17"},
    {gene: "TP53", log2fc: -1.1, pval: 0.23, chr: "17"},
    {gene: "EGFR", log2fc: 3.8, pval: 0.000001, chr: "7"},
    {gene: "MYC", log2fc: 1.9, pval: 0.04, chr: "8"},
    {gene: "KRAS", log2fc: -0.3, pval: 0.67, chr: "12"},
] |> to_table()

# Count genes per chromosome
let chr_counts = data
    |> group_by("chr")
    |> summarize(|key, subtable| {
        chr: key,
        gene_count: nrow(subtable)
    })
println(chr_counts)

Expected output:

chr | gene_count
7   | 1
8   | 1
12  | 1
17  | 2

# Mean fold change per chromosome
let chr_means = data
    |> group_by("chr")
    |> summarize(|key, subtable| {
        chr: key,
        mean_fc: col_mean(subtable, "log2fc"),
        n_genes: nrow(subtable)
    })
println(chr_means)

Expected output:

chr | mean_fc | n_genes
7   | 3.8     | 1
8   | 1.9     | 1
12  | -0.3    | 1
17  | 0.65    | 2

The summarize function can compute any aggregation you want. Use col_mean(), col_sum(), col_min(), col_max(), col_stdev() for numeric columns, and nrow() for counts.

Quick Counts with count_by

For the common case of just counting groups, count_by() is a shortcut:

let chr_counts = data |> count_by("chr")
println(chr_counts)

Expected output:

chr | count
7   | 1
8   | 1
12  | 1
17  | 2

Joining Tables

Joins connect two tables by matching rows on a shared key column. This is how you annotate results with metadata, link identifiers across databases, or combine measurements from different experiments.

Setting Up Two Tables

let results = [
    {gene: "BRCA1", log2fc: 2.4, pval: 0.001},
    {gene: "TP53", log2fc: -1.1, pval: 0.23},
    {gene: "EGFR", log2fc: 3.8, pval: 0.000001},
    {gene: "MYC", log2fc: 1.9, pval: 0.04},
    {gene: "KRAS", log2fc: -0.3, pval: 0.67},
] |> to_table()

let annotations = [
    {gene: "BRCA1", full_name: "BRCA1 DNA repair", pathway: "DNA repair"},
    {gene: "TP53", full_name: "Tumor protein p53", pathway: "Apoptosis"},
    {gene: "EGFR", full_name: "EGF receptor", pathway: "Signaling"},
    {gene: "MYC", full_name: "MYC proto-oncogene", pathway: "Cell cycle"},
    {gene: "PTEN", full_name: "Phosphatase tensin homolog", pathway: "Signaling"},
] |> to_table()

Note that KRAS is in results but not annotations, and PTEN is in annotations but not results.

Inner Join

Keeps only rows present in both tables.

let annotated = inner_join(results, annotations, "gene")
println(annotated)
println(f"Inner join: {nrow(annotated)} rows")

Expected output:

gene  | log2fc | pval     | full_name         | pathway
BRCA1 | 2.4    | 0.001    | BRCA1 DNA repair  | DNA repair
TP53  | -1.1   | 0.23     | Tumor protein p53 | Apoptosis
EGFR  | 3.8    | 0.000001 | EGF receptor      | Signaling
MYC   | 1.9    | 0.04     | MYC proto-oncogene | Cell cycle
Inner join: 4 rows

KRAS is dropped (no annotation). PTEN is dropped (no result).

Left Join

Keeps all rows from the left table. Where the right table has no match, those columns are nil.

let full = left_join(results, annotations, "gene")
println(full)
println(f"Left join: {nrow(full)} rows")

Expected output:

gene  | log2fc | pval     | full_name         | pathway
BRCA1 | 2.4    | 0.001    | BRCA1 DNA repair  | DNA repair
TP53  | -1.1   | 0.23     | Tumor protein p53 | Apoptosis
EGFR  | 3.8    | 0.000001 | EGF receptor      | Signaling
MYC   | 1.9    | 0.04     | MYC proto-oncogene | Cell cycle
KRAS  | -0.3   | 0.67     | nil               | nil
Left join: 5 rows

KRAS is kept with nil annotations. PTEN is dropped (not in results).

Anti Join

Returns rows from the left table that have no match in the right table. This is the “what’s missing?” join.

let missing = anti_join(results, annotations, "gene")
println(missing)
println(f"Missing annotations: {nrow(missing)} genes")

Expected output:

gene  | log2fc | pval
KRAS  | -0.3   | 0.67
Missing annotations: 1 genes

Semi Join

Returns rows from the left table that do have a match in the right table, but without adding columns from the right table. It is a filter, not a column merger.

let has_annotation = semi_join(results, annotations, "gene")
println(has_annotation)

Expected output:

gene  | log2fc | pval
BRCA1 | 2.4    | 0.001
TP53  | -1.1   | 0.23
EGFR  | 3.8    | 0.000001
MYC   | 1.9    | 0.04

All Join Types at a Glance

inner_join(A, B, key):  A ∩ B     — only matching rows
left_join(A, B, key):   all A     — all of A, matching from B (nil where missing)
right_join(A, B, key):  all B     — all of B, matching from A (nil where missing)
outer_join(A, B, key):  A ∪ B     — all rows from both (nil where missing)
anti_join(A, B, key):   A - B     — rows in A with no match in B
semi_join(A, B, key):   A ∩∃ B    — rows in A that have a match in B (no extra columns)

When to use which:

Reshaping: Pivot Wider and Longer

Biological data comes in two shapes. Wide format has one row per entity (e.g., one row per gene, one column per sample). Long format has one row per measurement (e.g., one row per gene-sample combination).

Long to Wide: pivot_wider

You have expression measurements in long (tidy) format:

let long = [
    {gene: "BRCA1", sample: "S1", expression: 5.2},
    {gene: "BRCA1", sample: "S2", expression: 8.1},
    {gene: "TP53", sample: "S1", expression: 3.4},
    {gene: "TP53", sample: "S2", expression: 7.6},
] |> to_table()

println("Long format:")
println(long)

Expected output:

Long format:
gene  | sample | expression
BRCA1 | S1     | 5.2
BRCA1 | S2     | 8.1
TP53  | S1     | 3.4
TP53  | S2     | 7.6

pivot_wider() spreads the sample names into columns:

let wide = long |> pivot_wider("sample", "expression")
println("Wide format:")
println(wide)

Expected output:

Wide format:
gene  | S1  | S2
BRCA1 | 5.2 | 8.1
TP53  | 3.4 | 7.6

The first argument (piped) is the table. The second argument is the column whose values become new column names. The third argument is the column whose values fill those new columns. All other columns (here, gene) become the row identifiers.

Wide to Long: pivot_longer

Going the other direction, pivot_longer() gathers columns back into rows:

let back_to_long = wide |> pivot_longer(["S1", "S2"], "sample", "expression")
println("Back to long:")
println(back_to_long)

Expected output:

Back to long:
gene  | sample | expression
BRCA1 | S1     | 5.2
BRCA1 | S2     | 8.1
TP53  | S1     | 3.4
TP53  | S2     | 7.6

The first argument (piped) is the table. The second argument is a list of column names to gather. The third argument is the name for the new “names” column. The fourth argument is the name for the new “values” column.

The Visual Transformation

PIVOT WIDER                         PIVOT LONGER

gene  | sample | expr               gene  | S1  | S2
------+--------+-----    ====>      ------+-----+-----
BRCA1 | S1     | 5.2                BRCA1 | 5.2 | 8.1
BRCA1 | S2     | 8.1                TP53  | 3.4 | 7.6
TP53  | S1     | 3.4
TP53  | S2     | 7.6     <====

  3 columns, 4 rows          2+N columns, 2 rows
  (one row per measurement)   (one row per gene)

When to use which:

• Pivot wider when you need a matrix for computation (e.g., gene-by-sample expression matrix for heatmaps, PCA, clustering)
• Pivot longer when you need tidy data for filtering, grouping, and plotting (e.g., faceted plots, group_by + summarize)

Window Functions

Window functions compute a value for each row based on its position or neighbors, without collapsing the table.

Row Numbers and Ranks

let data = [
    {gene: "BRCA1", pval: 0.001},
    {gene: "EGFR", pval: 0.000001},
    {gene: "MYC", pval: 0.04},
    {gene: "TP53", pval: 0.23},
] |> to_table()

# Add row numbers
let numbered = data |> row_number()
println(numbered)

Expected output:

gene  | pval     | row_number
BRCA1 | 0.001    | 1
EGFR  | 0.000001 | 2
MYC   | 0.04     | 3
TP53  | 0.23     | 4

# Rank by p-value
let ranked = data |> rank("pval")
println(ranked)

Expected output:

gene  | pval     | rank
BRCA1 | 0.001    | 2
EGFR  | 0.000001 | 1
MYC   | 0.04     | 3
TP53  | 0.23     | 4

Cumulative Functions

let data = [
    {gene: "A", count: 10},
    {gene: "B", count: 25},
    {gene: "C", count: 15},
    {gene: "D", count: 30},
] |> to_table()

# Cumulative sum
let with_cumsum = data |> cumsum("count")
println(with_cumsum)

Expected output:

gene | count | cumsum
A    | 10    | 10
B    | 25    | 35
C    | 15    | 50
D    | 30    | 80

Rolling Mean

Smooths noisy data by averaging over a sliding window.

let timeseries = [
    {day: 1, value: 10.0},
    {day: 2, value: 12.0},
    {day: 3, value: 8.0},
    {day: 4, value: 15.0},
    {day: 5, value: 11.0},
    {day: 6, value: 14.0},
] |> to_table()

let smoothed = timeseries |> rolling_mean("value", 3)
println(smoothed)

Expected output:

day | value | rolling_mean
1   | 10.0  | 10.0
2   | 12.0  | 11.0
3   | 8.0   | 10.0
4   | 15.0  | 11.666...
5   | 11.0  | 11.333...
6   | 14.0  | 13.333...

The third argument is the window size. The first few rows use a smaller window (whatever data is available).

Lag and Lead

Access values from previous or next rows — useful for computing changes between consecutive measurements.

let data = [
    {day: 1, expression: 2.0},
    {day: 2, expression: 4.5},
    {day: 3, expression: 3.8},
    {day: 4, expression: 6.1},
] |> to_table()

# Previous day's value
let with_lag = data |> lag("expression")
println(with_lag)

Expected output:

day | expression | lag
1   | 2.0        | nil
2   | 4.5        | 2.0
3   | 3.8        | 4.5
4   | 6.1        | 3.8

Complete Example: Expression Analysis Pipeline

This is the kind of analysis you will do repeatedly in practice: load data, annotate it, filter it, summarize it, and export the results.

Requires CLI: This example uses file I/O not available in the browser. Run with bl run.

# Complete table analysis pipeline
# Run init.bl first to generate CSV files in data/

# Step 1: Read expression results and gene annotations
let expr = csv("data/expression.csv")
let gene_info = csv("data/gene_info.csv")

println(f"Expression data: {nrow(expr)} genes x {ncol(expr)} columns")
println(f"Gene info: {nrow(gene_info)} annotations")

# Step 2: Add derived columns
let analyzed = expr
    |> mutate("significant", |r| r.padj < 0.05)
    |> mutate("direction", |r| if r.log2fc > 0 { "up" } else { "down" })
    |> mutate("neg_log_p", |r| -1.0 * log10(r.pval))

# Step 3: Count by direction among significant genes
let direction_counts = analyzed
    |> filter(|r| r.significant)
    |> count_by("direction")
println("Significant genes by direction:")
println(direction_counts)

# Step 4: Annotate with gene info
let annotated = left_join(analyzed, gene_info, "gene")

# Step 5: Top 10 most significant genes
let top10 = annotated
    |> filter(|r| r.significant)
    |> arrange("padj")
    |> head(10)
    |> select("gene", "log2fc", "padj", "pathway")
println("Top 10 significant genes:")
println(top10)

# Step 6: Summary statistics per pathway
let pathway_summary = annotated
    |> filter(|r| r.significant)
    |> group_by("pathway")
    |> summarize(|key, subtable| {
        pathway: key,
        n_genes: nrow(subtable),
        mean_fc: col_mean(subtable, "log2fc")
    })
println("Pathway summary:")
println(pathway_summary)

# Step 7: Export
write_csv(annotated, "results/annotated_results.csv")
println("Results saved to results/annotated_results.csv")

This pipeline reads data, enriches it, filters it, summarizes it, and exports it — all in a single readable chain of piped operations. Each step is self-documenting.

Table Operations Cheat Sheet

Structure

Column Operations

Row Operations

Aggregation

Joins

Reshaping

Window Functions

I/O

Exercises

1. Fold change calculator. Create a table of 10 genes with columns: gene, expression_control, expression_treated. Add a fold_change column (treated / control), then filter to keep only genes where fold change is greater than 2.0.

2. Annotation join. Create a results table (gene, pval) and an annotations table (gene, pathway, description). Use left_join to annotate the results, then filter to keep only genes in the “Apoptosis” pathway.

3. Wide to long and back. Create a wide expression matrix with columns gene, sample_A, sample_B, sample_C. Pivot it to long format. Then compute the mean expression per gene using group_by and summarize.

4. Variant counting. Create a table of variants with columns chr, pos, ref_allele, alt_allele, quality. Use count_by("chr") to count variants per chromosome, then sort by count descending.

5. Full pipeline. Build a complete pipeline that: reads the expression CSV (from init.bl), adds a significance column (padj < 0.05), joins with gene info, filters to significant genes only, groups by pathway, counts genes per pathway, sorts by count descending, and writes the result to a new CSV.

Key Takeaways

• Tables are the central data structure for analysis results. Most bioinformatics output is tabular.
• select/filter/mutate/arrange cover 80% of table operations. Master these four first.
• group_by + summarize is the split-apply-combine pattern. It is how you compute summary statistics per category.
• Joins connect related datasets. Learn inner_join and left_join first — they handle most annotation and linking tasks.
• Pivot wider/longer reshapes between wide format (for computation) and long format (for grouping and plotting).
• Chain operations with pipes for readable analysis code. Each pipe step does one thing, and the data flows top to bottom.
• Window functions (row_number, rank, cumsum, rolling_mean, lag, lead) add context-aware columns without collapsing the table.

What’s Next

Tomorrow we compare sequences — GC content, k-mers, dotplots, motif searching, and multi-species lookups. Day 11 takes the sequence skills from Days 3-4 and scales them up to comparative analysis.

Day 11: Sequence Comparison

What You’ll Learn

• How to compare sequences by base composition and GC content
• How k-mer decomposition enables alignment-free similarity
• How dotplots visually reveal similarity, repeats, and rearrangements
• How to find exact motifs including restriction enzyme recognition sites
• Why reverse complement matters for double-stranded DNA
• How to analyze codon usage bias across genes
• How to compare genes across species using Ensembl APIs

The Problem

Two sequences sit on your screen. Are they related? How similar? Where do they differ? Sequence comparison is the foundation of evolutionary biology, variant detection, and functional prediction.

Some comparisons are quick: does this gene have unusually high GC content? Others are structural: do these two sequences share long stretches of similarity? And some are functional: does this promoter contain a known transcription factor binding site?

Today you will build a toolkit for answering all of these questions, starting from the simplest metric — base composition — and working up to multi-species gene comparison.

Base Composition Analysis

The simplest way to compare two sequences is to count their nucleotides. GC content — the fraction of bases that are G or C — varies dramatically across organisms, from ~25% in some parasites to ~70% in thermophilic bacteria. It is a quick first-pass metric: if two sequences have wildly different GC content, they likely come from different organisms or genomic regions.

let seqs = [
    {name: "E. coli",   seq: dna"GCGCATCGATCGATCGCG"},
    {name: "Human",     seq: dna"ATATCGATCGATATATAT"},
    {name: "Thermus",   seq: dna"GCGCGCGCGCGCGCGCGC"},
]
for s in seqs {
    let gc = round(gc_content(s.seq) * 100, 1)
    let counts = base_counts(s.seq)
    println(f"{s.name}: GC={gc}%, A={counts.A}, T={counts.T}, G={counts.G}, C={counts.C}")
}

Expected output:

E. coli: GC=61.1%, A=2, T=2, G=5, C=6
Human: GC=27.8%, A=7, T=7, G=2, C=3
Thermus: GC=100.0%, A=0, T=0, G=9, C=9

gc_content() returns a float between 0.0 and 1.0. Multiplying by 100 gives a percentage. base_counts() returns a record with fields A, T, G, and C.

Notice how the three example sequences span a wide GC range: the Thermus fragment is entirely GC (thermophilic organisms use GC-rich DNA for thermal stability), while the human fragment is AT-rich (common in non-coding regions).

K-mer Analysis

A k-mer is a subsequence of length k. Decomposing a sequence into k-mers is the foundation of alignment-free comparison — instead of aligning two sequences end to end, you compare their k-mer content.

Here is how k-mers work. Given a sequence, a sliding window of size k moves one base at a time:

Sequence: A T C G A T C G
           |---|                 → ATC
             |---|               → TCG
               |---|             → CGA
                 |---|           → GAT
                   |---|         → ATC
                     |---|       → TCG

3-mers:   ATC  TCG  CGA  GAT  ATC  TCG

Each position produces one k-mer. A sequence of length L contains L - k + 1 k-mers.

Extracting K-mers

let seq = dna"ATCGATCGATCG"
let kmers_list = kmers(seq, 3)
println(f"Sequence: {seq}")
println(f"3-mers: {kmers_list}")

Expected output:

Sequence: ATCGATCGATCG
3-mers: [ATC, TCG, CGA, GAT, ATC, TCG, CGA, GAT, ATC, TCG]

K-mer Frequency

Counting how often each k-mer appears reveals sequence composition at a deeper level than single-base counts.

let seq = dna"ATCGATCGATCG"
let freq = kmer_count(seq, 3)
println(f"3-mer frequencies: {freq}")

Expected output:

3-mer frequencies: {ATC: 3, TCG: 3, CGA: 2, GAT: 2}

Alignment-Free Similarity with K-mers

Two sequences that share many k-mers are likely similar, even without performing a formal alignment. The Jaccard similarity measures this: the size of the intersection divided by the size of the union of the two k-mer sets.

let seq1 = dna"ATCGATCGATCGATCG"
let seq2 = dna"ATCGATCGTTTTGATCG"

let k1 = set(kmers(seq1, 5))
let k2 = set(kmers(seq2, 5))

let shared = intersection(k1, k2)
let total = union(k1, k2)
let jaccard = len(shared) / len(total)
println(f"Shared 5-mers: {len(shared)}")
println(f"Total unique 5-mers: {len(total)}")
println(f"K-mer similarity: {round(jaccard * 100, 1)}%")

Expected output:

Shared 5-mers: 6
Total unique 5-mers: 17
K-mer similarity: 35.3%

Jaccard similarity ranges from 0% (no shared k-mers) to 100% (identical k-mer sets). It is fast to compute, works on sequences of different lengths, and does not require alignment. Tools like Mash and Sourmash use this principle for large-scale genome comparison.

Dotplots — Visual Sequence Comparison

A dotplot is the oldest and most intuitive method for comparing two sequences. The idea is simple:

• Place sequence 1 along the X axis
• Place sequence 2 along the Y axis
• Put a dot at position (i, j) if the bases at position i and j match

The resulting pattern reveals structural relationships at a glance:

let seq1 = dna"ATCGATCGATCG"
let seq2 = dna"ATCGTTGATCG"
dotplot(seq1, seq2)

The dotplot() function generates an SVG visualization. You can customize it:

dotplot(seq1, seq2, window: 3, title: "Pairwise comparison")

The window parameter sets the match window size. A window of 1 shows every single-base match (noisy). A window of 3 or larger filters out random matches, leaving only meaningful stretches of similarity.

Self-Dotplots

Comparing a sequence against itself is a powerful way to find internal repeats. Any repeated region appears as a parallel diagonal line offset from the main diagonal.

let repeat_seq = dna"ATCGATCGATCGATCG"
dotplot(repeat_seq, repeat_seq, window: 3, title: "Self-comparison: internal repeats")

The main diagonal (where the sequence matches itself perfectly) will always be present. Parallel lines above or below the diagonal indicate tandem repeats.

Motif Finding

A motif is a short sequence pattern with biological significance. Start codons, stop codons, restriction enzyme recognition sites, and transcription factor binding sites are all motifs.

Finding Exact Motifs

let seq = dna"ATGATCGATGATCGATGATCG"
let atg_sites = find_motif(seq, "ATG")
println(f"ATG positions: {atg_sites}")

Expected output:

ATG positions: [0, 9, 18]

Positions are zero-indexed. Each value is the start position where the motif begins in the sequence.

Restriction Enzyme Sites

Restriction enzymes cut DNA at specific recognition sequences. Finding these sites is essential for cloning, Southern blotting, and restriction fragment analysis.

let seq = dna"ATCGGAATTCGATCGGGATCCATCG"
let ecori = find_motif(seq, "GAATTC")
let bamhi = find_motif(seq, "GGATCC")
println(f"EcoRI sites: {ecori}")
println(f"BamHI sites: {bamhi}")

Expected output:

EcoRI sites: [4]
BamHI sites: [14]

Common restriction enzymes and their recognition sequences:

Reverse Complement and Strand Awareness

DNA is double-stranded. A motif on the forward strand has a corresponding motif on the reverse strand. When you search for a binding site, you must check both strands — the protein does not care which strand it binds.

let forward = dna"ATGCGATCGATCG"
let revcomp = reverse_complement(forward)
println(f"Forward:  5'-{forward}-3'")
println(f"RevComp:  5'-{revcomp}-3'")

Expected output:

Forward:  5'-ATGCGATCGATCG-3'
RevComp:  5'-CGATCGATCGCAT-3'

Searching Both Strands

let seq = dna"ATCGGAATTCGATCG"
let motif = "GAATTC"
let fwd_hits = find_motif(seq, motif)
let rev_hits = find_motif(reverse_complement(seq), motif)
println(f"Forward strand hits: {fwd_hits}")
println(f"Reverse strand hits: {rev_hits}")

Expected output:

Forward strand hits: [4]
Reverse strand hits: [1]

EcoRI’s recognition sequence (GAATTC) is a palindrome — its reverse complement is also GAATTC. This means EcoRI cuts both strands at the same site. Not all restriction enzymes are palindromic, but most Type II enzymes are.

Codon Analysis

Codons are triplets of nucleotides that encode amino acids. Different organisms prefer different codons for the same amino acid — a phenomenon called codon usage bias. Highly expressed genes tend to use preferred codons for faster translation.

let gene = dna"ATGGCTGCTTCTGATAAATGA"
let usage = codon_usage(gene)
println(f"Codon usage: {usage}")

Expected output:

Codon usage: {ATG: 1, GCT: 1, GCT: 1, TCT: 1, GAT: 1, AAA: 1, TGA: 1}

Comparing Codon Bias Between Species

Different organisms have evolved different codon preferences. E. coli prefers GCG for alanine, while humans prefer GCC. Comparing codon usage can reveal whether a gene has been horizontally transferred or synthetically designed.

let human_gene = dna"ATGGCTGCTTCTGATAAATGA"
let ecoli_gene = dna"ATGGCAGCGAGCGATAAATGA"
let human_usage = codon_usage(human_gene)
let ecoli_usage = codon_usage(ecoli_gene)
println(f"Human codons: {human_usage}")
println(f"E. coli codons: {ecoli_usage}")

Expected output:

Human codons: {ATG: 1, GCT: 1, GCT: 1, TCT: 1, GAT: 1, AAA: 1, TGA: 1}
E. coli codons: {ATG: 1, GCA: 1, GCG: 1, AGC: 1, GAT: 1, AAA: 1, TGA: 1}

Notice how both genes encode roughly similar proteins but use different codons: the human gene uses GCT (alanine) where E. coli uses GCA and GCG.

Multi-Species Comparison via APIs

Comparing a gene across species reveals evolutionary conservation. Genes that are highly conserved across distant species are usually functionally important.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: internet connection

let species = [
    {name: "Human", species: "homo_sapiens"},
    {name: "Mouse", species: "mus_musculus"},
    {name: "Zebrafish", species: "danio_rerio"},
]

let results = species |> map(|sp| {
    let gene = ensembl_symbol(sp.species, "BRCA1")
    let protein = ensembl_sequence(gene.id, type: "protein")
    {name: sp.name, gene_id: gene.id, protein_len: len(protein.seq)}
})

let comparison = results |> to_table()
println(comparison)

Expected output (values depend on current Ensembl release):

name      | gene_id            | protein_len
Human     | ENSG00000012048    | 1863
Mouse     | ENSMUSG00000017146 | 1812
Zebrafish | ENSDARG00000076256 | 1679

The BRCA1 protein is conserved across vertebrates but gets progressively shorter in more distant species — zebrafish BRCA1 is about 10% shorter than human BRCA1. This kind of comparison is a first step toward understanding which regions of the protein are functionally essential (the conserved parts) versus dispensable (the parts that vary).

Building a Similarity Matrix

When you have more than two sequences, pairwise comparison produces a similarity matrix — a table where each cell contains the similarity between two sequences.

let sequences = [
    {name: "seq1", seq: dna"ATCGATCGATCGATCG"},
    {name: "seq2", seq: dna"ATCGATCGTTTTGATCG"},
    {name: "seq3", seq: dna"GCGCGCGCGCGCGCGC"},
]

let results = []
for i in range(0, len(sequences)) {
    for j in range(0, len(sequences)) {
        let k1 = set(kmers(sequences[i].seq, 5))
        let k2 = set(kmers(sequences[j].seq, 5))
        let shared = len(intersection(k1, k2))
        let total = len(union(k1, k2))
        let sim = if total > 0 { round(shared / total, 3) } else { 0.0 }
        results = push(results, {
            seq1: sequences[i].name,
            seq2: sequences[j].name,
            similarity: sim
        })
    }
}
let matrix = results |> to_table()
println(matrix)

Expected output:

seq1 | seq2 | similarity
seq1 | seq1 | 1.0
seq1 | seq2 | 0.353
seq1 | seq3 | 0.0
seq2 | seq1 | 0.353
seq2 | seq2 | 1.0
seq2 | seq3 | 0.0
seq3 | seq1 | 0.0
seq3 | seq2 | 0.0
seq3 | seq3 | 1.0

The matrix confirms what you would expect: seq1 and seq2 share some similarity (they have overlapping subsequences), but seq3 (all GC) shares nothing with either.

Reading a similarity matrix:

• The diagonal is always 1.0 (every sequence is identical to itself)
• The matrix is symmetric (similarity of A to B equals similarity of B to A)
• Values near 0.0 mean unrelated sequences; values near 1.0 mean nearly identical sequences

Complete Example: Gene Comparison Report

This script ties together everything from today — base composition, k-mers, motif finding, and API-based cross-species comparison — into a single analysis.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Compare TP53 protein sequence properties across species
# requires: internet connection (optional: NCBI_API_KEY for higher rate limits)

fn compare_gene(gene_symbol, species_list) {
    let results = []
    for sp in species_list {
        try {
            let gene = ensembl_symbol(sp.species, gene_symbol)
            let cds = ensembl_sequence(gene.id, type: "cdna")
            let prot = ensembl_sequence(gene.id, type: "protein")
            results = push(results, {
                species: sp.name,
                cds_length: len(cds.seq),
                protein_length: len(prot.seq),
                gc: round(gc_content(cds.seq) * 100, 1)
            })
        } catch e {
            println(f"  Skipping {sp.name}: {e}")
        }
    }
    results |> to_table()
}

let species = [
    {name: "Human", species: "homo_sapiens"},
    {name: "Mouse", species: "mus_musculus"},
    {name: "Chicken", species: "gallus_gallus"},
]

let comparison = compare_gene("TP53", species)
println(comparison)

Expected output (values depend on current Ensembl release):

species | cds_length | protein_length | gc
Human   | 1182       | 393            | 48.2
Mouse   | 1176       | 391            | 49.1
Chicken | 1113       | 370            | 52.8

TP53 (the “guardian of the genome”) is highly conserved across vertebrates. The protein length varies by only ~6%, but GC content differs more — chicken TP53 has higher GC content, consistent with the generally higher GC content of bird genomes.

Exercises

1. GC content ranking. Create an array of 5 DNA sequences with different compositions. Calculate GC content for each and sort them from highest to lowest using sort_by and reverse.

2. Start and stop codons. Given the sequence dna"ATGCGATCGATGATCGTAGATCGATGATCGTGAATCG", find all start codons (ATG) and all stop codons (TAA, TAG, TGA). Print the positions of each.

3. Self-dotplot for repeats. Create a sequence that contains a repeated motif (e.g., dna"ATCGATCGATCGATCG") and use dotplot() to compare it against itself. How many parallel diagonals do you see?

4. K-mer similarity at different k values. Compare two related sequences at k=3, k=5, and k=7. How does increasing k affect the Jaccard similarity? Why?

5. Cross-species comparison. Use the Ensembl API to compare BRCA1 across human, mouse, and zebrafish. Build a table with columns for species, CDS length, protein length, and GC content.

Key Takeaways

• GC content and base composition are quick first-pass comparisons between sequences
• K-mers enable alignment-free similarity measurement — fast and effective for large-scale comparisons
• Dotplots visually reveal similarity, insertions, deletions, and repeats at a glance
• find_motif() searches for exact patterns including restriction enzyme recognition sites
• Reverse complement is essential — biology uses both DNA strands, and many binding sites are palindromic
• Codon usage bias varies across organisms and reveals evolutionary and functional signatures
• API-based multi-species comparison reveals evolutionary conservation of genes and proteins

What’s Next

Tomorrow: finding variants in genomes — VCF analysis, variant filtering, and clinical interpretation.

Day 12: Finding Variants in Genomes

What You’ll Learn

• How to read and explore VCF files with read_vcf()
• How to classify variants by type: SNP, insertion, deletion, MNV
• What the transition/transversion ratio means and why it matters
• How to filter variants by quality metrics
• How to annotate variants using Ensembl VEP
• The basics of clinical variant interpretation (ACMG/AMP framework)

The Problem

A clinical sequencing lab returns a VCF file with 4 million variants. Your patient’s diagnosis depends on finding the 1–3 variants that actually cause disease. Filtering 4 million down to a handful requires understanding variant types, quality metrics, population frequencies, and clinical databases.

Today you will build the tools and intuition for this process — from loading raw VCF files to classifying, filtering, and annotating variants. The dataset is small (48 variants) so you can see every step clearly, but the techniques scale to millions of variants.

What Are Variants?

A variant is any position where a genome differs from the reference sequence. Variants come in several types:

Reference:  ...A T C G A T C G A T C G...
                        *
SNP:        ...A T C G A T T G A T C G...    (C -> T at one position)

Reference:  ...A T C G A - - T C G A T C G...
Insertion:  ...A T C G A A A T C G A T C G...  (AA inserted)

Reference:  ...A T C G A T C G A T C G...
Deletion:   ...A T C G - - - G A T C G...    (ATC deleted)

Reference:  ...A T C G A T C G A T C G...
MNV:        ...A T C G T T C G A T C G...    (AT -> TT, multi-nucleotide)

• SNP (Single Nucleotide Polymorphism): one base changed. The most common variant type.
• Insertion: bases added that are not in the reference.
• Deletion: bases present in the reference are missing.
• MNV (Multi-Nucleotide Variant): multiple adjacent bases changed simultaneously.

Insertions and deletions are collectively called indels. They are harder to detect accurately than SNPs because they shift the reading frame of the sequencer’s alignment.

Reading and Exploring VCF Files

VCF (Variant Call Format) is the standard file format for storing variant data. Each row describes one variant: its chromosome, position, reference allele, alternate allele, quality score, and filter status.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: data/variants.vcf in working directory (run init.bl first)
let variants = read_vcf("data/variants.vcf")
println(f"Total variants: {len(variants)}")

Expected output:

Total variants: 48

read_vcf() returns a list of Variant values. Each variant has properties you can access with dot notation:

let v = first(variants)
println(f"Chrom: {v.chrom}")
println(f"Position: {v.pos}")
println(f"ID: {v.id}")
println(f"Ref: {v.ref}, Alt: {v.alt}")
println(f"Quality: {v.qual}")
println(f"Filter: {v.filter}")

Expected output:

Chrom: chr1
Position: 14907
ID: rs6682375
Ref: A, Alt: G
Quality: 45.3
Filter: PASS

The key fields are:

Variant Classification

BioLang’s Variant values have built-in properties for classification. You do not need to write your own classification function — the runtime does it for you:

let v = first(variants)
println(f"Type: {v.variant_type}")    # "Snp", "Indel", "Mnp", or "Other"
println(f"Is SNP? {v.is_snp}")        # true or false
println(f"Is indel? {v.is_indel}")    # true or false

Expected output:

Type: Snp
Is SNP? true
Is indel? false

Use these properties with filter() to separate variants by type:

let snps = variants |> filter(|v| v.is_snp) |> collect()
let indels = variants |> filter(|v| v.is_indel) |> collect()
println(f"SNPs: {len(snps)}")
println(f"Indels: {len(indels)}")

Expected output:

SNPs: 38
Indels: 10

You can also inspect individual variants with their type:

let first_ten = variants |> take(10) |> map(|v| {
    chrom: v.chrom, pos: v.pos,
    ref: v.ref, alt: v.alt,
    type: v.variant_type
})
for item in first_ten {
    println(f"  {item.chrom}:{item.pos} {item.ref}>{item.alt} ({item.type})")
}

Expected output:

  chr1:14907 A>G (Snp)
  chr1:69511 A>G (Snp)
  chr1:817186 G>A (Snp)
  chr1:949654 C>T (Snp)
  chr1:984971 G>A (Snp)
  chr1:1018704 T>C (Snp)
  chr1:1110294 G>A (Snp)
  chr1:1234567 ATG>A (Indel)
  chr1:1567890 C>CTAG (Indel)
  chr1:2045678 A>T (Snp)

Transition/Transversion Ratio

Not all SNPs are equally likely. There are two categories:

• Transitions (Ts): purine-to-purine or pyrimidine-to-pyrimidine changes. A↔G and C↔T. These are chemically more likely because the molecular shape is similar.
• Transversions (Tv): purine-to-pyrimidine or vice versa. A↔C, A↔T, G↔C, G↔T. These require a bigger structural change.

         Transitions (Ts)
     A  <===============>  G       (purines)

     C  <===============>  T       (pyrimidines)

         Transversions (Tv)
     A  <------>  C     A  <------>  T
     G  <------>  C     G  <------>  T

Because transitions are chemically favored, the expected Ts/Tv ratio for real biological variants is approximately 2.0–2.1 for whole-genome sequencing. A significantly lower ratio (say, 1.0) suggests many false-positive variant calls — the errors are random and equally likely to be transitions or transversions.

BioLang computes this in one call:

let ratio = tstv_ratio(variants)
println(f"Ts/Tv ratio: {round(ratio, 2)}")

Expected output:

Ts/Tv ratio: 1.92

You can also use the per-variant properties to count manually:

let ts_count = variants |> filter(|v| v.is_snp and v.is_transition) |> count()
let tv_count = variants |> filter(|v| v.is_snp and v.is_transversion) |> count()
println(f"Transitions: {ts_count}")
println(f"Transversions: {tv_count}")

Expected output:

Transitions: 25
Transversions: 13

The .is_transition and .is_transversion properties are only meaningful for SNPs. For indels and MNVs, both return false.

Quality Filtering

Raw variant calls contain many false positives. The first step in any analysis is filtering. A typical filtering cascade:

In BioLang:

# Filter by PASS status
let passed = variants |> filter(|v| v.filter == "PASS") |> collect()
println(f"PASS variants: {len(passed)} / {len(variants)}")

# Add quality threshold
let high_quality = variants
    |> filter(|v| v.filter == "PASS")
    |> filter(|v| v.qual >= 30)
    |> collect()
println(f"PASS + quality >= 30: {len(high_quality)}")

Expected output:

PASS variants: 41 / 48
PASS + quality >= 30: 41

It is informative to examine what was filtered out:

let low_qual = variants |> filter(|v| v.filter != "PASS") |> collect()
println(f"Filtered out (non-PASS): {len(low_qual)}")
for lq in low_qual {
    println(f"  {lq.chrom}:{lq.pos} {lq.ref}>{lq.alt} qual={lq.qual} filter={lq.filter}")
}

Expected output:

Filtered out (non-PASS): 7
  chr1:984971 G>A qual=12.5 filter=LowQual
  chr1:2045678 A>T qual=8.1 filter=LowQual
  chr2:6123456 T>C qual=15.2 filter=LowDP
  chr3:4567890 T>A qual=10.4 filter=LowQual
  chr7:5678901 A>C qual=9.7 filter=LowQual
  chr11:5678901 G>T qual=14.3 filter=LowDP
  chrX:5678901 C>A qual=11.8 filter=LowQual

Notice that the filtered variants have low quality scores (all under 16) and were flagged as either LowQual (low confidence) or LowDP (low read depth). These are exactly the variants you want to remove — they are likely sequencing errors, not real biological variation.

Variant Summary and Statistics

For a quick overview, variant_summary() computes all key statistics in one call:

let summary = variant_summary(variants)
println(f"Total alleles: {summary.total}")
println(f"  SNPs: {summary.snp}")
println(f"  Indels: {summary.indel}")
println(f"  MNPs: {summary.mnp}")
println(f"  Transitions: {summary.transitions}")
println(f"  Transversions: {summary.transversions}")
println(f"  Ts/Tv ratio: {round(summary.ts_tv_ratio, 2)}")
println(f"  Multiallelic: {summary.multiallelic}")

Expected output:

Total alleles: 48
  SNPs: 38
  Indels: 10
  MNPs: 0
  Transitions: 25
  Transversions: 13
  Ts/Tv ratio: 1.92
  Multiallelic: 0

The het/hom ratio measures the balance between heterozygous calls (one copy of the variant) and homozygous-alternate calls (both copies). For a diploid organism like humans, the expected ratio is roughly 1.5–2.0.

let hh_ratio = het_hom_ratio(variants)
println(f"Het/Hom ratio: {round(hh_ratio, 2)}")

let het_count = variants |> filter(|v| v.is_het) |> count()
let hom_count = variants |> filter(|v| v.is_hom_alt) |> count()
println(f"Heterozygous: {het_count}")
println(f"Homozygous alt: {hom_count}")

Expected output:

Het/Hom ratio: 4.33
Heterozygous: 39
Homozygous alt: 9

Our small test dataset has a higher-than-expected het/hom ratio because we deliberately included more heterozygous variants. In a real whole-genome dataset, this ratio is a useful quality indicator — an abnormally high or low ratio may indicate contamination or incorrect variant calling.

Chromosome Distribution

Knowing how variants are distributed across chromosomes helps spot problems. An unexpected spike on one chromosome might indicate a copy number variant or a systematic alignment issue.

let by_chrom = variants
    |> map(|v| {chrom: v.chrom, type: v.variant_type})
    |> to_table()
    |> group_by("chrom")
    |> summarize(|chrom, rows| {chrom: chrom, count: len(rows)})
println(by_chrom)

Expected output:

chrom | count
chr1  | 10
chr11 | 5
chr17 | 5
chr2  | 7
chr3  | 6
chr5  | 5
chr7  | 5
chrX  | 5

In a real dataset, the variant count would be roughly proportional to chromosome length. Chromosome 1 (the longest) would have the most variants, and chromosome 21 (the shortest autosome) would have the fewest.

Variant Annotation with Ensembl VEP

Knowing that a variant exists is only the first step. To understand its biological significance, you need to annotate it: determine which gene it falls in, what effect it has on the protein, and whether it has been seen before in clinical databases.

The Ensembl Variant Effect Predictor (VEP) does this. BioLang wraps the Ensembl REST API in a single function call:

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: internet connection
let annotation = ensembl_vep("17:7577120:G:A")
let result = first(annotation)
println(f"Allele string: {result.allele_string}")
println(f"Most severe consequence: {result.most_severe_consequence}")

let tcs = result.transcript_consequences
if len(tcs) > 0 {
    let tc = first(tcs)
    println(f"Gene: {tc.gene_id}")
    println(f"Impact: {tc.impact}")
    println(f"Consequences: {tc.consequences}")
}

The ensembl_vep() function takes a string in the format "chrom:pos:ref:alt" and returns a list of annotation results. Each result contains:

VEP classifies consequences by severity. From most to least severe:

For batch annotation, wrap the call in try/catch to handle network errors gracefully:

let annotated = variants |> take(5) |> map(|v| {
    chrom: v.chrom, pos: v.pos, ref: v.ref, alt: v.alt,
    annotation: try { ensembl_vep(f"{v.chrom}:{v.pos}:{v.ref}:{v.alt}") } catch e { nil }
})

Note: The Ensembl REST API has rate limits (15 requests per second without an API key). For large-scale annotation, use the standalone VEP command-line tool instead.

Clinical Variant Interpretation

Finding and annotating variants is a technical problem. Interpreting their clinical significance is a medical one. The standard framework is the ACMG/AMP guidelines (American College of Medical Genetics / Association for Molecular Pathology), which classify variants into five tiers:

The classification uses several types of evidence:

• Population frequency: If a variant is common in healthy populations (e.g., >1% in gnomAD), it is unlikely to cause rare disease.
• Computational predictions: Tools like SIFT, PolyPhen-2, and CADD predict whether a protein change is damaging.
• Functional data: Laboratory experiments showing the variant disrupts protein function.
• Segregation: Whether the variant co-occurs with disease in families.
• Clinical databases: ClinVar aggregates clinical interpretations from laboratories worldwide.

Important: Clinical variant interpretation requires specialized training. The code in this chapter teaches the computational steps — reading VCF files, filtering, and annotating — but the medical interpretation of results should always involve a trained clinical geneticist or genetic counselor.

Complete Variant Analysis Pipeline

Here is the full pipeline, from raw VCF to classified, filtered results:

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Complete Variant Analysis Pipeline
# requires: data/variants.vcf in working directory

println("=== Variant Analysis Pipeline ===\n")

# Step 1: Load
let variants = read_vcf("data/variants.vcf")
println(f"1. Total variants: {len(variants)}")

# Step 2: Quality filtering
let passed = variants
    |> filter(|v| v.filter == "PASS")
    |> filter(|v| v.qual >= 30)
    |> collect()
println(f"2. After filtering: {len(passed)} variants")

# Step 3: Classify
let snps = passed |> filter(|v| v.is_snp) |> count()
let indels = passed |> filter(|v| v.is_indel) |> count()
println(f"3. SNPs: {snps}, Indels: {indels}")

# Step 4: Ts/Tv ratio
let ratio = tstv_ratio(passed)
println(f"4. Ts/Tv ratio: {round(ratio, 2)}")

# Step 5: Chromosome distribution
let by_chrom = passed
    |> map(|v| {chrom: v.chrom, type: v.variant_type})
    |> to_table()
    |> group_by("chrom")
    |> summarize(|chrom, rows| {chrom: chrom, count: len(rows)})
println(f"\n5. Variants per chromosome:")
println(by_chrom)

# Step 6: Export
let results = passed |> map(|v| {
    chrom: v.chrom, pos: v.pos, id: v.id,
    ref: v.ref, alt: v.alt,
    qual: v.qual, type: v.variant_type
}) |> to_table()
write_csv(results, "results/classified_variants.csv")
println(f"\n6. Results saved to results/classified_variants.csv")
println("\n=== Pipeline complete ===")

Expected output:

=== Variant Analysis Pipeline ===

1. Total variants: 48
2. After filtering: 41 variants
3. SNPs: 31, Indels: 10
4. Ts/Tv ratio: 1.92

5. Variants per chromosome:
chrom | count
chr1  | 8
chr11 | 4
chr17 | 5
chr2  | 6
chr3  | 5
chr5  | 5
chr7  | 4
chrX  | 4

6. Results saved to results/classified_variants.csv

=== Pipeline complete ===

This pipeline reduces 48 raw variants to 41 high-confidence calls, classifies them, checks the quality metric (Ts/Tv near 2.0 — good), and exports the results. In a clinical setting, the next steps would be frequency filtering (against gnomAD), functional annotation (VEP), and manual review of candidates.

Exercises

1. SNP-to-indel ratio: Load the VCF file and calculate the ratio of SNPs to indels. A typical whole-genome ratio is about 10:1. How does our test data compare?

2. Classify transitions and transversions: Write a function that takes a variant and returns "transition" or "transversion" (or "not_snp" for indels). Apply it to all variants and print the counts.

3. Region filter: Filter variants to chromosome chr17 between positions 7,500,000 and 42,000,000. This spans the TP53 and BRCA1 genes. How many variants fall in this region?

4. VEP annotation: Annotate 5 variants from your VCF using ensembl_vep() and print the predicted consequence for each. Which has the highest impact?

5. Summary report: Build a report that shows: total variants, variants per chromosome, SNP/indel counts, Ts/Tv ratio, and het/hom ratio. Export it as a CSV table.

Key Takeaways

• Variants are differences from the reference genome: SNPs, insertions, deletions, and multi-nucleotide variants.
• Quality filtering is the first step in any variant analysis — remove low-confidence calls before doing anything else.
• The Ts/Tv ratio (~2.0 for whole genome) is a quick quality check for your variant calls.
• VEP annotation predicts the biological effect of each variant, from benign intronic changes to damaging frameshift mutations.
• Clinical interpretation follows the ACMG/AMP framework and requires domain expertise — code can filter and annotate, but a human expert interprets.
• The goal of variant analysis: start with millions of raw calls, filter down to the few that matter for your biological question.

What’s Next

Week 3 starts tomorrow with Day 13: Gene Expression and RNA-seq. You will move from DNA variants to measuring which genes are active — how much RNA each gene produces, and how expression changes between conditions. This is the foundation of transcriptomics and differential expression analysis.

Day 13: Gene Expression and RNA-seq

What You’ll Learn

• What gene expression is and why it matters
• How RNA-seq measures expression by counting reads
• How to work with count matrices (genes x samples)
• Why normalization is essential and how CPM and TPM work
• How to perform differential expression analysis between conditions
• What log2 fold change means and how to interpret it
• How to correct for multiple testing with Benjamini-Hochberg
• How to create volcano plots and MA plots

The Problem

A cancer researcher has RNA-seq data from 6 patients — 3 tumor samples and 3 normal. Which genes are overactive in tumors? Which are silenced? Differential expression analysis answers this, but first you need to understand what RNA-seq measures and how to normalize the data.

Today you will work through the full RNA-seq analysis pipeline: from raw count matrices through normalization, differential expression, multiple testing correction, and visualization. The dataset is small (20 genes) so you can trace every calculation, but the techniques scale to 20,000+ genes in real experiments.

What Is Gene Expression?

Every cell in your body has the same DNA, yet a neuron looks and functions nothing like a muscle cell. The difference is gene expression — which genes are turned on and how strongly.

• Expression = how much mRNA a gene produces at a given moment.
• High expression = the gene is active, producing many mRNA copies. Example: GAPDH in most cells.
• Low or no expression = the gene is silent. Example: hemoglobin genes in skin cells.
• Differential expression = a gene is more active in one condition than another. Example: an oncogene overexpressed in tumor tissue.

Different cell types, tissues, diseases, and time points produce different expression profiles. Measuring these differences is the goal of RNA-seq.

RNA-seq: Measuring Expression

RNA-seq is the standard technology for measuring gene expression across the genome. The workflow has several steps, each producing a different data format:

The key idea: the number of reads that map to a gene is proportional to how much mRNA that gene produced. More mRNA means more reads. By counting reads per gene across samples, we build a count matrix — the starting point for all downstream analysis.

But raw counts are not directly comparable:

• A 10,000 bp gene captures more reads than a 500 bp gene, even at the same expression level (length bias).
• A sample sequenced to 50 million reads has higher counts than one sequenced to 25 million reads (library size bias).

Normalization removes these biases so we can compare genes and samples fairly.

Count Matrices

A count matrix has genes as rows and samples as columns. Each cell contains the number of reads mapped to that gene in that sample.

# Create a count matrix from records
let counts = [
    {gene: "BRCA1", normal_1: 120, normal_2: 135, normal_3: 128, tumor_1: 340, tumor_2: 380, tumor_3: 355},
    {gene: "TP53",  normal_1: 450, normal_2: 420, normal_3: 440, tumor_1: 890, tumor_2: 920, tumor_3: 850},
    {gene: "GAPDH", normal_1: 5000, normal_2: 5200, normal_3: 4800, tumor_1: 5100, tumor_2: 4900, tumor_3: 5300},
    {gene: "MYC",   normal_1: 80,  normal_2: 75,  normal_3: 85,  tumor_1: 450, tumor_2: 480, tumor_3: 420},
    {gene: "ACTB",  normal_1: 3000, normal_2: 3100, normal_3: 2900, tumor_1: 3050, tumor_2: 2950, tumor_3: 3100},
] |> to_table()

println(f"Genes: {nrow(counts)}")
println(f"Columns: {colnames(counts)}")
println(counts)

Expected output:

Genes: 5
Columns: ["gene", "normal_1", "normal_2", "normal_3", "tumor_1", "tumor_2", "tumor_3"]
gene    normal_1  normal_2  normal_3  tumor_1  tumor_2  tumor_3
BRCA1   120       135       128       340      380      355
TP53    450       420       440       890      920      850
GAPDH   5000      5200      4800      5100     4900     5300
MYC     80        75        85        450      480      420
ACTB    3000      3100      2900      3050     2950     3100

In practice, count matrices come from tools like featureCounts or HTSeq, and you would load them from a CSV file:

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: data/counts.csv in working directory (run init.bl first)
let counts = csv("data/counts.csv")
println(f"Genes: {nrow(counts)}")
println(f"Samples: {ncol(counts) - 1}")
println(counts |> head(5))

Expected output:

Genes: 20
Samples: 6
gene    normal_1  normal_2  normal_3  tumor_1  tumor_2  tumor_3
BRCA1   120       135       128       340      380      355
TP53    450       420       440       890      920      850
GAPDH   5000      5200      4800      5100     4900     5300
MYC     80        75        85        450      480      420
ACTB    3000      3100      2900      3050     2950     3100

Normalization: Why and How

The Problem

Imagine two genes:

Gene A has 4x more reads than Gene B, but it is also 20x longer. Per unit length, Gene B is actually expressed at a higher level. Raw counts are misleading.

Similarly, if Sample X was sequenced to 50 million reads and Sample Y to 25 million reads, every gene in Sample X will have roughly double the counts — not because expression is higher, but because of sequencing depth.

CPM: Counts Per Million

CPM corrects for library size (total number of reads per sample). It answers: “Out of every million reads, how many mapped to this gene?”

Formula: CPM = (count / total reads in sample) x 1,000,000

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# CPM normalization
# requires: data/counts.csv in working directory
let counts = csv("data/counts.csv")
let normalized_cpm = cpm(counts)
println("CPM normalized (first 5 genes):")
println(normalized_cpm |> head(5))

Expected output:

CPM normalized (first 5 genes):
gene    normal_1    normal_2    normal_3    tumor_1    tumor_2    tumor_3
BRCA1   5765.2      6311.5      6111.5      14475.9    16174.9    15191.3
TP53    21619.5     19630.3     21002.4     37889.0    39163.5    36369.3
GAPDH   240217.1    243034.7    229095.5    217107.5   208617.1   226786.8
MYC     3843.3      3505.5      4057.6      19156.6    20432.3    17972.8
ACTB    144130.2    144875.9    138431.6    129848.3   125558.6   132638.9

CPM is good for comparing the same gene across samples but does not account for gene length.

TPM: Transcripts Per Million

TPM corrects for both gene length and library size. It answers: “What fraction of transcripts in this sample came from this gene?”

Steps:

1. Divide each count by gene length (in kilobases) to get reads per kilobase (RPK).
2. Sum all RPK values in the sample.
3. Divide each RPK by the sum and multiply by 1,000,000.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# TPM normalization (needs gene lengths)
# requires: data/counts.csv, data/gene_lengths.csv in working directory
let counts = csv("data/counts.csv")
let gene_lengths = csv("data/gene_lengths.csv")
let normalized_tpm = tpm(counts, gene_lengths)
println("TPM normalized (first 5 genes):")
println(normalized_tpm |> head(5))

Expected output:

TPM normalized (first 5 genes):
gene    normal_1    normal_2    normal_3    tumor_1    tumor_2    tumor_3
BRCA1   3214.8      3518.9      3401.5      8150.2     9116.3     8550.1
TP53    26971.3     24483.4     26179.1     47620.3    48967.0    45584.2
GAPDH   238641.5    241543.0    226895.7    216413.8   207590.7   225700.3
MYC     9607.1      8759.6      10130.4     48220.9    51524.5    45301.2
ACTB    143201.7    143969.6    137264.8    129348.5   124933.4   131946.3

TPM is preferred for most analyses because it accounts for gene length. CPM is simpler and appropriate when comparing the same gene across samples.

FPKM/RPKM (Older Methods)

FPKM (Fragments Per Kilobase of transcript per Million mapped reads) and RPKM (Reads Per Kilobase per Million) were early normalization methods. They divide by library size first, then by gene length. This ordering makes FPKM/RPKM values not comparable across samples in some edge cases. TPM fixes this problem by normalizing in the opposite order. You may encounter FPKM in older datasets, but use TPM for new analyses.

Exploratory Analysis

Before differential expression, inspect your data for obvious problems.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: data/counts.csv in working directory
let counts = csv("data/counts.csv")

# Check library sizes (total reads per sample)
let samples = ["normal_1", "normal_2", "normal_3", "tumor_1", "tumor_2", "tumor_3"]
let sample_sums = samples
    |> map(|s| {sample: s, total: col(counts, s) |> sum()})
    |> to_table()
println("Library sizes:")
println(sample_sums)

Expected output:

Library sizes:
sample    total
normal_1  20813
normal_2  21389
normal_3  20958
tumor_1   23486
tumor_2   23497
tumor_3   23376

Library sizes should be roughly similar. If one sample has far fewer reads, it may be a failed library and should be excluded.

# Mean expression per gene across conditions
let gene_means = counts
    |> mutate("normal_mean", |r| round((r.normal_1 + r.normal_2 + r.normal_3) / 3.0, 1))
    |> mutate("tumor_mean", |r| round((r.tumor_1 + r.tumor_2 + r.tumor_3) / 3.0, 1))
    |> select("gene", "normal_mean", "tumor_mean")
println("Mean expression per gene:")
println(gene_means)

Expected output:

Mean expression per gene:
gene      normal_mean  tumor_mean
BRCA1     127.7        358.3
TP53      436.7        886.7
GAPDH     5000.0       5100.0
MYC       80.0         450.0
ACTB      3000.0       3033.3
VEGFA     200.0        620.0
EGFR      310.0        780.0
CDH1      520.0        155.0
RB1       380.0        115.0
PTEN      290.0        90.0
APC       150.0        50.0
KRAS      95.0         420.0
HER2      60.0         540.0
BCL2      340.0        120.0
CDKN2A    260.0        70.0
MDM2      180.0        500.0
PIK3CA    110.0        370.0
TERT      15.0         310.0
IL6       45.0         380.0
TNF       55.0         120.0

Genes like GAPDH and ACTB show similar expression in both conditions — they are housekeeping genes. Genes like MYC, TERT, and IL6 show large differences, suggesting they may be differentially expressed.

Differential Expression

Differential expression analysis identifies genes whose expression differs significantly between two conditions. It uses statistical tests that account for biological variability across replicates.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: data/counts.csv in working directory
let counts = csv("data/counts.csv")

# Run differential expression analysis
let de_results = diff_expr(counts,
    control: ["normal_1", "normal_2", "normal_3"],
    treatment: ["tumor_1", "tumor_2", "tumor_3"]
)
println(f"DE results: {nrow(de_results)} genes")
println(de_results |> head(5))

Expected output:

DE results: 20 genes
gene    log2fc    pvalue      padj        mean_ctrl  mean_treat
TERT    4.37      0.000012    0.000240    15.0       310.0
MYC     2.49      0.000035    0.000350    80.0       450.0
HER2    3.17      0.000041    0.000273    60.0       540.0
IL6     3.08      0.000058    0.000290    45.0       380.0
KRAS    2.14      0.000089    0.000356    95.0       420.0

The result table includes:

• log2fc: log2 fold change (positive = higher in treatment/tumor)
• pvalue: raw p-value from the statistical test
• padj: p-value adjusted for multiple testing (Benjamini-Hochberg)
• mean_ctrl: mean expression in control samples
• mean_treat: mean expression in treatment samples

# Filter significant results
let significant = de_results
    |> filter(|r| r.padj < 0.05 and abs(r.log2fc) > 1.0)
    |> arrange("padj")

println(f"\nSignificant DE genes (|log2FC| > 1, padj < 0.05):")
println(significant)

# Count up vs down regulated
let up = significant |> filter(|r| r.log2fc > 0) |> nrow()
let down = significant |> filter(|r| r.log2fc < 0) |> nrow()
println(f"Upregulated in tumor: {up}")
println(f"Downregulated in tumor: {down}")

Expected output:

Significant DE genes (|log2FC| > 1, padj < 0.05):
gene      log2fc    pvalue      padj        mean_ctrl  mean_treat
TERT      4.37      0.000012    0.000240    15.0       310.0
HER2      3.17      0.000041    0.000273    60.0       540.0
IL6       3.08      0.000058    0.000290    45.0       380.0
MYC       2.49      0.000035    0.000350    80.0       450.0
KRAS      2.14      0.000089    0.000356    95.0       420.0
PIK3CA    1.75      0.000150    0.000500    110.0      370.0
MDM2      1.47      0.000210    0.000600    180.0      500.0
VEGFA     1.63      0.000180    0.000514    200.0      620.0
EGFR      1.33      0.000320    0.000800    310.0      780.0
TP53      1.02      0.000450    0.001000    436.7      886.7
CDKN2A    -1.89     0.000095    0.000380    260.0      70.0
APC       -1.58     0.000120    0.000400    150.0      50.0
PTEN      -1.69     0.000110    0.000393    290.0      90.0
CDH1      -1.75     0.000085    0.000356    520.0      155.0
RB1       -1.72     0.000130    0.000433    380.0      115.0
BCL2      -1.50     0.000200    0.000571    340.0      120.0

Upregulated in tumor: 10
Downregulated in tumor: 6

The upregulated genes (MYC, TERT, HER2, KRAS, EGFR, VEGFA) are well-known oncogenes. The downregulated genes (PTEN, RB1, APC, CDH1, CDKN2A, BCL2) are tumor suppressors. This pattern is biologically consistent with cancer biology.

Fold Change

Fold change measures how much a gene’s expression changes between conditions. We use the log2 scale because it makes increases and decreases symmetric:

On the linear scale, a 2x increase is +100% but a 2x decrease is only -50%. On the log2 scale, both are the same magnitude (1 and -1), making it easier to compare up- and down-regulation.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Manual fold change calculation
# requires: data/counts.csv in working directory
let counts = csv("data/counts.csv")

let fc_table = counts
    |> mutate("normal_mean", |r| (r.normal_1 + r.normal_2 + r.normal_3) / 3.0)
    |> mutate("tumor_mean", |r| (r.tumor_1 + r.tumor_2 + r.tumor_3) / 3.0)
    |> mutate("log2fc", |r| log2(r.tumor_mean / r.normal_mean))
    |> select("gene", "normal_mean", "tumor_mean", "log2fc")

println("Fold changes:")
println(fc_table |> head(10))

Expected output:

Fold changes:
gene    normal_mean  tumor_mean  log2fc
BRCA1   127.7        358.3       1.49
TP53    436.7        886.7       1.02
GAPDH   5000.0       5100.0      0.03
MYC     80.0         450.0       2.49
ACTB    3000.0       3033.3      0.02
VEGFA   200.0        620.0       1.63
EGFR    310.0        780.0       1.33
CDH1    520.0        155.0       -1.75
RB1     380.0        115.0       -1.72
PTEN    290.0        90.0        -1.69

Notice: GAPDH and ACTB have log2FC near 0 (housekeeping genes, stable expression). MYC has log2FC = 2.49, meaning it is about 5.6x higher in tumors. CDH1 has log2FC = -1.75, meaning it is about 3.4x lower in tumors (a tumor suppressor being silenced).

Visualization

Volcano Plot

The volcano plot is the classic differential expression visualization. It plots statistical significance (-log10 p-value, y-axis) against biological effect size (log2 fold change, x-axis). Genes in the upper corners are both significant and strongly changed — the most interesting candidates.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: data/counts.csv in working directory
let counts = csv("data/counts.csv")
let de_results = diff_expr(counts,
    control: ["normal_1", "normal_2", "normal_3"],
    treatment: ["tumor_1", "tumor_2", "tumor_3"]
)

# Basic volcano plot
volcano(de_results)

# With thresholds highlighted
volcano(de_results, fc_threshold: 1.0, p_threshold: 0.05, title: "Tumor vs Normal")

The plot marks genes as:

• Red (upper right): significantly upregulated (high log2FC, low p-value)
• Blue (upper left): significantly downregulated (negative log2FC, low p-value)
• Gray (center/bottom): not significant or small effect

MA Plot

The MA plot shows the relationship between average expression (x-axis) and fold change (y-axis). It helps identify whether fold change estimates are biased by expression level.

# MA plot
ma_plot(de_results)

In a well-behaved experiment, the cloud of points should be centered on log2FC = 0 across all expression levels. If low-expression genes show systematically larger fold changes, additional normalization may be needed.

Multiple Testing Correction

When you test 20,000 genes for differential expression at p < 0.05, you expect 1,000 false positives purely by chance (0.05 x 20,000 = 1,000). Multiple testing correction adjusts p-values to control the false discovery rate.

The Benjamini-Hochberg method is the standard correction. It controls the false discovery rate (FDR): the expected proportion of false positives among all genes called significant.

# Why correction matters
let raw_pvals = [0.001, 0.01, 0.03, 0.04, 0.049, 0.06, 0.1]
let adjusted = p_adjust(raw_pvals, "BH")
println("Raw vs Adjusted p-values:")
for i in range(0, len(raw_pvals)) {
    println(f"  {raw_pvals[i]} -> {round(adjusted[i], 4)}")
}

Expected output:

Raw vs Adjusted p-values:
  0.001 -> 0.007
  0.01 -> 0.035
  0.03 -> 0.07
  0.04 -> 0.07
  0.049 -> 0.0686
  0.06 -> 0.07
  0.1 -> 0.1

Notice how some p-values that were below 0.05 (raw) become above 0.05 after correction. This removes likely false positives.

Rules of thumb:

• Always use adjusted p-values (padj) when testing many genes.
• FDR < 0.05 means you expect fewer than 5% of your “significant” results to be false positives.
• FDR < 0.01 is a more stringent threshold for high-confidence results.
• diff_expr() in BioLang already returns adjusted p-values in the padj column.

Complete RNA-seq Pipeline

Putting it all together into a single script:

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Complete RNA-seq Differential Expression Pipeline
# requires: data/counts.csv, data/gene_lengths.csv in working directory

println("=== RNA-seq Differential Expression Pipeline ===\n")

# Step 1: Load data
let counts = csv("data/counts.csv")
println(f"1. Loaded {nrow(counts)} genes x {ncol(counts) - 1} samples")

# Step 2: Check library sizes
let samples = ["normal_1", "normal_2", "normal_3", "tumor_1", "tumor_2", "tumor_3"]
let lib_sizes = samples
    |> map(|s| {sample: s, total: col(counts, s) |> sum()})
    |> to_table()
println("2. Library sizes:")
println(lib_sizes)

# Step 3: Normalize
let gene_lengths = csv("data/gene_lengths.csv")
let norm = tpm(counts, gene_lengths)
println(f"3. TPM normalization complete")

# Step 4: Differential expression
let de = diff_expr(counts,
    control: ["normal_1", "normal_2", "normal_3"],
    treatment: ["tumor_1", "tumor_2", "tumor_3"]
)

# Step 5: Filter significant
let sig = de
    |> filter(|r| r.padj < 0.05 and abs(r.log2fc) > 1.0)
    |> arrange("padj")

let up = sig |> filter(|r| r.log2fc > 0) |> nrow()
let down = sig |> filter(|r| r.log2fc < 0) |> nrow()
println(f"4. Significant: {nrow(sig)} genes ({up} up, {down} down)")

# Step 6: Show top results
println("\n   Top upregulated:")
let top_up = sig |> filter(|r| r.log2fc > 0) |> head(5)
println(top_up)

println("\n   Top downregulated:")
let top_down = sig |> filter(|r| r.log2fc < 0) |> head(5)
println(top_down)

# Step 7: Visualize
println("\n5. Generating volcano plot...")
volcano(de, fc_threshold: 1.0, p_threshold: 0.05, title: "Tumor vs Normal DE")

# Step 8: Export
write_csv(sig, "results/significant_genes.csv")
println(f"6. Results saved: results/significant_genes.csv")
println("\n=== Pipeline complete ===")

Exercises

1. Build a count matrix. Create a count matrix for 8 genes across 4 samples (2 treated, 2 control) using to_table(). Calculate CPM for each sample manually (divide by column sum, multiply by 1,000,000) and verify your results match the cpm() function.

2. Compute fold change. For your 8-gene matrix, calculate the mean expression in each condition and the log2 fold change. Which genes have the largest positive fold change? Which have the largest negative?

3. Differential expression. Load data/counts.csv and run diff_expr(). How many genes have |log2FC| > 2? What are they? Why might a stricter threshold (|log2FC| > 2) be preferred over |log2FC| > 1?

4. Volcano plot interpretation. Generate a volcano plot from the differential expression results. Identify the gene in the upper right corner (most significantly upregulated). Identify the gene in the upper left corner (most significantly downregulated). What are their biological roles?

5. Multiple testing. Generate a list of 100 random p-values between 0 and 1. Apply Benjamini-Hochberg correction with p_adjust(). How many are significant at raw p < 0.05? How many remain significant at adjusted p < 0.05? What does this tell you about false positives?

Key Takeaways

• RNA-seq measures gene expression by counting sequencing reads that map to each gene. More reads = higher expression.
• Raw counts need normalization. CPM corrects for library size (sequencing depth). TPM corrects for both gene length and library size. Use TPM for cross-gene comparisons.
• Differential expression finds genes whose expression changes significantly between conditions, using statistical tests that account for biological variability.
• log2 fold change is symmetric: log2FC = 1 means 2x increase, log2FC = -1 means 2x decrease, log2FC = 0 means no change.
• Always correct for multiple testing. Testing 20,000 genes at p < 0.05 generates about 1,000 false positives by chance. Benjamini-Hochberg correction controls the false discovery rate.
• Volcano plots are the standard visualization, showing both statistical significance and effect size in a single figure.

What’s Next

Tomorrow: statistics for bioinformatics — hypothesis testing, p-values, and when to use which test. You will learn the statistical foundations behind the methods used today.

Day 14: Statistics for Bioinformatics

What You’ll Learn

• How to compute descriptive statistics and summarize data before testing
• What p-values actually mean (and what they do not mean)
• How to compare two groups with t-tests (independent, paired, one-sample)
• When to use non-parametric tests like Wilcoxon rank-sum
• How to compare three or more groups with ANOVA
• How to measure correlation (Pearson, Spearman, Kendall)
• How to fit a simple linear regression model
• Why multiple testing correction is critical in genomics
• How to test categorical associations with chi-square and Fisher’s exact test
• How to choose the right statistical test for your data

The Problem

Your experiment shows gene X is 2.3x higher in tumor samples. But is that real, or just random noise? With only 3 replicates, how confident can you be? Statistics separates genuine biological signals from experimental noise.

Yesterday you ran a differential expression pipeline that used t-tests, p-values, and FDR correction behind the scenes. Today you will learn how those methods work, when to use each one, and — just as importantly — when not to use them. Every bioinformatician needs this foundation because nearly every biological conclusion depends on a statistical claim.

Descriptive Statistics First

Before running any test, look at your data. Descriptive statistics tell you the shape, center, and spread of your measurements. Skipping this step is one of the most common mistakes in bioinformatics.

let expression = [5.2, 8.1, 3.4, 6.7, 4.1, 9.3, 7.5, 2.8]

println(f"Mean:     {round(mean(expression), 2)}")
println(f"Median:   {round(median(expression), 2)}")
println(f"Stdev:    {round(stdev(expression), 2)}")
println(f"Variance: {round(variance(expression), 2)}")
println(f"Min:      {min(expression)}")
println(f"Max:      {max(expression)}")
println(f"Range:    {max(expression) - min(expression)}")
println(f"Q25:      {round(quantile(expression, 0.25), 2)}")
println(f"Q75:      {round(quantile(expression, 0.75), 2)}")

Expected output:

Mean:     5.89
Median:   5.95
Stdev:    2.32
Variance: 5.37
Min:      2.8
Max:      9.3
Range:    6.5
Q25:      3.93
Q75:      7.65

What to look for:

• Mean vs median: If they are far apart, the data may be skewed. Here they are close (5.89 vs 5.95), suggesting roughly symmetric data.
• Standard deviation: Gives a sense of how spread out the data is. Here stdev = 2.32 on a mean of 5.89 means moderate variability.
• Range and quartiles: Min/max reveal outliers. The interquartile range (Q75 - Q25 = 3.72) captures the middle 50%.

For a table with multiple columns, describe() gives a quick overview:

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

let data = csv("data/experiment.csv")
println(describe(data))

Expected output:

stat      control_1  control_2  control_3  treated_1  treated_2  treated_3
count     15         15         15         15         15         15
mean      48.73      50.73      49.47      64.6       67.47      65.8
stdev     26.29      27.63      26.44      29.68      32.61      30.95
min       8.0        9.0        8.0        14.0       15.0       14.0
q25       28.0       28.0       30.0       41.0       42.0       40.0
median    48.0       50.0       47.0       64.0       67.0       68.0
q75       68.0       74.0       72.0       89.0       93.0       88.0
max       95.0       97.0       93.0       110.0      118.0      115.0

Always examine your data before testing. If the mean and median diverge wildly, or the standard deviation is enormous relative to the mean, a t-test may not be appropriate.

P-values: What They Mean (and Don’t Mean)

The p-value is the most misunderstood statistic in science. Let us be precise:

P-value = the probability of observing a result this extreme (or more extreme) if there is no real effect.

That is it. The p-value answers: “If the null hypothesis were true (no difference, no correlation, no effect), how surprising would my data be?”

What a p-value is NOT:

The 0.05 threshold is a convention, not a law of nature. Ronald Fisher suggested it as a rough guide in the 1920s. A result with p = 0.049 is not fundamentally different from p = 0.051.

Always report effect size alongside p-value. A drug that lowers blood pressure by 0.1 mmHg might be “statistically significant” with 100,000 patients (tiny p-value) but biologically meaningless. Conversely, a 30% reduction in tumor size might be biologically important even if p = 0.07 with a small pilot study.

In genomics, you will see p-values as small as 10^-50 or smaller. These extreme values arise because the effects are large and the data are abundant, not because the statistics are fundamentally different.

The t-test — Comparing Two Groups

The t-test is the workhorse of biological statistics. It asks: “Are these two groups drawn from populations with different means?”

Independent two-sample t-test

Use this when you have two separate groups of subjects:

# Two-sample t-test: are tumor and normal expression different?
let normal = [5.2, 4.8, 5.1, 4.9, 5.3]
let tumor = [8.1, 7.9, 8.5, 7.6, 8.3]

let result = ttest(normal, tumor)
println(f"t-statistic: {round(result.statistic, 3)}")
println(f"p-value: {result.pvalue}")
println(f"Significant: {result.pvalue < 0.05}")

Expected output:

t-statistic: -18.908
p-value: 0.0
Significant: true

The t-statistic of -18.9 is very large in magnitude, meaning the groups are far apart relative to their variability. The p-value is essentially zero — these groups are clearly different.

Assumptions of the t-test:

1. Data are roughly normally distributed (or sample size > 30)
2. The two groups are independent
3. Variances are similar (BioLang uses Welch’s t-test by default, which relaxes this)

Paired t-test

Use this when you measure the same subjects under two conditions:

# Paired t-test: same patients, before vs after treatment
let before = [10.2, 8.5, 12.1, 9.8, 11.3]
let after = [7.1, 6.2, 8.5, 7.0, 8.8]

let result = ttest_paired(before, after)
println(f"Paired t-test p-value: {result.pvalue}")

Expected output:

Paired t-test p-value: 0.0001

Why paired? Because patient-to-patient variability is removed. Patient 1’s “before” and “after” are linked. The test focuses on the difference within each patient, not the absolute values.

One-sample t-test

Use this to test whether a sample’s mean differs from a specific value:

# One-sample t-test: is this different from a known value?
let observed = [2.1, 1.9, 2.3, 2.0, 2.2]
let result = ttest_one(observed, 2.0)
println(f"One-sample p-value: {result.pvalue}")

Expected output:

One-sample p-value: 0.3739

Here p = 0.37, meaning we have no evidence that the mean differs from 2.0. The small deviations (1.9, 2.1, 2.3) are consistent with random noise around 2.0.

When the t-test Doesn’t Work: Non-parametric Tests

The t-test assumes your data are approximately normally distributed. Biological data often are not — think of gene expression counts, survival times, or ranked categories. Non-parametric tests make no distributional assumptions.

# Wilcoxon rank-sum (Mann-Whitney U): doesn't assume normality
let control = [1.2, 3.5, 2.1, 4.8, 1.5]
let treated = [5.2, 8.1, 6.3, 9.5, 7.2]

let result = wilcoxon(control, treated)
println(f"Wilcoxon p-value: {result.pvalue}")

Expected output:

Wilcoxon p-value: 0.0079

The Wilcoxon test works by ranking all values from both groups combined, then asking whether one group’s ranks are systematically higher. It is less powerful than the t-test when data are normal, but more reliable when they are not.

When to use Wilcoxon instead of t-test:

• Small sample sizes (n < 10 per group)
• Skewed distributions (many small values, few large ones)
• Outliers present
• Ordinal data (rankings, scores)
• When you are unsure whether normality holds

Decision Guide: Choosing the Right Comparison Test

If you are unsure whether your data are normal, the non-parametric test is the safer choice. You pay a small price in statistical power, but you avoid making a potentially invalid assumption.

ANOVA — Comparing Multiple Groups

When you have three or more groups, do not run multiple t-tests (control vs low dose, control vs high dose, low vs high). That inflates your false positive rate. ANOVA tests all groups simultaneously.

# Three treatment groups
let control = [5.0, 4.8, 5.2, 4.9]
let low_dose = [6.5, 7.1, 6.8, 6.3]
let high_dose = [9.2, 8.8, 9.5, 9.0]

let result = anova([control, low_dose, high_dose])
println(f"ANOVA F-statistic: {round(result.statistic, 2)}")
println(f"ANOVA p-value: {result.pvalue}")

Expected output:

ANOVA F-statistic: 107.29
p-value: 0.0

The F-statistic compares the variance between groups to the variance within groups. A large F means the group means are more spread out than you would expect from within-group variability alone.

Important: ANOVA tells you “at least one group differs” but not which groups differ. To find out which specific pairs are different, you would follow up with pairwise t-tests (applying multiple testing correction):

# Follow-up: which pairs differ?
let pairs = [
    {name: "control vs low", result: ttest(control, low_dose)},
    {name: "control vs high", result: ttest(control, high_dose)},
    {name: "low vs high", result: ttest(low_dose, high_dose)},
]

# Collect raw p-values and adjust
let raw_ps = pairs |> map(|p| p.result.pvalue)
let adj_ps = p_adjust(raw_ps, "BH")

for i in range(0, len(pairs)) {
    println(f"  {pairs[i].name}: p = {round(adj_ps[i], 4)}")
}

Expected output:

  control vs low: p = 0.0001
  control vs high: p = 0.0
  low vs high: p = 0.0

All three pairs are significantly different even after correction. The dose-response pattern is clear.

Correlation

Correlation measures the strength and direction of the relationship between two variables. In bioinformatics, you might ask: “Do these two genes tend to go up and down together across samples?”

Pearson correlation

Measures linear relationships. Returns a single number between -1 and +1:

# Pearson correlation
let gene_a = [2.1, 3.5, 4.2, 5.8, 6.1, 7.3]
let gene_b = [1.8, 3.2, 3.9, 5.5, 6.4, 7.0]

let r = cor(gene_a, gene_b)
println(f"Pearson r: {round(r, 3)}")

Expected output:

Pearson r: 0.998

An r of 0.998 indicates a near-perfect positive linear relationship. As gene A increases, gene B increases proportionally.

Interpreting correlation coefficients:

Spearman rank correlation

Measures monotonic relationships (not necessarily linear). More robust to outliers:

# Spearman (rank-based, for non-linear relationships)
let rho = spearman(gene_a, gene_b)
println(f"Spearman rho: {round(rho.statistic, 3)}")
println(f"Spearman p-value: {rho.pvalue}")

Expected output:

Spearman rho: 1.0
Spearman p-value: 0.0

Spearman works by converting values to ranks first, then computing Pearson r on the ranks. It detects any monotonic relationship, even if the relationship is curved.

Kendall tau

Another rank-based measure, often preferred for small sample sizes:

# Kendall tau
let tau = kendall(gene_a, gene_b)
println(f"Kendall tau: {round(tau.statistic, 3)}")
println(f"Kendall p-value: {tau.pvalue}")

Expected output:

Kendall tau: 1.0
Kendall p-value: 0.0

Which correlation to use:

• Pearson: When the relationship is linear and data are normally distributed
• Spearman: When the relationship might be non-linear, or data have outliers
• Kendall: For small samples or when many values are tied

Warning: Correlation does not imply causation. Two genes may be correlated because they are both regulated by a third factor, or because they respond to the same environmental condition.

Linear Regression

Regression goes beyond correlation: it builds a predictive model. “If gene A’s expression is 5.0, what do we predict gene B’s expression to be?”

# Simple linear regression: does gene A predict gene B?
let x = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
let y = [2.1, 3.9, 6.2, 7.8, 10.1, 12.3]

let model = lm(x, y)
println(f"Slope: {round(model.slope, 3)}")
println(f"Intercept: {round(model.intercept, 3)}")
println(f"R-squared: {round(model.r_squared, 3)}")
println(f"p-value: {model.pvalue}")

Expected output:

Slope: 2.046
Intercept: -0.01
R-squared: 0.999
p-value: 0.0

Interpreting the output:

• Slope = 2.046: For every 1-unit increase in x, y increases by about 2.05.
• Intercept = -0.01: When x = 0, the predicted y is approximately 0.
• R-squared = 0.999: The model explains 99.9% of the variance in y. Values closer to 1.0 indicate a better fit.
• p-value: Tests whether the slope is significantly different from zero. Here it is essentially zero, confirming a strong relationship.

Example: predicting drug response from expression

# Gene expression vs drug sensitivity (IC50)
let expression = [1.5, 3.2, 4.8, 6.1, 7.9, 9.5]
let ic50 = [85.0, 72.0, 58.0, 45.0, 31.0, 18.0]

let model = lm(expression, ic50)
println(f"Slope: {round(model.slope, 3)}")
println(f"R-squared: {round(model.r_squared, 3)}")
println(f"p-value: {model.pvalue}")

Expected output:

Slope: -8.357
R-squared: 0.999
p-value: 0.0

The negative slope tells us that higher expression of this gene predicts lower IC50 (greater drug sensitivity). This kind of analysis is the foundation of pharmacogenomics.

Multiple Testing Correction (Critical for Genomics)

This is the single most important statistical concept in genomics. When you test many hypotheses simultaneously, false positives accumulate.

The problem: If you test 20,000 genes at p < 0.05, you expect 20,000 x 0.05 = 1,000 false positives by chance alone, even if no gene is truly differentially expressed. That is 1,000 genes that look significant but are not.

# The multiple testing problem
# Testing 20,000 genes at p < 0.05 -> expect 1,000 false positives!

let raw_pvals = [0.001, 0.005, 0.01, 0.03, 0.04, 0.049, 0.06, 0.1, 0.5, 0.9]

# Benjamini-Hochberg (FDR) -- most common in genomics
let bh = p_adjust(raw_pvals, "BH")

# Bonferroni -- most conservative
let bonf = p_adjust(raw_pvals, "bonferroni")

println("Raw       | BH        | Bonferroni")
println("----------|-----------|----------")
for i in range(0, len(raw_pvals)) {
    println(f"{raw_pvals[i]}    | {round(bh[i], 4)}   | {round(bonf[i], 4)}")
}

Expected output:

Raw       | BH        | Bonferroni
----------|-----------|----------
0.001    | 0.01   | 0.01
0.005    | 0.025   | 0.05
0.01    | 0.0333   | 0.1
0.03    | 0.075   | 0.3
0.04    | 0.08   | 0.4
0.049    | 0.0817   | 0.49
0.06    | 0.0857   | 0.6
0.1    | 0.125   | 1.0
0.5    | 0.5556   | 1.0
0.9    | 0.9   | 1.0

Understanding the Methods

Bonferroni correction multiplies each p-value by the number of tests. It is the most conservative method — very few false positives, but many real effects are missed.

Benjamini-Hochberg (BH) controls the False Discovery Rate (FDR). At FDR < 0.05, you expect fewer than 5% of your “significant” results to be false positives. This is the standard in genomics because it balances sensitivity and specificity.

Key observations from the table above:

• Raw p = 0.001 survives both corrections (a strong signal stays strong).
• Raw p = 0.03 is significant by raw p-value but NOT by BH (FDR = 0.075) — this was likely noise.
• Raw p = 0.049 (barely significant) has BH-adjusted p = 0.082 — no longer significant.
• Bonferroni is much harsher: only the two smallest p-values survive at the 0.05 level.

When to use which:

Chi-square and Fisher’s Exact Test

These tests are for categorical data — counts of items in categories, not continuous measurements.

Chi-square goodness-of-fit test

# Chi-square goodness-of-fit: do observed counts match expected?
# Example: are mutations distributed equally across 4 gene regions?

let observed = [30, 15, 25, 10]
let expected = [20, 20, 20, 20]
let result = chi_square(observed, expected)
println(f"Chi-square statistic: {round(result.statistic, 2)}")
println(f"Chi-square p-value: {result.pvalue}")

Expected output:

Chi-square statistic: 12.5
Chi-square p-value: 0.0059

The p-value of 0.0059 indicates that the observed mutation counts differ significantly from a uniform distribution across the four regions. Some regions are mutation hotspots.

Fisher’s exact test

For small sample sizes (any cell count < 5), use Fisher’s exact test instead:

# Fisher's exact test: for small sample sizes
#
#                  Responded    Didn't respond
# Mutated             8               2
# Wild-type            1               9

let result = fisher_exact(8, 2, 1, 9)
println(f"Fisher's exact p-value: {result.pvalue}")

Expected output:

Fisher's exact p-value: 0.0014

Fisher’s exact test computes the exact probability rather than relying on an approximation. With small numbers, the chi-square approximation breaks down, so Fisher’s test is preferred.

Choosing the Right Test

Use this reference table when you are unsure which test to apply:

Complete Example: Experiment Analysis

Let us put everything together. You have expression data from 15 genes measured across 6 samples (3 control, 3 treated). The goal: which genes respond to treatment?

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Complete statistical analysis of an experiment
# Requires: data/experiment.csv (run init.bl first)

println("=== Complete Experiment Analysis ===\n")

# Step 1: Load and describe data
let data = csv("data/experiment.csv")
println("Step 1: Data overview")
println(f"  Genes: {nrow(data)}")
println(describe(data))
println("")

# Step 2: Per-gene descriptive statistics
let control_cols = ["control_1", "control_2", "control_3"]
let treated_cols = ["treated_1", "treated_2", "treated_3"]

let gene_stats = []
for i in range(0, nrow(data)) {
    let gene = col(data, "gene")[i]
    let ctrl_vals = control_cols |> map(|c| col(data, c)[i])
    let trt_vals = treated_cols |> map(|c| col(data, c)[i])

    let ctrl_mean = mean(ctrl_vals)
    let trt_mean = mean(trt_vals)
    let fc = trt_mean / ctrl_mean
    let log2fc = log2(fc)

    # t-test per gene
    let test = ttest(ctrl_vals, trt_vals)

    gene_stats = gene_stats + [{
        gene: gene,
        ctrl_mean: round(ctrl_mean, 1),
        trt_mean: round(trt_mean, 1),
        log2fc: round(log2fc, 2),
        pvalue: test.p_value,
    }]
}

let results = to_table(gene_stats)

# Step 3: Multiple testing correction
let raw_ps = col(results, "pvalue")
let adj_ps = p_adjust(raw_ps, "BH")

println("Step 2: Per-gene test results (with FDR correction)")
println("gene       | ctrl_mean | trt_mean | log2fc | raw_p    | adj_p")
println("-----------|-----------|----------|--------|----------|------")
for i in range(0, nrow(results)) {
    let g = col(results, "gene")[i]
    let cm = col(results, "ctrl_mean")[i]
    let tm = col(results, "trt_mean")[i]
    let lfc = col(results, "log2fc")[i]
    let rp = round(raw_ps[i], 4)
    let ap = round(adj_ps[i], 4)
    println(f"{g}  | {cm}  | {tm}  | {lfc}  | {rp}  | {ap}")
}

# Step 4: Filter significant genes
let sig_count = 0
let up_count = 0
let down_count = 0
for i in range(0, len(adj_ps)) {
    if adj_ps[i] < 0.05 {
        sig_count = sig_count + 1
        if col(results, "log2fc")[i] > 0 {
            up_count = up_count + 1
        } else {
            down_count = down_count + 1
        }
    }
}

println(f"\nStep 3: Significant genes (FDR < 0.05): {sig_count}")
println(f"  Upregulated: {up_count}")
println(f"  Downregulated: {down_count}")

# Step 5: Correlation between control replicates (quality check)
let ctrl1 = col(data, "control_1")
let ctrl2 = col(data, "control_2")
let r = cor(ctrl1, ctrl2)
println(f"\nStep 4: Replicate correlation (control_1 vs control_2): r = {round(r, 3)}")

# Step 6: Linear model: does control expression predict treated expression?
let ctrl_means = []
let trt_means = []
for i in range(0, nrow(data)) {
    let cv = control_cols |> map(|c| col(data, c)[i])
    let tv = treated_cols |> map(|c| col(data, c)[i])
    ctrl_means = ctrl_means + [mean(cv)]
    trt_means = trt_means + [mean(tv)]
}

let model = lm(ctrl_means, trt_means)
println(f"\nStep 5: Linear model (control -> treated)")
println(f"  Slope: {round(model.slope, 3)}")
println(f"  R-squared: {round(model.r_squared, 3)}")

println("\n=== Analysis complete ===")

Expected output:

=== Complete Experiment Analysis ===

Step 1: Data overview
  Genes: 15
stat      control_1  control_2  control_3  treated_1  treated_2  treated_3
count     15         15         15         15         15         15
mean      48.73      50.73      49.47      64.6       67.47      65.8
stdev     26.29      27.63      26.44      29.68      32.61      30.95
min       8.0        9.0        8.0        14.0       15.0       14.0
q25       28.0       28.0       30.0       41.0       42.0       40.0
median    48.0       50.0       47.0       64.0       67.0       68.0
q75       68.0       74.0       72.0       89.0       93.0       88.0
max       95.0       97.0       93.0       110.0      118.0      115.0

Step 2: Per-gene test results (with FDR correction)
gene       | ctrl_mean | trt_mean | log2fc | raw_p    | adj_p
-----------|-----------|----------|--------|----------|------
GENE01  | 8.3  | 14.3  | 0.78  | 0.0199  | 0.0498
GENE02  | 22.0  | 24.7  | 0.17  | 0.5834  | 0.6563
GENE03  | 95.0  | 114.3  | 0.27  | 0.0462  | 0.0866
GENE04  | 30.0  | 42.3  | 0.5  | 0.019  | 0.0498
GENE05  | 48.0  | 64.3  | 0.42  | 0.0105  | 0.0394
GENE06  | 68.0  | 89.7  | 0.4  | 0.0138  | 0.0414
GENE07  | 42.0  | 14.7  | -1.52  | 0.0024  | 0.018
GENE08  | 74.0  | 93.0  | 0.33  | 0.0262  | 0.0561
GENE09  | 12.0  | 42.3  | 1.82  | 0.0015  | 0.018
GENE10  | 55.0  | 68.0  | 0.31  | 0.0725  | 0.1088
GENE11  | 28.0  | 40.7  | 0.54  | 0.0225  | 0.0498
GENE12  | 38.0  | 52.7  | 0.47  | 0.0238  | 0.0498
GENE13  | 85.0  | 112.3  | 0.4  | 0.0095  | 0.0394
GENE14  | 58.0  | 60.3  | 0.06  | 0.7352  | 0.7352
GENE15  | 68.0  | 55.7  | -0.29  | 0.1252  | 0.1627

Step 3: Significant genes (FDR < 0.05): 9
  Upregulated: 7
  Downregulated: 2

Step 4: Replicate correlation (control_1 vs control_2): r = 0.998

Step 5: Linear model (control -> treated)
  Slope: 1.181
  R-squared: 0.933

=== Analysis complete ===

Interpreting these results:

• 9 out of 15 genes are significantly changed after FDR correction (several border-line raw p-values did not survive correction — see GENE03 with raw p = 0.046 but adj p = 0.087).
• GENE09 is strongly upregulated (log2FC = 1.82, nearly 4x increase).
• GENE07 is strongly downregulated (log2FC = -1.52, about 3x decrease).
• The high replicate correlation (r = 0.998) confirms good data quality.
• The regression slope of 1.18 tells us treated expression is on average 18% higher than control, but with gene-specific variation (R-squared = 0.93).

Exercises

1. Generate and test. Create two groups of 20 random values — control with values around 50 (e.g., 40-60 range) and treated with values around 55 (e.g., 45-65 range). Run a t-test. Is the difference significant? Try increasing the gap between groups or adding more samples. How does each change affect the p-value?

2. Correlation analysis. Pick any two numeric columns from data/experiment.csv and compute Pearson, Spearman, and Kendall correlations. Are the values similar? When might they diverge?

3. ANOVA follow-up. Create three groups: low = [10, 12, 11, 13], mid = [15, 14, 16, 15], high = [15, 16, 14, 15]. Run ANOVA. Then run pairwise t-tests with BH correction. Which pairs are significantly different? Is mid vs high significant?

4. Multiple testing in practice. Generate a list of 100 p-values: 90 drawn uniformly from [0.1, 1.0] (no effect) and 10 set to small values like 0.001-0.01 (real effects). Apply BH correction at FDR < 0.05. Do all 10 real effects survive? Do any false positives sneak through?

5. Regression prediction. Using the expression and IC50 data from the linear regression section, predict the IC50 for a new sample with expression = 5.0 using the model’s slope and intercept. What is the predicted IC50? How confident are you in this prediction (hint: check R-squared)?

Key Takeaways

• Always examine descriptive statistics before hypothesis testing. Know your data’s shape, center, and spread before running any test.
• P-values tell you about noise, not importance — report effect sizes too. A tiny p-value with a tiny effect is not biologically interesting.
• Use the right test for your data: parametric (t-test, ANOVA) if data are roughly normal, non-parametric (Wilcoxon) otherwise.
• Multiple testing correction is mandatory in genomics — use Benjamini-Hochberg (FDR). Without it, thousands of false positives will contaminate your results.
• Correlation does not equal causation, but it is a useful starting point for identifying co-regulated genes and pathways.
• Statistics quantifies uncertainty — it does not eliminate it. A significant result means the data are unlikely under the null hypothesis, not that you have proven a biological mechanism.

What’s Next

Tomorrow: publication-quality visualization — making figures that tell a story. You will learn how to create plots that are clear, accurate, and ready for a manuscript.

Day 15: Publication-Quality Visualization

What You’ll Learn

• Why choosing the right plot is the most important visualization decision
• How to create scatter plots, histograms, bar charts, and boxplots in BioLang
• How to use bioinformatics-specific plots: volcano, MA, Manhattan, heatmap, genome track
• How to produce quick ASCII visualizations for terminal work
• How to export SVG figures for publication and presentation
• How to use sparklines, dotplots, quality plots, and coverage charts
• Design principles that make figures clear, honest, and journal-ready

The Problem

Your analysis is done, but the reviewer says “Figure 3 is unclear.” Visualization is how you communicate results. The right plot makes your finding obvious; the wrong plot hides it. Today you learn to make figures that journals accept and audiences understand.

Yesterday you ran statistical tests to determine which genes are significantly differentially expressed. But a table of p-values does not tell a story — a volcano plot does. A list of GWAS hits does not show genomic context — a Manhattan plot does. Visualization turns numbers into insight.

BioLang includes 30+ built-in plot functions. They produce either ASCII output for quick terminal exploration or SVG for publication-quality figures. No external libraries, no R/Python interop, no dependencies to install.

Choosing the Right Plot

Before writing any code, decide what you are showing. The data type determines the plot type.

Rule of thumb:

• One continuous variable? Histogram or density.
• Two continuous variables? Scatter plot (with plot).
• One categorical, one continuous? Boxplot or bar chart.
• Matrix of values? Heatmap.
• Differential expression results? Volcano or MA plot.
• GWAS hits across the genome? Manhattan plot.
• Genomic features at a locus? Genome track.
• Sequencing quality? Quality plot.

Basic Plots

Scatter Plot

The scatter plot is the workhorse of data visualization. Use it whenever you have two continuous variables and want to see their relationship.

let data = [
    {x: 1.0, y: 2.1}, {x: 2.0, y: 3.9}, {x: 3.0, y: 6.2},
    {x: 4.0, y: 7.8}, {x: 5.0, y: 10.1},
] |> to_table()

plot(data, {x: "x", y: "y", title: "Gene Expression Correlation"})

The plot function takes a table and an options record. The x and y fields name the columns to plot. When data shows a clear linear trend like this, you know correlation is strong before computing any statistic.

Histogram

Histograms show the distribution of a single variable. Use them to check whether data is normal, skewed, or bimodal — something you should always do before running parametric tests.

let values = [2.1, 3.5, 4.2, 5.8, 6.1, 7.3, 3.8, 5.5, 4.9, 6.7, 3.2, 5.1]
histogram(values, {bins: 6, title: "Expression Distribution"})

Expected output (ASCII):

Expression Distribution

2.00 -  3.00 | ██████████           2
3.00 -  3.87 | ██████████████████   3
3.87 -  4.73 | ██████████           2
4.73 -  5.60 | ██████████████████   3
5.60 -  6.47 | █████                1
6.47 -  7.33 | █████                1

The default output is ASCII — it works in any terminal, over SSH, in log files. For publication, add format: "svg" (covered below).

Bar Chart

Bar charts compare discrete categories. They are the right choice when you have counts or totals for named groups.

let data = [
    {category: "SNP", count: 3500},
    {category: "Insertion", count: 450},
    {category: "Deletion", count: 520},
    {category: "MNV", count: 30},
]
bar_chart(data)

Expected output:

SNP       | ████████████████████████████████████████  3500
Insertion | █████                                      450
Deletion  | ██████                                     520
MNV       | ▏                                           30

The visual immediately tells you SNPs dominate — something that is less obvious staring at a column of numbers.

Boxplot

Boxplots show the distribution of values across groups: median, quartiles, and outliers at a glance. They are better than bar charts for distributions because they show spread, not just a single summary number.

# boxplot() accepts a Table — renders one boxplot per numeric column
let groups = table({
    control: [5.2, 4.8, 5.1, 4.9, 5.3, 5.0],
    treated: [8.1, 7.9, 8.5, 7.6, 8.3, 8.0],
    resistant: [5.5, 5.3, 5.8, 5.1, 5.6, 5.4]
})
boxplot(groups)

Expected output:

control   |    ├──[█|█]──┤          4.80 .. 5.30  median=5.05
treated   |              ├──[█|█]──┤ 7.60 .. 8.50  median=8.05
resistant |     ├──[█|█]──┤         5.10 .. 5.80  median=5.45

The treated group is clearly elevated. The resistant group overlaps with control — exactly the kind of visual insight a reviewer needs.

Bioinformatics-Specific Plots

Volcano Plot

The volcano plot is the standard visualization for differential expression results. It plots fold change (x-axis) against statistical significance (y-axis), making it easy to identify genes that are both large in effect and statistically significant.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: data/de_results.csv (run init.bl first)
let de = csv("data/de_results.csv")
volcano(de, {fc_threshold: 1.0, p_threshold: 0.05, title: "Tumor vs Normal"})

The function expects columns named log2fc (or log2FoldChange) and padj (or pvalue). Points are colored by significance: genes passing both thresholds are highlighted, non-significant genes are dimmed.

MA Plot

The MA plot (Bland-Altman plot for genomics) shows mean expression (x-axis) versus log fold change (y-axis). It reveals whether fold change depends on expression level — a sign of normalization problems.

let de = csv("data/de_results.csv")
ma_plot(de, {title: "MA Plot - Tumor vs Normal"})

In a well-normalized dataset, the cloud of points is centered at y=0 across all expression levels. A trend away from zero at low expression suggests the need for better normalization.

Manhattan Plot

Manhattan plots display GWAS results across the genome. Each point is a variant; the y-axis shows -log10(p-value). Peaks that rise above the genome-wide significance line mark associated loci.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: data/gwas_results.csv (run init.bl first)
let gwas = csv("data/gwas_results.csv")
manhattan(gwas, {title: "GWAS Results"})

The function expects columns chr, pos, and pvalue. Chromosomes alternate colors. A horizontal line marks the genome-wide significance threshold (5e-8).

Heatmap

Heatmaps visualize matrix data — gene expression across samples, correlation matrices, or any row-by-column numeric data. Color intensity encodes value.

let matrix = [
    {gene: "BRCA1", S1: 2.4, S2: 3.1, S3: 1.8},
    {gene: "TP53", S1: -1.2, S2: -0.8, S3: -1.5},
    {gene: "EGFR", S1: 4.1, S2: 3.8, S3: 4.5},
    {gene: "MYC", S1: 1.9, S2: 2.2, S3: 1.7},
] |> to_table()
heatmap(matrix, {title: "Expression Heatmap"})

Expected output (ASCII):

Expression Heatmap

       S1     S2     S3
BRCA1  ▓▓▓    ████   ▓▓
TP53   ░░     ░      ░░░
EGFR   █████  ████   █████
MYC    ▓▓     ▓▓▓    ▓▓

Darker blocks = higher values. The pattern is immediately visible: EGFR is highly expressed, TP53 is down. For publication, use format: "svg" to get a proper color-coded heatmap.

Genome Track

Genome tracks display genomic features along a chromosomal region. Use them to show gene models, variants, regulatory elements, or any feature with coordinates.

let features = [
    {chrom: "chr17", start: 43044295, end: 43125483, name: "BRCA1", strand: "+"},
    {chrom: "chr17", start: 43170245, end: 43176514, name: "NBR2", strand: "-"},
    {chrom: "chr17", start: 43104956, end: 43104960, name: "variant1", strand: "+"},
] |> to_table()
genome_track(features, {title: "BRCA1 Locus"})

The function renders a linear representation of the region with features drawn at their coordinates. Gene bodies, point mutations, and regulatory regions are distinguishable by size and annotation.

ASCII vs SVG Output

BioLang plot functions produce ASCII by default. This is ideal for quick exploration — it works in any terminal, renders instantly, and needs no graphics setup. For publication, switch to SVG.

# ASCII output (default --- works everywhere)
bar_chart(data)

# SVG output (for publications, presentations, web)
bar_chart(data, {format: "svg"})

# Save SVG to file
let svg = bar_chart(data, {format: "svg"})
save_svg(svg, "figures/variant_types.svg")

Why SVG?

• Vector format: infinite resolution at any zoom level
• Small file size compared to raster images
• Editable in Inkscape, Illustrator, or any text editor
• Most journals accept SVG directly or convert it to PDF
• Web-friendly: renders in any browser

The save_svg function writes the SVG string to a file. The save_plot function does the same — they are aliases.

# These are equivalent
save_svg(svg_string, "figures/plot.svg")
save_plot(svg_string, "figures/plot.svg")

Sparklines for Quick Inline Visualization

Sparklines are tiny inline charts — a single line of Unicode block characters that fit inside a sentence or log message. Use them for quick visual scans of trends.

let values = [3, 5, 2, 8, 4, 7, 1, 6]
println(sparkline(values))

Expected output:

▃▅▂█▄▇▁▆

Each character represents one value. The tallest block is the maximum (8 = █), the shortest is the minimum (1 = ▁). Sparklines are useful in reports, dashboards, and pipeline logs where you want a quick visual without a full chart.

# Per-base quality across a read
let quals = [30, 32, 35, 34, 33, 31, 28, 25, 22, 18]
println(f"Quality: {sparkline(quals)}")

Output:

Quality: ▆▇██▇▆▅▃▂▁

The quality drop-off at the read end is immediately visible.

Dotplot for Sequence Comparison

Dotplots compare two sequences by marking positions where they match. Diagonal lines indicate regions of similarity; breaks in the diagonal reveal insertions, deletions, or rearrangements.

let seq1 = dna"ATCGATCGATCG"
let seq2 = dna"ATCGTTGATCG"
dotplot(seq1, seq2, {window: 3, title: "Pairwise Comparison"})

The window parameter controls the k-mer size used for matching. Larger windows reduce noise but may miss short matches. A window of 3-5 is typical for short sequences; 10-20 for longer genomic comparisons.

Quality Plot for Sequencing Data

Quality plots show per-base quality scores across read positions. They are the first thing you should look at when evaluating sequencing data.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

let reads = read_fastq("data/reads.fastq")
let first_read = reads |> first()
quality_plot(first_read.qual)

The plot shows quality scores (Phred scale) for each position in the read. Good data has scores above 30 across most positions. A characteristic drop-off at the 3’ end is normal for Illumina data and is the reason we trim reads.

For a dataset-level view, you would typically compute mean quality per position across many reads and plot that.

Coverage Visualization

Coverage plots show read depth across a genomic region. They reveal whether sequencing is uniform or has gaps and peaks.

# coverage() accepts List of [start, end] pairs
let intervals = [
    [100, 300],
    [200, 500],
    [250, 400],
    [600, 800],
]
coverage(intervals)

Expected output:

100       200       300       400       500       600       700       800
|         |         |         |         |         |         |         |
▁▁▁▁▁▁▁▁▁▁██████████▓▓▓▓▓▓▓▓▓▓██████████▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁██████████████████▁

The height (or density character) reflects how many intervals overlap at each position. The gap between 500-600 indicates no coverage — a potential problem if that region contains your target gene.

Customization Options

Most plot functions accept an options record as their second argument. Common options work across plot types:

# Title and dimensions
plot(data, {x: "x", y: "y",
    title: "Gene Expression Correlation",
    width: 800, height: 600})

# SVG format
histogram(values, {bins: 10, title: "Distribution", format: "svg"})

# Volcano with custom thresholds
volcano(de, {fc_threshold: 1.5, p_threshold: 0.01, format: "svg"})

Options that are not recognized by a particular plot function are silently ignored, so you do not need to remember exactly which options each function supports.

Saving Figures

# Generate SVG and save in one step
let vol = volcano(de, {format: "svg", title: "Differential Expression"})
save_svg(vol, "figures/volcano.svg")

# Or more concisely via pipe
volcano(de, {format: "svg", title: "Differential Expression"})
    |> save_svg("figures/volcano.svg")

Plot Gallery

This table lists every plot function available in BioLang, what it does, and when to use it.

Design Principles for Scientific Figures

Good figures follow consistent rules. These principles apply regardless of which tool you use.

1. Label all axes with units. “Expression (log2 TPM)” is informative. “Values” is not.

2. Use colorblind-safe palettes. About 8% of men have some form of color vision deficiency. Avoid red-green contrasts. BioLang’s default palette is colorblind-safe.

3. Do not use pie charts. Bar charts are always clearer. The human eye is poor at comparing angles but good at comparing lengths.

4. Show data points alongside summaries. A boxplot shows the distribution. A bar chart with error bars hides it. Two very different distributions can produce the same mean and standard error.

5. Use SVG for publications. Raster formats (PNG, JPEG) lose quality when resized. SVG is vector — it looks sharp at any size and any DPI. Most journals accept SVG, PDF, or EPS.

6. One figure, one message. Every figure should answer one question. If you need to tell two stories, make two figures.

7. Consistent styling across panels. Use the same axis ranges, font sizes, and color coding across related panels so they can be compared directly.

Complete Example: Multi-Panel Figure

This example generates a complete set of figures from differential expression results, ready for a publication supplement.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Generate a complete set of figures for a publication
# requires: data/de_results.csv (run init.bl first)

let de = csv("data/de_results.csv")

# Figure 1: Volcano plot
let vol = volcano(de, {format: "svg", title: "A) Differential Expression"})
save_svg(vol, "figures/fig1_volcano.svg")
println("Saved figures/fig1_volcano.svg")

# Figure 2: MA plot
let ma = ma_plot(de, {format: "svg", title: "B) MA Plot"})
save_svg(ma, "figures/fig2_ma.svg")
println("Saved figures/fig2_ma.svg")

# Figure 3: Expression heatmap of top genes
let top = de |> filter(|r| r.padj < 0.01) |> arrange("padj") |> head(20)
let hm = heatmap(top, {format: "svg", title: "C) Top 20 DE Genes"})
save_svg(hm, "figures/fig3_heatmap.svg")
println("Saved figures/fig3_heatmap.svg")

# Figure 4: Summary bar chart
let up_count = de |> filter(|r| r.padj < 0.05 and r.log2fc > 1.0) |> nrow()
let down_count = de |> filter(|r| r.padj < 0.05 and r.log2fc < -1.0) |> nrow()
let ns_count = nrow(de) - up_count - down_count

let summary = [
    {category: "Up", count: up_count},
    {category: "Down", count: down_count},
    {category: "NS", count: ns_count},
]
bar_chart(summary)

println("All figures saved to figures/")

This script produces four coordinated figures. The volcano plot shows the overall landscape. The MA plot checks for normalization artifacts. The heatmap focuses on the top hits. The bar chart gives a simple summary count. Together, they tell a complete story.

Exercises

1. Histogram of GC content. Generate 100 random GC content values (between 0.3 and 0.7) and create a histogram with 10 bins. What shape do you expect?

2. Volcano plot with export. Load the DE results from data/de_results.csv, create a volcano plot with fc_threshold: 1.5 and p_threshold: 0.01, and save it as SVG.

3. Boxplot comparison. Create three groups of expression values (control, low dose, high dose) with 8 values each. Make a boxplot. Do the groups look different?

4. Genome track. Create a table with 5 genes on chromosome 17, each with start/end coordinates and strand. Display them as a genome track.

5. Heatmap from expression matrix. Create a 6-gene by 4-sample expression matrix as a table and visualize it as a heatmap. Which gene has the highest expression?

Key Takeaways

• Choose the right plot for your data type — distributions, comparisons, relationships, and genomic data each have dedicated plot types.
• BioLang has 30+ plot functions built in — no external libraries, no installation, no Python/R interop needed.
• ASCII plots for exploration, SVG for publication — the same function produces both; just add format: "svg".
• save_svg() and save_plot() export to files — pipe your SVG string directly to a file path.
• Label axes, use clear titles, avoid pie charts — follow the design principles and reviewers will thank you.
• Visualization is communication — your plot should tell the story without needing explanation.

What’s Next

Tomorrow: pathway and enrichment analysis — finding the biological meaning behind your gene lists. You have a set of differentially expressed genes; now you will ask what pathways and functions they share.

Day 16: Pathway and Enrichment Analysis

What You’ll Learn

• Why enrichment analysis is the bridge between gene lists and biological meaning
• How Over-Representation Analysis (ORA) uses Fisher’s exact test to find enriched terms
• How Gene Set Enrichment Analysis (GSEA) uses ranked lists to detect subtle coordinated shifts
• How to read GMT files and query GO, KEGG, Reactome, and STRING databases
• How to build interaction networks from your gene lists
• How to run a complete enrichment pipeline from DE results to biological interpretation

The Problem

Differential expression gave you 500 significantly changed genes. But what do they mean together? Are they all in the same pathway? Do they share a function? A list of gene names is not biology — it is a phone book. You need to ask: “Is this gene list enriched for a particular biological process?”

Enrichment analysis answers that question. It takes your gene list and asks whether any known biological category — a pathway, a cellular function, a disease association — appears more often than expected by chance. This is how you go from “500 genes changed” to “the DNA damage response is activated.”

What Is Enrichment Analysis?

Think of it as a marble analogy. You have a bag with 1000 marbles: 100 red, 900 blue. You pull 50 marbles at random. You would expect about 5 red ones (10%). But you pulled 25 red marbles. Red is “enriched” in your draw — something non-random is going on.

The same logic applies to genes. Your genome has ~20,000 genes. Only 200 are annotated as “DNA repair.” Your DE list has 500 genes. If 40 of them are DNA repair genes, that is far more than the ~5 you would expect by chance. DNA repair is enriched.

Two Approaches

There are two main strategies for enrichment analysis, and they answer slightly different questions.

ORA (Over-Representation Analysis): Binary. A gene is either “in the list” or “not in the list.” You define a cutoff (e.g., padj < 0.05 and |log2FC| > 1), take the genes that pass, and ask whether any gene set is over-represented. Uses Fisher’s exact test (hypergeometric distribution). Fast and intuitive, but throws away information — a gene with padj = 0.049 is “in” and padj = 0.051 is “out.”

GSEA (Gene Set Enrichment Analysis): Ranked. Uses all genes ranked by their fold change (or any other metric). Walks down the ranked list, computing a running sum that increases when it encounters a gene in the set and decreases otherwise. Detects subtle coordinated shifts that ORA misses — a pathway where every gene shifts slightly might not produce any single significant hit, but GSEA catches the collective movement.

Gene Set Databases

Before running enrichment, you need gene set databases — curated collections that group genes by shared function, pathway, or property.

Gene Ontology (GO)

The most widely used annotation system. Organizes gene function into three namespaces:

• Biological Process (BP): What the gene does in the cell (e.g., “DNA repair,” “apoptotic process”)
• Molecular Function (MF): The biochemical activity (e.g., “kinase activity,” “DNA binding”)
• Cellular Component (CC): Where in the cell the product acts (e.g., “nucleus,” “mitochondrion”)

GO is a directed acyclic graph: terms are linked from specific to general. “Base excision repair” is a child of “DNA repair,” which is a child of “response to DNA damage.”

KEGG

The Kyoto Encyclopedia of Genes and Genomes. Focuses on metabolic and signaling pathways drawn as maps. KEGG pathways show how proteins interact in specific processes (e.g., “p53 signaling pathway,” “cell cycle”). Good for understanding mechanism.

Reactome

A curated, peer-reviewed pathway database. Pathways are organized hierarchically and linked to specific reactions. More detailed than KEGG for signaling cascades and immune pathways.

MSigDB Hallmark Gene Sets

The Molecular Signatures Database curates gene sets for computational biology. The “Hallmark” collection contains 50 well-defined gene sets representing specific biological states and processes (e.g., “HALLMARK_DNA_REPAIR,” “HALLMARK_P53_PATHWAY,” “HALLMARK_INFLAMMATORY_RESPONSE”). These are particularly useful for cancer biology.

Reading Gene Sets

Gene sets are commonly distributed in GMT (Gene Matrix Transposed) format. Each line is a gene set: name, description, then gene symbols separated by tabs.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Load gene sets from a GMT file
let gene_sets = read_gmt("data/hallmark.gmt")
println(f"Gene sets loaded: {len(gene_sets)}")

# gene_sets is a Map: set_name -> List of gene symbols
# Examine a specific set
let dna_repair = gene_sets["HALLMARK_DNA_REPAIR"]
println(f"DNA repair genes: {len(dna_repair)}")
println(f"First 5: {dna_repair |> take(5)}")

Expected output:

Gene sets loaded: 8
DNA repair genes: 15
First 5: [BRCA1, BRCA2, RAD51, ATM, ATR]

The read_gmt() function returns a Map where each key is a gene set name and each value is a list of gene symbols. This is the format that enrich() and gsea() expect.

Over-Representation Analysis (ORA)

ORA asks: “Are my DE genes enriched for any gene set?” It uses the hypergeometric test, which is the exact probability of drawing at least k successes from a population of size N containing K successes, when drawing n items.

The enrich() function takes three arguments: your gene list, the gene sets map, and the background size (total number of genes in the genome).

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Define DE genes (from a differential expression experiment)
let de_genes = ["BRCA1", "RAD51", "ATM", "CHEK2", "TP53", "MDM2",
                "CDKN1A", "EGFR", "KRAS", "MYC", "BCL2", "BAX",
                "CASP3", "CASP9", "PTEN", "RB1", "E2F1", "CDK4"]

# Load gene sets
let gene_sets = read_gmt("data/hallmark.gmt")

# Run ORA with background size of 20,000 (approximate human gene count)
let results = enrich(de_genes, gene_sets, 20000)
println(f"Total terms tested: {nrow(results)}")

# Filter for significant results and sort by FDR
let sig = results |> filter(|r| r.fdr < 0.05) |> arrange("fdr")
println(f"\nSignificant terms (FDR < 0.05): {nrow(sig)}")
println(sig)

Expected output:

Total terms tested: 8

Significant terms (FDR < 0.05): 3
term                     overlap  p_value   fdr       genes
HALLMARK_P53_PATHWAY     6        0.00001   0.00008   TP53,MDM2,CDKN1A,BAX,PTEN,RB1
HALLMARK_DNA_REPAIR      4        0.00023   0.00092   BRCA1,RAD51,ATM,CHEK2
HALLMARK_APOPTOSIS       4        0.00031   0.00083   BCL2,BAX,CASP3,CASP9

The output table has five columns:

• term: the gene set name
• overlap: how many of your genes are in this set
• p_value: raw hypergeometric p-value
• fdr: Benjamini-Hochberg adjusted p-value
• genes: which of your genes overlapped

Note: ora() is an alias for enrich() — they call the same function.

Gene Set Enrichment Analysis (GSEA)

GSEA does not use a cutoff. Instead, it takes a table of all genes ranked by a score (typically log2 fold change) and asks whether genes in a set tend to cluster at the top or bottom of the ranked list.

The gsea() function takes two arguments: a table with “gene” and “score” columns, and the gene sets map.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Load the full ranked gene list (all genes, not just significant ones)
let ranked = csv("data/ranked_genes.csv")
println(f"Total ranked genes: {nrow(ranked)}")
println(ranked |> head(5))

# Load gene sets
let gene_sets = read_gmt("data/hallmark.gmt")

# Run GSEA
let gsea_results = gsea(ranked, gene_sets)
println(f"\nGSEA results: {nrow(gsea_results)}")

# Filter for significant results
let gsea_sig = gsea_results |> filter(|r| r.fdr < 0.25)
println(f"Significant terms (FDR < 0.25): {nrow(gsea_sig)}")
println(gsea_sig)

Expected output:

Total ranked genes: 100
gene    score
EGFR    3.12
ERBB2   2.91
KRAS    2.67
CDKN2A  2.53
BRCA1   2.45

GSEA results: 8
Significant terms (FDR < 0.25): 4
term                       es      nes     p_value  fdr     leading_edge
HALLMARK_P53_PATHWAY       0.72    1.85    0.001    0.004   TP53,MDM2,CDKN1A,BAX,PTEN,RB1
HALLMARK_DNA_REPAIR        0.68    1.72    0.003    0.008   BRCA1,RAD51,ATM,CHEK2,ATR
HALLMARK_APOPTOSIS         0.55    1.41    0.012    0.032   BCL2,BAX,CASP3,CASP9
HALLMARK_CELL_CYCLE       -0.48    -1.23   0.045    0.12    CDK4,E2F1,CCND1,CDK2

The GSEA output table has six columns:

• term: the gene set name
• es: enrichment score (positive = enriched at top of ranked list, negative = enriched at bottom)
• nes: normalized enrichment score (ES normalized to null distribution)
• p_value: permutation-based p-value
• fdr: Benjamini-Hochberg adjusted p-value
• leading_edge: the genes driving the enrichment signal

Why FDR < 0.25 for GSEA? The GSEA authors (Subramanian et al. 2005) recommended a more lenient FDR cutoff because the permutation-based test is conservative. Many publications use FDR < 0.25, though FDR < 0.05 is stricter and also common.

GO Term Analysis

The Gene Ontology provides structured annotations for every gene. You can look up what a term means and what annotations a protein has.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: internet connection
# Look up what a GO term means
let term = go_term("GO:0006281")
println(f"Term: {term.name}")
println(f"Namespace: {term.aspect}")
println(f"Definition: {term.definition}")

Expected output:

Term: DNA repair
Namespace: biological_process
Definition: The process of restoring DNA after damage...

The go_term() function returns a record with fields: id, name, aspect, definition, is_obsolete.

# requires: internet connection
# Get GO annotations for a protein (using UniProt accession)
let annotations = go_annotations("P38398")  # BRCA1
println(f"Total annotations: {len(annotations)}")

# Classify by namespace
let bp = annotations |> filter(|a| a.aspect == "biological_process")
let mf = annotations |> filter(|a| a.aspect == "molecular_function")
let cc = annotations |> filter(|a| a.aspect == "cellular_component")
println(f"Biological processes: {len(bp)}")
println(f"Molecular functions: {len(mf)}")
println(f"Cellular components: {len(cc)}")

# Show biological process annotations
for a in bp |> take(5) {
    println(f"  {a.go_id}: {a.go_name} [{a.evidence}]")
}

Expected output:

Total annotations: 25
Biological processes: 12
Molecular functions: 8
Cellular components: 5
  GO:0006281: DNA repair [IDA]
  GO:0006302: double-strand break repair [IDA]
  GO:0006974: cellular response to DNA damage stimulus [IEA]
  GO:0010165: response to X-ray [IMP]
  GO:0045893: positive regulation of transcription [IDA]

Each annotation record has fields: go_id, go_name, aspect, evidence, gene_product_id.

KEGG Pathway Analysis

KEGG provides metabolic and signaling pathway maps. You can search for pathways and retrieve their details.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: internet connection
# Search for DNA repair pathways
let kegg_result = kegg_find("pathway", "DNA repair")
println(f"DNA repair pathways found: {len(kegg_result)}")
for entry in kegg_result |> take(5) {
    println(f"  {entry.id}: {entry.description}")
}

Expected output:

DNA repair pathways found: 4
  hsa03410: Base excision repair
  hsa03420: Nucleotide excision repair
  hsa03430: Mismatch repair
  hsa03440: Homologous recombination

# requires: internet connection
# Get details for a specific pathway
let pathway = kegg_get("hsa03410")  # Base excision repair
println(pathway)

Expected output:

ENTRY       hsa03410                    Pathway
NAME        Base excision repair - Homo sapiens (human)
...

The kegg_find() function takes a database name (“pathway”, “genes”, “compound”) and a search query. It returns a list of records with id and description fields. The kegg_get() function returns the raw KEGG flat-file text for an entry.

You can also use kegg_link() to find cross-references between KEGG databases:

# requires: internet connection
# Find genes linked to a pathway
let genes_in_pathway = kegg_link("genes", "hsa03410")
println(f"Genes in base excision repair: {len(genes_in_pathway)}")

Reactome Pathways

Reactome provides curated biological pathway data. You can look up which pathways a gene participates in.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: internet connection
# Find pathways for BRCA1
let pathways = reactome_pathways("BRCA1")
println(f"BRCA1 pathways: {len(pathways)}")
for p in pathways |> take(5) {
    println(f"  [{p.id}] {p.name}")
}

Expected output:

BRCA1 pathways: 12
  [R-HSA-73894] DNA Repair
  [R-HSA-5685942] HDR through Homologous Recombination (HRR)
  [R-HSA-5693532] DNA Double-Strand Break Repair
  [R-HSA-69473] G2/M DNA damage checkpoint
  [R-HSA-73886] Chromosome Maintenance

Each pathway record has fields: id, name, species.

# requires: internet connection
# Search Reactome for a topic
let results = reactome_search("apoptosis")
println(f"Apoptosis entries: {len(results)}")
for r in results |> take(3) {
    println(f"  [{r.id}] {r.name} ({r.species})")
}

Visualizing Enrichment Results

A bar chart of the top enriched terms is the standard visualization for enrichment results.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Visualize top enriched terms from ORA
let gene_sets = read_gmt("data/hallmark.gmt")
let de_genes = ["BRCA1", "RAD51", "ATM", "CHEK2", "TP53", "MDM2",
                "CDKN1A", "EGFR", "KRAS", "MYC", "BCL2", "BAX",
                "CASP3", "CASP9", "PTEN", "RB1", "E2F1", "CDK4"]

let results = enrich(de_genes, gene_sets, 20000)
let top_terms = results
    |> filter(|r| r.fdr < 0.05)
    |> arrange("fdr")
    |> head(10)

# Create a bar chart of overlap counts
let chart_data = top_terms |> map(|r| {category: r.term, count: r.overlap})
bar_chart(chart_data)

Expected output:

HALLMARK_P53_PATHWAY   ██████████████████████████████ 6
HALLMARK_DNA_REPAIR    ████████████████████ 4
HALLMARK_APOPTOSIS     ████████████████████ 4

Network Context with STRING

Your enriched genes do not act in isolation. STRING is a database of known and predicted protein-protein interactions. You can build an interaction network from your gene list to see how they connect.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# requires: internet connection
# Get protein interactions for DNA repair genes
let dna_repair_genes = ["BRCA1", "RAD51", "ATM", "CHEK2", "TP53"]
let network = string_network(dna_repair_genes, 9606)  # 9606 = Homo sapiens
println(f"Interactions found: {len(network)}")

for edge in network |> take(5) {
    println(f"  {edge.protein_a} -- {edge.protein_b} (score: {edge.score})")
}

Expected output:

Interactions found: 8
  BRCA1 -- RAD51 (score: 0.999)
  BRCA1 -- ATM (score: 0.998)
  BRCA1 -- CHEK2 (score: 0.997)
  ATM -- TP53 (score: 0.999)
  ATM -- CHEK2 (score: 0.999)

Each interaction record has fields: protein_a, protein_b, score.

You can build a graph from these interactions to analyze network properties:

# requires: internet connection
# Build a graph from STRING interactions
let dna_repair_genes = ["BRCA1", "RAD51", "ATM", "CHEK2", "TP53"]
let network = string_network(dna_repair_genes, 9606)

let g = graph()
for edge in network {
    let g = add_edge(g, edge.protein_a, edge.protein_b)
}
println(f"Nodes: {node_count(g)}, Edges: {edge_count(g)}")

# Find the most connected gene (highest degree)
let gene_nodes = nodes(g)
for gene in gene_nodes {
    println(f"  {gene}: {degree(g, gene)} connections")
}

Expected output:

Nodes: 5, Edges: 8
  ATM: 4 connections
  BRCA1: 3 connections
  TP53: 3 connections
  CHEK2: 2 connections
  RAD51: 2 connections

The most connected node (highest degree) is often a hub gene — a central regulator in the pathway. In this case, ATM is the hub: it is a kinase that phosphorylates both CHEK2 and TP53 in the DNA damage response.

Complete Enrichment Pipeline

Here is a full pipeline that takes DE results, runs both ORA and GSEA, queries pathway databases, and exports the results.

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

# Complete Pathway Enrichment Pipeline
# Requires: data/de_results.csv, data/hallmark.gmt, data/ranked_genes.csv
# (run init.bl first)

println("=== Enrichment Analysis Pipeline ===\n")

# Step 1: Load DE results and extract significant genes
let de = csv("data/de_results.csv")
println(f"1. Total genes in DE results: {nrow(de)}")

let sig_genes = de
    |> filter(|r| r.padj < 0.05 and abs(r.log2fc) > 1.0)
    |> col("gene")
    |> collect()
println(f"   Significant DE genes (|log2FC| > 1, padj < 0.05): {len(sig_genes)}")

# Step 2: Load gene sets
let gene_sets = read_gmt("data/hallmark.gmt")
println(f"\n2. Gene sets loaded: {len(gene_sets)}")

# Step 3: Over-Representation Analysis
let ora_results = enrich(sig_genes, gene_sets, 20000)
let ora_sig = ora_results |> filter(|r| r.fdr < 0.05) |> arrange("fdr")
println(f"\n3. ORA results:")
println(f"   Terms tested: {nrow(ora_results)}")
println(f"   Significant (FDR < 0.05): {nrow(ora_sig)}")
println(ora_sig |> head(5))

# Step 4: Gene Set Enrichment Analysis
let ranked = csv("data/ranked_genes.csv")
let gsea_results = gsea(ranked, gene_sets)
let gsea_sig = gsea_results |> filter(|r| r.fdr < 0.25)
println(f"\n4. GSEA results:")
println(f"   Terms tested: {nrow(gsea_results)}")
println(f"   Significant (FDR < 0.25): {nrow(gsea_sig)}")
println(gsea_sig |> head(5))

# Step 5: Compare ORA and GSEA
let ora_terms = ora_sig |> col("term") |> collect()
let gsea_terms = gsea_sig |> col("term") |> collect()
println(f"\n5. Comparison:")
println(f"   ORA significant terms: {ora_terms}")
println(f"   GSEA significant terms: {gsea_terms}")

# Step 6: Export results
write_csv(ora_sig, "results/ora_results.csv")
write_csv(gsea_sig, "results/gsea_results.csv")
println(f"\n6. Results saved:")
println(f"   results/ora_results.csv")
println(f"   results/gsea_results.csv")

println("\n=== Pipeline complete ===")

Expected output:

=== Enrichment Analysis Pipeline ===

1. Total genes in DE results: 50
   Significant DE genes (|log2FC| > 1, padj < 0.05): 20

2. Gene sets loaded: 8

3. ORA results:
   Terms tested: 8
   Significant (FDR < 0.05): 3
   term                     overlap  p_value   fdr       genes
   HALLMARK_P53_PATHWAY     5        0.00003   0.00024   TP53,MDM2,CDKN2A,RB1,PTEN
   HALLMARK_DNA_REPAIR      4        0.00018   0.00072   BRCA1,BRCA2,ATM,RAD51
   HALLMARK_APOPTOSIS       3        0.00095   0.0025    BCL2,BAX,CASP3

4. GSEA results:
   Terms tested: 8
   Significant (FDR < 0.25): 4
   term                       es      nes     p_value  fdr     leading_edge
   HALLMARK_P53_PATHWAY       0.71    1.82    0.001    0.005   TP53,MDM2,CDKN2A,RB1,PTEN
   HALLMARK_DNA_REPAIR        0.65    1.68    0.004    0.011   BRCA1,BRCA2,ATM,RAD51,ATR
   HALLMARK_APOPTOSIS         0.52    1.35    0.015    0.04    BCL2,BAX,CASP3
   HALLMARK_CELL_CYCLE       -0.45   -1.18    0.048    0.13    CDK4,E2F1,CCND1

5. Comparison:
   ORA significant terms: [HALLMARK_P53_PATHWAY, HALLMARK_DNA_REPAIR, HALLMARK_APOPTOSIS]
   GSEA significant terms: [HALLMARK_P53_PATHWAY, HALLMARK_DNA_REPAIR, HALLMARK_APOPTOSIS, HALLMARK_CELL_CYCLE]

6. Results saved:
   results/ora_results.csv
   results/gsea_results.csv

=== Pipeline complete ===

Notice that GSEA detected HALLMARK_CELL_CYCLE as significant even though ORA did not. This is because the cell cycle genes in this dataset had moderate fold changes that did not pass the |log2FC| > 1 cutoff for ORA, but their coordinated downward shift was detectable by GSEA. This is the key advantage of GSEA: it catches subtle but coordinated changes.

Exercises

1. Count gene set membership. Load the GMT file and count how many gene sets contain “TP53.” (Hint: iterate over the map and check if each list contains the gene.)

2. Run ORA on a custom gene list. Pick 15 genes from the DE results and run enrich(). How do the results change compared to using all significant genes?

3. Compare ORA and GSEA. Run both methods on the same data. Do they agree on the top pathways? Which method finds more significant terms?

4. GO annotation classifier. Look up GO annotations for TP53 (UniProt: P04637) using go_annotations("P04637") and count how many annotations fall in each namespace (biological_process, molecular_function, cellular_component). (Requires internet.)

5. Network hub analysis. Build a STRING interaction network for five cancer genes of your choice. Find the gene with the highest degree (most connections). Is it biologically meaningful that this gene is the hub? (Requires internet.)

Key Takeaways

• Enrichment analysis finds biological themes in gene lists — it is the bridge between statistics and biology.
• ORA (Fisher’s exact test) is simple, fast, and intuitive. It uses a binary gene list and the hypergeometric distribution.
• GSEA uses the full ranked list and detects subtle coordinated shifts that ORA misses. Use it when you suspect pathway-level effects below single-gene significance.
• GO, KEGG, and Reactome are complementary. GO provides broad functional classification. KEGG shows pathway maps. Reactome offers detailed reaction-level curation. Use multiple databases for a complete picture.
• Always correct for multiple testing. With hundreds of terms tested, raw p-values are meaningless. Use FDR (Benjamini-Hochberg) adjusted p-values.
• Network context (STRING) shows how your genes interact physically. Hub genes with many connections are often key regulators.
• GMT format is the standard for gene set distribution. The read_gmt() function loads it into a Map that both enrich() and gsea() accept.

What’s Next

Tomorrow: protein analysis — UniProt entries, domain architecture, sequence features, and structural context for the proteins your enrichment analysis highlighted.

Day 17: Protein Analysis

What You’ll Learn

• How to work with protein sequences and understand amino acid properties
• How to query UniProt for protein information, features, domains, and GO terms
• How to access 3D structure data from the PDB
• How to analyze amino acid composition and k-mer profiles
• How to compare orthologs across species and assess mutation impact

The Problem

You found a missense mutation in EGFR. Does it affect the protein? Is it in a critical domain? What does the structure look like? Protein analysis connects genetic variants to functional consequences. DNA tells you what changed; protein analysis tells you why it matters.

Every gene encodes a protein (or several), and the protein is what actually does the work in the cell. A single amino acid change can destroy enzyme activity, disrupt a binding interface, or destabilize the entire fold. To understand the impact of a variant, you need to know the protein: its domains, its structure, its function, and the properties of the amino acids involved.

Protein Sequence Basics

Proteins are chains of amino acids. Where DNA uses a 4-letter alphabet (A, T, G, C), proteins use a 20-letter alphabet. Each amino acid has distinct chemical properties that determine how the protein folds and functions.

Amino Acid Properties
=====================

Hydrophobic: A, V, L, I, M, F, W, P    (pack in the protein interior)
Polar:       S, T, N, Q, Y, C          (surface, form hydrogen bonds)
Positive:    K, R, H                    (basic, often bind DNA/RNA)
Negative:    D, E                       (acidic, often in catalytic sites)
Special:     G (flexible), P (rigid)

Protein structure has four levels:

Levels of Protein Structure
============================

Primary    →  Amino acid sequence (MEEPQSD...)
Secondary  →  Local folding: alpha helices, beta sheets
Tertiary   →  Complete 3D fold of one chain
Quaternary →  Multiple chains assembled together

Each level builds on the previous one. The primary sequence determines everything else — change one amino acid, and the entire fold can be disrupted.

BioLang has a native protein literal type, just like DNA and RNA:

let p53 = protein"MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGPDEAPRMPEAAPPVAPAPAAPTPAAPAPAPSWPLSSSVPSQKTYPQGLNGTVNLPGRNSFEV"
println(f"Length: {len(p53)} amino acids")
println(f"Type: {type(p53)}")

Expected output:

Length: 120 amino acids
Type: Protein

The protein"..." literal validates that every character is a valid amino acid code. Just as dna"ATCG" ensures valid nucleotides, protein"MEEP..." ensures valid residues.

UniProt: The Protein Knowledge Base

UniProt is the single most important protein database. It assigns each protein a stable accession number (like P04637 for human TP53) and aggregates information from hundreds of sources: sequence, function, domains, GO annotations, disease associations, post-translational modifications, and cross-references to every other major database.

Looking Up a Protein

# requires: internet connection

# Look up a protein by accession
let entry = uniprot_entry("P04637")  # TP53
println(f"Protein: {entry.name}")
println(f"Gene: {entry.gene_names}")
println(f"Organism: {entry.organism}")
println(f"Length: {entry.sequence_length} aa")
println(f"Function: {substr(entry.function, 0, 80)}...")

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

Protein: Cellular tumor antigen p53
Gene: [TP53, P53]
Organism: Homo sapiens (Human)
Length: 393 aa
Function: Acts as a tumor suppressor in many tumor types; induces growth arrest or apop...

The uniprot_entry() function returns a record with fields: accession, name, organism, sequence_length, gene_names (a list), and function.

Getting the Protein Sequence

# requires: internet connection

# Get the FASTA sequence as a string
let fasta = uniprot_fasta("P04637")
println(f"First 60 residues: {substr(fasta, 0, 60)}")
println(f"Full length: {len(fasta)} aa")

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

First 60 residues: MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGP
Full length: 393 aa

The uniprot_fasta() function returns the raw amino acid sequence as a string.

Searching UniProt

# requires: internet connection

# Search UniProt for human kinases in the reviewed (SwissProt) database
let results = uniprot_search("kinase AND organism_id:9606 AND reviewed:true")
println(f"Human kinases in SwissProt: {len(results)}")
println(f"First 3: {results |> take(3)}")

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

Human kinases in SwissProt: 518
First 3: [{accession: P00533, name: Epidermal growth factor receptor, ...}, ...]

Protein Features and Domains

Proteins are not uniform chains — they contain distinct regions (domains) that perform specific functions. A kinase domain phosphorylates substrates. A DNA-binding domain recognizes specific sequences. A transmembrane domain anchors the protein in the membrane.

UniProt annotates these features with precise locations. The uniprot_features() function returns a list of records, each with type, description, and location fields.

# requires: internet connection

let features = uniprot_features("P04637")
println(f"Total features: {len(features)}")

# Count by type
let types = features |> map(|f| f.type) |> frequencies()
println(f"Feature types: {types}")

# Find domains
let domains = features |> filter(|f| f.type == "Domain")
for d in domains {
    println(f"  Domain: {d.description} ({d.location})")
}

# Find binding sites
let binding = features |> filter(|f| f.type == "Binding site")
println(f"\nBinding sites: {len(binding)}")
for b in binding {
    println(f"  {b.description} ({b.location})")
}

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

Total features: 68
Feature types: {Chain: 1, Domain: 3, DNA binding: 1, Region: 4, ...}
  Domain: Transactivation domain 1 (1..43)
  Domain: Proline-rich region (63..97)
  Domain: Tetramerization domain (323..356)

Binding sites: 4
  Zinc (176)
  Zinc (179)
  Zinc (238)
  Zinc (242)

Why Features Matter for Variant Interpretation

When you find a missense mutation, the first question is: where in the protein is it? A mutation in a flexible loop might be tolerated. A mutation in the DNA-binding domain that disrupts a zinc-coordinating residue is almost certainly pathogenic. Features give you this context.

# requires: internet connection

# Check if a mutation position falls in a domain
let features = uniprot_features("P04637")
let domains = features |> filter(|f| f.type == "Domain")

# TP53 R248W is one of the most common cancer mutations
let mutation_pos = 248
println(f"Mutation at position {mutation_pos}")
println(f"Domains in TP53:")
for d in domains {
    println(f"  {d.description}: {d.location}")
}
println("Position 248 falls in the DNA-binding domain (102-292)")
println("This is a hotspot mutation that disrupts DNA contact")

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

Mutation at position 248
Domains in TP53:
  Transactivation domain 1: 1..43
  Proline-rich region: 63..97
  Tetramerization domain: 323..356
Position 248 falls in the DNA-binding domain (102-292)
This is a hotspot mutation that disrupts DNA contact

GO Terms for Protein Function

Gene Ontology (GO) terms classify what a protein does at three levels: Biological Process (what it participates in), Molecular Function (what biochemical activity it has), and Cellular Component (where in the cell it acts). You encountered GO briefly in Day 16. Here we focus on protein-level annotation.

# requires: internet connection

let go_terms = uniprot_go("P04637")
println(f"GO annotations: {len(go_terms)}")

# Group by aspect
let bp = go_terms |> filter(|t| t.aspect == "biological_process") |> len()
let mf = go_terms |> filter(|t| t.aspect == "molecular_function") |> len()
let cc = go_terms |> filter(|t| t.aspect == "cellular_component") |> len()
println(f"Biological Process: {bp}")
println(f"Molecular Function: {mf}")
println(f"Cellular Component: {cc}")

# Show some specific terms
let functions = go_terms |> filter(|t| t.aspect == "molecular_function")
println(f"\nMolecular functions:")
for f in functions |> take(5) {
    println(f"  {f.id}: {f.term}")
}

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

GO annotations: 142
Biological Process: 98
Molecular Function: 24
Cellular Component: 20

Molecular functions:
  GO:0003700: DNA-binding transcription factor activity
  GO:0003677: DNA binding
  GO:0005515: protein binding
  GO:0046982: protein heterodimerization activity
  GO:0042802: identical protein binding

GO terms tell you the functional context. If a protein has “kinase activity” (MF), participates in “signal transduction” (BP), and localizes to the “plasma membrane” (CC), you have a clear picture of a membrane-associated signaling kinase.

PDB: 3D Protein Structures

The Protein Data Bank (PDB) contains experimentally determined 3D structures of proteins, solved by X-ray crystallography, cryo-EM, or NMR. Resolution matters: lower numbers mean sharper detail. A 1.5 Angstrom structure shows individual atoms; a 4.0 Angstrom structure shows overall shape but not side-chain detail.

# requires: internet connection

let structure = pdb_entry("1TUP")  # TP53 DNA-binding domain
println(f"Title: {structure.title}")
println(f"Resolution: {structure.resolution} angstrom")
println(f"Method: {structure.method}")
println(f"Release date: {structure.release_date}")
println(f"Organism: {structure.organism}")

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

Title: CRYSTAL STRUCTURE OF THE TETRAMERIZATION DOMAIN OF THE TUMOR SUPPRESSOR P53
Resolution: 1.7 angstrom
Method: X-RAY DIFFRACTION
Release date: 1995-10-15
Organism: Homo sapiens

Searching for Structures

# requires: internet connection

# Search for all structures of a protein
let p53_structures = pdb_search("TP53")
println(f"TP53 structures in PDB: {len(p53_structures)}")
println(f"First 5 IDs: {p53_structures |> take(5)}")

# Look up a specific structure for more detail
let best = pdb_entry(first(p53_structures))
println(f"\nFirst hit: {best.id}")
println(f"  Title: {best.title}")
println(f"  Method: {best.method}")
println(f"  Resolution: {best.resolution}")

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

TP53 structures in PDB: 385
First 5 IDs: [1TUP, 1TSR, 1UOL, 2AC0, 2AHI]

First hit: 1TUP
  Title: CRYSTAL STRUCTURE OF THE TETRAMERIZATION DOMAIN OF THE TUMOR SUPPRESSOR P53
  Method: X-RAY DIFFRACTION
  Resolution: 1.7

Getting the Protein Sequence from PDB

# requires: internet connection

# Get the amino acid sequence from a PDB entry (entity 1)
let seq = pdb_sequence("1TUP", 1)
println(f"Type: {type(seq)}")
println(f"Length: {len(seq)} residues")
println(f"Sequence: {substr(str(seq), 0, 50)}...")

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

Type: Protein
Length: 60 residues
Sequence: PQHLRVEGNLHAEYLDDKQTKFISLHGNVQLGDSSVKFKSNEDLRNEEGF...

The pdb_sequence() function takes a PDB ID and an entity number (typically 1 for the main protein chain) and returns a Protein value.

Amino Acid Composition Analysis

The amino acid composition of a protein tells you a lot about its character. Membrane proteins are enriched in hydrophobic residues. DNA-binding proteins are enriched in positively charged residues (K, R). Intrinsically disordered regions tend to be enriched in charged and polar residues and depleted in hydrophobic ones.

let seq = protein"MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDD"
let counts = base_counts(seq)
println(f"Amino acid counts: {counts}")

Expected output:

Amino acid counts: {A: 2, D: 5, E: 4, F: 1, K: 1, L: 7, M: 2, N: 2, P: 7, Q: 3, S: 4, T: 1, V: 2, W: 1}

Despite its name, base_counts() works on all BioLang sequence types — DNA, RNA, and Protein. It returns a map of character frequencies.

Classifying by Chemical Properties

let seq = protein"MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDD"
let counts = base_counts(seq)

# Classify each amino acid by chemical property
fn classify_aa(aa) {
    match aa {
        "A" | "V" | "L" | "I" | "M" | "F" | "W" | "P" => "hydrophobic",
        "S" | "T" | "N" | "Q" | "Y" | "C" => "polar",
        "K" | "R" | "H" => "positive",
        "D" | "E" => "negative",
        _ => "other"
    }
}

# Count by property group
let residues = split(str(seq), "")
let groups = residues |> map(|aa| classify_aa(aa)) |> frequencies()
println(f"Property distribution: {groups}")

# Calculate percentages
let total = len(residues)
for group in ["hydrophobic", "polar", "negative", "positive"] {
    let count = groups[group]
    let pct = round(count / total * 100, 1)
    println(f"  {group}: {count}/{total} ({pct}%)")
}

Expected output:

Property distribution: {hydrophobic: 20, polar: 10, negative: 9, positive: 1}
  hydrophobic: 20/48 (41.7%)
  polar: 10/48 (20.8%)
  negative: 9/48 (18.8%)
  positive: 1/48 (2.1%)

A high fraction of hydrophobic residues is expected in globular proteins (they form the core). The very low positive charge here reflects this fragment of TP53 being the transactivation domain, which is acidic (lots of D and E).

K-mer Analysis of Proteins

Just as DNA k-mers reveal motifs and repeat patterns (Day 5), protein k-mers can identify sequence motifs and conserved patterns. Dipeptide and tripeptide frequencies are used in machine learning models that predict protein localization, solubility, and function.

# Protein k-mers reveal motifs and domain signatures
let seq = protein"MEEPQSDPSVEPPLSQETFSDLWKLL"
let trimers = kmers(seq, 3)
println(f"Protein 3-mers: {len(trimers)}")
println(f"First 5 trimers: {trimers |> take(5)}")

# Count dipeptide frequencies
let dipeptides = kmer_count(seq, 2)
println(f"\nDipeptide counts (top 10):")
println(dipeptides |> head(10))

Expected output:

Protein 3-mers: 24
First 5 trimers: [MEE, EEP, EPQ, PQS, QSD]

Dipeptide counts (top 10):
EP: 2
PL: 2
PS: 2
SD: 2
SQ: 1
...

Certain dipeptides are over-represented in specific structural contexts. For example, “PP” is common in proline-rich regions that resist folding, while “LV” and “IL” clusters are typical of hydrophobic cores.

Comparing Proteins Across Species

Orthologous proteins — the same gene in different species — reveal what evolution has preserved. Highly conserved positions are functionally critical. Variable positions are tolerant of change. Comparing orthologs is one of the most powerful ways to predict whether a mutation is damaging.

# requires: internet connection

# Compare TP53 across species
let accessions = ["P04637", "Q00366", "O09185"]  # Human, Chicken, Mouse
let names = ["Human", "Chicken", "Mouse"]

let proteins = []
for i in range(0, len(accessions)) {
    let entry = uniprot_entry(accessions[i])
    proteins = proteins + [{
        species: names[i],
        accession: entry.accession,
        name: entry.name,
        organism: entry.organism,
        length: entry.sequence_length
    }]
}

let comparison = proteins |> to_table()
println("TP53 Orthologs:")
println(comparison)

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

TP53 Orthologs:
species   accession  name                       organism                length
Human     P04637     Cellular tumor antigen p53  Homo sapiens (Human)    393
Chicken   Q00366     Cellular tumor antigen p53  Gallus gallus (Chicken) 367
Mouse     O09185     Cellular tumor antigen p53  Mus musculus (Mouse)    387

The lengths differ slightly between species, but the core structure is conserved. The DNA-binding domain (roughly residues 100-290 in human) is the most highly conserved region, reflecting its critical function.

Protein Mutation Impact

When you find a missense variant, the question is: does this amino acid change matter? The answer depends on several factors:

1. Where in the protein is the mutation? (domain, active site, surface?)
2. What property changed? (charge, size, hydrophobicity?)
3. How conserved is this position? (conserved = important)

Assessing Property Changes

# Assess the impact of a point mutation
let normal = protein"MEEPQSDPSVEPPLSQE"
let mutant = protein"MEEPQSDPSVEPPLSRE"  # Q16R: glutamine → arginine

# Compare the changed residue
let normal_aa = substr(str(normal), 15, 1)
let mutant_aa = substr(str(mutant), 15, 1)
println(f"Position 16: {normal_aa} -> {mutant_aa}")

fn classify_aa(aa) {
    match aa {
        "A" | "V" | "L" | "I" | "M" | "F" | "W" | "P" => "hydrophobic",
        "S" | "T" | "N" | "Q" | "Y" | "C" => "polar",
        "K" | "R" | "H" => "positive",
        "D" | "E" => "negative",
        _ => "other"
    }
}

let normal_class = classify_aa(normal_aa)
let mutant_class = classify_aa(mutant_aa)
println(f"Property: {normal_class} -> {mutant_class}")

if normal_class != mutant_class {
    println("WARNING: Property change detected --- likely functional impact")
} else {
    println("Same property class --- may be tolerated")
}

Expected output:

Position 16: Q -> R
Property: polar -> positive
WARNING: Property change detected --- likely functional impact

A polar-to-positive change introduces a new charge. This is the kind of change most likely to disrupt protein function, especially if it occurs at a conserved position in a functional domain.

Using Ensembl VEP for Variant Assessment

For real variant assessment, the Variant Effect Predictor (VEP) integrates multiple lines of evidence: conservation, structural data, and known disease associations.

# requires: internet connection

# Assess a known pathogenic EGFR mutation
let vep = ensembl_vep("7:55249071:C:T")  # EGFR variant
println(f"Consequence: {vep.consequence}")
println(f"Gene: {vep.gene}")
println(f"Impact: {vep.impact}")

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

Consequence: missense_variant
Gene: EGFR
Impact: MODERATE

Complete Protein Analysis Pipeline

This pipeline brings together everything from this chapter: UniProt lookup, feature extraction, GO annotation, and PDB structure search. It produces a comprehensive report for any protein given its UniProt accession.

# Complete Protein Analysis Report
# requires: internet connection

fn protein_report(accession) {
    println(f"\n{'=' * 50}")
    println(f"Protein Report: {accession}")
    println(f"{'=' * 50}\n")

    # Basic info
    let entry = uniprot_entry(accession)
    println(f"Name: {entry.name}")
    println(f"Gene: {entry.gene_names}")
    println(f"Organism: {entry.organism}")
    println(f"Length: {entry.sequence_length} aa")

    # Get sequence and analyze composition
    let fasta = uniprot_fasta(accession)
    let residues = split(fasta, "")
    let total = len(residues)

    fn classify_aa(aa) {
        match aa {
            "A" | "V" | "L" | "I" | "M" | "F" | "W" | "P" => "hydrophobic",
            "S" | "T" | "N" | "Q" | "Y" | "C" => "polar",
            "K" | "R" | "H" => "positive",
            "D" | "E" => "negative",
            _ => "other"
        }
    }

    let groups = residues |> map(|aa| classify_aa(aa)) |> frequencies()
    println(f"\nComposition:")
    for group in ["hydrophobic", "polar", "negative", "positive"] {
        let count = groups[group]
        let pct = round(count / total * 100, 1)
        println(f"  {group}: {pct}%")
    }

    # Domains
    let features = uniprot_features(accession)
    let domains = features |> filter(|f| f.type == "Domain")
    println(f"\nDomains ({len(domains)}):")
    for d in domains {
        println(f"  {d.description}: {d.location}")
    }

    # GO terms
    let go = uniprot_go(accession)
    let bp = go |> filter(|t| t.aspect == "biological_process") |> len()
    let mf = go |> filter(|t| t.aspect == "molecular_function") |> len()
    let cc = go |> filter(|t| t.aspect == "cellular_component") |> len()
    println(f"\nGO annotations: {len(go)} total")
    println(f"  Biological Process: {bp}")
    println(f"  Molecular Function: {mf}")
    println(f"  Cellular Component: {cc}")

    # PDB structures
    let structures = pdb_search(first(entry.gene_names))
    println(f"\nPDB structures: {len(structures)}")
    if len(structures) > 0 {
        let top = pdb_entry(first(structures))
        println(f"  Best: {top.id} - {top.method}, {top.resolution} angstrom")
    }
}

# Generate reports for key cancer proteins
let targets = ["P04637", "P00533", "P01116"]  # TP53, EGFR, KRAS
for acc in targets {
    protein_report(acc)
}

Requires CLI: This example uses file I/O / network APIs not available in the browser. Run with bl run.

Expected output:

==================================================
Protein Report: P04637
==================================================

Name: Cellular tumor antigen p53
Gene: [TP53, P53]
Organism: Homo sapiens (Human)
Length: 393 aa

Composition:
  hydrophobic: 35.4%
  polar: 28.2%
  negative: 10.7%
  positive: 14.2%

Domains (3):
  Transactivation domain 1: 1..43
  Proline-rich region: 63..97
  Tetramerization domain: 323..356

GO annotations: 142 total
  Biological Process: 98
  Molecular Function: 24
  Cellular Component: 20

PDB structures: 385
  Best: 1TUP - X-RAY DIFFRACTION, 1.7 angstrom

==================================================
Protein Report: P00533
==================================================

Name: Epidermal growth factor receptor
Gene: [EGFR, ERBB1, HER1]
Organism: Homo sapiens (Human)
Length: 1210 aa

Composition:
  hydrophobic: 38.1%
  polar: 24.5%
  negative: 11.3%
  positive: 13.8%

Domains (4):
  Furin-like cysteine rich domain: 177..338
  Furin-like cysteine rich domain: 481..621
  Protein kinase domain: 712..979
  Receptor L domain: 57..167

GO annotations: 96 total
  Biological Process: 62
  Molecular Function: 18
  Cellular Component: 16

PDB structures: 290
  Best: 1NQL - X-RAY DIFFRACTION, 2.5 angstrom

==================================================
Protein Report: P01116
==================================================

Name: GTPase KRas
Gene: [KRAS]
Organism: Homo sapiens (Human)
Length: 189 aa

Composition:
  hydrophobic: 34.9%
  polar: 25.9%
  negative: 14.8%
  positive: 13.2%

Domains (0):

GO annotations: 78 total
  Biological Process: 52
  Molecular Function: 14
  Cellular Component: 12

PDB structures: 620
  Best: 4OBE - X-RAY DIFFRACTION, 1.2 angstrom

Exercises

1. Insulin deep dive. Look up insulin (P01308) in UniProt and list its domains, features, and GO terms. How many PDB structures exist for it?

2. Composition comparison. Get the amino acid sequences for a membrane protein (e.g., EGFR, P00533) and a nuclear protein (e.g., TP53, P04637). Compare their hydrophobic/polar/charged ratios. Which has more hydrophobic residues, and why?

3. Structure search. Find all PDB structures for EGFR using pdb_search(). Pick the first result and look up its resolution and method. How does cryo-EM resolution compare to X-ray crystallography?

4. K-mer motifs. Use kmers() and kmer_count() to analyze protein 3-mers in the first 100 residues of TP53 (get the sequence with uniprot_fasta("P04637")). Are there any repeated tripeptides?

5. Ortholog comparison. Build a protein comparison table for BRCA1 across three species: human (P38398), mouse (P48754), and chicken (F1NLG5). Compare their lengths and domain counts.

Key Takeaways

• UniProt is the primary protein knowledge base — accession numbers are stable identifiers that never change, even as annotation improves.
• Protein features map function to sequence — domains, binding sites, and active sites explain what each region of the protein does.
• GO terms classify function at three levels — biological process, molecular function, and cellular component give complementary views.
• PDB structures show the 3D shape — resolution matters; lower numbers mean more reliable atomic detail.
• Amino acid properties determine protein behavior — hydrophobicity, charge, and size all affect folding, binding, and catalysis.
• Mutations in critical domains have the highest impact — a change in an active site or binding interface is far more damaging than one in a flexible loop.

What’s Next

Tomorrow: Day 18 — Genomic Coordinates and Intervals. BED operations, overlap queries, coordinate systems (0-based vs 1-based), and the interval arithmetic that underlies every genome browser and variant annotation tool.

Day 18: Genomic Coordinates and Intervals

What You’ll Learn

• Why coordinate systems are the #1 source of bioinformatics bugs
• The difference between 0-based half-open (BED) and 1-based inclusive (VCF, GFF) coordinates
• How to create and manipulate genomic intervals
• How interval trees enable fast overlap queries on millions of regions
• How to filter variants by genomic region (exonic vs intronic)
• How to read and write BED files, and work with GFF annotations

The Problem

Your exome capture kit targets 200,000 regions. Your variant caller found 50,000 variants. Which variants fall inside targeted regions? Which exons overlap regulatory elements? Genomic interval operations answer these questions in milliseconds.

Genomic coordinates are deceptively simple — a chromosome name, a start position, and an end position. But the way those positions are counted differs between file formats, and getting it wrong means your analysis is off by one base. That one base can be the difference between “variant in exon” and “variant in intron.” At genome scale, you cannot check these by eye. You need fast, correct interval operations.

Coordinate Systems

This is the single most important concept in this chapter. If you get coordinates wrong, every downstream analysis is silently incorrect.

Position:   1  2  3  4  5  6  7  8  9 10
Sequence:   A  T  C  G  A  T  C  G  A  T

1-based inclusive (VCF, GFF, SAM):
  "positions 3-7" = C G A T C  (5 bases)
  start=3, end=7, length = end - start + 1 = 5

0-based half-open (BED, BAM index, UCSC):
  "positions 2-7" = C G A T C  (5 bases, same region!)
  start=2, end=7, length = end - start = 5
  start is included, end is EXCLUDED

The key rules: