biocorpus — a sequence-first biology pretraining corpus

21.1 M documents · ~9.15 B tokens (Llama-3 / marin-community/marin-tokenizer). Every document places a biological sequence first, then its annotation — so an autoregressive model learns to read structure/function from sequence (recognition), and, for the curated slice emitted in both directions, to generate sequence from a specification (design).

This is not a scrape of papers or abstracts. Each record is a real UniProtKB or Ensembl entry, rendered into plain text with only substantive, learnable annotation kept — no markup tags, no bookkeeping, no uncharacterized filler.

Composition

The protein backbone is deduplicated at the UniRef50 level (50% identity): we keep the UniProtKB entry of each cluster representative, so the corpus spans the whole protein universe once, without the ~10× redundancy of raw TrEMBL. Cross-file exact-sequence dedup removes any residual duplicates (0 found — representatives are unique by construction).

Quality gate

A protein is included only if it carries genuinely learnable annotation — one of:

• tier A (biological): function, catalytic activity, subcellular localization, pathway, disease, or GO terms; or
• tier B (structural): a domain boundary, active/binding site, signal peptide, transmembrane span, PTM, disulfide, repeat, …

Entries whose only annotation is a disordered/coiled-coil region — the bulk of "Uncharacterized protein" TrEMBL entries — are dropped. Genomic records are intrinsically annotated (exon structure, UTRs, splice motifs) and always kept.

Annotation coverage of the kept set:

Document format

>tr:Q977Q6 Methionine aminopeptidase [uncultured crenarchaeote 4B7]
MTFDNYIKAGKIAGEIRENVRKTDWVGKTVYEICEYVENEIKKRGAKCAFPVNTSINEVAAHYTAEPNDEIT…

Methionine aminopeptidase — UniProtKB/TrEMBL Q977Q6 — is a 225-residue protein from
uncultured crenarchaeote 4B7 (NCBI taxon 44557).
Catalytic activity: Reaction=Release of N-terminal amino acids, preferentially methionine…; EC=3.4.11.18
GO annotations: cytoplasm (component); initiator methionyl aminopeptidase activity (function);
  metal ion binding (function); metalloexopeptidase activity (function); proteolysis (process)
Sequence features: 1 domain [6-194 (Peptidase M24)]
Keywords: Aminopeptidase; Hydrolase; Metal-binding; Protease
Lineage: Archaea > Nitrososphaerota > Nitrososphaeria > Nitrosopumilales > environmental samples

Central-dogma records show one transcript as genomic pre-mRNA (exons uppercase, introns lowercase) → spliced mRNA (with 5′UTR/CDS/3′UTR boundaries) → translated protein, and are verified: the mRNA equals the spliced exons and translate(CDS) equals the protein.

Each line of the JSONL is one document with fields: id, source, accession, entity_type, seq_type, seq_len, organism, taxid, gene, name, annotations, sequence, ordering, and the rendered text. Shards are shuffled.

Sources, licensing, provenance

• UniProtKB (Swiss-Prot + TrEMBL) and UniRef50 — UniProt Consortium, CC-BY 4.0.
• Ensembl (human GRCh38, release 112) gene models and sequence — EMBL-EBI, no restriction.

Built by a local flat-file join (no per-record web requests): UniRef50 FASTA supplies the representative set; each representative's annotation is rendered straight from the UniProtKB flat file. Fully reproducible from the builder in the biocorpus repo (builders/bio_pretrain/).