tinystories

Llama2 inference in one TypeScript file

🎈 A series of lightweight GPT models featuring TinyGPT Base (~51M params) and TinyGPT2 (~95M params). Fast, creative text generation trained on whimsical stories.

Codebase for Linguistic Collapse: Neural Collapse in (Large) Language Models [NeurIPS 2024] [arXiv:2405.17767]

Dataset Generation Code for SimpleStories

Implementation of a small GPT-style transformer from scratch in PyTorch. Learn how Large Language Models work by building, training, generating text, and visualizing attention.

Train and run a small Llama 2 model from scratch on the TinyStories dataset.

Train GPT style model on tinystories dataset

Code implementation for our paper "BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training" as part of the 2024 BabyLM Challenge

Small transformer trained from scratch

A 30.72M parameter LLM running dense on an $8 ESP32-S3. Every stored parameter is multiplied on every token. 0.95 tok/s, no cloud, no SD card.

This project fine-tunes GPT-2, a popular pre-trained transformer model, to generate short stories using the TinyStories dataset. The goal is to teach GPT-2 to produce creative and coherent stories based on prompts.

Specialized agentic coding LLM — 3B, pre-trained from scratch, runs locally on Apple Silicon. No API key, no data leaving your machine, zero cost per call.

A ~30M GPT small language model trained from scratch on TinyStories — GPT-2-compatible so the ember inference server runs it.

A complete LLM inference engine from scratch in ~1400 lines of dependency-free C — transformer, BPE tokenizer, sampler, hand-rolled thread pool, int8 quantization. No OpenMP, no BLAS, no API keys.

A 29M-parameter language model written and trained from scratch in PyTorch, running in your browser with ONNX Runtime Web.

LLM Architecure, Open Source

The whole male Drosophila CNS connectome (166,700 neurons) used uncut as the spiking core of a language model. Scope, numbers, honest controls, pipeline animation.

A 27.8M-parameter GPT trained from scratch on TinyStories — transformer, tokenizer pipeline and training loop written by hand in PyTorch. Trained weights included.

Run Next-Concept Modeling (NCM) on the Apple Silicon GPU with MLX: HazyResearch's released checkpoints with verified parity to PyTorch, plus from-scratch training and benchmarks.

A 110M-parameter Llama-style transformer trained from scratch on the TinyStories dataset, optimized for high-throughput training on 4GB VRAM consumer GPUs. The project features a custom asynchronous CUDA-stream prefetcher and KV-cache inference, achieving 10k+ TPS on an RTX 3050.