What I've built in open source
A short list of my open-source work on LLM tooling, mostly in Hugging Face tokenizers and transformers. Every speedup below keeps the output identical to the original code, and each one links to the benchmarks.
1. Faster BPE training in tokenizers
Training a BPE tokenizer with HF tokenizers is slow on large corpora, and especially slow on languages like Chinese. Yikai Liao's parallel trainer (#2501) is fast but about 14,500 lines. I wrote a compact version, about 470 lines, that keeps the three ideas the speed comes from: fixed slot coordinates, lazily checked per-pair occurrence lists, and ordered-prefix batching of merges. The vocab and merges are byte-identical to the stock trainer.
main: 3.5–19x faster end to end, 9–26x on the training step alone, 54–61% less peak memory. Yikai's own harness, 512 MiB corpora (code, English, Chinese), vocab 100k, 8 workers.2. Encoding one long text on every core, and streaming files
HF tokenizers encodes a single long input on one thread, and encoding a big file from Python means loading it into one string and building per-token objects, which costs gigabytes of RAM. I split long inputs at safe points and encode the pieces in parallel. A new Tokenizer.encode_file() streams a file: Rust threads read it with pread and hand back token ids as zero-copy numpy arrays, so memory stays constant. The ids are identical to the original.
main from Python: 2.7–9.5x faster, 92–94% less memory (code: 14.3 s / 3.96 GB → 1.5 s / 0.29 GB).3. Thread control in tokenizers 1.0
tokenizers 1.0 builds its own thread pool and ignores RAYON_NUM_THREADS, and Python has no way to set the thread count, so on shared machines it silently takes every core. I restored the environment variable and exposed set_num_threads in Python.
4. toki
My fork of HF tokenizers, where the three changes above were developed and benchmarked at larger scale: thread counts, vocab sizes, 11 pre-tokenizers, 10 normalizers, 1 and 10 GB corpora across four datasets.
5. Faster loading of quantized models in transformers
When transformers quantizes a model while loading it (for example bitsandbytes 4-bit), it reads checkpoint tensors one at a time; on network filesystems that runs at 0.3–0.7 GB/s, far below what the disk can do. Worker threads now read whole tensors ahead of the quantizer, within a memory budget.
6. Model inspector: running any model without its weights
from_config() needs the full weights in memory, about 1 TB for a 500B model in bf16, and the meta device uses no memory but can't run a forward pass. from_pretrained(dummy=True) builds the real model from its config with procedurally generated weights, so it runs at near-zero memory. You can test loading, quantization, parallelism and generation code on a 500B architecture on one machine.
7. NF4 grouped GEMM for MoE experts
There is no grouped 4-bit kernel for mixture-of-experts layers on A100/H100: transformers leaves fused experts in bf16, and looping bitsandbytes over experts is slow. I wrote a kernel that runs all experts of an MoE layer in one launch on 4-bit NF4 weights, published in Triton on the Hugging Face kernel hub, with a CUDA version on top.
nf4-grouped-gemm on the kernel hub
8. lazycomb
itertools.product, permutations and combinations copy their inputs first, so nesting them over anything large runs out of memory before the first element. lazycomb gives drop-in, itertools-compatible versions in Python and Rust that never evaluate their inputs. You can iterate any composition of combinatorial groups, with size, random access (x[i]) and ranking (.index(x)) in O(1) over spaces of 10275+ elements.