NeuroText Bigram
Exploring language modeling from first principles by implementing both a statistical Bigram model and a neural-network-based Bigram model in PyTorch to learn how machines generate text one character at a time.
Media
Overview
NeuroText Bigram is an educational language modeling project that explores how machines can learn to generate text using only character-to-character relationships.
The project implements two different approaches to Bigram Language Modeling:
- A statistical model based on character frequency counts and probability distributions.
- A neural-network-based model trained with gradient descent in PyTorch.
By comparing these approaches, the project demonstrates the progression from traditional probabilistic models to neural language models—the same conceptual journey that eventually leads to modern Transformer-based architectures.
Motivation
Before building large language models such as GPT, it's important to understand the simpler models that inspired them.
This project focuses on answering questions like:
- How can a machine predict the next character?
- How are probability distributions created from text?
- How does a neural network learn these probabilities?
- Why does training reduce prediction error over time?
Building these models from scratch provides valuable intuition for understanding modern LLMs.
Implementations
Statistical Bigram Model
The first implementation constructs a character transition matrix by counting how frequently every character follows another character.
Using Laplace smoothing, these counts are converted into probability distributions, allowing the model to generate entirely new names through random sampling.
Features:
- Character vocabulary generation