NeuroText Bigram

Overview

NeuroText Bigram is an educational language modeling project that explores how machines can learn to generate text using only character-to-character relationships.

The project implements two different approaches to Bigram Language Modeling:

• A statistical model based on character frequency counts and probability distributions.
• A neural-network-based model trained with gradient descent in PyTorch.

By comparing these approaches, the project demonstrates the progression from traditional probabilistic models to neural language models—the same conceptual journey that eventually leads to modern Transformer-based architectures.

---

Motivation

Before building large language models such as GPT, it's important to understand the simpler models that inspired them.

This project focuses on answering questions like:

• How can a machine predict the next character?
• How are probability distributions created from text?
• How does a neural network learn these probabilities?
• Why does training reduce prediction error over time?

Building these models from scratch provides valuable intuition for understanding modern LLMs.

---

Implementations

Statistical Bigram Model

The first implementation constructs a character transition matrix by counting how frequently every character follows another character.

Using Laplace smoothing, these counts are converted into probability distributions, allowing the model to generate entirely new names through random sampling.

Features:

• Character vocabulary generation
• Bigram frequency matrix
• Probability normalization
• Laplace smoothing
• Character-by-character text generation
• Negative log-likelihood evaluation

---

Neural Bigram Model

The second implementation replaces explicit counting with a simple neural network.

Each input character is represented using one-hot encoding, and a trainable weight matrix learns the transition probabilities through gradient descent.

Although extremely small compared to modern LLMs, this model demonstrates the fundamental principles of neural language modeling.

Features:

• One-hot encoded inputs
• Trainable weight matrix
• Forward propagation
• Softmax probability prediction
• Gradient descent optimization
• Regularized loss function
• Learned text generation

---

Dataset