NeuroText MLP
Building a character-level Multi-Layer Perceptron (MLP) language model from scratch in PyTorch. This project extends the Bigram model by learning character embeddings and predicting the next character using multiple previous characters as context.
Overview
NeuroText MLP is the next step in my journey of building language models from scratch.
After implementing statistical and neural Bigram models, this project introduces a Multi-Layer Perceptron (MLP) language model that learns richer representations of text through character embeddings and hidden layers.
Instead of predicting the next character using only the previous one, the model learns from multiple preceding characters, enabling it to capture more meaningful patterns in language.
Motivation
Bigram models are excellent for understanding probability and language modeling fundamentals, but they have a significant limitation—they only consider one previous character when making predictions.
The goal of this project is to overcome that limitation by introducing:
- Character embeddings
- Context windows
- Hidden layers
- Non-linear activations
- Gradient-based optimization
These ideas form the foundation of more advanced neural language models.
Architecture
The model follows a simple neural language modeling pipeline:
Previous Characters
│
▼
Character Embeddings
│
▼
Embedding Concatenation
│
▼
Hidden Layer (MLP)
│
▼
Output Layer
│
▼
Probability Distribution
│
▼
Next Character Prediction
Current Features
- Character-level vocabulary generation
- Context window of previous characters
- Learnable embedding matrix
- Multi-Layer Perceptron
- Tanh activation
- Cross-entropy loss
- Mini-batch gradient descent
- Random parameter initialization
- Training loop implemented in PyTorch
What I Learned
This project helped me understand several core deep learning concepts, including:
- Character embeddings
- Dense neural networks
- Context-based language modeling
- Mini-batch training
- Backpropagation
- Cross-entropy optimization
- Parameter initialization
- Gradient descent
More importantly, it demonstrated why neural networks outperform simple statistical language models.
Key Takeaway
This project represents the transition from simple probabilistic language models to neural language models.
By introducing embeddings and hidden layers, the model begins to learn semantic relationships between characters instead of relying solely on frequency counts. These concepts serve as an essential bridge toward modern Transformer architectures and GPT-style language models.
Future Vision
NeuroText MLP is part of a larger series of educational AI projects focused on understanding language models from first principles.
The long-term roadmap includes progressively more advanced architectures, eventually leading to a complete GPT-style implementation built entirely from scratch.