GPT Mini Engine
Building a GPT-style language model from scratch in Python to understand every core component of modern LLMs. The project currently implements a character-level tokenizer and will gradually evolve into a complete Transformer-based language model.
Media
Overview
Modern Large Language Models like GPT often feel like a black box. Instead of relying on existing frameworks, I wanted to understand how these models work from first principles by implementing every major component myself.
GPT Mini Engine is my long-term learning project where I build a GPT-style language model step by step, starting with the tokenizer and gradually progressing toward a complete Transformer architecture capable of generating text.
This repository documents that journey.
Current Progress
At the moment, the project includes a fully functional character-level tokenizer capable of:
- Building a vocabulary from raw text
- Creating character-to-index and index-to-character mappings
- Encoding text into integer tokens
- Decoding tokens back into readable text
- Converting the dataset into PyTorch tensors for model training
Although this is only the first stage, it forms the foundation for every future component of the language model.
Why I'm Building This
Rather than simply using APIs from existing LLM providers, I want to understand the mathematics and engineering behind language models.
This project helps me explore concepts such as:
- Tokenization
- Language Modeling
- Neural Networks
- Attention Mechanisms
- Transformer Architecture
- Training Pipelines
- Text Generation
By implementing these systems from scratch, I gain a much deeper understanding of how modern AI models actually work.