GPT Mini Engine

Overview

Modern Large Language Models like GPT often feel like a black box. Instead of relying on existing frameworks, I wanted to understand how these models work from first principles by implementing every major component myself.

GPT Mini Engine is my long-term learning project where I build a GPT-style language model step by step, starting with the tokenizer and gradually progressing toward a complete Transformer architecture capable of generating text.

This repository documents that journey.

---

Current Progress

At the moment, the project includes a fully functional character-level tokenizer capable of:

• Building a vocabulary from raw text
• Creating character-to-index and index-to-character mappings
• Encoding text into integer tokens
• Decoding tokens back into readable text
• Converting the dataset into PyTorch tensors for model training

Although this is only the first stage, it forms the foundation for every future component of the language model.

---

Why I'm Building This

Rather than simply using APIs from existing LLM providers, I want to understand the mathematics and engineering behind language models.

This project helps me explore concepts such as:

• Tokenization
• Language Modeling
• Neural Networks
• Attention Mechanisms
• Transformer Architecture
• Training Pipelines
• Text Generation

By implementing these systems from scratch, I gain a much deeper understanding of how modern AI models actually work.

---

Roadmap

This project is being developed incrementally.

Completed

✅ Character-level Tokenizer

Currently Working On

🚧 Dataset batching 🚧 Training pipeline

Planned

• Bigram Language Model
• Self-Attention
• Multi-Head Attention
• Transformer Blocks
• Positional Embeddings
• Feed Forward Networks
• Layer Normalization
• Dropout
• GPT-style Decoder
• Text Generation
• Checkpoint Saving
• Byte Pair Encoding (BPE)
• GPT-2 Style Improvements

---

Project Philosophy

This isn't intended to compete with production LLMs. Instead, it's an educational project focused on understanding how GPT models work internally by rebuilding each component from scratch.

Every completed milestone represents another step toward a complete language model.

---

Github Link : https://github.com/sayan-dey12/gpt-mini-engine

---