AI-Powered Personalized News Aggregator

The system is divided into separate stages so that scraping, content processing, AI generation, curation, and delivery remain independent components.

3. Content Sources

The system collects AI-related content from 22 sources across RSS feeds and YouTube channels.

RSS Sources

The project currently integrates 16 RSS feeds from major AI companies, research organizations, developers, and AI communities.

YouTube Sources

The project also collects content from 6 AI-focused YouTube channels:

• Dave Ebbelaar
• Matthew Berman
• Matt Wolfe
• The AI Daily Brief
• AI Explained
• Two Minute Papers

This combination allows the system to collect information from both official AI sources and independent technical content creators.

4. AI Agents

The project uses three main AI agents, each responsible for a different part of the pipeline.

1. Digest Agent

The Digest Agent converts processed content into a concise and useful digest.

It receives the article content or YouTube transcript and generates:

• A concise title
• A short summary
• Important technical information
• Practical and actionable insights

The agent is instructed to focus on meaningful content and avoid unnecessary elements such as navigation, promotional content, social links, and other irrelevant page information.

Processed Content
       │
       ▼
  Digest Agent
       │
       ▼
Title + Summary

2. Curator Agent

The Curator Agent determines how relevant each generated digest is to the user.

It uses a structured user profile containing information such as:

• Technical background
• Areas of interest
• Preferred content style
• Expertise level
• Learning preferences

The agent then analyzes the generated digests and assigns relevance scores and rankings.

User Profile ──────┐
                   │
                   ▼
              Curator Agent
                   ▲
                   │
             Generated Digests
                   │
                   ▼
          Ranked AI Content

This allows the system to prioritize content that is more useful to the specific user instead of treating every AI article equally.

3. Email Agent

The Email Agent prepares the final personalized email experience.

It generates the introduction for the email and works with the ranked articles to produce the final digest.

Ranked Content
      │
      ▼
 Email Agent
      │
      ▼
Personalized Email
      │
      ▼
    Resend
      │
      ▼
  User Inbox

5. YouTube Transcript Handling

One important part of the pipeline is handling situations where a YouTube transcript is not available.

The system does not depend completely on transcripts for digest generation.

Transcript Processing Flow

YouTube Video
      │
      ▼
Try to Get Transcript
      │
      ▼
┌───────────────────────┐
│ Transcript Available? │
└───────────┬───────────┘
            │
       ┌────┴────┐
      YES        NO
       │          │
       ▼          ▼
  Use Transcript  Check Failure
       │          │
       │     ┌────┴─────────────┐
       │     │                  │
       │   Temporary        IP Blocked
       │    Failure              │
       │     │                   │
       │     ▼                   ▼
       │   Retry             Stop Retry
       │     │
       │     ▼
       │   Retry
       │     │
       │     ▼
       │   Retry
       │
       └──────────────┬──────────────┘
                      ▼
                Digest Generation

Retry Strategy

When transcript retrieval fails because of a temporary problem, the system can retry the transcript retrieval up to three times.

Attempt 1
   │
   ├── Success → Continue
   │
   └── Failure
         ↓
Attempt 2
   │
   ├── Success → Continue
   │
   └── Failure
         ↓
Attempt 3
   │
   ├── Success → Continue
   │
   └── Failure → Continue without transcript

However, an IP-blocked condition is treated differently.

If the system detects that the IP is blocked, there is no reason to repeatedly retry the same request. The retry loop is immediately stopped and the pipeline continues using the available metadata.

No Transcript Fallback

If a transcript cannot be obtained, the system can still generate a digest using the information available from the YouTube video:

YouTube Video
      │
      ├── Title
      │
      ├── Description
      │
      └── Transcript
            │
            ▼
      Transcript Available?
         /          \
       YES           NO
        │             │
        ▼             ▼
   Full Content    Title +
                   Description
        │             │
        └──────┬──────┘
               ▼
         Digest Agent
               │
               ▼
          AI Digest

When the transcript is unavailable, the title and meaningful description are used as the fallback context.

The model is instructed to generate the digest only from the information available and not invent details that are not supported by the source.

This allows the pipeline to continue processing useful content even when transcript extraction is unavailable.

6. Key Features

• Multi-source AI content collection

  • RSS feeds
  • YouTube channels

• Automated content processing

  • RSS → Markdown
  • YouTube → Transcript

• LLM-powered digest generation

  • Structured summaries
  • Technical insights
  • Actionable information

• Personalized content curation

  • User-specific profile
  • Relevance scoring
  • Ranking

• Automated email generation

  • Personalized introduction
  • Ranked articles
  • Original source links

• Automated delivery

  • Email delivery through Resend

• Fault-tolerant transcript processing

  • Retry handling
  • IP-block detection
  • Title + description fallback

7. Content Pipeline

The complete processing workflow can be represented as:

         DISCOVER
            │
            ▼
     RSS + YouTube
            │
            ▼
        SCRAPING
            │
            ▼
       PostgreSQL
            │
            ▼
       PROCESSING
       ┌────┴────┐
       │         │
      RSS       YouTube
       │         │
       ▼         ▼
   Markdown   Transcript
       │         │
       └────┬────┘
            ▼
      UNDERSTAND
            │
            ▼
       Digest Agent
            │
            ▼
      PERSONALIZE
            │
            ▼
      Curator Agent
            │
            ▼
        RANKED DATA
            │
            ▼
        EMAIL
            │
            ▼
       Email Agent
            │
            ▼
          Resend
            │
            ▼
       User Inbox

8. Database Layer

PostgreSQL acts as the persistent storage layer of the application.

The system stores:

• YouTube videos
• RSS articles
• Processed content
• Generated digests

SQLAlchemy is used as the database ORM and repository layer.

This separates the persistent data layer from the scraping, processing, and AI components.

9. Tech Stack

Backend

• Python
• SQLAlchemy
• Pydantic
• PostgreSQL
• uv

AI & Agents

• OpenRouter
• LLM APIs
• Structured LLM Outputs
• AI Agents

Content Sources & Processing

• RSS Feeds
• YouTube
• YouTube Transcript API
• Markdown Processing

Infrastructure & Delivery

• Docker — PostgreSQL container
• Resend — Email delivery
• Environment-based Configuration

10. Design Philosophy

The project is built around the idea of turning information overload into personalized knowledge.

Instead of manually visiting multiple AI websites, reading dozens of articles, watching videos, and deciding what is relevant, the system automates the entire process:

Discover
   ↓
Process
   ↓
Understand
   ↓
Personalize
   ↓
Deliver

The project also demonstrates how different AI engineering concepts can work together in a practical application:

LLMs + AI Agents + Backend Systems + Databases + APIs + Automation

The focus is on building a complete AI-powered system rather than using an LLM as an isolated feature.