Major changes:
- Remove PRAW dependency and Reddit API credentials
- Add no-OAuth Reddit scraper using public .json endpoints
- No Reddit API keys required - simpler setup!
New scraper features:
- Uses Reddit's public .json endpoints (www.reddit.com/r/subreddit.json)
- Configurable rate limiting via request_delay setting
- Automatic retry with exponential backoff
- Fetches posts and comments without authentication
Files changed:
- reddit/scraper.py (new) - No-OAuth Reddit scraper
- reddit/subreddit.py - Updated to use scraper instead of PRAW
- requirements.txt - Removed praw dependency
- utils/.config.template.toml - Removed Reddit credentials
- config.example.toml - Updated with scraper settings
- docker-entrypoint.sh - Updated for no-auth setup
- docker-compose.yml - Removed Reddit credential env vars
- main.py - Updated exception handling
Limitations:
- Subject to Reddit's rate limiting (configurable delay)
- ~1000 post cap per subreddit listing
- Some comments may be missing in large threads
https://claude.ai/code/session_01HLLH3WjpmRzvaoY6eYSFAD