Finding jobs on Indeed is time-consuming. Job seekers manually search, scroll, and track listings across multiple sessions. There's no way to aggregate, search, and track Indeed jobs in one place, and no tools to match your resume against available positions or track your application pipeline.
A complete job board website that scrapes real job listings from Indeed.com using Playwright (headless Chromium) and displays them with a searchable, filterable interface. The app scrapes Indeed by launching a real Chromium browser that bypasses Cloudflare's JavaScript challenges, parses job cards (title, company, location, salary, links) from the HTML, and stores them in a database. Searches then serve instantly from the DB. Key features: - Live Indeed scraping via Playwright (headless Chromium bypasses Cloudflare) - Instant DB-backed search (no loading delays per request) - Job detail pages with direct apply links to Indeed - User authentication (JWT cookies + bcrypt password hashing) - Save favorite jobs and save searches - Application tracking pipeline (Bookmarked -> Applied -> Interview -> Offer -> Rejected) - Analytics dashboard with job stats, top companies/locations bar charts, salary insights - Resume keyword matcher (paste resume, get match % per job with scored results) - RSS feed for subscribing to new jobs - CSV export of tracked applications - Dark mode toggle with localStorage persistence - Graceful fallback to free legal job APIs (Adzuna, Jooble, Arbeitnow) when Indeed blocks - 2Captcha integration for CAPTCHA solving with daily budget guard - Rate limiting, result caching, anti-bot stealth measures - 24 pytest tests passing - CI/CD via GitHub Actions (daily auto-refresh + deploy to GitHub Pages) - Dockerfile + docker-compose for containerized deployment - Cloudflare Tunnel for live public access
1. Cloudflare blocking: Indeed uses Cloudflare to block datacenter IPs and non-browser clients. Solved by using Playwright with a real Chromium browser + stealth measures (user agent spoofing, anti-automation flag removal, randomized delays). 2. CAPTCHA encounters: Built a 2Captcha integration with daily budget guards that auto-detects captcha type (reCAPTCHA, hCaptcha, Cloudflare Turnstile) and solves it within the browser session. 3. Scraping timeout: Live scraping takes 30-60s, exceeding request timeouts. Solved by pre-populating the DB from local scraping (residential IP) and serving searches instantly from the DB. 4. Deployment without root/card: The Plesk server can't run Python+Chromium (no root), and cloud platforms require a credit card. Solved by deploying via GitHub Pages (free) + Cloudflare Tunnel (free, no card). 5. Indeed selector changes: Indeed changes their HTML markup frequently. Built the parser with multiple selector fallbacks per field and unit tests against saved HTML fixtures so it's easy to fix and verify offline.
- Anti-bot evasion: How Cloudflare detects bots (JS challenges, IP reputation, TLS fingerprinting) and how to work around them ethically with real browser automation. - Source-agnostic architecture: Designing an abstraction layer (JobFetcher ABC) so each job source normalizes to one schema, making it trivial to add/remove providers. - Graceful degradation: Building fallback chains so the app never shows an empty screen — if Indeed is blocked, it falls through to legal APIs. - Deployment tradeoffs: Understanding what each hosting platform can/can't do (Plesk = PHP only, Render = needs card, GitHub Pages = static only, Cloudflare Tunnel = exposes local apps for free). - DB-first design pattern: Pre-populate from slow operations, serve fast from cache/DB. - Test-driven scraping: Using saved HTML fixtures so parser tests run without network.
- Add email alerts for new matching jobs (cron + SMTP) - Build a job recommendation engine using saved favorites + semantic similarity - Add interview prep questions generated from job descriptions - Mobile-responsive PWA with push notifications - Permanent domain via Cloudflare Named Tunnel (instead of trycloudflare random URLs) - Indeed scraping via residential proxy rotation for higher reliability - Multi-country Indeed support (UK, DE, AU) - Salary prediction model based on job title + location
Great job tackling a complex scraping problem and building a full‑stack feature set with authentication, analytics, and CI/CD. To strengthen the project, provide clearer evidence that the FastAPI backend runs in the live URL (GitHub Pages can only host static content) and align the reported time spent with the work done. Adding more detailed setup docs and discussing the ethical considerations of bypassing Cloudflare would also improve the submission.