Fix JWT_SECRET env var, setup database, update spider web and DAILY BUGLE colors to #e62020

2026-06-24 15:08:48 +02:00
commit 1e332fb50d
942 changed files with 447438 additions and 0 deletions
@@ -0,0 +1,71 @@
+# Scrapers - Configuration & Status
+
+## Facebook Scraper
+
+**File:** `rust-ai/src/main.rs`  
+**Lines:** ~70-85
+
+Target URL (line 72):
+```rust
+let url = "https://www.facebook.com/search/top/?q=need%20website%20create";
+```
+
+**Status:** Uses direct connection (no proxy) — your home IP. Facebook blocks datacenter IPs. May work from a residential connection.
+
+**Test with curl:**
+```
+curl.exe -s "https://www.facebook.com/search/top/?q=need%20website%20create" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" --max-time 10 2>$null
+```
+
+**Expected log output when blocked:**
+```
+ERROR crm_ai: Facebook scraper error: error sending request for url (https://www.facebook.com/search/top/?q=need%20website%20create)
+```
+
+## Reddit Scraper
+
+**File:** `rust-ai/src/main.rs`  
+**Lines:** ~90-120
+
+Uses `old.reddit.com` (older design that allows scraping without captchas).
+
+Search queries (currently 6, lines 93-100):
+- `r/southafrica` — "need website", "web developer"
+- Global — '"need a website"', "website quote"
+- `r/forhire` — "website"
+- `r/smallbusiness` — "website"
+
+**Status:** Working. Reddit results appear as `INFO LEAD:` entries in the server log.
+
+**Test with curl:**
+```
+curl.exe -s "https://old.reddit.com/r/southafrica/search?q=need+website&sort=new&restrict_sr=on&t=week" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
+```
+
+**Expected log output when working:**
+```
+INFO crm_ai: LEAD: [Post Title] -> https://old.reddit.com/r/.../...
+```
+
+## Background Loop
+
+**File:** `rust-ai/src/main.rs`  
+**Lines:** ~370-383
+
+Both scrapers run together every 60-180 seconds on a `spawn_blocking` thread.
+
+## What You Need to Do
+
+### Facebook
+- Try running from your home PC instead — your residential IP may not be blocked
+- If still blocked, you need residential proxies (BrightData, IPRoyal, Oxylabs)
+- Configure them in the `PROXIES` array at line 25
+
+### Reddit  
+- Works out of the box via `old.reddit.com`
+- If it stops working, Reddit IP-blocked you — use proxies or switch to Playwright
+
+### To add more search queries
+Edit the `searches` array in `run_reddit_scraper()` (line 93).
+
+