Scrapers - Configuration & Status

Facebook Scraper

File: rust-ai/src/main.rs
Lines: ~70-85

Target URL (line 72):

let url = "https://www.facebook.com/search/top/?q=need%20website%20create";

Status: Uses direct connection (no proxy) — your home IP. Facebook blocks datacenter IPs. May work from a residential connection.

Test with curl:

curl.exe -s "https://www.facebook.com/search/top/?q=need%20website%20create" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" --max-time 10 2>$null

Expected log output when blocked:

ERROR crm_ai: Facebook scraper error: error sending request for url (https://www.facebook.com/search/top/?q=need%20website%20create)

Reddit Scraper

File: rust-ai/src/main.rs
Lines: ~90-120

Uses old.reddit.com (older design that allows scraping without captchas).

Search queries (currently 6, lines 93-100):

r/southafrica — "need website", "web developer"
Global — '"need a website"', "website quote"
r/forhire — "website"
r/smallbusiness — "website"

Status: Working. Reddit results appear as INFO LEAD: entries in the server log.

Test with curl:

curl.exe -s "https://old.reddit.com/r/southafrica/search?q=need+website&sort=new&restrict_sr=on&t=week" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

Expected log output when working:

INFO crm_ai: LEAD: [Post Title] -> https://old.reddit.com/r/.../...

Background Loop

File: rust-ai/src/main.rs
Lines: ~370-383

Both scrapers run together every 60-180 seconds on a spawn_blocking thread.

What You Need to Do

Facebook

Try running from your home PC instead — your residential IP may not be blocked
If still blocked, you need residential proxies (BrightData, IPRoyal, Oxylabs)
Configure them in the PROXIES array at line 25

Works out of the box via old.reddit.com
If it stops working, Reddit IP-blocked you — use proxies or switch to Playwright

To add more search queries

Edit the searches array in run_reddit_scraper() (line 93).

2.1 KiB Raw Permalink Blame History