# Scrapers - Configuration & Status

## Facebook Scraper

**File:** `rust-ai/src/main.rs`  
**Lines:** ~70-85

Target URL (line 72):
```rust
let url = "https://www.facebook.com/search/top/?q=need%20website%20create";
```

**Status:** Uses direct connection (no proxy) — your home IP. Facebook blocks datacenter IPs. May work from a residential connection.

**Test with curl:**
```
curl.exe -s "https://www.facebook.com/search/top/?q=need%20website%20create" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" --max-time 10 2>$null
```

**Expected log output when blocked:**
```
ERROR crm_ai: Facebook scraper error: error sending request for url (https://www.facebook.com/search/top/?q=need%20website%20create)
```

## Reddit Scraper

**File:** `rust-ai/src/main.rs`  
**Lines:** ~90-120

Uses `old.reddit.com` (older design that allows scraping without captchas).

Search queries (currently 6, lines 93-100):
- `r/southafrica` — "need website", "web developer"
- Global — '"need a website"', "website quote"
- `r/forhire` — "website"
- `r/smallbusiness` — "website"

**Status:** Working. Reddit results appear as `INFO LEAD:` entries in the server log.

**Test with curl:**
```
curl.exe -s "https://old.reddit.com/r/southafrica/search?q=need+website&sort=new&restrict_sr=on&t=week" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
```

**Expected log output when working:**
```
INFO crm_ai: LEAD: [Post Title] -> https://old.reddit.com/r/.../...
```

## Background Loop

**File:** `rust-ai/src/main.rs`  
**Lines:** ~370-383

Both scrapers run together every 60-180 seconds on a `spawn_blocking` thread.

## What You Need to Do

### Facebook
- Try running from your home PC instead — your residential IP may not be blocked
- If still blocked, you need residential proxies (BrightData, IPRoyal, Oxylabs)
- Configure them in the `PROXIES` array at line 25

### Reddit  
- Works out of the box via `old.reddit.com`
- If it stops working, Reddit IP-blocked you — use proxies or switch to Playwright

### To add more search queries
Edit the `searches` array in `run_reddit_scraper()` (line 93).