Got facebook to work test tomorrow with new accounts please

2026-06-22 22:00:56 +02:00
parent 6c88dcca7b
commit b81f391df5
806 changed files with 421635 additions and 138 deletions
@@ -1,60 +1,71 @@
-# Facebook Scraper - Configuration Needed
+# Scrapers - Configuration & Status

-## Proxy Configuration
+## Facebook Scraper

 **File:** `rust-ai/src/main.rs`  
-**Lines:** 25-27
-
-The scraper needs real proxy URLs. The dummy placeholder `"http://0.0.0.0:0"` will fail to connect (expected — no crash, just error logs):
+**Lines:** ~70-85

+Target URL (line 72):
 ```rust
-const PROXIES: &[&str] = &[
-    "http://0.0.0.0:0",  // <- Replace with real proxy(es)
-];
+let url = "https://www.facebook.com/search/top/?q=need%20website%20create";
 ```

-Replace with your actual proxies, e.g.:
-```rust
-const PROXIES: &[&str] = &[
-    "http://user:pass@192.168.1.1:8080",
-    "http://user:pass@192.168.1.2:8080",
-];
+**Status:** Uses direct connection (no proxy) — your home IP. Facebook blocks datacenter IPs. May work from a residential connection.
+
+**Test with curl:**
+```
+curl.exe -s "https://www.facebook.com/search/top/?q=need%20website%20create" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" --max-time 10 2>$null
 ```

-## User Agents (Optional)
+**Expected log output when blocked:**
+```
+ERROR crm_ai: Facebook scraper error: error sending request for url (https://www.facebook.com/search/top/?q=need%20website%20create)
+```
+
+## Reddit Scraper

 **File:** `rust-ai/src/main.rs`  
-**Lines:** 32-36
+**Lines:** ~90-120

-Add more realistic user agents here if needed.
+Uses `old.reddit.com` (older design that allows scraping without captchas).

-## Scraper Target URL
+Search queries (currently 6, lines 93-100):
+- `r/southafrica` — "need website", "web developer"
+- Global — '"need a website"', "website quote"
+- `r/forhire` — "website"
+- `r/smallbusiness` — "website"
+
+**Status:** Working. Reddit results appear as `INFO LEAD:` entries in the server log.
+
+**Test with curl:**
+```
+curl.exe -s "https://old.reddit.com/r/southafrica/search?q=need+website&sort=new&restrict_sr=on&t=week" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
+```
+
+**Expected log output when working:**
+```
+INFO crm_ai: LEAD: [Post Title] -> https://old.reddit.com/r/.../...
+```
+
+## Background Loop

 **File:** `rust-ai/src/main.rs`  
-**Line:** 68
+**Lines:** ~370-383

-Currently fetches `https://www.facebook.com/search/top/?q=need%20website%20create`. Change if needed.
+Both scrapers run together every 60-180 seconds on a `spawn_blocking` thread.

-## Background Thread
+## What You Need to Do

-**File:** `rust-ai/src/main.rs`  
-**Lines:** 355-371
+### Facebook
+- Try running from your home PC instead — your residential IP may not be blocked
+- If still blocked, you need residential proxies (BrightData, IPRoyal, Oxylabs)
+- Configure them in the `PROXIES` array at line 25

-Runs in a `tokio::task::spawn_blocking` thread (changed from `tokio::spawn` because the scraper uses `reqwest::blocking::Client` + `thread::sleep`).
+### Reddit  
+- Works out of the box via `old.reddit.com`
+- If it stops working, Reddit IP-blocked you — use proxies or switch to Playwright

-## Expected Behavior Until Configured
+### To add more search queries
+Edit the `searches` array in `run_reddit_scraper()` (line 93).

-Until real proxies are set, the Rust server will log this every 1-5 seconds (not a crash):
-```
-ERROR crm_ai: Facebook scraper error: error sending request for url (https://www.facebook.com/messages)
-```
-Once you add working proxies, these errors stop and actual scraping begins.
-
-## How it works
-
- Runs every 1-5 seconds on a background blocking thread (line 355-371)
- Rotates through proxies and user agents for each request
- Parses conversation HTML via `scraper` crate
- Designed to detect job posts like "need a website" and notify sales reps
- Error handling added to prevent server crash (uses `match` instead of `.unwrap()` at line 69)