Added logic to the 4 languages so it can search each language at a time
Build & Auto-Repair / build (push) Has been cancelled
Build & Auto-Repair / build (push) Has been cancelled
This commit is contained in:
+10
-3
@@ -40,12 +40,19 @@ The scraper lives at `browser-use-service/main.py` port 3008.
|
||||
- 6-7 non-English quick searches (Afrikaans, isiXhosa, isiZulu — 2 queries each per category)
|
||||
- Total: ~14 searches per scrape, completed in 2-4 minutes
|
||||
4. **Quick searches** — load page, double-scroll, extract visible posts (~12-18s each)
|
||||
5. **Date filter** — only posts within 7 days are considered
|
||||
6. **2-pass classification (dead-accurate)**:
|
||||
5. **Date filter** — only posts within **2 days** are considered. Anything older is discarded. Fresh leads only.
|
||||
6. **Stealth mechanics**:
|
||||
- Random viewport dimensions (1280×800 to 1920×1080) — never the same size twice
|
||||
- Variable delays between searches (5-12 seconds) with mouse idle actions mixed in
|
||||
- Human-like scroll patterns: scroll down, pause, sometimes scroll back up, sometimes return to top
|
||||
- Canvas/WebGL/audio fingerprint spoofing via injected init scripts
|
||||
- Random decoy page visits (e.g., Facebook Groups) between searches
|
||||
- Profile directory is temp-copied and cleaned up after each scrape
|
||||
- Detection signal monitoring (checkpoint, login pages, security challenges)
|
||||
7. **2-pass classification (dead-accurate)**:
|
||||
- **Pass 1 (AI)**: Ollama classifies each post as LEAD or NOT using a strict prompt per category. This is the primary filter and most accurate.
|
||||
- **Pass 2 (Keyword)**: Only posts matching BOTH a target term AND a request term are kept. Requires multi-word phrases — standalone words like "need", "want", "help" are NOT used as they cause false positives. Aggressive reject list catches service offers, self-promotions, portfolio posts, learning-requests, and existing-site issues.
|
||||
- **No loose fill**: Unlike the old approach, there is NO third pass that accepts posts matching EITHER term. Every returned lead has passed both AI and/or strict keyword validation. If fewer than 5 posts pass, that means only genuine leads are returned — no noise to pad the count.
|
||||
7. **Anti-detection** — random delays, human-like scrolling, user-agent rotation, proxy support
|
||||
8. **Scrape timing** — 3-6 minutes for a complete run. Returns 5-10 leads with high confidence.
|
||||
|
||||
### Lead Categories
|
||||
|
||||
Reference in New Issue
Block a user