CRM_ENVIROMENT/CRM_ENVR
7
0
Fork 0
mirror of https://git.coastit.co.za/caitlin/CRM_ENVR.git synced 2026-07-10 03:05:43 +02:00
Code Issues Packages Projects Releases Wiki Activity

Added finishing touch on other languages

Browse Source
This commit is contained in:
Ace
2026-07-08 14:24:03 +02:00
parent d77ff2b965
commit dba4c84cd5
6 changed files with 232 additions and 250 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/ai/ai.md
+14 -12
View File
@@ -34,12 +34,14 @@ The scraper lives at `browser-use-service/main.py` port 3008.
### How It Works
1. **Browser detection** — tries Firefox profile first, then Chromium-based (Chrome/Opera/Edge), falls back to browser-use Agent
2. **Profile paths** — configured via env vars (`FX_PROFILE`, `CHROME_PROFILE`, `OPERA_PROFILE`, `EDGE_PROFILE`) or auto-detected on first run
3. **Search dispatch** — per scrape run:
- 1 English primary search (full scroll with human-like delays)
- 2-3 English supplementary searches (quick searches)
- 6-7 non-English quick searches (Afrikaans, isiXhosa, isiZulu — 2 queries each per category)
- Total: ~14 searches per scrape, completed in 2-4 minutes
4. **Quick searches** — load page, double-scroll, extract visible posts (~12-18s each)
3. **4-phase language pipeline** (English → Afrikaans → Xhosa → Zulu):
- **Phase 1 (English)**: User's selected query + 2-3 supplementary English searches from the English search pool. First query gets full human-like scroll, rest use quick search. This phase does the heavy lifting.
- **Phase 2 (Afrikaans)**: 2 Afrikaans queries targeting Afrikaans-speaking communities.
- **Phase 3 (isiXhosa)**: 2 Xhosa queries targeting Xhosa-speaking communities.
- **Phase 4 (isiZulu)**: 2 Zulu queries targeting Zulu-speaking communities.
- After all phases: pipeline check (date filter 2 days → AI + keyword classification → sort by freshness). Newest leads ranked first.
- Each phase extracts posts, deduplicates against all prior phases, then passes through a stealth delay (5-12s + mouse idle) before the next phase.
4. **Quick searches** — load page, double-scroll, extract visible posts (~12-18s each). Scroll-back behavior (35% chance to scroll up) and random return-to-top (25% chance) for stealth.
5. **Date filter** — only posts within **2 days** are considered. Anything older is discarded. Fresh leads only.
6. **Stealth mechanics**:
- Random viewport dimensions (1280×800 to 1920×1080) — never the same size twice
@@ -70,12 +72,12 @@ Two categories, selectable when starting a scrape:
- Request terms: same as website category — must co-occur with a target keyword
- Strict reject: people offering tutoring, educational products, homeschool programs, free trials, general study tips
### Multi-Language Support
Searches in 4 South African languages:
- English — 1 primary + 2-3 supplementary queries
- Afrikaans — 2 queries (e.g., "ek benodig n webwerf", "ek soek n privaat onderwyser")
- isiXhosa — 2 queries (e.g., "ndidinga iwebhusayithi yeshishini", "ndifuna utitshala womntwana wam")
- isiZulu — 2 queries (e.g., "ngidinga iwebhusayithi yebhizinisi", "ngifuna umfundisi wengane")
### Multi-Language Pipeline (Phase Order)
4 South African languages in structured phases:
- **Phase 1 (English)**: primary query + supplementary English searches
- **Phase 2 (Afrikaans)**: 2 queries targeting Afrikaans speakers
- **Phase 3 (isiXhosa)**: 2 queries targeting Xhosa speakers
- **Phase 4 (isiZulu)**: 2 queries targeting Zulu speakers
### Output Format
Each lead returned includes: