feat: Enhance provider components with detailed documentation and comments
- Added comprehensive comments and JSDoc-style documentation to NotificationProvider, ThemeProvider, UserProvider, and WebsiteThemeProvider for better clarity and maintainability. - Improved type definitions in index.ts for better code understanding and usage. - Introduced Docker support with Dockerfiles for various services including AI server, signaling server, and browser-use service. - Created a docker-compose.yml file to orchestrate multiple services including PostgreSQL, AI, scraper, and frontend. - Added a startup guide (startup.txt) for setting up the CRM environment on Ubuntu with Docker. - Included a .dockerignore file to exclude unnecessary files from Docker builds.
This commit is contained in:
+444
@@ -0,0 +1,444 @@
|
||||
╔══════════════════════════════════════════════════════════════════════╗
|
||||
║ CRM ENVR — Full Startup Guide (Ubuntu 24.04) ║
|
||||
║ Multi-service Docker Deployment ║
|
||||
╚══════════════════════════════════════════════════════════════════════╝
|
||||
|
||||
Architecture (7 services):
|
||||
db postgres:16-alpine 5432 PostgreSQL database
|
||||
ollama ollama/ollama 11434 Local AI model server
|
||||
migrate (ai-server image) — Runs DB migrations once
|
||||
ai ai-server/Dockerfile 3001 AI chat + setup API
|
||||
scraper browser-use/Dockerfile 3008 Facebook lead scraper
|
||||
next root Dockerfile 3006 Next.js frontend UI
|
||||
signaling Dockerfile.signaling 3007 WebRTC signaling for chats
|
||||
|
||||
Required: 8GB RAM, 4 CPU cores, 20GB+ free disk
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
1. INSTALL DOCKER & DEPENDENCIES
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
# Update system
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
|
||||
# Install Docker (official method)
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sudo sh get-docker.sh
|
||||
sudo usermod -aG docker $USER
|
||||
newgrp docker
|
||||
|
||||
# Verify
|
||||
docker --version
|
||||
docker compose version
|
||||
|
||||
# Install git (if cloning from repo)
|
||||
sudo apt install -y git
|
||||
|
||||
# Install tools (optional but helpful)
|
||||
sudo apt install -y htop net-tools ufw
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
2. GET THE PROJECT FILES
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
Option A: Copy from local machine to VPS (if project is on your PC)
|
||||
─────────────────────────────────────────────────────────────────────
|
||||
# On your Windows PC (PowerShell):
|
||||
# Install rsync via WSL or use scp
|
||||
scp -r C:\CoastIT Projects\CRM_ENVR\CRM_ENVR user@YOUR_VPS_IP:~/crm-envr
|
||||
|
||||
# On your VPS:
|
||||
cd ~/crm-envr
|
||||
|
||||
Option B: Clone from git repo (if hosted)
|
||||
─────────────────────────────────────────────────────────────────────
|
||||
git clone <YOUR_REPO_URL> ~/crm-envr
|
||||
cd ~/crm-envr
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
3. CONFIGURE ENVIRONMENT
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
# Create env file from template
|
||||
cp .env.docker .env
|
||||
nano .env
|
||||
|
||||
# ── Required: Change these ──
|
||||
JWT_SECRET=<GENERATE_A_RANDOM_STRING_HERE>
|
||||
# Generate one: openssl rand -hex 32
|
||||
|
||||
# ── Set your VPS IP for external access ──
|
||||
NEXT_PUBLIC_SCRAPER_URL=http://YOUR_VPS_IP:3008
|
||||
# This lets friends' browsers reach the scraper API
|
||||
# If testing locally: http://localhost:3008
|
||||
|
||||
# ── Optional overrides ──
|
||||
DB_PASSWORD=crm # Change for production
|
||||
AI_MODEL=dolphin-llama3:8b # Ollama model for AI chat
|
||||
CLASSIFY_MODEL=dolphin-llama3:8b # Ollama model for lead classification
|
||||
SELECTED_BROWSER=firefox # Browser for scraping (firefox/chrome/edge/opera)
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
4. BUILD & START ALL SERVICES
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
# First build: downloads base images, installs deps, builds frontend
|
||||
# This takes 10-30 minutes the first time
|
||||
docker compose build
|
||||
|
||||
# Start everything in the background
|
||||
docker compose up -d
|
||||
|
||||
# Watch startup logs
|
||||
docker compose logs -f
|
||||
|
||||
# Check all services are running
|
||||
docker compose ps
|
||||
|
||||
# Expected output (all should show "Up" or "Completed" for migrate):
|
||||
# Name Status
|
||||
# crm-envr-db-1 Up (healthy)
|
||||
# crm-envr-ollama-1 Up (healthy)
|
||||
# crm-envr-migrate-1 Exited (0) ← This is normal, ran once
|
||||
# crm-envr-ai-1 Up
|
||||
# crm-envr-scraper-1 Up
|
||||
# crm-envr-next-1 Up
|
||||
# crm-envr-signaling-1 Up
|
||||
|
||||
# If something fails, check its logs:
|
||||
docker compose logs <service-name>
|
||||
# Example: docker compose logs scraper
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
5. POST-SETUP — AI MODEL
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
# Pull the AI model into the Ollama container (one time, 4-5GB download)
|
||||
docker compose exec ollama ollama pull dolphin-llama3:8b
|
||||
|
||||
# Verify model is loaded
|
||||
docker compose exec ollama ollama list
|
||||
|
||||
# Expected output:
|
||||
# NAME ID SIZE
|
||||
# dolphin-llama3:8b <hash> 4.5 GB
|
||||
|
||||
# Test the AI server
|
||||
curl http://localhost:3001/health
|
||||
# Expected: {"status":"ok"}
|
||||
|
||||
# Test the setup status
|
||||
curl http://localhost:3001/setup/status | python3 -m json.tool
|
||||
# Expected: ollama_running: true, model_available: true
|
||||
|
||||
# If model_available is false, wait for the pull to finish and retry
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
6. POST-SETUP — FACEBOOK SCRAPING
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
The scraper needs a Facebook-logged-in browser profile to work.
|
||||
On a VPS (no GUI browser), there are TWO approaches:
|
||||
|
||||
─── Option A: Copy your local Firefox profile to the VPS (recommended) ──
|
||||
|
||||
Step 1: On your Windows PC, locate your Firefox profile:
|
||||
C:\Users\USER-PC\AppData\Roaming\Mozilla\Firefox\Profiles\*.default-release
|
||||
|
||||
Step 2: Zip it:
|
||||
Compress-Archive -Path "C:\Users\USER-PC\AppData\Roaming\Mozilla\Firefox\Profiles\*.default-release\*" -DestinationPath firefox-profile.zip
|
||||
|
||||
Step 3: Copy to VPS:
|
||||
scp firefox-profile.zip user@YOUR_VPS_IP:~/firefox-profile.zip
|
||||
|
||||
Step 4: On VPS, extract to a persistent location:
|
||||
mkdir -p ~/browser-profiles/firefox
|
||||
unzip ~/firefox-profile.zip -d ~/browser-profiles/firefox
|
||||
|
||||
Step 5: Stop services, add profile mount, restart:
|
||||
docker compose down
|
||||
# Edit docker-compose.yml: add volume to the scraper service
|
||||
# volumes:
|
||||
# - ~/browser-profiles/firefox:/root/.mozilla/firefox/profile:ro
|
||||
nano docker-compose.yml
|
||||
|
||||
docker compose up -d
|
||||
|
||||
Step 6: Set the profile path in .env:
|
||||
FX_PROFILE=/root/.mozilla/firefox/profile
|
||||
# Then restart: docker compose restart scraper
|
||||
|
||||
─── Option B: Use Agent fallback (no profile needed, AI-driven) ──
|
||||
|
||||
The scraper automatically falls back to browser-use Agent
|
||||
(AI-powered navigation) when no logged-in profile is found.
|
||||
This is slower but works without a real browser profile.
|
||||
|
||||
No setup needed — it just works.
|
||||
|
||||
─── Option C: Chrome/Edge on VPS (if you have a desktop environment) ──
|
||||
|
||||
Install a browser and log into Facebook once:
|
||||
sudo apt install -y firefox
|
||||
# Then manually log into Facebook and keep the profile
|
||||
|
||||
─── Verify scraper is working ──
|
||||
|
||||
# Test the scraper health endpoint
|
||||
curl http://localhost:3008/health
|
||||
|
||||
# Check profile detection
|
||||
curl http://localhost:3008/setup/profile | python3 -m json.tool
|
||||
# Shows detected browser profiles
|
||||
|
||||
# Do a test scrape (note: this takes 2-5 minutes)
|
||||
curl -X POST "http://localhost:3008/scrape/facebook?force=true&query=I%20need%20a%20website" | python3 -m json.tool
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
7. DATABASE SETUP (automatic)
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
No manual DB setup needed — the migrate service runs automatically:
|
||||
• Creates the crm database (via POSTGRES_DB env var)
|
||||
• Runs all SQL migrations from database/migrations/
|
||||
• Creates the set_session_user_context() function
|
||||
• Creates all tables (users, leads, conversations, etc.)
|
||||
|
||||
If you need to reset the database:
|
||||
docker compose down -v # WARNING: DELETES ALL DATA
|
||||
docker compose up -d # Fresh start
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
8. FIREWALL & SECURITY
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
# Allow SSH
|
||||
sudo ufw allow 22/tcp
|
||||
|
||||
# Allow frontend (share with friends)
|
||||
sudo ufw allow 3006/tcp
|
||||
|
||||
# Allow scraper API (needed by frontend JS)
|
||||
sudo ufw allow 3008/tcp
|
||||
|
||||
# Optional: allow AI server setup page
|
||||
sudo ufw allow 3001/tcp
|
||||
|
||||
# Optional: allow signaling server
|
||||
sudo ufw allow 3007/tcp
|
||||
|
||||
# Enable firewall
|
||||
sudo ufw enable
|
||||
sudo ufw status
|
||||
|
||||
# For production: use a reverse proxy (Caddy/Nginx) on port 80/443
|
||||
# instead of exposing raw ports. Let me know if you need this.
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
9. ACCESSING THE APP
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
# From your VPS itself:
|
||||
http://localhost:3006
|
||||
|
||||
# From other machines / friends:
|
||||
http://YOUR_VPS_IP:3006
|
||||
|
||||
# The first time you visit, the splash page will show you all services.
|
||||
# If everything is green, click "Continue to App" to set up your
|
||||
# admin account and log in.
|
||||
|
||||
# Splash page (setup wizard):
|
||||
http://YOUR_VPS_IP:3001/splash
|
||||
|
||||
# Viewing leads (after scraping):
|
||||
Use the AI Assistant panel in the app to search for leads.
|
||||
Select a job category (Website Creation or Tutoring) and click search.
|
||||
Results appear within 2-5 minutes.
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
10. COMMON COMMANDS
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
# Stop all services
|
||||
docker compose down
|
||||
|
||||
# Stop and delete all data (volumes)
|
||||
docker compose down -v
|
||||
|
||||
# Restart a specific service
|
||||
docker compose restart <service>
|
||||
|
||||
# View logs (follow)
|
||||
docker compose logs -f <service>
|
||||
|
||||
# Pull the latest code and rebuild
|
||||
git pull
|
||||
docker compose build --no-cache <service>
|
||||
docker compose up -d
|
||||
|
||||
# Rebuild everything from scratch
|
||||
docker compose down
|
||||
docker compose build --no-cache
|
||||
docker compose up -d
|
||||
|
||||
# Check disk usage
|
||||
docker system df
|
||||
|
||||
# Clean up unused Docker data
|
||||
docker system prune -a
|
||||
|
||||
# Enter a running container
|
||||
docker compose exec <service> /bin/bash
|
||||
|
||||
# For the scraper container (uses sh, not bash):
|
||||
docker compose exec scraper /bin/sh
|
||||
|
||||
# View live resource usage
|
||||
docker stats
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
11. TROUBLESHOOTING
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
─── "port is already allocated" ──
|
||||
Something else is using port 3006/3008/3001.
|
||||
Check: sudo lsof -i :3006
|
||||
Kill: sudo kill <PID>
|
||||
|
||||
─── "No function matches the given name" ──
|
||||
Migrations haven't run. Check the migrate container:
|
||||
docker compose logs migrate
|
||||
If it failed, run manually:
|
||||
docker compose run --rm migrate
|
||||
|
||||
─── "Model not found" or "model_available: false" ──
|
||||
The Ollama model hasn't been pulled yet:
|
||||
docker compose exec ollama ollama pull dolphin-llama3:8b
|
||||
Wait for it to finish, then restart the AI server:
|
||||
docker compose restart ai
|
||||
|
||||
─── "Scraper not reachable" ──
|
||||
Scraper container might be restarting. Check logs:
|
||||
docker compose logs scraper
|
||||
Common cause: not enough memory for Playwright browsers
|
||||
|
||||
─── "Facebook login page detected" ──
|
||||
No logged-in browser profile available. The scraper will
|
||||
automatically fall back to the Agent (AI-powered) path.
|
||||
Or set up a profile using Option A in Section 6.
|
||||
|
||||
─── "Next.js build failed" ──
|
||||
The @next/swc-win32-x64-msvc package is Windows-only and will
|
||||
be skipped on Linux. If the build fails, check:
|
||||
docker compose logs next-build
|
||||
Common fix: add Linux SWC package
|
||||
docker compose run --rm next npm install @next/swc-linux-x64-gnu --save-dev
|
||||
|
||||
─── Container keeps restarting ──
|
||||
Check why: docker compose logs <service>
|
||||
If it's the scraper, it might be out of memory:
|
||||
docker compose logs scraper | grep -i "memory\|killed\|oom"
|
||||
|
||||
─── Slow scraping / Agent timeout ──
|
||||
The browser-use Agent fallback is slow on 8GB RAM.
|
||||
Close other services or upgrade to 16GB for better performance.
|
||||
|
||||
─── Scraper always returns empty leads ──
|
||||
The date filter is 2 days max. If no recent posts match,
|
||||
you'll get empty results. This is by design (precision over quantity).
|
||||
Try again later when fresh posts appear, or force a wider search:
|
||||
curl -X POST "http://localhost:3008/scrape/facebook?force=true&query=I%20need%20a%20website"
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
12. FILE REFERENCE
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
docker-compose.yml Orchestrates all 7 services
|
||||
Dockerfile Next.js frontend (multi-stage build)
|
||||
ai-server/Dockerfile AI server + PostgreSQL client
|
||||
browser-use-service/Dockerfile Python scraper with Playwright
|
||||
Dockerfile.signaling WebRTC signaling server
|
||||
.dockerignore Files excluded from Docker builds
|
||||
.env.docker Template env vars for Docker deployment
|
||||
.env Your actual env vars (create from .env.docker)
|
||||
|
||||
ai-server/index.mjs AI chat + setup wizard API
|
||||
browser-use-service/main.py Facebook scraper (FastAPI + Playwright)
|
||||
src/ Next.js frontend source
|
||||
database/migrations/ SQL migration files (run automatically)
|
||||
data/ai/ AI instructions and job definitions
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
13. UPDATING THE APP
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
# If using git:
|
||||
git pull
|
||||
docker compose build --no-cache next ai scraper
|
||||
docker compose up -d
|
||||
|
||||
# If copying files manually:
|
||||
# Copy updated project files to VPS
|
||||
docker compose build --no-cache next ai scraper
|
||||
docker compose up -d
|
||||
|
||||
# To update the AI model to a different one:
|
||||
docker compose exec ollama ollama pull <new-model-name>
|
||||
# Then update .env: AI_MODEL=<new-model-name>
|
||||
docker compose restart ai scraper
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
14. PRODUCTION REVERSE PROXY (for port 80/443)
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
If you want to serve everything on standard ports (80/443 with HTTPS):
|
||||
|
||||
# Install Caddy (simplest, auto HTTPS)
|
||||
sudo apt install -y caddy
|
||||
|
||||
# Create config: /etc/caddy/Caddyfile
|
||||
# ──────────────────────────────
|
||||
# yourdomain.com {
|
||||
# # Next.js frontend
|
||||
# handle_path / {
|
||||
# reverse_proxy next:3006
|
||||
# }
|
||||
# # API routes used by frontend JS
|
||||
# handle_path /api/* {
|
||||
# reverse_proxy next:3006
|
||||
# }
|
||||
# # Scraper API (needed by browser JS)
|
||||
# handle /scrape/* {
|
||||
# reverse_proxy scraper:3008
|
||||
# }
|
||||
# handle /health {
|
||||
# reverse_proxy scraper:3008
|
||||
# }
|
||||
# }
|
||||
# ──────────────────────────────
|
||||
|
||||
# Reload Caddy
|
||||
sudo caddy reload
|
||||
|
||||
For this setup, NEXT_PUBLIC_SCRAPER_URL would need to be a relative path
|
||||
(e.g., just "" or "/") which requires a small code change. Ask me if needed.
|
||||
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
END — You're all set. The app should be running at http://YOUR_VPS_IP:3006
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
Reference in New Issue
Block a user