Ace
d35c806d5b
- Added comprehensive comments and JSDoc-style documentation to NotificationProvider, ThemeProvider, UserProvider, and WebsiteThemeProvider for better clarity and maintainability. - Improved type definitions in index.ts for better code understanding and usage. - Introduced Docker support with Dockerfiles for various services including AI server, signaling server, and browser-use service. - Created a docker-compose.yml file to orchestrate multiple services including PostgreSQL, AI, scraper, and frontend. - Added a startup guide (startup.txt) for setting up the CRM environment on Ubuntu with Docker. - Included a .dockerignore file to exclude unnecessary files from Docker builds.
445 lines
19 KiB
Plaintext
445 lines
19 KiB
Plaintext
╔══════════════════════════════════════════════════════════════════════╗
|
|
║ CRM ENVR — Full Startup Guide (Ubuntu 24.04) ║
|
|
║ Multi-service Docker Deployment ║
|
|
╚══════════════════════════════════════════════════════════════════════╝
|
|
|
|
Architecture (7 services):
|
|
db postgres:16-alpine 5432 PostgreSQL database
|
|
ollama ollama/ollama 11434 Local AI model server
|
|
migrate (ai-server image) — Runs DB migrations once
|
|
ai ai-server/Dockerfile 3001 AI chat + setup API
|
|
scraper browser-use/Dockerfile 3008 Facebook lead scraper
|
|
next root Dockerfile 3006 Next.js frontend UI
|
|
signaling Dockerfile.signaling 3007 WebRTC signaling for chats
|
|
|
|
Required: 8GB RAM, 4 CPU cores, 20GB+ free disk
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
1. INSTALL DOCKER & DEPENDENCIES
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
# Update system
|
|
sudo apt update && sudo apt upgrade -y
|
|
|
|
# Install Docker (official method)
|
|
curl -fsSL https://get.docker.com -o get-docker.sh
|
|
sudo sh get-docker.sh
|
|
sudo usermod -aG docker $USER
|
|
newgrp docker
|
|
|
|
# Verify
|
|
docker --version
|
|
docker compose version
|
|
|
|
# Install git (if cloning from repo)
|
|
sudo apt install -y git
|
|
|
|
# Install tools (optional but helpful)
|
|
sudo apt install -y htop net-tools ufw
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
2. GET THE PROJECT FILES
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
Option A: Copy from local machine to VPS (if project is on your PC)
|
|
─────────────────────────────────────────────────────────────────────
|
|
# On your Windows PC (PowerShell):
|
|
# Install rsync via WSL or use scp
|
|
scp -r C:\CoastIT Projects\CRM_ENVR\CRM_ENVR user@YOUR_VPS_IP:~/crm-envr
|
|
|
|
# On your VPS:
|
|
cd ~/crm-envr
|
|
|
|
Option B: Clone from git repo (if hosted)
|
|
─────────────────────────────────────────────────────────────────────
|
|
git clone <YOUR_REPO_URL> ~/crm-envr
|
|
cd ~/crm-envr
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
3. CONFIGURE ENVIRONMENT
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
# Create env file from template
|
|
cp .env.docker .env
|
|
nano .env
|
|
|
|
# ── Required: Change these ──
|
|
JWT_SECRET=<GENERATE_A_RANDOM_STRING_HERE>
|
|
# Generate one: openssl rand -hex 32
|
|
|
|
# ── Set your VPS IP for external access ──
|
|
NEXT_PUBLIC_SCRAPER_URL=http://YOUR_VPS_IP:3008
|
|
# This lets friends' browsers reach the scraper API
|
|
# If testing locally: http://localhost:3008
|
|
|
|
# ── Optional overrides ──
|
|
DB_PASSWORD=crm # Change for production
|
|
AI_MODEL=dolphin-llama3:8b # Ollama model for AI chat
|
|
CLASSIFY_MODEL=dolphin-llama3:8b # Ollama model for lead classification
|
|
SELECTED_BROWSER=firefox # Browser for scraping (firefox/chrome/edge/opera)
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
4. BUILD & START ALL SERVICES
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
# First build: downloads base images, installs deps, builds frontend
|
|
# This takes 10-30 minutes the first time
|
|
docker compose build
|
|
|
|
# Start everything in the background
|
|
docker compose up -d
|
|
|
|
# Watch startup logs
|
|
docker compose logs -f
|
|
|
|
# Check all services are running
|
|
docker compose ps
|
|
|
|
# Expected output (all should show "Up" or "Completed" for migrate):
|
|
# Name Status
|
|
# crm-envr-db-1 Up (healthy)
|
|
# crm-envr-ollama-1 Up (healthy)
|
|
# crm-envr-migrate-1 Exited (0) ← This is normal, ran once
|
|
# crm-envr-ai-1 Up
|
|
# crm-envr-scraper-1 Up
|
|
# crm-envr-next-1 Up
|
|
# crm-envr-signaling-1 Up
|
|
|
|
# If something fails, check its logs:
|
|
docker compose logs <service-name>
|
|
# Example: docker compose logs scraper
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
5. POST-SETUP — AI MODEL
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
# Pull the AI model into the Ollama container (one time, 4-5GB download)
|
|
docker compose exec ollama ollama pull dolphin-llama3:8b
|
|
|
|
# Verify model is loaded
|
|
docker compose exec ollama ollama list
|
|
|
|
# Expected output:
|
|
# NAME ID SIZE
|
|
# dolphin-llama3:8b <hash> 4.5 GB
|
|
|
|
# Test the AI server
|
|
curl http://localhost:3001/health
|
|
# Expected: {"status":"ok"}
|
|
|
|
# Test the setup status
|
|
curl http://localhost:3001/setup/status | python3 -m json.tool
|
|
# Expected: ollama_running: true, model_available: true
|
|
|
|
# If model_available is false, wait for the pull to finish and retry
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
6. POST-SETUP — FACEBOOK SCRAPING
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
The scraper needs a Facebook-logged-in browser profile to work.
|
|
On a VPS (no GUI browser), there are TWO approaches:
|
|
|
|
─── Option A: Copy your local Firefox profile to the VPS (recommended) ──
|
|
|
|
Step 1: On your Windows PC, locate your Firefox profile:
|
|
C:\Users\USER-PC\AppData\Roaming\Mozilla\Firefox\Profiles\*.default-release
|
|
|
|
Step 2: Zip it:
|
|
Compress-Archive -Path "C:\Users\USER-PC\AppData\Roaming\Mozilla\Firefox\Profiles\*.default-release\*" -DestinationPath firefox-profile.zip
|
|
|
|
Step 3: Copy to VPS:
|
|
scp firefox-profile.zip user@YOUR_VPS_IP:~/firefox-profile.zip
|
|
|
|
Step 4: On VPS, extract to a persistent location:
|
|
mkdir -p ~/browser-profiles/firefox
|
|
unzip ~/firefox-profile.zip -d ~/browser-profiles/firefox
|
|
|
|
Step 5: Stop services, add profile mount, restart:
|
|
docker compose down
|
|
# Edit docker-compose.yml: add volume to the scraper service
|
|
# volumes:
|
|
# - ~/browser-profiles/firefox:/root/.mozilla/firefox/profile:ro
|
|
nano docker-compose.yml
|
|
|
|
docker compose up -d
|
|
|
|
Step 6: Set the profile path in .env:
|
|
FX_PROFILE=/root/.mozilla/firefox/profile
|
|
# Then restart: docker compose restart scraper
|
|
|
|
─── Option B: Use Agent fallback (no profile needed, AI-driven) ──
|
|
|
|
The scraper automatically falls back to browser-use Agent
|
|
(AI-powered navigation) when no logged-in profile is found.
|
|
This is slower but works without a real browser profile.
|
|
|
|
No setup needed — it just works.
|
|
|
|
─── Option C: Chrome/Edge on VPS (if you have a desktop environment) ──
|
|
|
|
Install a browser and log into Facebook once:
|
|
sudo apt install -y firefox
|
|
# Then manually log into Facebook and keep the profile
|
|
|
|
─── Verify scraper is working ──
|
|
|
|
# Test the scraper health endpoint
|
|
curl http://localhost:3008/health
|
|
|
|
# Check profile detection
|
|
curl http://localhost:3008/setup/profile | python3 -m json.tool
|
|
# Shows detected browser profiles
|
|
|
|
# Do a test scrape (note: this takes 2-5 minutes)
|
|
curl -X POST "http://localhost:3008/scrape/facebook?force=true&query=I%20need%20a%20website" | python3 -m json.tool
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
7. DATABASE SETUP (automatic)
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
No manual DB setup needed — the migrate service runs automatically:
|
|
• Creates the crm database (via POSTGRES_DB env var)
|
|
• Runs all SQL migrations from database/migrations/
|
|
• Creates the set_session_user_context() function
|
|
• Creates all tables (users, leads, conversations, etc.)
|
|
|
|
If you need to reset the database:
|
|
docker compose down -v # WARNING: DELETES ALL DATA
|
|
docker compose up -d # Fresh start
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
8. FIREWALL & SECURITY
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
# Allow SSH
|
|
sudo ufw allow 22/tcp
|
|
|
|
# Allow frontend (share with friends)
|
|
sudo ufw allow 3006/tcp
|
|
|
|
# Allow scraper API (needed by frontend JS)
|
|
sudo ufw allow 3008/tcp
|
|
|
|
# Optional: allow AI server setup page
|
|
sudo ufw allow 3001/tcp
|
|
|
|
# Optional: allow signaling server
|
|
sudo ufw allow 3007/tcp
|
|
|
|
# Enable firewall
|
|
sudo ufw enable
|
|
sudo ufw status
|
|
|
|
# For production: use a reverse proxy (Caddy/Nginx) on port 80/443
|
|
# instead of exposing raw ports. Let me know if you need this.
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
9. ACCESSING THE APP
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
# From your VPS itself:
|
|
http://localhost:3006
|
|
|
|
# From other machines / friends:
|
|
http://YOUR_VPS_IP:3006
|
|
|
|
# The first time you visit, the splash page will show you all services.
|
|
# If everything is green, click "Continue to App" to set up your
|
|
# admin account and log in.
|
|
|
|
# Splash page (setup wizard):
|
|
http://YOUR_VPS_IP:3001/splash
|
|
|
|
# Viewing leads (after scraping):
|
|
Use the AI Assistant panel in the app to search for leads.
|
|
Select a job category (Website Creation or Tutoring) and click search.
|
|
Results appear within 2-5 minutes.
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
10. COMMON COMMANDS
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
# Stop all services
|
|
docker compose down
|
|
|
|
# Stop and delete all data (volumes)
|
|
docker compose down -v
|
|
|
|
# Restart a specific service
|
|
docker compose restart <service>
|
|
|
|
# View logs (follow)
|
|
docker compose logs -f <service>
|
|
|
|
# Pull the latest code and rebuild
|
|
git pull
|
|
docker compose build --no-cache <service>
|
|
docker compose up -d
|
|
|
|
# Rebuild everything from scratch
|
|
docker compose down
|
|
docker compose build --no-cache
|
|
docker compose up -d
|
|
|
|
# Check disk usage
|
|
docker system df
|
|
|
|
# Clean up unused Docker data
|
|
docker system prune -a
|
|
|
|
# Enter a running container
|
|
docker compose exec <service> /bin/bash
|
|
|
|
# For the scraper container (uses sh, not bash):
|
|
docker compose exec scraper /bin/sh
|
|
|
|
# View live resource usage
|
|
docker stats
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
11. TROUBLESHOOTING
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
─── "port is already allocated" ──
|
|
Something else is using port 3006/3008/3001.
|
|
Check: sudo lsof -i :3006
|
|
Kill: sudo kill <PID>
|
|
|
|
─── "No function matches the given name" ──
|
|
Migrations haven't run. Check the migrate container:
|
|
docker compose logs migrate
|
|
If it failed, run manually:
|
|
docker compose run --rm migrate
|
|
|
|
─── "Model not found" or "model_available: false" ──
|
|
The Ollama model hasn't been pulled yet:
|
|
docker compose exec ollama ollama pull dolphin-llama3:8b
|
|
Wait for it to finish, then restart the AI server:
|
|
docker compose restart ai
|
|
|
|
─── "Scraper not reachable" ──
|
|
Scraper container might be restarting. Check logs:
|
|
docker compose logs scraper
|
|
Common cause: not enough memory for Playwright browsers
|
|
|
|
─── "Facebook login page detected" ──
|
|
No logged-in browser profile available. The scraper will
|
|
automatically fall back to the Agent (AI-powered) path.
|
|
Or set up a profile using Option A in Section 6.
|
|
|
|
─── "Next.js build failed" ──
|
|
The @next/swc-win32-x64-msvc package is Windows-only and will
|
|
be skipped on Linux. If the build fails, check:
|
|
docker compose logs next-build
|
|
Common fix: add Linux SWC package
|
|
docker compose run --rm next npm install @next/swc-linux-x64-gnu --save-dev
|
|
|
|
─── Container keeps restarting ──
|
|
Check why: docker compose logs <service>
|
|
If it's the scraper, it might be out of memory:
|
|
docker compose logs scraper | grep -i "memory\|killed\|oom"
|
|
|
|
─── Slow scraping / Agent timeout ──
|
|
The browser-use Agent fallback is slow on 8GB RAM.
|
|
Close other services or upgrade to 16GB for better performance.
|
|
|
|
─── Scraper always returns empty leads ──
|
|
The date filter is 2 days max. If no recent posts match,
|
|
you'll get empty results. This is by design (precision over quantity).
|
|
Try again later when fresh posts appear, or force a wider search:
|
|
curl -X POST "http://localhost:3008/scrape/facebook?force=true&query=I%20need%20a%20website"
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
12. FILE REFERENCE
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
docker-compose.yml Orchestrates all 7 services
|
|
Dockerfile Next.js frontend (multi-stage build)
|
|
ai-server/Dockerfile AI server + PostgreSQL client
|
|
browser-use-service/Dockerfile Python scraper with Playwright
|
|
Dockerfile.signaling WebRTC signaling server
|
|
.dockerignore Files excluded from Docker builds
|
|
.env.docker Template env vars for Docker deployment
|
|
.env Your actual env vars (create from .env.docker)
|
|
|
|
ai-server/index.mjs AI chat + setup wizard API
|
|
browser-use-service/main.py Facebook scraper (FastAPI + Playwright)
|
|
src/ Next.js frontend source
|
|
database/migrations/ SQL migration files (run automatically)
|
|
data/ai/ AI instructions and job definitions
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
13. UPDATING THE APP
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
# If using git:
|
|
git pull
|
|
docker compose build --no-cache next ai scraper
|
|
docker compose up -d
|
|
|
|
# If copying files manually:
|
|
# Copy updated project files to VPS
|
|
docker compose build --no-cache next ai scraper
|
|
docker compose up -d
|
|
|
|
# To update the AI model to a different one:
|
|
docker compose exec ollama ollama pull <new-model-name>
|
|
# Then update .env: AI_MODEL=<new-model-name>
|
|
docker compose restart ai scraper
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
14. PRODUCTION REVERSE PROXY (for port 80/443)
|
|
═══════════════════════════════════════════════════════════════════════
|
|
|
|
If you want to serve everything on standard ports (80/443 with HTTPS):
|
|
|
|
# Install Caddy (simplest, auto HTTPS)
|
|
sudo apt install -y caddy
|
|
|
|
# Create config: /etc/caddy/Caddyfile
|
|
# ──────────────────────────────
|
|
# yourdomain.com {
|
|
# # Next.js frontend
|
|
# handle_path / {
|
|
# reverse_proxy next:3006
|
|
# }
|
|
# # API routes used by frontend JS
|
|
# handle_path /api/* {
|
|
# reverse_proxy next:3006
|
|
# }
|
|
# # Scraper API (needed by browser JS)
|
|
# handle /scrape/* {
|
|
# reverse_proxy scraper:3008
|
|
# }
|
|
# handle /health {
|
|
# reverse_proxy scraper:3008
|
|
# }
|
|
# }
|
|
# ──────────────────────────────
|
|
|
|
# Reload Caddy
|
|
sudo caddy reload
|
|
|
|
For this setup, NEXT_PUBLIC_SCRAPER_URL would need to be a relative path
|
|
(e.g., just "" or "/") which requires a small code change. Ask me if needed.
|
|
|
|
|
|
═══════════════════════════════════════════════════════════════════════
|
|
END — You're all set. The app should be running at http://YOUR_VPS_IP:3006
|
|
═══════════════════════════════════════════════════════════════════════
|