╔══════════════════════════════════════════════════════════════════════╗
║              CRM ENVR — Full Startup Guide (Ubuntu 24.04)          ║
║                  Multi-service Docker Deployment                    ║
╚══════════════════════════════════════════════════════════════════════╝

Architecture (7 services):
  db        postgres:16-alpine     5432   PostgreSQL database
  ollama    ollama/ollama          11434  Local AI model server
  migrate   (ai-server image)      —      Runs DB migrations once
  ai        ai-server/Dockerfile   3001   AI chat + setup API
  scraper   browser-use/Dockerfile 3008   Facebook lead scraper
  next      root Dockerfile        3006   Next.js frontend UI
  signaling Dockerfile.signaling   3007   WebRTC signaling for chats

Required: 8GB RAM, 4 CPU cores, 20GB+ free disk


═══════════════════════════════════════════════════════════════════════
1. INSTALL DOCKER & DEPENDENCIES
═══════════════════════════════════════════════════════════════════════

# Update system
sudo apt update && sudo apt upgrade -y

# Install Docker (official method)
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER
newgrp docker

# Verify
docker --version
docker compose version

# Install git (if cloning from repo)
sudo apt install -y git

# Install tools (optional but helpful)
sudo apt install -y htop net-tools ufw


═══════════════════════════════════════════════════════════════════════
2. GET THE PROJECT FILES
═══════════════════════════════════════════════════════════════════════

Option A: Copy from local machine to VPS (if project is on your PC)
─────────────────────────────────────────────────────────────────────
  # On your Windows PC (PowerShell):
  # Install rsync via WSL or use scp
  scp -r C:\CoastIT Projects\CRM_ENVR\CRM_ENVR user@YOUR_VPS_IP:~/crm-envr

  # On your VPS:
  cd ~/crm-envr

Option B: Clone from git repo (if hosted)
─────────────────────────────────────────────────────────────────────
  git clone <YOUR_REPO_URL> ~/crm-envr
  cd ~/crm-envr


═══════════════════════════════════════════════════════════════════════
3. CONFIGURE ENVIRONMENT
═══════════════════════════════════════════════════════════════════════

# Create env file from template
cp .env.docker .env
nano .env

# ── Required: Change these ──
JWT_SECRET=<GENERATE_A_RANDOM_STRING_HERE>
  # Generate one: openssl rand -hex 32

# ── Set your VPS IP for external access ──
NEXT_PUBLIC_SCRAPER_URL=http://YOUR_VPS_IP:3008
  # This lets friends' browsers reach the scraper API
  # If testing locally: http://localhost:3008

# ── Optional overrides ──
DB_PASSWORD=crm                    # Change for production
AI_MODEL=dolphin-llama3:8b         # Ollama model for AI chat
CLASSIFY_MODEL=dolphin-llama3:8b   # Ollama model for lead classification
SELECTED_BROWSER=firefox           # Browser for scraping (firefox/chrome/edge/opera)


═══════════════════════════════════════════════════════════════════════
4. BUILD & START ALL SERVICES
═══════════════════════════════════════════════════════════════════════

# First build: downloads base images, installs deps, builds frontend
# This takes 10-30 minutes the first time
docker compose build

# Start everything in the background
docker compose up -d

# Watch startup logs
docker compose logs -f

# Check all services are running
docker compose ps

# Expected output (all should show "Up" or "Completed" for migrate):
#   Name                 Status
#   crm-envr-db-1        Up (healthy)
#   crm-envr-ollama-1    Up (healthy)
#   crm-envr-migrate-1   Exited (0)  ← This is normal, ran once
#   crm-envr-ai-1        Up
#   crm-envr-scraper-1   Up
#   crm-envr-next-1      Up
#   crm-envr-signaling-1 Up

# If something fails, check its logs:
docker compose logs <service-name>
# Example: docker compose logs scraper


═══════════════════════════════════════════════════════════════════════
5. POST-SETUP — AI MODEL
═══════════════════════════════════════════════════════════════════════

# Pull the AI model into the Ollama container (one time, 4-5GB download)
docker compose exec ollama ollama pull dolphin-llama3:8b

# Verify model is loaded
docker compose exec ollama ollama list

# Expected output:
#   NAME                    ID              SIZE
#   dolphin-llama3:8b       <hash>          4.5 GB

# Test the AI server
curl http://localhost:3001/health
# Expected: {"status":"ok"}

# Test the setup status
curl http://localhost:3001/setup/status | python3 -m json.tool
# Expected: ollama_running: true, model_available: true

# If model_available is false, wait for the pull to finish and retry


═══════════════════════════════════════════════════════════════════════
6. POST-SETUP — FACEBOOK SCRAPING
═══════════════════════════════════════════════════════════════════════

The scraper needs a Facebook-logged-in browser profile to work.
On a VPS (no GUI browser), there are TWO approaches:

─── Option A: Copy your local Firefox profile to the VPS (recommended) ──

  Step 1: On your Windows PC, locate your Firefox profile:
    C:\Users\USER-PC\AppData\Roaming\Mozilla\Firefox\Profiles\*.default-release

  Step 2: Zip it:
    Compress-Archive -Path "C:\Users\USER-PC\AppData\Roaming\Mozilla\Firefox\Profiles\*.default-release\*" -DestinationPath firefox-profile.zip

  Step 3: Copy to VPS:
    scp firefox-profile.zip user@YOUR_VPS_IP:~/firefox-profile.zip

  Step 4: On VPS, extract to a persistent location:
    mkdir -p ~/browser-profiles/firefox
    unzip ~/firefox-profile.zip -d ~/browser-profiles/firefox

  Step 5: Stop services, add profile mount, restart:
    docker compose down
    # Edit docker-compose.yml: add volume to the scraper service
    #   volumes:
    #     - ~/browser-profiles/firefox:/root/.mozilla/firefox/profile:ro
    nano docker-compose.yml

    docker compose up -d

  Step 6: Set the profile path in .env:
    FX_PROFILE=/root/.mozilla/firefox/profile
    # Then restart: docker compose restart scraper

─── Option B: Use Agent fallback (no profile needed, AI-driven) ──

  The scraper automatically falls back to browser-use Agent
  (AI-powered navigation) when no logged-in profile is found.
  This is slower but works without a real browser profile.

  No setup needed — it just works.

─── Option C: Chrome/Edge on VPS (if you have a desktop environment) ──

  Install a browser and log into Facebook once:
    sudo apt install -y firefox
    # Then manually log into Facebook and keep the profile

─── Verify scraper is working ──

  # Test the scraper health endpoint
  curl http://localhost:3008/health

  # Check profile detection
  curl http://localhost:3008/setup/profile | python3 -m json.tool
  # Shows detected browser profiles

  # Do a test scrape (note: this takes 2-5 minutes)
  curl -X POST "http://localhost:3008/scrape/facebook?force=true&query=I%20need%20a%20website" | python3 -m json.tool


═══════════════════════════════════════════════════════════════════════
7. DATABASE SETUP (automatic)
═══════════════════════════════════════════════════════════════════════

No manual DB setup needed — the migrate service runs automatically:
  • Creates the crm database (via POSTGRES_DB env var)
  • Runs all SQL migrations from database/migrations/
  • Creates the set_session_user_context() function
  • Creates all tables (users, leads, conversations, etc.)

If you need to reset the database:
  docker compose down -v    # WARNING: DELETES ALL DATA
  docker compose up -d      # Fresh start


═══════════════════════════════════════════════════════════════════════
8. FIREWALL & SECURITY
═══════════════════════════════════════════════════════════════════════

# Allow SSH
sudo ufw allow 22/tcp

# Allow frontend (share with friends)
sudo ufw allow 3006/tcp

# Allow scraper API (needed by frontend JS)
sudo ufw allow 3008/tcp

# Optional: allow AI server setup page
sudo ufw allow 3001/tcp

# Optional: allow signaling server
sudo ufw allow 3007/tcp

# Enable firewall
sudo ufw enable
sudo ufw status

# For production: use a reverse proxy (Caddy/Nginx) on port 80/443
# instead of exposing raw ports. Let me know if you need this.


═══════════════════════════════════════════════════════════════════════
9. ACCESSING THE APP
═══════════════════════════════════════════════════════════════════════

# From your VPS itself:
  http://localhost:3006

# From other machines / friends:
  http://YOUR_VPS_IP:3006

# The first time you visit, the splash page will show you all services.
# If everything is green, click "Continue to App" to set up your
# admin account and log in.

# Splash page (setup wizard):
  http://YOUR_VPS_IP:3001/splash

# Viewing leads (after scraping):
  Use the AI Assistant panel in the app to search for leads.
  Select a job category (Website Creation or Tutoring) and click search.
  Results appear within 2-5 minutes.


═══════════════════════════════════════════════════════════════════════
10. COMMON COMMANDS
═══════════════════════════════════════════════════════════════════════

# Stop all services
docker compose down

# Stop and delete all data (volumes)
docker compose down -v

# Restart a specific service
docker compose restart <service>

# View logs (follow)
docker compose logs -f <service>

# Pull the latest code and rebuild
git pull
docker compose build --no-cache <service>
docker compose up -d

# Rebuild everything from scratch
docker compose down
docker compose build --no-cache
docker compose up -d

# Check disk usage
docker system df

# Clean up unused Docker data
docker system prune -a

# Enter a running container
docker compose exec <service> /bin/bash

# For the scraper container (uses sh, not bash):
docker compose exec scraper /bin/sh

# View live resource usage
docker stats


═══════════════════════════════════════════════════════════════════════
11. TROUBLESHOOTING
═══════════════════════════════════════════════════════════════════════

─── "port is already allocated" ──
  Something else is using port 3006/3008/3001.
  Check: sudo lsof -i :3006
  Kill: sudo kill <PID>

─── "No function matches the given name" ──
  Migrations haven't run. Check the migrate container:
    docker compose logs migrate
  If it failed, run manually:
    docker compose run --rm migrate

─── "Model not found" or "model_available: false" ──
  The Ollama model hasn't been pulled yet:
    docker compose exec ollama ollama pull dolphin-llama3:8b
  Wait for it to finish, then restart the AI server:
    docker compose restart ai

─── "Scraper not reachable" ──
  Scraper container might be restarting. Check logs:
    docker compose logs scraper
  Common cause: not enough memory for Playwright browsers

─── "Facebook login page detected" ──
  No logged-in browser profile available. The scraper will
  automatically fall back to the Agent (AI-powered) path.
  Or set up a profile using Option A in Section 6.

─── "Next.js build failed" ──
  The @next/swc-win32-x64-msvc package is Windows-only and will
  be skipped on Linux. If the build fails, check:
    docker compose logs next-build
  Common fix: add Linux SWC package
    docker compose run --rm next npm install @next/swc-linux-x64-gnu --save-dev

─── Container keeps restarting ──
  Check why: docker compose logs <service>
  If it's the scraper, it might be out of memory:
    docker compose logs scraper | grep -i "memory\|killed\|oom"

─── Slow scraping / Agent timeout ──
  The browser-use Agent fallback is slow on 8GB RAM.
  Close other services or upgrade to 16GB for better performance.

─── Scraper always returns empty leads ──
  The date filter is 2 days max. If no recent posts match,
  you'll get empty results. This is by design (precision over quantity).
  Try again later when fresh posts appear, or force a wider search:
    curl -X POST "http://localhost:3008/scrape/facebook?force=true&query=I%20need%20a%20website"


═══════════════════════════════════════════════════════════════════════
12. FILE REFERENCE
═══════════════════════════════════════════════════════════════════════

docker-compose.yml          Orchestrates all 7 services
Dockerfile                  Next.js frontend (multi-stage build)
ai-server/Dockerfile        AI server + PostgreSQL client
browser-use-service/Dockerfile  Python scraper with Playwright
Dockerfile.signaling        WebRTC signaling server
.dockerignore               Files excluded from Docker builds
.env.docker                 Template env vars for Docker deployment
.env                        Your actual env vars (create from .env.docker)

ai-server/index.mjs         AI chat + setup wizard API
browser-use-service/main.py Facebook scraper (FastAPI + Playwright)
src/                        Next.js frontend source
database/migrations/        SQL migration files (run automatically)
data/ai/                    AI instructions and job definitions


═══════════════════════════════════════════════════════════════════════
13. UPDATING THE APP
═══════════════════════════════════════════════════════════════════════

# If using git:
  git pull
  docker compose build --no-cache next ai scraper
  docker compose up -d

# If copying files manually:
  # Copy updated project files to VPS
  docker compose build --no-cache next ai scraper
  docker compose up -d

# To update the AI model to a different one:
  docker compose exec ollama ollama pull <new-model-name>
  # Then update .env: AI_MODEL=<new-model-name>
  docker compose restart ai scraper


═══════════════════════════════════════════════════════════════════════
14. PRODUCTION REVERSE PROXY (for port 80/443)
═══════════════════════════════════════════════════════════════════════

If you want to serve everything on standard ports (80/443 with HTTPS):

# Install Caddy (simplest, auto HTTPS)
sudo apt install -y caddy

# Create config: /etc/caddy/Caddyfile
# ──────────────────────────────
# yourdomain.com {
#     # Next.js frontend
#     handle_path / {
#         reverse_proxy next:3006
#     }
#     # API routes used by frontend JS
#     handle_path /api/* {
#         reverse_proxy next:3006
#     }
#     # Scraper API (needed by browser JS)
#     handle /scrape/* {
#         reverse_proxy scraper:3008
#     }
#     handle /health {
#         reverse_proxy scraper:3008
#     }
# }
# ──────────────────────────────

# Reload Caddy
sudo caddy reload

For this setup, NEXT_PUBLIC_SCRAPER_URL would need to be a relative path
(e.g., just "" or "/") which requires a small code change. Ask me if needed.


═══════════════════════════════════════════════════════════════════════
END — You're all set. The app should be running at http://YOUR_VPS_IP:3006
═══════════════════════════════════════════════════════════════════════