godcrm/deploy/RUNBOOK.md

GOD CRM — Deployment Runbook

ADR-064 Phase 3, Task 13 | Last updated: 2026-01-31

---

1. Pre-Deploy Checklist

• All tests pass: npm test -- --run
• Build succeeds: npm run build
• TypeScript checks: npx tsc --noEmit
• Environment variables set (see backend/.env.example)
• Database backup taken: pg_dump godcrm_prod > /root/backups/godcrm-$(date +%Y%m%d-%H%M).sql
• No secrets in codebase: grep -r "sk-proj\|GOCSPX" backend/ returns 0

---

2. Environments

Env	URL	Branch	Port	DB	Directory
PROD	https://crm.hltrn.cc	main	5000	godcrm_prod	/root/prod/business-crm
DEV	https://devcrm.hltrn.cc	develop	5001	godcrm	/root/workspace/business-crm

---

3. Deploy to DEV

cd /root/workspace/business-crm
git pull origin develop
npm ci
npm run build

# Copy frontend
rm -rf /var/www/business-crm-dev/dist/*
cp -r dist/* /var/www/business-crm-dev/dist/
chown -R www-data:www-data /var/www/business-crm-dev/dist/

# Restart backend
systemctl restart business-crm-dev
systemctl status business-crm-dev

# Verify
curl -s https://devcrm.hltrn.cc/api/health | jq .

---

4. Deploy to PROD

# 1. Backup database FIRST
pg_dump godcrm_prod > /root/backups/godcrm-$(date +%Y%m%d-%H%M).sql

# 2. Pull and build
cd /root/prod/business-crm
git pull origin main
npm ci --production
npm run build

# 3. Copy frontend
rm -rf /var/www/business-crm/dist/*
cp -r dist/* /var/www/business-crm/dist/
chown -R www-data:www-data /var/www/business-crm/dist/

# 4. Restart backend
systemctl restart business-crm
systemctl status business-crm

# 5. Verify
curl -s https://crm.hltrn.cc/api/health | jq .

---

5. Rollback Procedure

Backend Rollback

cd /root/prod/business-crm
git log --oneline -5           # Find last good commit
git checkout <commit-hash>     # Revert to good commit
npm ci --production
systemctl restart business-crm

Database Rollback

# Stop backend first
systemctl stop business-crm

# Restore from backup
psql -U godcrm -d godcrm_prod < /root/backups/godcrm-YYYYMMDD-HHMM.sql

# Restart
systemctl start business-crm

Frontend-Only Rollback

# If only frontend broke, restore from previous build
cd /root/prod/business-crm
git checkout HEAD~1 -- dist/
cp -r dist/* /var/www/business-crm/dist/
chown -R www-data:www-data /var/www/business-crm/dist/

---

6. Monitoring Endpoints

Endpoint	Auth	Purpose
GET /api/health	None	Basic health (status, version, uptime, DB)
GET /api/health/deep	Admin	Full system status (DB pool, memory, disk)

Health Check Commands

# Basic health
curl -s https://crm.hltrn.cc/api/health | jq .

# Deep health (requires admin token)
curl -s -H "Authorization: Bearer $TOKEN" https://crm.hltrn.cc/api/health/deep | jq .

---

7. Log Locations

Service	Log Path	Rotation
Backend (Pino)	journalctl -u business-crm	systemd journal
Nginx access	/var/log/nginx/crm.hltrn.cc.access.log	logrotate daily
Nginx error	/var/log/nginx/crm.hltrn.cc.error.log	logrotate daily
PostgreSQL	/var/log/postgresql/postgresql-16-main.log	logrotate weekly

Useful Log Commands

# Live backend logs
journalctl -u business-crm -f

# Last 100 errors
journalctl -u business-crm --since "1 hour ago" -p err

# Nginx errors
tail -f /var/log/nginx/crm.hltrn.cc.error.log

---

8. Database Backup Schedule

Type	Frequency	Retention	Script
Daily	02:00 MSK	7 days	cron: pg_dump godcrm_prod > /root/backups/daily/
Manual	On demand	30 days	POST /api/v3/system/backups/create (admin)

Cron Entry

0 2 * * * pg_dump -U godcrm godcrm_prod --compress=6 -f /root/backups/daily/godcrm_$(date +\%Y\%m\%d).sql && find /root/backups/daily/ -name "*.sql" -mtime +7 -delete

---

9. Emergency Contacts

Role	Contact
System Admin	Check server via SSH
Database	PostgreSQL on localhost:5432

Emergency Commands

# Check if backend is running
systemctl status business-crm

# Check if nginx is running
systemctl status nginx

# Check PostgreSQL
pg_isready -U godcrm

# Check disk space
df -h /

# Check memory
free -m

# Kill runaway process
systemctl restart business-crm

---

10. Required Environment Variables

See backend/.env.example for full list. Critical ones:

Variable	Required	Description
NODE_ENV	Yes	production or development
JWT_SECRET	Yes	Min 64 chars, random
MASTER_ENCRYPTION_KEY	Yes	64 hex chars for AES-256
POSTGRES_PASSWORD	Yes	DB password
CORS_ORIGINS	Prod only	Comma-separated allowed origins
OPENAI_API_KEY	Optional	For AI features
SMTP_*	Optional	For email features