f89e074dd1
Governed substrate for autonomous agents: scoped identity (passports), audited actions, MCP workspace. Infra IPs and secrets redacted for public release.
208 lines
4.9 KiB
Markdown
208 lines
4.9 KiB
Markdown
# GOD CRM — Deployment Runbook
|
|
|
|
> ADR-064 Phase 3, Task 13 | Last updated: 2026-01-31
|
|
|
|
---
|
|
|
|
## 1. Pre-Deploy Checklist
|
|
|
|
- [ ] All tests pass: `npm test -- --run`
|
|
- [ ] Build succeeds: `npm run build`
|
|
- [ ] TypeScript checks: `npx tsc --noEmit`
|
|
- [ ] Environment variables set (see `backend/.env.example`)
|
|
- [ ] Database backup taken: `pg_dump godcrm_prod > /root/backups/godcrm-$(date +%Y%m%d-%H%M).sql`
|
|
- [ ] No secrets in codebase: `grep -r "sk-proj\|GOCSPX" backend/` returns 0
|
|
|
|
---
|
|
|
|
## 2. Environments
|
|
|
|
| Env | URL | Branch | Port | DB | Directory |
|
|
|-----|-----|--------|------|----|-----------|
|
|
| PROD | https://crm.hltrn.cc | `main` | 5000 | `godcrm_prod` | `/root/prod/business-crm` |
|
|
| DEV | https://devcrm.hltrn.cc | `develop` | 5001 | `godcrm` | `/root/workspace/business-crm` |
|
|
|
|
---
|
|
|
|
## 3. Deploy to DEV
|
|
|
|
```bash
|
|
cd /root/workspace/business-crm
|
|
git pull origin develop
|
|
npm ci
|
|
npm run build
|
|
|
|
# Copy frontend
|
|
rm -rf /var/www/business-crm-dev/dist/*
|
|
cp -r dist/* /var/www/business-crm-dev/dist/
|
|
chown -R www-data:www-data /var/www/business-crm-dev/dist/
|
|
|
|
# Restart backend
|
|
systemctl restart business-crm-dev
|
|
systemctl status business-crm-dev
|
|
|
|
# Verify
|
|
curl -s https://devcrm.hltrn.cc/api/health | jq .
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Deploy to PROD
|
|
|
|
```bash
|
|
# 1. Backup database FIRST
|
|
pg_dump godcrm_prod > /root/backups/godcrm-$(date +%Y%m%d-%H%M).sql
|
|
|
|
# 2. Pull and build
|
|
cd /root/prod/business-crm
|
|
git pull origin main
|
|
npm ci --production
|
|
npm run build
|
|
|
|
# 3. Copy frontend
|
|
rm -rf /var/www/business-crm/dist/*
|
|
cp -r dist/* /var/www/business-crm/dist/
|
|
chown -R www-data:www-data /var/www/business-crm/dist/
|
|
|
|
# 4. Restart backend
|
|
systemctl restart business-crm
|
|
systemctl status business-crm
|
|
|
|
# 5. Verify
|
|
curl -s https://crm.hltrn.cc/api/health | jq .
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Rollback Procedure
|
|
|
|
### Backend Rollback
|
|
```bash
|
|
cd /root/prod/business-crm
|
|
git log --oneline -5 # Find last good commit
|
|
git checkout <commit-hash> # Revert to good commit
|
|
npm ci --production
|
|
systemctl restart business-crm
|
|
```
|
|
|
|
### Database Rollback
|
|
```bash
|
|
# Stop backend first
|
|
systemctl stop business-crm
|
|
|
|
# Restore from backup
|
|
psql -U godcrm -d godcrm_prod < /root/backups/godcrm-YYYYMMDD-HHMM.sql
|
|
|
|
# Restart
|
|
systemctl start business-crm
|
|
```
|
|
|
|
### Frontend-Only Rollback
|
|
```bash
|
|
# If only frontend broke, restore from previous build
|
|
cd /root/prod/business-crm
|
|
git checkout HEAD~1 -- dist/
|
|
cp -r dist/* /var/www/business-crm/dist/
|
|
chown -R www-data:www-data /var/www/business-crm/dist/
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Monitoring Endpoints
|
|
|
|
| Endpoint | Auth | Purpose |
|
|
|----------|------|---------|
|
|
| `GET /api/health` | None | Basic health (status, version, uptime, DB) |
|
|
| `GET /api/health/deep` | Admin | Full system status (DB pool, memory, disk) |
|
|
|
|
### Health Check Commands
|
|
```bash
|
|
# Basic health
|
|
curl -s https://crm.hltrn.cc/api/health | jq .
|
|
|
|
# Deep health (requires admin token)
|
|
curl -s -H "Authorization: Bearer $TOKEN" https://crm.hltrn.cc/api/health/deep | jq .
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Log Locations
|
|
|
|
| Service | Log Path | Rotation |
|
|
|---------|----------|----------|
|
|
| Backend (Pino) | `journalctl -u business-crm` | systemd journal |
|
|
| Nginx access | `/var/log/nginx/crm.hltrn.cc.access.log` | logrotate daily |
|
|
| Nginx error | `/var/log/nginx/crm.hltrn.cc.error.log` | logrotate daily |
|
|
| PostgreSQL | `/var/log/postgresql/postgresql-16-main.log` | logrotate weekly |
|
|
|
|
### Useful Log Commands
|
|
```bash
|
|
# Live backend logs
|
|
journalctl -u business-crm -f
|
|
|
|
# Last 100 errors
|
|
journalctl -u business-crm --since "1 hour ago" -p err
|
|
|
|
# Nginx errors
|
|
tail -f /var/log/nginx/crm.hltrn.cc.error.log
|
|
```
|
|
|
|
---
|
|
|
|
## 8. Database Backup Schedule
|
|
|
|
| Type | Frequency | Retention | Script |
|
|
|------|-----------|-----------|--------|
|
|
| Daily | 02:00 MSK | 7 days | cron: `pg_dump godcrm_prod > /root/backups/daily/` |
|
|
| Manual | On demand | 30 days | `POST /api/v3/system/backups/create` (admin) |
|
|
|
|
### Cron Entry
|
|
```cron
|
|
0 2 * * * pg_dump -U godcrm godcrm_prod --compress=6 -f /root/backups/daily/godcrm_$(date +\%Y\%m\%d).sql && find /root/backups/daily/ -name "*.sql" -mtime +7 -delete
|
|
```
|
|
|
|
---
|
|
|
|
## 9. Emergency Contacts
|
|
|
|
| Role | Contact |
|
|
|------|---------|
|
|
| System Admin | Check server via SSH |
|
|
| Database | PostgreSQL on localhost:5432 |
|
|
|
|
### Emergency Commands
|
|
```bash
|
|
# Check if backend is running
|
|
systemctl status business-crm
|
|
|
|
# Check if nginx is running
|
|
systemctl status nginx
|
|
|
|
# Check PostgreSQL
|
|
pg_isready -U godcrm
|
|
|
|
# Check disk space
|
|
df -h /
|
|
|
|
# Check memory
|
|
free -m
|
|
|
|
# Kill runaway process
|
|
systemctl restart business-crm
|
|
```
|
|
|
|
---
|
|
|
|
## 10. Required Environment Variables
|
|
|
|
See `backend/.env.example` for full list. Critical ones:
|
|
|
|
| Variable | Required | Description |
|
|
|----------|----------|-------------|
|
|
| `NODE_ENV` | Yes | `production` or `development` |
|
|
| `JWT_SECRET` | Yes | Min 64 chars, random |
|
|
| `MASTER_ENCRYPTION_KEY` | Yes | 64 hex chars for AES-256 |
|
|
| `POSTGRES_PASSWORD` | Yes | DB password |
|
|
| `CORS_ORIGINS` | Prod only | Comma-separated allowed origins |
|
|
| `OPENAI_API_KEY` | Optional | For AI features |
|
|
| `SMTP_*` | Optional | For email features |
|