L1 diagnostic steps for troubleshooting on-premise Odin AI deployments
This guide provides Level 1 (L1) diagnostic steps for troubleshooting issues in on-premise Odin AI deployments. These steps help identify common problems with containers, databases, services, and system resources.
Prerequisites: You need SSH access to the customer’s VM/server where Odin AI is deployed, and appropriate permissions to run Docker commands and access container logs.
Container Status Checks
Check All Container Status
First, verify which containers are running and their health status:
# Check all containers statusdocker ps -a# Check containers with health statusdocker ps --format "table {{.Names}}\t{{.Status}}\t{{.Health}}"# Check specific container statusdocker ps -a | grep <container_name>
Expected Containers:
web - Frontend application
api or fastapi_backend - Backend API server
worker or celery_worker - Celery worker(s)
redis - Redis cache
rabbitmq - RabbitMQ message queue
supabase-studio - Supabase Studio
supabase-kong - Kong API Gateway
supabase-auth - Auth service
supabase-db or postgres - PostgreSQL database
Other Supabase services (storage, meta, etc.)
What to Check:
All containers should be in “Up” status
No containers should be in “Restarting” or “Exited” state
Health checks should show “healthy” where applicable
Restart Failed Containers
If containers are stopped or restarting:
# Restart a specific containerdocker restart <container_name># Restart all containersdocker compose restart# Restart specific servicedocker compose restart <service_name>
Backend Container Logs
Check API Container Logs
The backend API container logs contain critical information about errors, database connections, and service issues:
# View recent logs (last 100 lines)docker logs --tail 100 api# Or for fastapi_backenddocker logs --tail 100 fastapi_backend# Follow logs in real-timedocker logs -f api# View logs with timestampsdocker logs -t api# View logs from specific timedocker logs --since 30m api# View logs between timestampsdocker logs --since "2024-01-01T00:00:00" --until "2024-01-01T23:59:59" api
# Test database connection from API containerdocker exec api python -c "import osfrom sqlalchemy import create_enginetry: engine = create_engine(os.environ.get('DATABASE_URL')) conn = engine.connect() print('Database connection successful') conn.close()except Exception as e: print(f'Database connection failed: {e}')"
Check Database Migrations
# Check if migrations are up to datedocker exec api alembic current# Check migration historydocker exec api alembic history# View pending migrationsdocker exec api alembic heads
# Test RabbitMQ connectiondocker exec api python -c "import pikatry: connection = pika.BlockingConnection( pika.ConnectionParameters('rabbitmq', 5672, '/', pika.PlainCredentials('user', 'password')) ) print('RabbitMQ connection successful') connection.close()except Exception as e: print(f'RabbitMQ connection failed: {e}')"
System Resources
Check Disk Space
Low disk space can cause database, storage, and container issues:
# Check overall disk usagedf -h# Check disk usage for Docker volumesdocker system df# Check specific volume usagedocker volume inspect <volume_name># Check directory sizesdu -sh /var/lib/docker/volumes/*du -sh ./supabase/docker/volumes/*# Check in-container disk usage (for database)docker exec supabase-db df -h
What to Check:
Root partition has sufficient space (>20% free recommended)
Docker volumes are not full
Database data directory has space
Supabase storage has space
Check Memory Usage
# Check system memoryfree -h# Check container memory usagedocker stats --no-stream# Check specific container memorydocker stats <container_name> --no-stream
What to Check:
System has available memory
Containers are not hitting memory limits
No OOM (Out of Memory) kills in logs
Check CPU Usage
# Check CPU usagetop# orhtop# Check container CPU usagedocker stats --no-stream
Network Connectivity
Check Container Network
# Check Docker networkdocker network ls# Inspect network configurationdocker network inspect <network_name># Check if containers can communicatedocker exec api ping -c 3 redisdocker exec api ping -c 3 rabbitmqdocker exec api ping -c 3 supabase-db
Check Port Availability
# Check if ports are in usenetstat -tulpn | grep -E "(3001|8001|6379|5672|5432|8000)"# Or using ssss -tulpn | grep -E "(3001|8001|6379|5672|5432|8000)"# Check port from containerdocker exec api curl -I http://localhost:8001/health
Environment Variables
Check Environment Configuration
# Check environment variables for a containerdocker exec api env | grep -E "(DATABASE|REDIS|RABBITMQ|API)"# Check .env files (if accessible)cat ./env/.env.server | grep -v "PASSWORD\|SECRET\|KEY" # Exclude sensitive datacat ./env/.env.web | grep -v "PASSWORD\|SECRET\|KEY"# Check environment from docker-composedocker compose config