mirror of https://github.com/kjanat/livedash-node.git synced 2026-01-16 12:12:09 +01:00

Files

Kaj Kowalski e2301725a3 feat: complete development environment setup and code quality improvements

- Set up pre-commit hooks with husky and lint-staged for automated code quality
- Improved TypeScript type safety by replacing 'any' types with proper generics
- Fixed markdown linting violations (MD030 spacing) across all documentation
- Fixed compound adjective hyphenation in technical documentation
- Fixed invalid JSON union syntax in API documentation examples
- Automated code formatting and linting on commit
- Enhanced error handling with better type constraints
- Configured biome and markdownlint for consistent code style
- All changes verified with successful production build

2025-07-13 14:44:05 +02:00

3.1 KiB

Raw Blame History

🚨 Database Connection Issues - Fixes Applied

Issues Identified

From your logs:

Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.neon.tech:5432`
[NODE-CRON] [WARN] missed execution! Possible blocking IO or high CPU

Root Causes

Multiple PrismaClient instances across schedulers
No connection retry logic for temporary failures
No connection pooling optimization for Neon
Aggressive scheduler intervals overwhelming database

Fixes Applied ✅

1. Connection Retry Logic (`lib/database-retry.ts`)

Automatic retry for connection errors
Exponential backoff (1s → 2s → 4s → 10s max)
Smart error detection (only retry connection issues)
Configurable retry attempts (default: 3 retries)

2. Enhanced Schedulers

Import Processor: Added retry wrapper around main processing
Session Processor: Added retry wrapper around AI processing
Graceful degradation when database is temporarily unavailable

3. Singleton Pattern Enforced

All schedulers now use import { prisma } from "./prisma.js"
No more separate new PrismaClient() instances
Shared connection pool across all operations

4. Neon-Specific Optimizations

Connection limit guidance: 15 connections (below Neon's 20 limit)
Extended timeouts: 30s for cold start handling
SSL mode requirements: sslmode=require for Neon
Application naming: For better monitoring

Immediate Actions Needed

1. Update Environment Variables

# Add to .env.local
USE_ENHANCED_POOLING=true
DATABASE_CONNECTION_LIMIT=15
DATABASE_POOL_TIMEOUT=30

# Update your DATABASE_URL to include:
DATABASE_URL="postgresql://user:pass@ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.neon.tech:5432/db?sslmode=require&connection_limit=15&pool_timeout=30"

2. Reduce Scheduler Frequency (Optional)

# Less aggressive intervals
CSV_IMPORT_INTERVAL="*/30 * * * *"        # Every 30 min (was 15)
IMPORT_PROCESSING_INTERVAL="*/10 * * * *"  # Every 10 min (was 5)
SESSION_PROCESSING_INTERVAL="0 */2 * * *"  # Every 2 hours (was 1)

3. Run Configuration Check

pnpm db:check

Expected Results

✅ Connection Stability: Automatic retry on temporary failures
✅ Resource Efficiency: Single shared connection pool
✅ Neon Optimization: Proper connection limits and timeouts
✅ Monitoring: Health check endpoint for visibility
✅ Graceful Degradation: Schedulers won't crash on DB issues

Monitoring

Health Endpoint: /api/admin/database-health
Connection Logs: Enhanced logging for pool events
Retry Logs: Detailed retry attempt logging
Error Classification: Retryable vs non-retryable errors

Files Modified

lib/database-retry.ts - New retry utilities
lib/importProcessor.ts - Added retry wrapper
lib/processingScheduler.ts - Added retry wrapper
docs/neon-database-optimization.md - Neon-specific guide
scripts/check-database-config.ts - Configuration checker

The connection issues should be significantly reduced with these fixes! 🎯

3.1 KiB Raw Blame History