livedash-node/docs/database-connection-pooling.md

# Database Connection Pooling Guide

This document explains how to optimize database connection pooling for better performance and resource management in the LiveDash application.

## Overview

The application now supports two connection pooling modes:

1.  **Standard Pooling**: Default Prisma client connection pooling
2.  **Enhanced Pooling**: Advanced PostgreSQL connection pooling with custom configuration

## Configuration

### Environment Variables

Add these variables to your `.env.local` file:

```bash
# Database Connection Pooling Configuration
DATABASE_CONNECTION_LIMIT=20          # Maximum connections in pool
DATABASE_POOL_TIMEOUT=10              # Idle timeout in seconds
USE_ENHANCED_POOLING=true             # Enable advanced pooling (production recommended)

# Optional: Add pool parameters to DATABASE_URL for additional control
DATABASE_URL="postgresql://user:pass@host:5432/db?connection_limit=20&pool_timeout=10"
```

### Pooling Modes

#### Standard Pooling (Default)

-   Uses Prisma's built-in connection pooling
-   Simpler configuration
-   Good for development and small-scale deployments

#### Enhanced Pooling (Recommended for Production)

-   Uses PostgreSQL native connection pooling with `@prisma/adapter-pg`
-   Advanced monitoring and health checks
-   Better resource management
-   Detailed connection metrics

## Implementation Details

### Fixed Issues

1.  **Multiple PrismaClient Instances**:
   -   ❌ Before: Each scheduler created its own PrismaClient
   -   ✅ After: All modules use singleton pattern from `lib/prisma.ts`

2.  **No Connection Management**:
   -   ❌ Before: No graceful shutdown or connection cleanup
   -   ✅ After: Proper cleanup on process termination

3.  **No Monitoring**:
   -   ❌ Before: No visibility into connection usage
   -   ✅ After: Health check endpoint and connection metrics

### Key Files Modified

-   `lib/prisma.ts` - Enhanced singleton with pooling options
-   `lib/database-pool.ts` - Advanced pooling configuration
-   `lib/processingScheduler.ts` - Fixed to use singleton
-   `lib/importProcessor.ts` - Fixed to use singleton
-   `lib/processingStatusManager.ts` - Fixed to use singleton
-   `lib/schedulers.ts` - Added graceful shutdown
-   `app/api/admin/database-health/route.ts` - Monitoring endpoint

## Monitoring

### Health Check Endpoint

Check database connection health:

```bash
curl -H "Authorization: Bearer your-token" \
     http://localhost:3000/api/admin/database-health
```

Response includes:

-   Connection status
-   Pool statistics (if enhanced pooling enabled)
-   Basic metrics (session counts, etc.)
-   Configuration details

### Connection Metrics

With enhanced pooling enabled, you'll see console logs for:

-   Connection acquisitions/releases
-   Pool size changes
-   Error events
-   Health check results

## Performance Benefits

### Before Optimization

-   Multiple connection pools (one per scheduler)
-   Potential connection exhaustion under load
-   No connection monitoring
-   Resource waste from idle connections

### After Optimization

-   Single shared connection pool
-   Configurable pool size and timeouts
-   Connection health monitoring
-   Graceful shutdown and cleanup
-   Better resource utilization

## Recommended Settings

### Development

```bash
DATABASE_CONNECTION_LIMIT=10
DATABASE_POOL_TIMEOUT=30
USE_ENHANCED_POOLING=false
```

### Production

```bash
DATABASE_CONNECTION_LIMIT=20
DATABASE_POOL_TIMEOUT=10
USE_ENHANCED_POOLING=true
```

### High-Load Production

```bash
DATABASE_CONNECTION_LIMIT=50
DATABASE_POOL_TIMEOUT=5
USE_ENHANCED_POOLING=true
```

## Troubleshooting

### Connection Pool Exhaustion

If you see "too many connections" errors:

1.  Increase `DATABASE_CONNECTION_LIMIT`
2.  Check for connection leaks in application code
3.  Monitor the health endpoint for pool statistics

### Slow Database Queries

If queries are timing out:

1.  Decrease `DATABASE_POOL_TIMEOUT`
2.  Check database query performance
3.  Consider connection pooling at the infrastructure level (PgBouncer)

### Memory Usage

If memory usage is high:

1.  Decrease `DATABASE_CONNECTION_LIMIT`
2.  Enable enhanced pooling for better resource management
3.  Monitor idle connection cleanup

## Best Practices

1.  **Always use the singleton**: Import `prisma` from `lib/prisma.ts`
2.  **Monitor connection usage**: Use the health endpoint regularly
3.  **Set appropriate limits**: Don't over-provision connections
4.  **Enable enhanced pooling in production**: Better resource management
5.  **Implement graceful shutdown**: Ensure connections are properly closed
6.  **Log connection events**: Monitor for issues and optimize accordingly

## Next Steps

Consider implementing:

1.  **Connection pooling middleware**: PgBouncer or similar
2.  **Read replicas**: For read-heavy workloads
3.  **Connection retry logic**: For handling temporary failures
4.  **Metrics collection**: Prometheus/Grafana for detailed monitoring