feat: implement comprehensive email system with rate limiting and extensive test suite

- Add robust email service with rate limiting and configuration management
- Implement shared rate limiter utility for consistent API protection
- Create comprehensive test suite for core processing pipeline
- Add API tests for dashboard metrics and authentication routes
- Fix date range picker infinite loop issue
- Improve session lookup in refresh sessions API
- Refactor session API routing with better code organization
- Update processing pipeline status monitoring
- Clean up leftover files and improve code formatting

docs/neon-database-optimization.md

@@ -5,6 +5,7 @@ This document provides specific recommendations for optimizing database connecti
 ## Current Issues Observed
 
 From your logs, we can see:
+
 ```
 Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.neon.tech:5432`
 [NODE-CRON] [WARN] missed execution at Sun Jun 29 2025 12:00:00 GMT+0200! Possible blocking IO or high CPU
@@ -13,16 +14,19 @@ Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.ne
 ## Root Causes
 
 ### 1. Neon Connection Limits
+
 - **Free Tier**: 20 concurrent connections
-- **Pro Tier**: 100 concurrent connections  
+- **Pro Tier**: 100 concurrent connections
 - **Multiple schedulers** can quickly exhaust connections
 
 ### 2. Connection Pooling Issues
+
 - Each scheduler was creating separate PrismaClient instances
 - No connection reuse between operations
 - No retry logic for temporary failures
 
 ### 3. Neon-Specific Challenges
+
 - **Auto-pause**: Databases pause after inactivity
 - **Cold starts**: First connection after pause takes longer
 - **Regional latency**: eu-central-1 may have variable latency
@@ -30,6 +34,7 @@ Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.ne
 ## Solutions Implemented
 
 ### 1. Fixed Multiple PrismaClient Instances ✅
+
 ```typescript
 // Before: Each file created its own client
 const prisma = new PrismaClient(); // ❌
@@ -39,30 +44,30 @@ import { prisma } from "./prisma.js"; // ✅
 ```
 
 ### 2. Added Connection Retry Logic ✅
+
 ```typescript
 // Automatic retry for connection errors
-await withRetry(
-  async () => await databaseOperation(),
-  {
-    maxRetries: 3,
-    initialDelay: 2000,
-    maxDelay: 10000,
-    backoffMultiplier: 2,
-  }
-);
+await withRetry(async () => await databaseOperation(), {
+  maxRetries: 3,
+  initialDelay: 2000,
+  maxDelay: 10000,
+  backoffMultiplier: 2,
+});
 ```
 
 ### 3. Enhanced Connection Pooling ✅
+
 ```typescript
 // Production-ready pooling with @prisma/adapter-pg
-USE_ENHANCED_POOLING=true
-DATABASE_CONNECTION_LIMIT=20
-DATABASE_POOL_TIMEOUT=10
+USE_ENHANCED_POOLING = true;
+DATABASE_CONNECTION_LIMIT = 20;
+DATABASE_POOL_TIMEOUT = 10;
 ```
 
 ## Neon-Specific Configuration
 
 ### Environment Variables
+
 ```bash
 # Optimized for Neon
 DATABASE_URL="postgresql://user:pass@ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.neon.tech:5432/db?sslmode=require&connection_limit=15"
@@ -79,6 +84,7 @@ SESSION_PROCESSING_INTERVAL="0 */2 * * *"  # Every 2 hours instead of 1
 ```
 
 ### Connection String Optimization
+
 ```bash
 # Add these parameters to your DATABASE_URL
 ?sslmode=require                    # Required for Neon
@@ -91,6 +97,7 @@ SESSION_PROCESSING_INTERVAL="0 */2 * * *"  # Every 2 hours instead of 1
 ## Monitoring & Troubleshooting
 
 ### 1. Health Check Endpoint
+
 ```bash
 # Check connection health
 curl -H "Authorization: Bearer your-token" \
@@ -98,11 +105,13 @@ curl -H "Authorization: Bearer your-token" \
 ```
 
 ### 2. Neon Dashboard Monitoring
+
 - Monitor "Active connections" in Neon dashboard
 - Check for connection spikes during scheduler runs
 - Review query performance and slow queries
 
 ### 3. Application Logs
+
 ```bash
 # Look for connection patterns
 grep "Database connection" logs/*.log
@@ -113,65 +122,77 @@ grep "retry" logs/*.log
 ## Performance Optimizations
 
 ### 1. Reduce Scheduler Frequency
+
 ```typescript
 // Current intervals may be too aggressive
-CSV_IMPORT_INTERVAL="*/15 * * * *"     // ➜ "*/30 * * * *"
-IMPORT_PROCESSING_INTERVAL="*/5 * * * *"  // ➜ "*/10 * * * *"  
-SESSION_PROCESSING_INTERVAL="0 * * * *"   // ➜ "0 */2 * * *"
+CSV_IMPORT_INTERVAL = "*/15 * * * *"; // ➜ "*/30 * * * *"
+IMPORT_PROCESSING_INTERVAL = "*/5 * * * *"; // ➜ "*/10 * * * *"
+SESSION_PROCESSING_INTERVAL = "0 * * * *"; // ➜ "0 */2 * * *"
 ```
 
 ### 2. Batch Size Optimization
+
 ```typescript
 // Reduce batch sizes to avoid long-running transactions
-CSV_IMPORT_BATCH_SIZE=50         // ➜ 25
-IMPORT_PROCESSING_BATCH_SIZE=50  // ➜ 25
-SESSION_PROCESSING_BATCH_SIZE=20 // ➜ 10
+CSV_IMPORT_BATCH_SIZE = 50; // ➜ 25
+IMPORT_PROCESSING_BATCH_SIZE = 50; // ➜ 25
+SESSION_PROCESSING_BATCH_SIZE = 20; // ➜ 10
 ```
 
 ### 3. Connection Keepalive
+
 ```typescript
 // Keep connections warm to avoid cold starts
 const prisma = new PrismaClient({
   datasources: {
     db: {
-      url: process.env.DATABASE_URL + "&keepalive=true"
-    }
-  }
+      url: process.env.DATABASE_URL + "&keepalive=true",
+    },
+  },
 });
 ```
 
 ## Troubleshooting Common Issues
 
 ### "Can't reach database server"
+
 **Causes:**
+
 - Neon database auto-paused
 - Connection limit exceeded
 - Network issues
 
 **Solutions:**
+
 1. Enable enhanced pooling: `USE_ENHANCED_POOLING=true`
 2. Reduce connection limit: `DATABASE_CONNECTION_LIMIT=15`
 3. Implement retry logic (already done)
 4. Check Neon dashboard for database status
 
 ### "Connection terminated"
+
 **Causes:**
+
 - Idle connection timeout
 - Neon maintenance
 - Long-running transactions
 
 **Solutions:**
+
 1. Increase pool timeout: `DATABASE_POOL_TIMEOUT=30`
 2. Add connection cycling
 3. Break large operations into smaller batches
 
 ### "Missed cron execution"
+
 **Causes:**
+
 - Blocking database operations
 - Scheduler overlap
 - High CPU usage
 
 **Solutions:**
+
 1. Reduce scheduler frequency
 2. Add concurrency limits
 3. Monitor scheduler execution time
@@ -179,6 +200,7 @@ const prisma = new PrismaClient({
 ## Recommended Production Settings
 
 ### For Neon Free Tier (20 connections)
+
 ```bash
 DATABASE_CONNECTION_LIMIT=15
 DATABASE_POOL_TIMEOUT=30
@@ -189,6 +211,7 @@ SESSION_PROCESSING_INTERVAL="0 */3 * * *"
 ```
 
 ### For Neon Pro Tier (100 connections)
+
 ```bash
 DATABASE_CONNECTION_LIMIT=50
 DATABASE_POOL_TIMEOUT=20
@@ -213,4 +236,4 @@ SESSION_PROCESSING_INTERVAL="0 */2 * * *"
 - [ ] Test health endpoint regularly
 - [ ] Set up alerts for connection failures
 
-With these optimizations, your Neon database connections should be much more stable and efficient!
+With these optimizations, your Neon database connections should be much more stable and efficient!