This commit introduces a range of improvements across the application:
- **Security:**
- Adds authentication to the CSP metrics endpoint.
- Hardens CSP bypass detection regex to prevent ReDoS attacks.
- Improves CORS headers for the CSP metrics API.
- Adds filtering for acknowledged alerts in security monitoring.
- **Performance:**
- Optimizes database connection pooling for NeonDB.
- Improves session fetching with abort controller.
- **Stability:**
- Adds error handling to the tRPC demo component.
- Fixes type inconsistencies in session data mapping.
- **Docs & DX:**
- Ignores files in git.
- Fixes a token placeholder in the documentation.
- Fix 36+ biome linting issues reducing errors/warnings from 227 to 191
- Replace explicit 'any' types with proper TypeScript interfaces
- Fix React hooks dependencies and useCallback patterns
- Resolve unused variables and parameter assignment issues
- Improve accessibility with proper label associations
- Add comprehensive API documentation for admin and security features
- Update README.md with accurate PostgreSQL setup and current tech stack
- Create complete documentation for audit logging, CSP monitoring, and batch processing
- Fix outdated project information and missing developer workflows
- Add robust email service with rate limiting and configuration management
- Implement shared rate limiter utility for consistent API protection
- Create comprehensive test suite for core processing pipeline
- Add API tests for dashboard metrics and authentication routes
- Fix date range picker infinite loop issue
- Improve session lookup in refresh sessions API
- Refactor session API routing with better code organization
- Update processing pipeline status monitoring
- Clean up leftover files and improve code formatting
🎯 SESSION POOLING PERFORMANCE BREAKTHROUGH!
✅ Critical Issues Fixed:
- Eliminated multiple PrismaClient instances across schedulers
- Fixed connection pool exhaustion risk in processing modules
- Implemented singleton pattern for all database connections
- Added graceful shutdown and connection cleanup
🚀 Enhanced Pooling Features:
- Dual-mode connection pooling (standard + enhanced)
- PostgreSQL native pooling with @prisma/adapter-pg
- Advanced connection monitoring and health checks
- Configurable pool limits and timeouts via environment variables
- Real-time connection statistics and metrics
📊 Performance Optimizations:
- Single shared connection pool across all schedulers
- Configurable connection limits (DATABASE_CONNECTION_LIMIT=20)
- Idle timeout management (DATABASE_POOL_TIMEOUT=10)
- Connection cycling and health validation
- Process termination signal handling
🛠️ New Infrastructure:
- lib/database-pool.ts - Advanced pooling configuration
- app/api/admin/database-health/route.ts - Connection monitoring
- Enhanced lib/prisma.ts with dual-mode support
- Comprehensive documentation in docs/database-connection-pooling.md
- Graceful shutdown handling in lib/schedulers.ts
🎛️ Environment Configuration:
- USE_ENHANCED_POOLING=true for production optimization
- DATABASE_CONNECTION_LIMIT for pool size control
- DATABASE_POOL_TIMEOUT for idle connection management
- Automatic enhanced pooling in production environments
📈 Expected Impact:
- Eliminates connection pool exhaustion under load
- Reduces memory footprint from idle connections
- Improves scheduler performance and reliability
- Enables better resource monitoring and debugging
- Supports horizontal scaling with proper connection management
Production-ready connection pooling with monitoring and health checks!
- Replace SQLite with PostgreSQL using Neon as provider
- Add environment-based database URL configuration
- Create separate test database setup with DATABASE_URL_TEST
- Reset migration history and generate fresh PostgreSQL schema
- Add comprehensive migration documentation
- Include database unit tests for connection validation
- Updated session processing commands in documentation for clarity.
- Removed transcript content fetching from session processing, allowing on-demand retrieval.
- Improved session metrics calculations and added new metrics for dashboard.
- Refactored processing scheduler to handle sessions in parallel with concurrency limits.
- Added manual trigger API for processing unprocessed sessions with admin checks.
- Implemented scripts for fetching and parsing transcripts, checking transcript content, and testing processing status.
- Updated Prisma schema to enforce default values for processed sessions.
- Added error handling and logging improvements throughout the processing workflow.
- Updated formatting in SessionDetails component for better readability.
- Enhanced documentation in scheduler-fixes.md to clarify issues and solutions.
- Improved error handling and logging in csvFetcher.js and processingScheduler.js.
- Standardized code formatting across various scripts and components for consistency.
- Added validation checks for CSV URLs and transcript content to prevent processing errors.
- Enhanced logging messages for better tracking of processing status and errors.
- Added MessageViewer component to display parsed messages in a chat-like format.
- Introduced new Message table in the database to store individual messages with timestamps, roles, and content.
- Updated Session model to include a relation to parsed messages.
- Created transcript parsing logic to convert raw transcripts into structured messages.
- Enhanced processing scheduler to handle sessions with parsed messages.
- Updated API endpoints to return parsed messages alongside session details.
- Added manual trigger commands for session refresh, transcript parsing, and processing.
- Improved user experience with color-coded message roles and timestamps in the UI.
- Documented the new scheduler workflow and transcript parsing implementation.
- Added processingScheduler.js and processingScheduler.ts to handle session transcript processing using OpenAI API.
- Implemented a new scheduler (scheduler.js and schedulers.ts) for refreshing sessions every 15 minutes.
- Updated Prisma migrations to add new fields for processed sessions, including questions, sentimentCategory, and summary.
- Created scripts (process_sessions.mjs and process_sessions.ts) for manual processing of unprocessed sessions.
- Enhanced server.js and server.mjs to initialize schedulers on server start.
Improves the dashboard with additional metrics and visualizations
to provide a more comprehensive overview of application usage and performance.
Adds new charts, including:
- Word cloud for category analysis
- Geographic map for user distribution (simulated data)
- Response time distribution chart
Refactors existing components for improved clarity and reusability,
including the introduction of a generic `MetricCard` component.
Improves error handling and user feedback during data refresh and
session loading.
Adds recommended VSCode extensions for ESLint and Prettier.
