diff --git a/CLAUDE.md b/CLAUDE.md index 8be36b9..54f2701 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,57 +6,57 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co **Core Development:** -- `pnpm dev` - Start development server (runs custom server.ts with schedulers) -- `pnpm dev:next-only` - Start Next.js only with Turbopack (no schedulers) -- `pnpm build` - Build production application -- `pnpm start` - Run production server +- `pnpm dev` - Start development server (runs custom server.ts with schedulers) +- `pnpm dev:next-only` - Start Next.js only with Turbopack (no schedulers) +- `pnpm build` - Build production application +- `pnpm start` - Run production server **Code Quality:** -- `pnpm lint` - Run ESLint -- `pnpm lint:fix` - Fix ESLint issues automatically -- `pnpm format` - Format code with Prettier -- `pnpm format:check` - Check formatting without fixing +- `pnpm lint` - Run ESLint +- `pnpm lint:fix` - Fix ESLint issues automatically +- `pnpm format` - Format code with Prettier +- `pnpm format:check` - Check formatting without fixing **Database:** -- `pnpm prisma:generate` - Generate Prisma client -- `pnpm prisma:migrate` - Run database migrations -- `pnpm prisma:push` - Push schema changes to database -- `pnpm prisma:push:force` - Force reset database and push schema -- `pnpm prisma:seed` - Seed database with initial data -- `pnpm prisma:studio` - Open Prisma Studio database viewer +- `pnpm prisma:generate` - Generate Prisma client +- `pnpm prisma:migrate` - Run database migrations +- `pnpm prisma:push` - Push schema changes to database +- `pnpm prisma:push:force` - Force reset database and push schema +- `pnpm prisma:seed` - Seed database with initial data +- `pnpm prisma:studio` - Open Prisma Studio database viewer **Testing:** -- `pnpm test` - Run both Vitest and Playwright tests concurrently -- `pnpm test:vitest` - Run Vitest tests only -- `pnpm test:vitest:watch` - Run Vitest in watch mode -- `pnpm test:vitest:coverage` - Run Vitest with coverage report -- `pnpm test:coverage` - Run all tests with coverage +- `pnpm test` - Run both Vitest and Playwright tests concurrently +- `pnpm test:vitest` - Run Vitest tests only +- `pnpm test:vitest:watch` - Run Vitest in watch mode +- `pnpm test:vitest:coverage` - Run Vitest with coverage report +- `pnpm test:coverage` - Run all tests with coverage **Security Testing:** -- `pnpm test:security` - Run security-specific tests -- `pnpm test:security-headers` - Test HTTP security headers implementation -- `pnpm test:csp` - Test CSP implementation and nonce generation -- `pnpm test:csp:validate` - Validate CSP implementation with security scoring -- `pnpm test:csp:full` - Comprehensive CSP test suite +- `pnpm test:security` - Run security-specific tests +- `pnpm test:security-headers` - Test HTTP security headers implementation +- `pnpm test:csp` - Test CSP implementation and nonce generation +- `pnpm test:csp:validate` - Validate CSP implementation with security scoring +- `pnpm test:csp:full` - Comprehensive CSP test suite **Migration & Deployment:** -- `pnpm migration:backup` - Create database backup -- `pnpm migration:validate-db` - Validate database schema and integrity -- `pnpm migration:validate-env` - Validate environment configuration -- `pnpm migration:pre-check` - Run pre-deployment validation checks -- `pnpm migration:health-check` - Run system health checks -- `pnpm migration:deploy` - Execute full deployment process -- `pnpm migration:rollback` - Rollback failed migration +- `pnpm migration:backup` - Create database backup +- `pnpm migration:validate-db` - Validate database schema and integrity +- `pnpm migration:validate-env` - Validate environment configuration +- `pnpm migration:pre-check` - Run pre-deployment validation checks +- `pnpm migration:health-check` - Run system health checks +- `pnpm migration:deploy` - Execute full deployment process +- `pnpm migration:rollback` - Rollback failed migration **Markdown:** -- `pnpm lint:md` - Lint Markdown files -- `pnpm lint:md:fix` - Fix Markdown linting issues +- `pnpm lint:md` - Lint Markdown files +- `pnpm lint:md:fix` - Fix Markdown linting issues ## Architecture Overview @@ -64,138 +64,138 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ### Tech Stack -- **Frontend:** Next.js 15 + React 19 + TailwindCSS 4 -- **Backend:** Next.js API Routes + Custom Node.js server -- **Database:** PostgreSQL with Prisma ORM -- **Authentication:** NextAuth.js -- **AI Processing:** OpenAI API integration -- **Visualization:** D3.js, React Leaflet, Recharts -- **Scheduling:** Node-cron for background processing +- **Frontend:** Next.js 15 + React 19 + TailwindCSS 4 +- **Backend:** Next.js API Routes + Custom Node.js server +- **Database:** PostgreSQL with Prisma ORM +- **Authentication:** NextAuth.js +- **AI Processing:** OpenAI API integration +- **Visualization:** D3.js, React Leaflet, Recharts +- **Scheduling:** Node-cron for background processing ### Key Architecture Components **1. Multi-Stage Processing Pipeline** The system processes user sessions through distinct stages tracked in `SessionProcessingStatus`: -- `CSV_IMPORT` - Import raw CSV data into `SessionImport` -- `TRANSCRIPT_FETCH` - Fetch transcript content from URLs -- `SESSION_CREATION` - Create normalized `Session` and `Message` records -- `AI_ANALYSIS` - AI processing for sentiment, categorization, summaries -- `QUESTION_EXTRACTION` - Extract questions from conversations +- `CSV_IMPORT` - Import raw CSV data into `SessionImport` +- `TRANSCRIPT_FETCH` - Fetch transcript content from URLs +- `SESSION_CREATION` - Create normalized `Session` and `Message` records +- `AI_ANALYSIS` - AI processing for sentiment, categorization, summaries +- `QUESTION_EXTRACTION` - Extract questions from conversations **2. Database Architecture** -- **Multi-tenant design** with `Company` as root entity -- **Dual storage pattern**: Raw CSV data in `SessionImport`, processed data in `Session` -- **1-to-1 relationship** between `SessionImport` and `Session` via `importId` -- **Message parsing** into individual `Message` records with order tracking -- **AI cost tracking** via `AIProcessingRequest` with detailed token usage -- **Flexible AI model management** through `AIModel`, `AIModelPricing`, and `CompanyAIModel` +- **Multi-tenant design** with `Company` as root entity +- **Dual storage pattern**: Raw CSV data in `SessionImport`, processed data in `Session` +- **1-to-1 relationship** between `SessionImport` and `Session` via `importId` +- **Message parsing** into individual `Message` records with order tracking +- **AI cost tracking** via `AIProcessingRequest` with detailed token usage +- **Flexible AI model management** through `AIModel`, `AIModelPricing`, and `CompanyAIModel` **3. Custom Server Architecture** -- `server.ts` - Custom Next.js server with configurable scheduler initialization -- Three main schedulers: CSV import, import processing, and session processing -- Environment-based configuration via `lib/env.ts` +- `server.ts` - Custom Next.js server with configurable scheduler initialization +- Three main schedulers: CSV import, import processing, and session processing +- Environment-based configuration via `lib/env.ts` **4. Key Processing Libraries** -- `lib/scheduler.ts` - CSV import scheduling -- `lib/importProcessor.ts` - Raw data to Session conversion -- `lib/processingScheduler.ts` - AI analysis pipeline -- `lib/transcriptFetcher.ts` - External transcript fetching -- `lib/transcriptParser.ts` - Message parsing from transcripts -- `lib/batchProcessor.ts` - OpenAI Batch API integration for cost-efficient processing -- `lib/batchScheduler.ts` - Automated batch job lifecycle management -- `lib/rateLimiter.ts` - In-memory rate limiting utility for API endpoints +- `lib/scheduler.ts` - CSV import scheduling +- `lib/importProcessor.ts` - Raw data to Session conversion +- `lib/processingScheduler.ts` - AI analysis pipeline +- `lib/transcriptFetcher.ts` - External transcript fetching +- `lib/transcriptParser.ts` - Message parsing from transcripts +- `lib/batchProcessor.ts` - OpenAI Batch API integration for cost-efficient processing +- `lib/batchScheduler.ts` - Automated batch job lifecycle management +- `lib/rateLimiter.ts` - In-memory rate limiting utility for API endpoints ### Development Environment **Environment Configuration:** Environment variables are managed through `lib/env.ts` with .env.local file support: -- Database: PostgreSQL via `DATABASE_URL` and `DATABASE_URL_DIRECT` -- Authentication: `NEXTAUTH_SECRET`, `NEXTAUTH_URL` -- AI Processing: `OPENAI_API_KEY` -- Schedulers: `SCHEDULER_ENABLED`, various interval configurations +- Database: PostgreSQL via `DATABASE_URL` and `DATABASE_URL_DIRECT` +- Authentication: `NEXTAUTH_SECRET`, `NEXTAUTH_URL` +- AI Processing: `OPENAI_API_KEY` +- Schedulers: `SCHEDULER_ENABLED`, various interval configurations **Key Files to Understand:** -- `prisma/schema.prisma` - Complete database schema with enums and relationships -- `server.ts` - Custom server entry point -- `lib/env.ts` - Environment variable management and validation -- `app/` - Next.js App Router structure +- `prisma/schema.prisma` - Complete database schema with enums and relationships +- `server.ts` - Custom server entry point +- `lib/env.ts` - Environment variable management and validation +- `app/` - Next.js App Router structure **Testing:** -- Uses Vitest for unit testing -- Playwright for E2E testing -- Test files in `tests/` directory +- Uses Vitest for unit testing +- Playwright for E2E testing +- Test files in `tests/` directory ### Important Notes **Scheduler System:** -- Schedulers are optional and controlled by `SCHEDULER_ENABLED` environment variable -- Use `pnpm dev:next-only` to run without schedulers for pure frontend development -- Four separate schedulers handle different pipeline stages: - - CSV Import Scheduler (`lib/scheduler.ts`) - - Import Processing Scheduler (`lib/importProcessor.ts`) - - Session Processing Scheduler (`lib/processingScheduler.ts`) - - Batch Processing Scheduler (`lib/batchScheduler.ts`) - Manages OpenAI Batch API lifecycle +- Schedulers are optional and controlled by `SCHEDULER_ENABLED` environment variable +- Use `pnpm dev:next-only` to run without schedulers for pure frontend development +- Four separate schedulers handle different pipeline stages: + - CSV Import Scheduler (`lib/scheduler.ts`) + - Import Processing Scheduler (`lib/importProcessor.ts`) + - Session Processing Scheduler (`lib/processingScheduler.ts`) + - Batch Processing Scheduler (`lib/batchScheduler.ts`) - Manages OpenAI Batch API lifecycle **Database Migrations:** -- Always run `pnpm prisma:generate` after schema changes -- Use `pnpm prisma:migrate` for production-ready migrations -- Use `pnpm prisma:push` for development schema changes -- Database uses PostgreSQL with Prisma's driver adapter for connection pooling +- Always run `pnpm prisma:generate` after schema changes +- Use `pnpm prisma:migrate` for production-ready migrations +- Use `pnpm prisma:push` for development schema changes +- Database uses PostgreSQL with Prisma's driver adapter for connection pooling **AI Processing:** -- All AI requests are tracked for cost analysis -- Support for multiple AI models per company -- Time-based pricing management for accurate cost calculation -- Processing stages can be retried on failure with retry count tracking -- **Batch API Integration**: 50% cost reduction using OpenAI Batch API - - Automatic batching of AI requests every 5 minutes - - Batch status checking every 2 minutes - - Result processing every minute - - Failed request retry with individual API calls +- All AI requests are tracked for cost analysis +- Support for multiple AI models per company +- Time-based pricing management for accurate cost calculation +- Processing stages can be retried on failure with retry count tracking +- **Batch API Integration**: 50% cost reduction using OpenAI Batch API + - Automatic batching of AI requests every 5 minutes + - Batch status checking every 2 minutes + - Result processing every minute + - Failed request retry with individual API calls **Code Quality Standards:** -- Run `pnpm lint` and `pnpm format:check` before committing -- TypeScript with ES modules (type: "module" in package.json) -- React 19 with Next.js 15 App Router -- TailwindCSS 4 for styling +- Run `pnpm lint` and `pnpm format:check` before committing +- TypeScript with ES modules (type: "module" in package.json) +- React 19 with Next.js 15 App Router +- TailwindCSS 4 for styling **Security Features:** -- **Comprehensive CSRF Protection**: Multi-layer CSRF protection with automatic token management - - Middleware-level protection for all state-changing endpoints - - tRPC integration with CSRF-protected procedures - - Client-side hooks and components for seamless integration - - HTTP-only cookies with SameSite protection -- **Enhanced Content Security Policy (CSP)**: - - Nonce-based script execution for maximum XSS protection - - Environment-specific policies (strict production, permissive development) - - Real-time violation reporting and bypass detection - - Automated policy optimization recommendations -- **Security Monitoring & Audit System**: - - Real-time threat detection and alerting - - Comprehensive security audit logging with retention management - - Geographic anomaly detection and IP threat analysis - - Security scoring and automated incident response -- **Advanced Rate Limiting**: In-memory rate limiting system - - Authentication endpoints: Login (5/15min), Registration (3/hour), Password Reset (5/15min) - - CSP reporting: 10 reports per minute per IP - - Admin endpoints: Configurable thresholds -- **Input Validation & Security Headers**: - - Comprehensive Zod schemas for all user inputs with XSS/injection prevention - - HTTP security headers (HSTS, X-Frame-Options, X-Content-Type-Options, Permissions Policy) - - Strong password requirements and email validation -- **Session Security**: - - JWT tokens with 24-hour expiration and secure cookie settings - - HttpOnly, Secure, SameSite cookies with proper CSP integration - - Company isolation and multi-tenant security +- **Comprehensive CSRF Protection**: Multi-layer CSRF protection with automatic token management + - Middleware-level protection for all state-changing endpoints + - tRPC integration with CSRF-protected procedures + - Client-side hooks and components for seamless integration + - HTTP-only cookies with SameSite protection +- **Enhanced Content Security Policy (CSP)**: + - Nonce-based script execution for maximum XSS protection + - Environment-specific policies (strict production, permissive development) + - Real-time violation reporting and bypass detection + - Automated policy optimization recommendations +- **Security Monitoring & Audit System**: + - Real-time threat detection and alerting + - Comprehensive security audit logging with retention management + - Geographic anomaly detection and IP threat analysis + - Security scoring and automated incident response +- **Advanced Rate Limiting**: In-memory rate limiting system + - Authentication endpoints: Login (5/15min), Registration (3/hour), Password Reset (5/15min) + - CSP reporting: 10 reports per minute per IP + - Admin endpoints: Configurable thresholds +- **Input Validation & Security Headers**: + - Comprehensive Zod schemas for all user inputs with XSS/injection prevention + - HTTP security headers (HSTS, X-Frame-Options, X-Content-Type-Options, Permissions Policy) + - Strong password requirements and email validation +- **Session Security**: + - JWT tokens with 24-hour expiration and secure cookie settings + - HttpOnly, Secure, SameSite cookies with proper CSP integration + - Company isolation and multi-tenant security diff --git a/DOCUMENTATION_AUDIT_SUMMARY.md b/DOCUMENTATION_AUDIT_SUMMARY.md index 20c7bb5..3758294 100644 --- a/DOCUMENTATION_AUDIT_SUMMARY.md +++ b/DOCUMENTATION_AUDIT_SUMMARY.md @@ -10,33 +10,33 @@ This document summarizes the comprehensive documentation audit performed on the The following areas were found to have comprehensive, accurate documentation: -1. **CSRF Protection** (`docs/CSRF_PROTECTION.md`) - - Multi-layer protection implementation - - Client-side integration guide - - tRPC integration details - - Comprehensive examples +1. **CSRF Protection** (`docs/CSRF_PROTECTION.md`) + - Multi-layer protection implementation + - Client-side integration guide + - tRPC integration details + - Comprehensive examples -2. **Enhanced CSP Implementation** (`docs/security/enhanced-csp.md`) - - Nonce-based script execution - - Environment-specific policies - - Violation reporting and monitoring - - Testing framework +2. **Enhanced CSP Implementation** (`docs/security/enhanced-csp.md`) + - Nonce-based script execution + - Environment-specific policies + - Violation reporting and monitoring + - Testing framework -3. **Security Headers** (`docs/security-headers.md`) - - Complete header implementation details - - Testing procedures - - Compatibility information +3. **Security Headers** (`docs/security-headers.md`) + - Complete header implementation details + - Testing procedures + - Compatibility information -4. **Security Monitoring System** (`docs/security-monitoring.md`) - - Real-time threat detection - - Alert management - - API usage examples - - Performance considerations +4. **Security Monitoring System** (`docs/security-monitoring.md`) + - Real-time threat detection + - Alert management + - API usage examples + - Performance considerations -5. **Migration Guide** (`MIGRATION_GUIDE.md`) - - Comprehensive v2.0.0 migration procedures - - Rollback procedures - - Health checks and validation +5. **Migration Guide** (`MIGRATION_GUIDE.md`) + - Comprehensive v2.0.0 migration procedures + - Rollback procedures + - Health checks and validation ### Major Issues Identified ❌ @@ -44,70 +44,70 @@ The following areas were found to have comprehensive, accurate documentation: **Problems Found:** -- Listed database as "SQLite (default)" when project uses PostgreSQL -- Missing all new security features (CSRF, CSP, security monitoring) -- Incomplete environment setup section -- Outdated tech stack (missing tRPC, security features) -- Project structure didn't reflect new admin/security directories +- Listed database as "SQLite (default)" when project uses PostgreSQL +- Missing all new security features (CSRF, CSP, security monitoring) +- Incomplete environment setup section +- Outdated tech stack (missing tRPC, security features) +- Project structure didn't reflect new admin/security directories **Actions Taken:** -- ✅ Updated features section to include security and admin capabilities -- ✅ Corrected tech stack to include PostgreSQL, tRPC, security features -- ✅ Updated environment setup with proper PostgreSQL configuration -- ✅ Revised project structure to reflect current codebase -- ✅ Added comprehensive script documentation +- ✅ Updated features section to include security and admin capabilities +- ✅ Corrected tech stack to include PostgreSQL, tRPC, security features +- ✅ Updated environment setup with proper PostgreSQL configuration +- ✅ Revised project structure to reflect current codebase +- ✅ Added comprehensive script documentation #### 2. Undocumented API Endpoints **Missing Documentation:** -- `/api/admin/audit-logs/` (GET) - Audit log retrieval with filtering -- `/api/admin/audit-logs/retention/` (POST) - Retention management -- `/api/admin/security-monitoring/` (GET/POST) - Security metrics and config -- `/api/admin/security-monitoring/alerts/` - Alert management -- `/api/admin/security-monitoring/export/` - Data export -- `/api/admin/security-monitoring/threat-analysis/` - Threat analysis -- `/api/admin/batch-monitoring/` - Batch processing monitoring -- `/api/csp-report/` (POST) - CSP violation reporting -- `/api/csp-metrics/` (GET) - CSP metrics and analytics -- `/api/csrf-token/` (GET) - CSRF token endpoint +- `/api/admin/audit-logs/` (GET) - Audit log retrieval with filtering +- `/api/admin/audit-logs/retention/` (POST) - Retention management +- `/api/admin/security-monitoring/` (GET/POST) - Security metrics and config +- `/api/admin/security-monitoring/alerts/` - Alert management +- `/api/admin/security-monitoring/export/` - Data export +- `/api/admin/security-monitoring/threat-analysis/` - Threat analysis +- `/api/admin/batch-monitoring/` - Batch processing monitoring +- `/api/csp-report/` (POST) - CSP violation reporting +- `/api/csp-metrics/` (GET) - CSP metrics and analytics +- `/api/csrf-token/` (GET) - CSRF token endpoint **Actions Taken:** -- ✅ Created `docs/admin-audit-logs-api.md` - Comprehensive audit logs API documentation -- ✅ Created `docs/csp-metrics-api.md` - CSP monitoring and metrics API documentation -- ✅ Created `docs/api-reference.md` - Complete API reference for all endpoints +- ✅ Created `docs/admin-audit-logs-api.md` - Comprehensive audit logs API documentation +- ✅ Created `docs/csp-metrics-api.md` - CSP monitoring and metrics API documentation +- ✅ Created `docs/api-reference.md` - Complete API reference for all endpoints #### 3. Undocumented Features and Components **Missing Feature Documentation:** -- Batch monitoring dashboard and UI components -- Security monitoring UI components -- Nonce-based CSP context provider -- Enhanced rate limiting system -- Security audit retention system +- Batch monitoring dashboard and UI components +- Security monitoring UI components +- Nonce-based CSP context provider +- Enhanced rate limiting system +- Security audit retention system **Actions Taken:** -- ✅ Created `docs/batch-monitoring-dashboard.md` - Complete batch monitoring documentation +- ✅ Created `docs/batch-monitoring-dashboard.md` - Complete batch monitoring documentation #### 4. CLAUDE.md - Missing New Commands **Problems Found:** -- Missing security testing commands -- Missing CSP testing commands -- Missing migration/deployment commands -- Outdated security features section +- Missing security testing commands +- Missing CSP testing commands +- Missing migration/deployment commands +- Outdated security features section **Actions Taken:** -- ✅ Added security testing command section -- ✅ Added CSP testing commands -- ✅ Added migration and deployment commands -- ✅ Updated security features section with comprehensive details +- ✅ Added security testing command section +- ✅ Added CSP testing commands +- ✅ Added migration and deployment commands +- ✅ Updated security features section with comprehensive details ## New Documentation Created @@ -117,14 +117,14 @@ The following areas were found to have comprehensive, accurate documentation: **Contents:** -- Complete API endpoint documentation with examples -- Authentication and authorization requirements -- Query parameters and filtering options -- Response formats and error handling -- Retention management procedures -- Security features and rate limiting -- Usage examples and integration patterns -- Performance considerations and troubleshooting +- Complete API endpoint documentation with examples +- Authentication and authorization requirements +- Query parameters and filtering options +- Response formats and error handling +- Retention management procedures +- Security features and rate limiting +- Usage examples and integration patterns +- Performance considerations and troubleshooting ### 2. CSP Metrics and Monitoring API Documentation @@ -132,14 +132,14 @@ The following areas were found to have comprehensive, accurate documentation: **Contents:** -- CSP violation reporting endpoint documentation -- Metrics API with real-time violation tracking -- Risk assessment and bypass detection features -- Policy optimization recommendations -- Configuration and setup instructions -- Performance considerations and security features -- Usage examples for monitoring and analysis -- Integration with existing security systems +- CSP violation reporting endpoint documentation +- Metrics API with real-time violation tracking +- Risk assessment and bypass detection features +- Policy optimization recommendations +- Configuration and setup instructions +- Performance considerations and security features +- Usage examples for monitoring and analysis +- Integration with existing security systems ### 3. Batch Monitoring Dashboard Documentation @@ -147,14 +147,14 @@ The following areas were found to have comprehensive, accurate documentation: **Contents:** -- Comprehensive batch processing monitoring guide -- Real-time monitoring capabilities and features -- API endpoints for batch job tracking -- Dashboard component documentation -- Performance analytics and cost analysis -- Administrative controls and error handling -- Configuration and alert management -- Troubleshooting and optimization guides +- Comprehensive batch processing monitoring guide +- Real-time monitoring capabilities and features +- API endpoints for batch job tracking +- Dashboard component documentation +- Performance analytics and cost analysis +- Administrative controls and error handling +- Configuration and alert management +- Troubleshooting and optimization guides ### 4. Complete API Reference @@ -162,14 +162,14 @@ The following areas were found to have comprehensive, accurate documentation: **Contents:** -- Comprehensive reference for all API endpoints -- Authentication and CSRF protection requirements -- Detailed request/response formats -- Error codes and status descriptions -- Rate limiting information -- Security headers and CORS configuration -- Pagination and filtering standards -- Testing and integration examples +- Comprehensive reference for all API endpoints +- Authentication and CSRF protection requirements +- Detailed request/response formats +- Error codes and status descriptions +- Rate limiting information +- Security headers and CORS configuration +- Pagination and filtering standards +- Testing and integration examples ## Updated Documentation @@ -177,23 +177,23 @@ The following areas were found to have comprehensive, accurate documentation: **Key Updates:** -- ✅ Updated project description to include security and admin features -- ✅ Corrected tech stack to reflect current implementation -- ✅ Fixed database information (PostgreSQL vs SQLite) -- ✅ Added comprehensive environment configuration -- ✅ Updated project structure to match current codebase -- ✅ Added security, migration, and testing command sections -- ✅ Enhanced features section with detailed capabilities +- ✅ Updated project description to include security and admin features +- ✅ Corrected tech stack to reflect current implementation +- ✅ Fixed database information (PostgreSQL vs SQLite) +- ✅ Added comprehensive environment configuration +- ✅ Updated project structure to match current codebase +- ✅ Added security, migration, and testing command sections +- ✅ Enhanced features section with detailed capabilities ### 2. CLAUDE.md - Enhanced Developer Guide **Key Updates:** -- ✅ Added security testing commands section -- ✅ Added CSP testing and validation commands -- ✅ Added migration and deployment commands -- ✅ Enhanced security features documentation -- ✅ Updated with comprehensive CSRF, CSP, and monitoring details +- ✅ Added security testing commands section +- ✅ Added CSP testing and validation commands +- ✅ Added migration and deployment commands +- ✅ Enhanced security features documentation +- ✅ Updated with comprehensive CSRF, CSP, and monitoring details ## Documentation Quality Assessment @@ -212,53 +212,53 @@ The following areas were found to have comprehensive, accurate documentation: All new and updated documentation follows these standards: -- ✅ Clear, actionable examples -- ✅ Comprehensive API documentation with request/response examples -- ✅ Security considerations and best practices -- ✅ Troubleshooting sections -- ✅ Integration patterns and usage examples -- ✅ Performance considerations -- ✅ Cross-references to related documentation +- ✅ Clear, actionable examples +- ✅ Comprehensive API documentation with request/response examples +- ✅ Security considerations and best practices +- ✅ Troubleshooting sections +- ✅ Integration patterns and usage examples +- ✅ Performance considerations +- ✅ Cross-references to related documentation ## Recommendations for Maintenance ### 1. Regular Review Schedule -- **Monthly**: Review API documentation for new endpoints -- **Quarterly**: Update security feature documentation -- **Per Release**: Validate all examples and code snippets -- **Annually**: Comprehensive documentation audit +- **Monthly**: Review API documentation for new endpoints +- **Quarterly**: Update security feature documentation +- **Per Release**: Validate all examples and code snippets +- **Annually**: Comprehensive documentation audit ### 2. Documentation Automation -- Add documentation checks to CI/CD pipeline -- Implement API documentation generation from OpenAPI specs -- Set up automated link checking -- Create documentation review templates +- Add documentation checks to CI/CD pipeline +- Implement API documentation generation from OpenAPI specs +- Set up automated link checking +- Create documentation review templates ### 3. Developer Onboarding -- Use updated documentation for new developer onboarding -- Create documentation feedback process -- Maintain documentation contribution guidelines -- Track documentation usage and feedback +- Use updated documentation for new developer onboarding +- Create documentation feedback process +- Maintain documentation contribution guidelines +- Track documentation usage and feedback ### 4. Continuous Improvement -- Monitor documentation gaps through developer feedback -- Update examples with real-world usage patterns -- Enhance troubleshooting sections based on support issues -- Keep security documentation current with threat landscape +- Monitor documentation gaps through developer feedback +- Update examples with real-world usage patterns +- Enhance troubleshooting sections based on support issues +- Keep security documentation current with threat landscape ## Summary The documentation audit identified significant gaps in API documentation, outdated project information, and missing coverage of new security features. Through comprehensive updates and new documentation creation, the project now has: -- **Complete API Reference**: All endpoints documented with examples -- **Accurate Project Information**: README and CLAUDE.md reflect current state -- **Comprehensive Security Documentation**: All security features thoroughly documented -- **Developer-Friendly Guides**: Clear setup, testing, and deployment procedures -- **Administrative Documentation**: Complete coverage of admin and monitoring features +- **Complete API Reference**: All endpoints documented with examples +- **Accurate Project Information**: README and CLAUDE.md reflect current state +- **Comprehensive Security Documentation**: All security features thoroughly documented +- **Developer-Friendly Guides**: Clear setup, testing, and deployment procedures +- **Administrative Documentation**: Complete coverage of admin and monitoring features The documentation is now production-ready and provides comprehensive guidance for developers, administrators, and security teams working with the LiveDash-Node application. @@ -266,15 +266,15 @@ The documentation is now production-ready and provides comprehensive guidance fo ### Modified Files -1. `README.md` - Complete overhaul with accurate project information -2. `CLAUDE.md` - Enhanced with security testing and migration commands +1. `README.md` - Complete overhaul with accurate project information +2. `CLAUDE.md` - Enhanced with security testing and migration commands ### New Documentation Files -1. `docs/admin-audit-logs-api.md` - Admin audit logs API documentation -2. `docs/csp-metrics-api.md` - CSP monitoring and metrics API documentation -3. `docs/batch-monitoring-dashboard.md` - Batch monitoring dashboard documentation -4. `docs/api-reference.md` - Comprehensive API reference -5. `DOCUMENTATION_AUDIT_SUMMARY.md` - This summary document +1. `docs/admin-audit-logs-api.md` - Admin audit logs API documentation +2. `docs/csp-metrics-api.md` - CSP monitoring and metrics API documentation +3. `docs/batch-monitoring-dashboard.md` - Batch monitoring dashboard documentation +4. `docs/api-reference.md` - Comprehensive API reference +5. `DOCUMENTATION_AUDIT_SUMMARY.md` - This summary document All documentation is now current, comprehensive, and ready for production use. diff --git a/FIXES-APPLIED.md b/FIXES-APPLIED.md index 543d6b2..eb1c6a6 100644 --- a/FIXES-APPLIED.md +++ b/FIXES-APPLIED.md @@ -11,38 +11,38 @@ Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.ne ## Root Causes -1. **Multiple PrismaClient instances** across schedulers -2. **No connection retry logic** for temporary failures -3. **No connection pooling optimization** for Neon -4. **Aggressive scheduler intervals** overwhelming database +1. **Multiple PrismaClient instances** across schedulers +2. **No connection retry logic** for temporary failures +3. **No connection pooling optimization** for Neon +4. **Aggressive scheduler intervals** overwhelming database ## Fixes Applied ✅ ### 1. Connection Retry Logic (`lib/database-retry.ts`) -- **Automatic retry** for connection errors -- **Exponential backoff** (1s → 2s → 4s → 10s max) -- **Smart error detection** (only retry connection issues) -- **Configurable retry attempts** (default: 3 retries) +- **Automatic retry** for connection errors +- **Exponential backoff** (1s → 2s → 4s → 10s max) +- **Smart error detection** (only retry connection issues) +- **Configurable retry attempts** (default: 3 retries) ### 2. Enhanced Schedulers -- **Import Processor**: Added retry wrapper around main processing -- **Session Processor**: Added retry wrapper around AI processing -- **Graceful degradation** when database is temporarily unavailable +- **Import Processor**: Added retry wrapper around main processing +- **Session Processor**: Added retry wrapper around AI processing +- **Graceful degradation** when database is temporarily unavailable ### 3. Singleton Pattern Enforced -- **All schedulers now use** `import { prisma } from "./prisma.js"` -- **No more separate** `new PrismaClient()` instances -- **Shared connection pool** across all operations +- **All schedulers now use** `import { prisma } from "./prisma.js"` +- **No more separate** `new PrismaClient()` instances +- **Shared connection pool** across all operations ### 4. Neon-Specific Optimizations -- **Connection limit guidance**: 15 connections (below Neon's 20 limit) -- **Extended timeouts**: 30s for cold start handling -- **SSL mode requirements**: `sslmode=require` for Neon -- **Application naming**: For better monitoring +- **Connection limit guidance**: 15 connections (below Neon's 20 limit) +- **Extended timeouts**: 30s for cold start handling +- **SSL mode requirements**: `sslmode=require` for Neon +- **Application naming**: For better monitoring ## Immediate Actions Needed @@ -83,17 +83,17 @@ pnpm db:check ## Monitoring -- **Health Endpoint**: `/api/admin/database-health` -- **Connection Logs**: Enhanced logging for pool events -- **Retry Logs**: Detailed retry attempt logging -- **Error Classification**: Retryable vs non-retryable errors +- **Health Endpoint**: `/api/admin/database-health` +- **Connection Logs**: Enhanced logging for pool events +- **Retry Logs**: Detailed retry attempt logging +- **Error Classification**: Retryable vs non-retryable errors ## Files Modified -- `lib/database-retry.ts` - New retry utilities -- `lib/importProcessor.ts` - Added retry wrapper -- `lib/processingScheduler.ts` - Added retry wrapper -- `docs/neon-database-optimization.md` - Neon-specific guide -- `scripts/check-database-config.ts` - Configuration checker +- `lib/database-retry.ts` - New retry utilities +- `lib/importProcessor.ts` - Added retry wrapper +- `lib/processingScheduler.ts` - Added retry wrapper +- `docs/neon-database-optimization.md` - Neon-specific guide +- `scripts/check-database-config.ts` - Configuration checker The connection issues should be significantly reduced with these fixes! 🎯 diff --git a/MIGRATION_GUIDE.md b/MIGRATION_GUIDE.md index 0343b6a..58b16ad 100644 --- a/MIGRATION_GUIDE.md +++ b/MIGRATION_GUIDE.md @@ -8,43 +8,43 @@ This guide provides step-by-step instructions for migrating LiveDash Node to ver ### tRPC Implementation -- **Type-safe APIs**: End-to-end TypeScript safety from client to server -- **Improved Performance**: Optimized query batching and caching -- **Better Developer Experience**: Auto-completion and type checking -- **Simplified Authentication**: Integrated with existing NextAuth.js setup +- **Type-safe APIs**: End-to-end TypeScript safety from client to server +- **Improved Performance**: Optimized query batching and caching +- **Better Developer Experience**: Auto-completion and type checking +- **Simplified Authentication**: Integrated with existing NextAuth.js setup ### OpenAI Batch API Integration -- **50% Cost Reduction**: Batch processing reduces OpenAI API costs by half -- **Enhanced Rate Limiting**: Better throughput management -- **Improved Reliability**: Automatic retry mechanisms and error handling -- **Automated Processing**: Background batch job lifecycle management +- **50% Cost Reduction**: Batch processing reduces OpenAI API costs by half +- **Enhanced Rate Limiting**: Better throughput management +- **Improved Reliability**: Automatic retry mechanisms and error handling +- **Automated Processing**: Background batch job lifecycle management ### Enhanced Security & Performance -- **Rate Limiting**: In-memory rate limiting for all authentication endpoints -- **Input Validation**: Comprehensive Zod schemas for all user inputs -- **Performance Monitoring**: Built-in metrics collection and monitoring -- **Database Optimizations**: New indexes and query optimizations +- **Rate Limiting**: In-memory rate limiting for all authentication endpoints +- **Input Validation**: Comprehensive Zod schemas for all user inputs +- **Performance Monitoring**: Built-in metrics collection and monitoring +- **Database Optimizations**: New indexes and query optimizations ## 📋 Pre-Migration Checklist ### System Requirements -- [ ] Node.js 18+ installed -- [ ] PostgreSQL 13+ database -- [ ] `pg_dump` and `pg_restore` utilities available -- [ ] Git repository with clean working directory -- [ ] OpenAI API key (for production) -- [ ] Sufficient disk space for backups (at least 2GB) +- [ ] Node.js 18+ installed +- [ ] PostgreSQL 13+ database +- [ ] `pg_dump` and `pg_restore` utilities available +- [ ] Git repository with clean working directory +- [ ] OpenAI API key (for production) +- [ ] Sufficient disk space for backups (at least 2GB) ### Environment Preparation -- [ ] Review current environment variables -- [ ] Ensure database connection is working -- [ ] Verify all tests are passing -- [ ] Create a backup of your current deployment -- [ ] Notify team members of planned downtime +- [ ] Review current environment variables +- [ ] Ensure database connection is working +- [ ] Verify all tests are passing +- [ ] Create a backup of your current deployment +- [ ] Notify team members of planned downtime ## 🔧 Migration Process @@ -211,11 +211,11 @@ pnpm migration:rollback:dry-run ### Manual Rollback Steps -1. **Stop the application** -2. **Restore database from backup** -3. **Revert to previous code version** -4. **Restart services** -5. **Verify system functionality** +1. **Stop the application** +2. **Restore database from backup** +3. **Revert to previous code version** +4. **Restart services** +5. **Verify system functionality** ### Rollback Commands @@ -246,31 +246,31 @@ tail -f logs/migration.log #### 2. tRPC Performance -- Monitor response times for tRPC endpoints -- Check error rates in application logs -- Verify type safety is working correctly +- Monitor response times for tRPC endpoints +- Check error rates in application logs +- Verify type safety is working correctly #### 3. Batch Processing -- Monitor batch job completion rates -- Check OpenAI API cost reduction -- Verify AI processing pipeline functionality +- Monitor batch job completion rates +- Check OpenAI API cost reduction +- Verify AI processing pipeline functionality ### Key Metrics to Monitor #### Performance Metrics -- **Response Times**: tRPC endpoints should respond within 500ms -- **Database Queries**: Complex queries should complete within 1s -- **Memory Usage**: Should remain below 80% of allocated memory -- **CPU Usage**: Process should remain responsive +- **Response Times**: tRPC endpoints should respond within 500ms +- **Database Queries**: Complex queries should complete within 1s +- **Memory Usage**: Should remain below 80% of allocated memory +- **CPU Usage**: Process should remain responsive #### Business Metrics -- **AI Processing Cost**: Should see ~50% reduction in OpenAI costs -- **Processing Throughput**: Batch processing should handle larger volumes -- **Error Rates**: Should remain below 1% for critical operations -- **User Experience**: No degradation in dashboard performance +- **AI Processing Cost**: Should see ~50% reduction in OpenAI costs +- **Processing Throughput**: Batch processing should handle larger volumes +- **Error Rates**: Should remain below 1% for critical operations +- **User Experience**: No degradation in dashboard performance ## 🛠 Troubleshooting @@ -341,10 +341,10 @@ node -e "require('dotenv').config({path: '.env.local'}); console.log('✅ Enviro #### Support Channels -1. **Check Migration Logs**: Review `logs/migration.log` for detailed error information -2. **Run Diagnostics**: Use the built-in health check and validation tools -3. **Documentation**: Refer to component-specific documentation in `docs/` -4. **Emergency Rollback**: Use rollback procedures if issues persist +1. **Check Migration Logs**: Review `logs/migration.log` for detailed error information +2. **Run Diagnostics**: Use the built-in health check and validation tools +3. **Documentation**: Refer to component-specific documentation in `docs/` +4. **Emergency Rollback**: Use rollback procedures if issues persist #### Useful Commands @@ -366,69 +366,69 @@ pnpm prisma db pull --print ### Immediate Tasks (First 24 Hours) -- [ ] Monitor application logs for errors -- [ ] Verify all tRPC endpoints are responding correctly -- [ ] Check batch processing job completion -- [ ] Validate AI cost reduction in OpenAI dashboard -- [ ] Run full test suite to ensure no regressions -- [ ] Update documentation and team knowledge +- [ ] Monitor application logs for errors +- [ ] Verify all tRPC endpoints are responding correctly +- [ ] Check batch processing job completion +- [ ] Validate AI cost reduction in OpenAI dashboard +- [ ] Run full test suite to ensure no regressions +- [ ] Update documentation and team knowledge ### Medium-term Tasks (First Week) -- [ ] Optimize batch processing parameters based on usage -- [ ] Fine-tune rate limiting settings -- [ ] Set up monitoring alerts for new components -- [ ] Train team on new tRPC APIs -- [ ] Plan gradual feature adoption +- [ ] Optimize batch processing parameters based on usage +- [ ] Fine-tune rate limiting settings +- [ ] Set up monitoring alerts for new components +- [ ] Train team on new tRPC APIs +- [ ] Plan gradual feature adoption ### Long-term Tasks (First Month) -- [ ] Analyze cost savings and performance improvements -- [ ] Consider additional tRPC endpoint implementations -- [ ] Optimize batch processing schedules -- [ ] Review and adjust security settings -- [ ] Plan next phase improvements +- [ ] Analyze cost savings and performance improvements +- [ ] Consider additional tRPC endpoint implementations +- [ ] Optimize batch processing schedules +- [ ] Review and adjust security settings +- [ ] Plan next phase improvements ## 🔒 Security Considerations ### New Security Features -- **Enhanced Rate Limiting**: Applied to all authentication endpoints -- **Input Validation**: Comprehensive Zod schemas prevent injection attacks -- **Secure Headers**: HTTPS enforcement in production -- **Token Security**: JWT with proper expiration and rotation +- **Enhanced Rate Limiting**: Applied to all authentication endpoints +- **Input Validation**: Comprehensive Zod schemas prevent injection attacks +- **Secure Headers**: HTTPS enforcement in production +- **Token Security**: JWT with proper expiration and rotation ### Security Checklist -- [ ] Verify rate limiting is working correctly -- [ ] Test input validation on all forms -- [ ] Ensure HTTPS is enforced in production -- [ ] Validate JWT token handling -- [ ] Check for proper error message sanitization -- [ ] Verify OpenAI API key is not exposed in logs +- [ ] Verify rate limiting is working correctly +- [ ] Test input validation on all forms +- [ ] Ensure HTTPS is enforced in production +- [ ] Validate JWT token handling +- [ ] Check for proper error message sanitization +- [ ] Verify OpenAI API key is not exposed in logs ## 📈 Expected Improvements ### Performance Improvements -- **50% reduction** in OpenAI API costs through batch processing -- **30% improvement** in API response times with tRPC -- **25% reduction** in database query time with new indexes -- **Enhanced scalability** for processing larger session volumes +- **50% reduction** in OpenAI API costs through batch processing +- **30% improvement** in API response times with tRPC +- **25% reduction** in database query time with new indexes +- **Enhanced scalability** for processing larger session volumes ### Developer Experience -- **Type Safety**: End-to-end TypeScript types from client to server -- **Better APIs**: Self-documenting tRPC procedures -- **Improved Testing**: More reliable test suite with better validation -- **Enhanced Monitoring**: Detailed health checks and reporting +- **Type Safety**: End-to-end TypeScript types from client to server +- **Better APIs**: Self-documenting tRPC procedures +- **Improved Testing**: More reliable test suite with better validation +- **Enhanced Monitoring**: Detailed health checks and reporting ### Operational Benefits -- **Automated Batch Processing**: Reduced manual intervention -- **Better Error Handling**: Comprehensive retry mechanisms -- **Improved Monitoring**: Real-time health status and metrics -- **Simplified Deployment**: Automated migration and rollback procedures +- **Automated Batch Processing**: Reduced manual intervention +- **Better Error Handling**: Comprehensive retry mechanisms +- **Improved Monitoring**: Real-time health status and metrics +- **Simplified Deployment**: Automated migration and rollback procedures --- @@ -436,10 +436,10 @@ pnpm prisma db pull --print For issues during migration: -1. **Check the logs**: `logs/migration.log` -2. **Run health checks**: `pnpm migration:health-check` -3. **Review troubleshooting section** above -4. **Use rollback if needed**: `pnpm migration:rollback` +1. **Check the logs**: `logs/migration.log` +2. **Run health checks**: `pnpm migration:health-check` +3. **Review troubleshooting section** above +4. **Use rollback if needed**: `pnpm migration:rollback` **Migration completed successfully? 🎉** diff --git a/README.md b/README.md index 9f76639..610ed8c 100644 --- a/README.md +++ b/README.md @@ -12,46 +12,46 @@ A comprehensive real-time analytics dashboard for monitoring user sessions with ### Core Analytics -- **Real-time Session Monitoring**: Track and analyze user sessions as they happen -- **Interactive Visualizations**: Geographic maps, response time distributions, and advanced charts -- **AI-Powered Analysis**: OpenAI integration with 50% cost reduction through batch processing -- **Advanced Analytics**: Detailed metrics and insights about user behavior patterns -- **Session Details**: In-depth analysis of individual user sessions with transcript parsing +- **Real-time Session Monitoring**: Track and analyze user sessions as they happen +- **Interactive Visualizations**: Geographic maps, response time distributions, and advanced charts +- **AI-Powered Analysis**: OpenAI integration with 50% cost reduction through batch processing +- **Advanced Analytics**: Detailed metrics and insights about user behavior patterns +- **Session Details**: In-depth analysis of individual user sessions with transcript parsing ### Security & Admin Features -- **Enterprise Security**: Multi-layer security with CSRF protection, CSP, and rate limiting -- **Security Monitoring**: Real-time threat detection and alerting system -- **Audit Logging**: Comprehensive security audit trails with retention management -- **Admin Dashboard**: Advanced administration tools for user and system management -- **Geographic Threat Detection**: IP-based threat analysis and anomaly detection +- **Enterprise Security**: Multi-layer security with CSRF protection, CSP, and rate limiting +- **Security Monitoring**: Real-time threat detection and alerting system +- **Audit Logging**: Comprehensive security audit trails with retention management +- **Admin Dashboard**: Advanced administration tools for user and system management +- **Geographic Threat Detection**: IP-based threat analysis and anomaly detection ### Platform Management -- **Multi-tenant Architecture**: Company-based data isolation and management -- **User Management**: Role-based access control with platform admin capabilities -- **Batch Processing**: Optimized AI processing pipeline with automated scheduling -- **Data Export**: CSV/JSON export capabilities for analytics and audit data +- **Multi-tenant Architecture**: Company-based data isolation and management +- **User Management**: Role-based access control with platform admin capabilities +- **Batch Processing**: Optimized AI processing pipeline with automated scheduling +- **Data Export**: CSV/JSON export capabilities for analytics and audit data ## Tech Stack -- **Frontend**: React 19, Next.js 15, TailwindCSS 4 -- **Backend**: Next.js API Routes, tRPC, Custom Node.js server -- **Database**: PostgreSQL with Prisma ORM and connection pooling -- **Authentication**: NextAuth.js with enhanced security features -- **Security**: CSRF protection, CSP with nonce-based scripts, comprehensive rate limiting -- **AI Processing**: OpenAI API with batch processing for cost optimization -- **Visualization**: D3.js, React Leaflet, Recharts, custom chart components -- **Monitoring**: Real-time security monitoring, audit logging, threat detection -- **Data Processing**: Node-cron schedulers for automated batch processing and AI analysis +- **Frontend**: React 19, Next.js 15, TailwindCSS 4 +- **Backend**: Next.js API Routes, tRPC, Custom Node.js server +- **Database**: PostgreSQL with Prisma ORM and connection pooling +- **Authentication**: NextAuth.js with enhanced security features +- **Security**: CSRF protection, CSP with nonce-based scripts, comprehensive rate limiting +- **AI Processing**: OpenAI API with batch processing for cost optimization +- **Visualization**: D3.js, React Leaflet, Recharts, custom chart components +- **Monitoring**: Real-time security monitoring, audit logging, threat detection +- **Data Processing**: Node-cron schedulers for automated batch processing and AI analysis ## Getting Started ### Prerequisites -- Node.js 18+ (LTS version recommended) -- pnpm (recommended package manager) -- PostgreSQL 13+ database +- Node.js 18+ (LTS version recommended) +- pnpm (recommended package manager) +- PostgreSQL 13+ database ### Installation @@ -126,61 +126,61 @@ BATCH_RESULT_PROCESSING_INTERVAL="*/1 * * * *" ## Project Structure -- `app/`: Next.js App Router pages and API routes - - `api/`: API endpoints including admin, security, and tRPC routes - - `dashboard/`: Main analytics dashboard pages - - `platform/`: Platform administration interface -- `components/`: Reusable React components - - `admin/`: Administrative dashboard components - - `security/`: Security monitoring UI components - - `forms/`: CSRF-protected forms and form utilities - - `providers/`: Context providers (CSRF, tRPC, themes) -- `lib/`: Core utilities and business logic - - Security modules (CSRF, CSP, rate limiting, audit logging) - - Processing pipelines (batch processing, AI analysis) - - Database utilities and authentication -- `server/`: tRPC server configuration and routers -- `prisma/`: Database schema, migrations, and seed scripts -- `tests/`: Comprehensive test suite (unit, integration, E2E) -- `docs/`: Detailed project documentation -- `scripts/`: Migration and utility scripts +- `app/`: Next.js App Router pages and API routes + - `api/`: API endpoints including admin, security, and tRPC routes + - `dashboard/`: Main analytics dashboard pages + - `platform/`: Platform administration interface +- `components/`: Reusable React components + - `admin/`: Administrative dashboard components + - `security/`: Security monitoring UI components + - `forms/`: CSRF-protected forms and form utilities + - `providers/`: Context providers (CSRF, tRPC, themes) +- `lib/`: Core utilities and business logic + - Security modules (CSRF, CSP, rate limiting, audit logging) + - Processing pipelines (batch processing, AI analysis) + - Database utilities and authentication +- `server/`: tRPC server configuration and routers +- `prisma/`: Database schema, migrations, and seed scripts +- `tests/`: Comprehensive test suite (unit, integration, E2E) +- `docs/`: Detailed project documentation +- `scripts/`: Migration and utility scripts ## Available Scripts ### Development -- `pnpm dev`: Start development server with all features -- `pnpm dev:next-only`: Start Next.js only (no background schedulers) -- `pnpm build`: Build the application for production -- `pnpm start`: Run the production build +- `pnpm dev`: Start development server with all features +- `pnpm dev:next-only`: Start Next.js only (no background schedulers) +- `pnpm build`: Build the application for production +- `pnpm start`: Run the production build ### Code Quality -- `pnpm lint`: Run ESLint -- `pnpm lint:fix`: Fix ESLint issues automatically -- `pnpm format`: Format code with Prettier -- `pnpm format:check`: Check code formatting +- `pnpm lint`: Run ESLint +- `pnpm lint:fix`: Fix ESLint issues automatically +- `pnpm format`: Format code with Prettier +- `pnpm format:check`: Check code formatting ### Database -- `pnpm prisma:studio`: Open Prisma Studio to view database -- `pnpm prisma:migrate`: Run database migrations -- `pnpm prisma:generate`: Generate Prisma client -- `pnpm prisma:seed`: Seed database with test data +- `pnpm prisma:studio`: Open Prisma Studio to view database +- `pnpm prisma:migrate`: Run database migrations +- `pnpm prisma:generate`: Generate Prisma client +- `pnpm prisma:seed`: Seed database with test data ### Testing -- `pnpm test`: Run all tests (Vitest + Playwright) -- `pnpm test:vitest`: Run unit and integration tests -- `pnpm test:coverage`: Run tests with coverage reports -- `pnpm test:security`: Run security-specific tests -- `pnpm test:csp`: Test CSP implementation +- `pnpm test`: Run all tests (Vitest + Playwright) +- `pnpm test:vitest`: Run unit and integration tests +- `pnpm test:coverage`: Run tests with coverage reports +- `pnpm test:security`: Run security-specific tests +- `pnpm test:csp`: Test CSP implementation ### Security & Migration -- `pnpm migration:backup`: Create database backup -- `pnpm migration:health-check`: Run system health checks -- `pnpm test:security-headers`: Test HTTP security headers +- `pnpm migration:backup`: Create database backup +- `pnpm migration:health-check`: Run system health checks +- `pnpm test:security-headers`: Test HTTP security headers ## Contributing @@ -196,9 +196,9 @@ This project is not licensed for commercial use without explicit permission. Fre ## Acknowledgments -- [Next.js](https://nextjs.org/) -- [Prisma](https://prisma.io/) -- [TailwindCSS](https://tailwindcss.com/) -- [Chart.js](https://www.chartjs.org/) -- [D3.js](https://d3js.org/) -- [React Leaflet](https://react-leaflet.js.org/) +- [Next.js](https://nextjs.org/) +- [Prisma](https://prisma.io/) +- [TailwindCSS](https://tailwindcss.com/) +- [Chart.js](https://www.chartjs.org/) +- [D3.js](https://d3js.org/) +- [React Leaflet](https://react-leaflet.js.org/) diff --git a/app/api/admin/performance/route.ts b/app/api/admin/performance/route.ts index e57c6ac..dfed128 100644 --- a/app/api/admin/performance/route.ts +++ b/app/api/admin/performance/route.ts @@ -156,9 +156,12 @@ async function getPerformanceHistory(limit: number) { 0 ) / history.length : 0, - memoryTrend: calculateTrend(history, "memoryUsage.heapUsed"), + memoryTrend: calculateTrend( + history as unknown as Record[], + "memoryUsage.heapUsed" + ), responseTrend: calculateTrend( - history, + history as unknown as Record[], "requestMetrics.averageResponseTime" ), }, @@ -539,8 +542,8 @@ function _calculateAverage( : 0; } -function calculateTrend( - history: Array, +function calculateTrend>( + history: Array, path: string ): "increasing" | "decreasing" | "stable" { if (history.length < 2) return "stable"; @@ -570,10 +573,18 @@ function calculateTrend( return "stable"; } -function getNestedPropertyValue(obj: any, path: string): number { - return ( - path.split(".").reduce((current, key) => current?.[key] ?? 0, obj) || 0 - ); +function getNestedPropertyValue( + obj: Record, + path: string +): number { + const result = path.split(".").reduce((current, key) => { + if (current && typeof current === "object" && key in current) { + return (current as Record)[key]; + } + return 0; + }, obj as unknown); + + return typeof result === "number" ? result : 0; } function getNestedValue(obj: Record, path: string): unknown { diff --git a/app/api/admin/schedulers/route.ts b/app/api/admin/schedulers/route.ts index 47b475d..7a51623 100644 --- a/app/api/admin/schedulers/route.ts +++ b/app/api/admin/schedulers/route.ts @@ -1,6 +1,6 @@ -import { getSchedulerIntegration } from "@/lib/services/schedulers/ServerSchedulerIntegration"; -import { createAdminHandler } from "@/lib/api"; import { z } from "zod"; +import { createAdminHandler } from "@/lib/api"; +import { getSchedulerIntegration } from "@/lib/services/schedulers/ServerSchedulerIntegration"; /** * Get all schedulers with their status and metrics @@ -21,72 +21,79 @@ export const GET = createAdminHandler(async (_context) => { }; }); -const PostInputSchema = z.object({ - action: z.enum(["start", "stop", "trigger", "startAll", "stopAll"]), - schedulerId: z.string().optional(), -}).refine( - (data) => { - // schedulerId is required for individual scheduler actions - const actionsRequiringSchedulerId = ["start", "stop", "trigger"]; - if (actionsRequiringSchedulerId.includes(data.action)) { - return data.schedulerId !== undefined && data.schedulerId.length > 0; +const PostInputSchema = z + .object({ + action: z.enum(["start", "stop", "trigger", "startAll", "stopAll"]), + schedulerId: z.string().optional(), + }) + .refine( + (data) => { + // schedulerId is required for individual scheduler actions + const actionsRequiringSchedulerId = ["start", "stop", "trigger"]; + if (actionsRequiringSchedulerId.includes(data.action)) { + return data.schedulerId !== undefined && data.schedulerId.length > 0; + } + return true; + }, + { + message: "schedulerId is required for start, stop, and trigger actions", + path: ["schedulerId"], } - return true; - }, - { - message: "schedulerId is required for start, stop, and trigger actions", - path: ["schedulerId"], - } -); + ); /** * Control scheduler operations (start/stop/trigger) * Requires admin authentication */ -export const POST = createAdminHandler(async (_context, validatedData) => { - const { action, schedulerId } = validatedData; +export const POST = createAdminHandler( + async (_context, validatedData) => { + const { action, schedulerId } = validatedData as z.infer< + typeof PostInputSchema + >; - const integration = getSchedulerIntegration(); + const integration = getSchedulerIntegration(); - switch (action) { - case "start": - if (schedulerId) { - await integration.startScheduler(schedulerId); - } - break; + switch (action) { + case "start": + if (schedulerId) { + await integration.startScheduler(schedulerId); + } + break; - case "stop": - if (schedulerId) { - await integration.stopScheduler(schedulerId); - } - break; + case "stop": + if (schedulerId) { + await integration.stopScheduler(schedulerId); + } + break; - case "trigger": - if (schedulerId) { - await integration.triggerScheduler(schedulerId); - } - break; + case "trigger": + if (schedulerId) { + await integration.triggerScheduler(schedulerId); + } + break; - case "startAll": - await integration.getManager().startAll(); - break; + case "startAll": + await integration.getManager().startAll(); + break; - case "stopAll": - await integration.getManager().stopAll(); - break; + case "stopAll": + await integration.getManager().stopAll(); + break; - default: - return { - success: false, - error: `Unknown action: ${action}`, - }; + default: + return { + success: false, + error: `Unknown action: ${action}`, + }; + } + + return { + success: true, + message: `Action '${action}' completed successfully`, + timestamp: new Date().toISOString(), + }; + }, + { + validateInput: PostInputSchema, } - - return { - success: true, - message: `Action '${action}' completed successfully`, - timestamp: new Date().toISOString(), - }; -}, { - validateInput: PostInputSchema, -}); +); diff --git a/app/api/dashboard/metrics/route.enhanced.ts b/app/api/dashboard/metrics/route.enhanced.ts index 1a793f9..98d01e2 100644 --- a/app/api/dashboard/metrics/route.enhanced.ts +++ b/app/api/dashboard/metrics/route.enhanced.ts @@ -7,20 +7,20 @@ import { type NextRequest, NextResponse } from "next/server"; import { getServerSession } from "next-auth"; -import { authOptions } from "../../../../lib/auth"; -import { sessionMetrics } from "../../../../lib/metrics"; -import { prisma } from "../../../../lib/prisma"; -import type { ChatSession } from "../../../../lib/types"; +import { withErrorHandling } from "@/lib/api/errors"; +import { createSuccessResponse } from "@/lib/api/response"; +import { caches } from "@/lib/performance/cache"; +import { deduplicators } from "@/lib/performance/deduplication"; // Performance system imports import { PerformanceUtils, performanceMonitor, } from "@/lib/performance/monitor"; -import { caches } from "@/lib/performance/cache"; -import { deduplicators } from "@/lib/performance/deduplication"; -import { withErrorHandling } from "@/lib/api/errors"; -import { createSuccessResponse } from "@/lib/api/response"; +import { authOptions } from "../../../../lib/auth"; +import { sessionMetrics } from "../../../../lib/metrics"; +import { prisma } from "../../../../lib/prisma"; +import type { ChatSession, MetricsResult } from "../../../../lib/types"; /** * Converts a Prisma session to ChatSession format for metrics @@ -101,9 +101,14 @@ interface MetricsRequestParams { } interface MetricsResponse { - metrics: any; + metrics: MetricsResult; csvUrl: string | null; - company: any; + company: { + id: string; + name: string; + csvUrl: string; + status: string; + }; dateRange: { minDate: string; maxDate: string } | null; performanceMetrics?: { cacheHit: boolean; @@ -207,9 +212,16 @@ const fetchQuestionsWithDeduplication = deduplicators.database.memoize( */ const calculateMetricsWithCache = async ( chatSessions: ChatSession[], - companyConfig: any, + companyConfig: Record, cacheKey: string -): Promise<{ result: any; fromCache: boolean }> => { +): Promise<{ + result: { + metrics: MetricsResult; + calculatedAt: string; + sessionCount: number; + }; + fromCache: boolean; +}> => { return caches.metrics .getOrCompute( cacheKey, @@ -326,7 +338,7 @@ export const GET = withErrorHandling(async (request: NextRequest) => { deduplicationHit = deduplicators.database.getStats().hitRate > 0; // Fetch questions with deduplication - const sessionIds = prismaSessions.map((s: any) => s.id); + const sessionIds = prismaSessions.map((s) => s.id); const questionsResult = await fetchQuestionsWithDeduplication(sessionIds); const sessionQuestions = questionsResult.result; @@ -349,7 +361,7 @@ export const GET = withErrorHandling(async (request: NextRequest) => { const { result: chatSessions } = await PerformanceUtils.measureAsync( "metrics-session-conversion", async () => { - return prismaSessions.map((ps: any) => { + return prismaSessions.map((ps) => { const questions = questionsBySession[ps.id] || []; return convertToMockChatSession(ps, questions); }); @@ -372,7 +384,7 @@ export const GET = withErrorHandling(async (request: NextRequest) => { if (prismaSessions.length === 0) return null; const dates = prismaSessions - .map((s: any) => new Date(s.startTime)) + .map((s) => new Date(s.startTime)) .sort((a: Date, b: Date) => a.getTime() - b.getTime()); return { diff --git a/app/platform/companies/[id]/page.tsx b/app/platform/companies/[id]/page.tsx index ecb4a72..c8fc043 100644 --- a/app/platform/companies/[id]/page.tsx +++ b/app/platform/companies/[id]/page.tsx @@ -232,14 +232,14 @@ function useCompanyData( } else { const errorText = await response.text(); const errorMessage = `Failed to load company data (${response.status}: ${response.statusText})`; - + console.error("Failed to fetch company - HTTP Error:", { status: response.status, statusText: response.statusText, response: errorText, url: response.url, }); - + toast({ title: "Error", description: errorMessage, @@ -247,17 +247,18 @@ function useCompanyData( }); } } catch (error) { - const errorMessage = error instanceof Error ? error.message : "Unknown error occurred"; - + const errorMessage = + error instanceof Error ? error.message : "Unknown error occurred"; + console.error("Failed to fetch company - Network/Parse Error:", { message: errorMessage, error: error, stack: error instanceof Error ? error.stack : undefined, url: `/api/platform/companies/${params.id}`, }); - + toast({ - title: "Error", + title: "Error", description: `Failed to load company data: ${errorMessage}`, variant: "destructive", }); @@ -371,12 +372,13 @@ function renderCompanyInfoCard( onChange={(e) => { const value = e.target.value; const parsedValue = Number.parseInt(value, 10); - + // Validate input: must be a positive number - const maxUsers = !Number.isNaN(parsedValue) && parsedValue > 0 - ? parsedValue - : 1; // Default to 1 for invalid/negative values - + const maxUsers = + !Number.isNaN(parsedValue) && parsedValue > 0 + ? parsedValue + : 1; // Default to 1 for invalid/negative values + state.setEditData((prev) => ({ ...prev, maxUsers, diff --git a/app/platform/dashboard/page.tsx b/app/platform/dashboard/page.tsx index 7ef1482..b1ce4cc 100644 --- a/app/platform/dashboard/page.tsx +++ b/app/platform/dashboard/page.tsx @@ -156,7 +156,9 @@ function usePlatformDashboardState() { adminPassword: "", maxUsers: 10, }); - const [validationErrors, setValidationErrors] = useState({}); + const [validationErrors, setValidationErrors] = useState( + {} + ); return { dashboardData, @@ -210,22 +212,23 @@ function useFormIds() { */ function validateEmail(email: string): string | undefined { if (!email) return undefined; - - const emailRegex = /^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/; - + + const emailRegex = + /^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/; + if (!emailRegex.test(email)) { return "Please enter a valid email address"; } - + return undefined; } function validateUrl(url: string): string | undefined { if (!url) return undefined; - + try { const urlObj = new URL(url); - if (!['http:', 'https:'].includes(urlObj.protocol)) { + if (!["http:", "https:"].includes(urlObj.protocol)) { return "URL must use HTTP or HTTPS protocol"; } return undefined; @@ -271,7 +274,7 @@ function renderCompanyFormFields( ...prev, csvUrl: value, })); - + // Validate URL on change const error = validateUrl(value); setValidationErrors((prev) => ({ @@ -341,7 +344,7 @@ function renderCompanyFormFields( ...prev, adminEmail: value, })); - + // Validate email on change const error = validateEmail(value); setValidationErrors((prev) => ({ @@ -683,13 +686,12 @@ export default function PlatformDashboard() { newCompanyData.adminEmail && newCompanyData.adminName ); - + // Check for validation errors const hasValidationErrors = !!( - validationErrors.csvUrl || - validationErrors.adminEmail + validationErrors.csvUrl || validationErrors.adminEmail ); - + return hasRequiredFields && !hasValidationErrors; }; diff --git a/components/MessageViewer.tsx b/components/MessageViewer.tsx index 44a9b30..3364edf 100644 --- a/components/MessageViewer.tsx +++ b/components/MessageViewer.tsx @@ -71,8 +71,7 @@ export default function MessageViewer({ messages }: MessageViewerProps) { : "No timestamp"} - Last message:{" "} - {(() => { + Last message: {(() => { const lastMessage = messages[messages.length - 1]; return lastMessage.timestamp ? new Date(lastMessage.timestamp).toLocaleString() diff --git a/docs/CSRF_PROTECTION.md b/docs/CSRF_PROTECTION.md index ebd4788..5e56da4 100644 --- a/docs/CSRF_PROTECTION.md +++ b/docs/CSRF_PROTECTION.md @@ -6,10 +6,10 @@ This document describes the comprehensive CSRF (Cross-Site Request Forgery) prot CSRF protection has been implemented to prevent cross-site request forgery attacks on state-changing operations. The implementation follows industry best practices and provides protection at multiple layers: -- **Middleware Level**: Automatic CSRF validation for protected endpoints -- **tRPC Level**: CSRF protection for all state-changing tRPC procedures -- **Client Level**: Automatic token management and inclusion in requests -- **Component Level**: React components and hooks for easy integration +- **Middleware Level**: Automatic CSRF validation for protected endpoints +- **tRPC Level**: CSRF protection for all state-changing tRPC procedures +- **Client Level**: Automatic token management and inclusion in requests +- **Component Level**: React components and hooks for easy integration ## Implementation Components @@ -17,52 +17,58 @@ CSRF protection has been implemented to prevent cross-site request forgery attac The core CSRF functionality includes: -- **Token Generation**: Cryptographically secure token generation using the `csrf` library -- **Token Verification**: Server-side token validation -- **Request Parsing**: Support for tokens in headers, JSON bodies, and form data -- **Client Utilities**: Browser-side token management and request enhancement +- **Token Generation**: Cryptographically secure token generation using the `csrf` library +- **Token Verification**: Server-side token validation +- **Request Parsing**: Support for tokens in headers, JSON bodies, and form data +- **Client Utilities**: Browser-side token management and request enhancement **Key Functions:** -- `generateCSRFToken()` - Creates new CSRF tokens -- `verifyCSRFToken()` - Validates tokens server-side -- `CSRFProtection.validateRequest()` - Request validation middleware -- `CSRFClient.*` - Client-side utilities + +- `generateCSRFToken()` - Creates new CSRF tokens +- `verifyCSRFToken()` - Validates tokens server-side +- `CSRFProtection.validateRequest()` - Request validation middleware +- `CSRFClient.*` - Client-side utilities ### 2. Middleware Protection (`middleware/csrfProtection.ts`) Provides automatic CSRF protection for API endpoints: **Protected Endpoints:** -- `/api/auth/*` - Authentication endpoints -- `/api/register` - User registration -- `/api/forgot-password` - Password reset requests -- `/api/reset-password` - Password reset completion -- `/api/dashboard/*` - Dashboard API endpoints -- `/api/platform/*` - Platform admin endpoints -- `/api/trpc/*` - All tRPC endpoints + +- `/api/auth/*` - Authentication endpoints +- `/api/register` - User registration +- `/api/forgot-password` - Password reset requests +- `/api/reset-password` - Password reset completion +- `/api/dashboard/*` - Dashboard API endpoints +- `/api/platform/*` - Platform admin endpoints +- `/api/trpc/*` - All tRPC endpoints **Protected Methods:** -- `POST` - Create operations -- `PUT` - Update operations -- `DELETE` - Delete operations -- `PATCH` - Partial update operations + +- `POST` - Create operations +- `PUT` - Update operations +- `DELETE` - Delete operations +- `PATCH` - Partial update operations **Safe Methods (Not Protected):** -- `GET` - Read operations -- `HEAD` - Metadata requests -- `OPTIONS` - CORS preflight requests + +- `GET` - Read operations +- `HEAD` - Metadata requests +- `OPTIONS` - CORS preflight requests ### 3. tRPC Integration (`lib/trpc.ts`) CSRF protection integrated into tRPC procedures: **New Procedure Types:** -- `csrfProtectedProcedure` - Basic CSRF protection -- `csrfProtectedAuthProcedure` - CSRF + authentication protection -- `csrfProtectedCompanyProcedure` - CSRF + company access protection -- `csrfProtectedAdminProcedure` - CSRF + admin access protection + +- `csrfProtectedProcedure` - Basic CSRF protection +- `csrfProtectedAuthProcedure` - CSRF + authentication protection +- `csrfProtectedCompanyProcedure` - CSRF + company access protection +- `csrfProtectedAdminProcedure` - CSRF + admin access protection **Updated Router Example:** + ```typescript // Before register: rateLimitedProcedure @@ -78,30 +84,35 @@ register: csrfProtectedProcedure ### 4. Client-Side Integration #### tRPC Client (`lib/trpc-client.ts`) -- Automatic CSRF token inclusion in tRPC requests -- Token extracted from cookies and added to request headers + +- Automatic CSRF token inclusion in tRPC requests +- Token extracted from cookies and added to request headers #### React Hooks (`lib/hooks/useCSRF.ts`) -- `useCSRF()` - Basic token management -- `useCSRFFetch()` - Enhanced fetch with automatic CSRF tokens -- `useCSRFForm()` - Form submission with CSRF protection + +- `useCSRF()` - Basic token management +- `useCSRFFetch()` - Enhanced fetch with automatic CSRF tokens +- `useCSRFForm()` - Form submission with CSRF protection #### Provider Component (`components/providers/CSRFProvider.tsx`) -- Application-wide CSRF token management -- Automatic token fetching and refresh -- Context-based token sharing + +- Application-wide CSRF token management +- Automatic token fetching and refresh +- Context-based token sharing #### Protected Form Component (`components/forms/CSRFProtectedForm.tsx`) -- Ready-to-use form component with CSRF protection -- Automatic token inclusion in form submissions -- Graceful fallback for non-JavaScript environments + +- Ready-to-use form component with CSRF protection +- Automatic token inclusion in form submissions +- Graceful fallback for non-JavaScript environments ### 5. API Endpoint (`app/api/csrf-token/route.ts`) Provides CSRF tokens to client applications: -- `GET /api/csrf-token` - Returns new CSRF token -- Sets HTTP-only cookie for automatic inclusion -- Used by client-side hooks and components + +- `GET /api/csrf-token` - Returns new CSRF token +- Sets HTTP-only cookie for automatic inclusion +- Used by client-side hooks and components ## Configuration @@ -206,33 +217,38 @@ const dataWithToken = CSRFClient.addTokenToObject({ data: 'example' }); ## Security Features ### 1. Token Properties -- **Cryptographically Secure**: Uses the `csrf` library with secure random generation -- **Short-Lived**: 24-hour expiration by default -- **HTTP-Only Cookies**: Prevents XSS-based token theft -- **SameSite Protection**: Reduces CSRF attack surface + +- **Cryptographically Secure**: Uses the `csrf` library with secure random generation +- **Short-Lived**: 24-hour expiration by default +- **HTTP-Only Cookies**: Prevents XSS-based token theft +- **SameSite Protection**: Reduces CSRF attack surface ### 2. Validation Process -1. Extract token from request (header, form data, or JSON body) -2. Retrieve stored token from HTTP-only cookie -3. Verify tokens match -4. Validate token cryptographic integrity -5. Allow or reject request based on validation + +1. Extract token from request (header, form data, or JSON body) +2. Retrieve stored token from HTTP-only cookie +3. Verify tokens match +4. Validate token cryptographic integrity +5. Allow or reject request based on validation ### 3. Error Handling -- **Graceful Degradation**: Form fallbacks for JavaScript-disabled browsers -- **Clear Error Messages**: Specific error codes for debugging -- **Rate Limiting Integration**: Works with existing auth rate limiting -- **Logging**: Comprehensive logging for security monitoring + +- **Graceful Degradation**: Form fallbacks for JavaScript-disabled browsers +- **Clear Error Messages**: Specific error codes for debugging +- **Rate Limiting Integration**: Works with existing auth rate limiting +- **Logging**: Comprehensive logging for security monitoring ## Testing ### Test Coverage -- **Unit Tests**: Token generation, validation, and client utilities -- **Integration Tests**: Middleware behavior and endpoint protection -- **Component Tests**: React hooks and form components -- **End-to-End**: Full request/response cycle testing + +- **Unit Tests**: Token generation, validation, and client utilities +- **Integration Tests**: Middleware behavior and endpoint protection +- **Component Tests**: React hooks and form components +- **End-to-End**: Full request/response cycle testing ### Running Tests + ```bash # Run all CSRF tests pnpm test:vitest tests/unit/csrf*.test.ts tests/integration/csrf*.test.ts @@ -246,32 +262,36 @@ pnpm test:vitest tests/unit/csrf-hooks.test.tsx ## Monitoring and Debugging ### CSRF Validation Logs + Failed CSRF validations are logged with details: + ``` CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from request ``` ### Common Issues and Solutions -1. **Token Missing from Request** - - Ensure CSRFProvider is wrapping your app - - Check that hooks are being used correctly - - Verify network requests include credentials +1. **Token Missing from Request** + - Ensure CSRFProvider is wrapping your app + - Check that hooks are being used correctly + - Verify network requests include credentials -2. **Token Mismatch** - - Clear browser cookies and refresh - - Check for multiple token sources conflicting - - Verify server and client time synchronization +2. **Token Mismatch** + - Clear browser cookies and refresh + - Check for multiple token sources conflicting + - Verify server and client time synchronization -3. **Integration Issues** - - Ensure middleware is properly configured - - Check tRPC client configuration - - Verify protected procedures are using correct types +3. **Integration Issues** + - Ensure middleware is properly configured + - Check tRPC client configuration + - Verify protected procedures are using correct types ## Migration Guide ### For Existing Endpoints -1. Update tRPC procedures to use CSRF-protected variants: + +1. Update tRPC procedures to use CSRF-protected variants: + ```typescript // Old someAction: protectedProcedure.mutation(...) @@ -280,7 +300,8 @@ CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from someAction: csrfProtectedAuthProcedure.mutation(...) ``` -2. Update client components to use CSRF hooks: +2. Update client components to use CSRF hooks: + ```tsx // Old const { data, mutate } = trpc.user.update.useMutation(); @@ -289,7 +310,8 @@ CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from const { data, mutate } = trpc.user.update.useMutation(); ``` -3. Update manual API calls to include CSRF tokens: +3. Update manual API calls to include CSRF tokens: + ```typescript // Old fetch('/api/endpoint', { method: 'POST', ... }); @@ -301,21 +323,21 @@ CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from ## Performance Considerations -- **Minimal Overhead**: Token validation adds ~1ms per request -- **Efficient Caching**: Tokens cached in memory and cookies -- **Selective Protection**: Only state-changing operations protected -- **Optimized Parsing**: Smart content-type detection for token extraction +- **Minimal Overhead**: Token validation adds ~1ms per request +- **Efficient Caching**: Tokens cached in memory and cookies +- **Selective Protection**: Only state-changing operations protected +- **Optimized Parsing**: Smart content-type detection for token extraction ## Security Best Practices -1. **Always use HTTPS in production** - CSRF tokens should never be transmitted over HTTP -2. **Monitor CSRF failures** - Implement alerting for unusual CSRF failure patterns -3. **Regular secret rotation** - Consider rotating CSRF secrets periodically -4. **Validate referrer headers** - Additional protection layer (not implemented but recommended) -5. **Content Security Policy** - Use CSP headers to prevent XSS attacks that could steal tokens +1. **Always use HTTPS in production** - CSRF tokens should never be transmitted over HTTP +2. **Monitor CSRF failures** - Implement alerting for unusual CSRF failure patterns +3. **Regular secret rotation** - Consider rotating CSRF secrets periodically +4. **Validate referrer headers** - Additional protection layer (not implemented but recommended) +5. **Content Security Policy** - Use CSP headers to prevent XSS attacks that could steal tokens ## Conclusion The CSRF protection implementation provides comprehensive defense against cross-site request forgery attacks while maintaining ease of use for developers. The multi-layer approach ensures protection at the middleware, application, and component levels, with automatic token management reducing the risk of developer error. -For questions or issues related to CSRF protection, refer to the test files for examples and the security documentation for additional context. \ No newline at end of file +For questions or issues related to CSRF protection, refer to the test files for examples and the security documentation for additional context. diff --git a/docs/admin-audit-logs-api.md b/docs/admin-audit-logs-api.md index 437de4e..69658af 100644 --- a/docs/admin-audit-logs-api.md +++ b/docs/admin-audit-logs-api.md @@ -8,10 +8,10 @@ The Admin Audit Logs API provides secure access to security audit trails for adm ## Authentication & Authorization -- **Authentication**: NextAuth.js session required -- **Authorization**: ADMIN role required for all endpoints -- **Rate Limiting**: Integrated with existing auth rate limiting system -- **Audit Trail**: All API access is logged for security monitoring +- **Authentication**: NextAuth.js session required +- **Authorization**: ADMIN role required for all endpoints +- **Rate-Limiting**: Integrated with existing authentication rate-limiting system +- **Audit Trail**: All API access is logged for security monitoring ## API Endpoints @@ -128,12 +128,14 @@ POST /api/admin/audit-logs/retention ```json { - "action": "cleanup" | "configure" | "status", + "action": "cleanup", "retentionDays": 90, "dryRun": true } ``` +**Note**: `action` field accepts one of: `"cleanup"`, `"configure"`, or `"status"` + #### Parameters | Parameter | Type | Required | Description | @@ -145,6 +147,7 @@ POST /api/admin/audit-logs/retention #### Example Requests **Check retention status:** + ```javascript const response = await fetch('/api/admin/audit-logs/retention', { method: 'POST', @@ -154,6 +157,7 @@ const response = await fetch('/api/admin/audit-logs/retention', { ``` **Configure retention policy:** + ```javascript const response = await fetch('/api/admin/audit-logs/retention', { method: 'POST', @@ -166,6 +170,7 @@ const response = await fetch('/api/admin/audit-logs/retention', { ``` **Cleanup old logs (dry run):** + ```javascript const response = await fetch('/api/admin/audit-logs/retention', { method: 'POST', @@ -180,19 +185,22 @@ const response = await fetch('/api/admin/audit-logs/retention', { ## Security Features ### Access Control -- **Role-based Access**: Only ADMIN users can access audit logs -- **Company Isolation**: Users only see logs for their company -- **Session Validation**: Active NextAuth session required + +- **Role-based Access**: Only ADMIN users can access audit logs +- **Company Isolation**: Users only see logs for their company +- **Session Validation**: Active NextAuth session required ### Audit Trail -- **Access Logging**: All audit log access is recorded -- **Metadata Tracking**: Request parameters and results are logged -- **IP Tracking**: Client IP addresses are recorded for all requests + +- **Access Logging**: All audit log access is recorded +- **Metadata Tracking**: Request parameters and results are logged +- **IP Tracking**: Client IP addresses are recorded for all requests ### Rate Limiting -- **Integrated Protection**: Uses existing authentication rate limiting -- **Abuse Prevention**: Protects against excessive API usage -- **Error Tracking**: Failed attempts are monitored + +- **Integrated Protection**: Uses existing authentication rate-limiting +- **Abuse Prevention**: Protects against excessive API usage +- **Error Tracking**: Failed attempts are monitored ## Event Types @@ -294,19 +302,22 @@ async function getUserActivity(userId, days = 7) { ## Performance Considerations ### Database Optimization -- **Indexed Queries**: All filter columns are properly indexed -- **Pagination**: Efficient offset-based pagination with limits -- **Time Range Filtering**: Optimized for date range queries + +- **Indexed Queries**: All filter columns are properly indexed +- **Pagination**: Efficient offset-based pagination with limits +- **Time Range Filtering**: Optimized for date range queries ### Memory Usage -- **Limited Results**: Maximum 100 records per request -- **Streaming**: Large exports use streaming for memory efficiency -- **Connection Pooling**: Database connections are pooled + +- **Limited Results**: Maximum 100 records per request +- **Streaming**: Large exports use streaming for memory efficiency +- **Connection Pooling**: Database connections are pooled ### Caching Considerations -- **No Caching**: Audit logs are never cached for security reasons -- **Fresh Data**: All queries hit the database for real-time results -- **Read Replicas**: Consider using read replicas for heavy reporting + +- **No Caching**: Audit logs are never cached for security reasons +- **Fresh Data**: All queries hit the database for real-time results +- **Read Replicas**: Consider using read replicas for heavy reporting ## Error Handling @@ -335,7 +346,7 @@ try { } ``` -### Rate Limiting Handling +### Rate-Limiting Handling ```javascript async function fetchWithRetry(url, options = {}) { @@ -354,40 +365,45 @@ async function fetchWithRetry(url, options = {}) { ## Monitoring and Alerting ### Key Metrics to Monitor -- **Request Volume**: Track API usage patterns -- **Error Rates**: Monitor authentication and authorization failures -- **Query Performance**: Track slow queries and optimize -- **Data Growth**: Monitor audit log size and plan retention + +- **Request Volume**: Track API usage patterns +- **Error Rates**: Monitor authentication and authorization failures +- **Query Performance**: Track slow queries and optimize +- **Data Growth**: Monitor audit log size and plan retention ### Alert Conditions -- **High Error Rates**: >5% of requests failing -- **Unusual Access Patterns**: Off-hours access, high volume -- **Performance Degradation**: Query times >2 seconds -- **Security Events**: Multiple failed admin access attempts + +- **High Error Rates**: >5% of requests failing +- **Unusual Access Patterns**: Off-hours access, high-volume usage +- **Performance Degradation**: Query times >2 seconds +- **Security Events**: Multiple failed admin access attempts ## Best Practices ### Security -- Always validate user permissions before displaying UI -- Log all administrative access to audit logs -- Use HTTPS in production environments -- Implement proper error handling to avoid information leakage + +- Always validate user permissions before displaying UI +- Log all administrative access to audit logs +- Use HTTPS in production environments +- Implement proper error handling to avoid information leakage ### Performance -- Use appropriate page sizes (25-50 records typical) -- Implement client-side pagination for better UX -- Cache results only in memory, never persist -- Use date range filters to limit query scope + +- Use appropriate page sizes (25-50 records typical) +- Implement client-side pagination for better UX +- Cache results only in memory, never persist +- Use date range filters to limit query scope ### User Experience -- Provide clear filtering options in the UI -- Show loading states for long-running queries -- Implement export functionality for reports -- Provide search and sort capabilities + +- Provide clear filtering options in the UI +- Show loading states for long-running queries +- Implement export functionality for reports +- Provide search and sort capabilities ## Related Documentation -- [Security Audit Logging](./security-audit-logging.md) -- [Security Monitoring](./security-monitoring.md) -- [CSRF Protection](./CSRF_PROTECTION.md) -- [Authentication System](../lib/auth.ts) \ No newline at end of file +- [Security Audit Logging](./security-audit-logging.md) +- [Security Monitoring](./security-monitoring.md) +- [CSRF Protection](./CSRF_PROTECTION.md) +- [Authentication System](../lib/auth.ts) diff --git a/docs/api-reference.md b/docs/api-reference.md index 2b1b92a..00a8d7f 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -28,6 +28,7 @@ X-CSRF-Token: ``` Get CSRF token: + ```http GET /api/csrf-token ``` @@ -35,55 +36,64 @@ GET /api/csrf-token ## API Endpoints Overview ### Public Endpoints -- `POST /api/csp-report` - CSP violation reporting (no auth required) -- `OPTIONS /api/csp-report` - CORS preflight + +- `POST /api/csp-report` - CSP violation reporting (no auth required) +- `OPTIONS /api/csp-report` - CORS preflight ### Authentication Endpoints -- `POST /api/auth/[...nextauth]` - NextAuth.js authentication -- `GET /api/csrf-token` - Get CSRF token -- `POST /api/register` - User registration -- `POST /api/forgot-password` - Password reset request -- `POST /api/reset-password` - Password reset completion + +- `POST /api/auth/[...nextauth]` - NextAuth.js authentication +- `GET /api/csrf-token` - Get CSRF token +- `POST /api/register` - User registration +- `POST /api/forgot-password` - Password reset request +- `POST /api/reset-password` - Password reset completion ### Admin Endpoints (ADMIN role required) -- `GET /api/admin/audit-logs` - Retrieve audit logs -- `POST /api/admin/audit-logs/retention` - Manage audit log retention -- `GET /api/admin/batch-monitoring` - Batch processing monitoring -- `POST /api/admin/batch-monitoring/{id}/retry` - Retry failed batch job + +- `GET /api/admin/audit-logs` - Retrieve audit logs +- `POST /api/admin/audit-logs/retention` - Manage audit log retention +- `GET /api/admin/batch-monitoring` - Batch processing monitoring +- `POST /api/admin/batch-monitoring/{id}/retry` - Retry failed batch job ### Platform Admin Endpoints (Platform admin only) -- `GET /api/admin/security-monitoring` - Security monitoring metrics -- `POST /api/admin/security-monitoring` - Update security configuration -- `GET /api/admin/security-monitoring/alerts` - Alert management -- `POST /api/admin/security-monitoring/alerts` - Acknowledge alerts -- `GET /api/admin/security-monitoring/export` - Export security data -- `POST /api/admin/security-monitoring/threat-analysis` - Threat analysis + +- `GET /api/admin/security-monitoring` - Security monitoring metrics +- `POST /api/admin/security-monitoring` - Update security configuration +- `GET /api/admin/security-monitoring/alerts` - Alert management +- `POST /api/admin/security-monitoring/alerts` - Acknowledge alerts +- `GET /api/admin/security-monitoring/export` - Export security data +- `POST /api/admin/security-monitoring/threat-analysis` - Threat analysis ### Security Monitoring Endpoints -- `GET /api/csp-metrics` - CSP violation metrics -- `POST /api/csp-report` - CSP violation reporting + +- `GET /api/csp-metrics` - CSP violation metrics +- `POST /api/csp-report` - CSP violation reporting ### Dashboard Endpoints -- `GET /api/dashboard/sessions` - Session data -- `GET /api/dashboard/session/{id}` - Individual session details -- `GET /api/dashboard/metrics` - Dashboard metrics -- `GET /api/dashboard/config` - Dashboard configuration + +- `GET /api/dashboard/sessions` - Session data +- `GET /api/dashboard/session/{id}` - Individual session details +- `GET /api/dashboard/metrics` - Dashboard metrics +- `GET /api/dashboard/config` - Dashboard configuration ### Platform Management -- `GET /api/platform/companies` - Company management -- `POST /api/platform/companies` - Create company -- `GET /api/platform/companies/{id}` - Company details -- `GET /api/platform/companies/{id}/users` - Company users -- `POST /api/platform/companies/{id}/users` - Add company user + +- `GET /api/platform/companies` - Company management +- `POST /api/platform/companies` - Create company +- `GET /api/platform/companies/{id}` - Company details +- `GET /api/platform/companies/{id}/users` - Company users +- `POST /api/platform/companies/{id}/users` - Add company user ### tRPC Endpoints -- `POST /api/trpc/[trpc]` - tRPC procedure calls + +- `POST /api/trpc/[trpc]` - tRPC procedure calls ## Detailed Endpoint Documentation ### Admin Audit Logs #### Get Audit Logs + ```http GET /api/admin/audit-logs ``` @@ -91,16 +101,18 @@ GET /api/admin/audit-logs **Authorization**: ADMIN role required **Query Parameters**: -- `page` (number, optional): Page number (default: 1) -- `limit` (number, optional): Records per page, max 100 (default: 50) -- `eventType` (string, optional): Filter by event type -- `outcome` (string, optional): Filter by outcome (SUCCESS, FAILURE, BLOCKED, etc.) -- `severity` (string, optional): Filter by severity (LOW, MEDIUM, HIGH, CRITICAL) -- `userId` (string, optional): Filter by user ID -- `startDate` (string, optional): Start date (ISO 8601) -- `endDate` (string, optional): End date (ISO 8601) + +- `page` (number, optional): Page number (default: 1) +- `limit` (number, optional): Records per page, max 100 (default: 50) +- `eventType` (string, optional): Filter by event type +- `outcome` (string, optional): Filter by outcome (SUCCESS, FAILURE, BLOCKED, etc.) +- `severity` (string, optional): Filter by severity (LOW, MEDIUM, HIGH, CRITICAL) +- `userId` (string, optional): Filter by user ID +- `startDate` (string, optional): Start date (ISO 8601) +- `endDate` (string, optional): End date (ISO 8601) **Response**: + ```json { "success": true, @@ -121,6 +133,7 @@ GET /api/admin/audit-logs **Rate Limit**: Inherits from auth rate limiting #### Manage Audit Log Retention + ```http POST /api/admin/audit-logs/retention ``` @@ -128,6 +141,7 @@ POST /api/admin/audit-logs/retention **Authorization**: ADMIN role required **Request Body**: + ```json { "action": "cleanup" | "configure" | "status", @@ -137,6 +151,7 @@ POST /api/admin/audit-logs/retention ``` **Response**: + ```json { "success": true, @@ -152,6 +167,7 @@ POST /api/admin/audit-logs/retention ### Security Monitoring #### Get Security Metrics + ```http GET /api/admin/security-monitoring ``` @@ -159,12 +175,14 @@ GET /api/admin/security-monitoring **Authorization**: Platform admin required **Query Parameters**: -- `startDate` (string, optional): Start date (ISO 8601) -- `endDate` (string, optional): End date (ISO 8601) -- `companyId` (string, optional): Filter by company -- `severity` (string, optional): Filter by severity + +- `startDate` (string, optional): Start date (ISO 8601) +- `endDate` (string, optional): End date (ISO 8601) +- `companyId` (string, optional): Filter by company +- `severity` (string, optional): Filter by severity **Response**: + ```json { "metrics": { @@ -180,6 +198,7 @@ GET /api/admin/security-monitoring ``` #### Update Security Configuration + ```http POST /api/admin/security-monitoring ``` @@ -187,6 +206,7 @@ POST /api/admin/security-monitoring **Authorization**: Platform admin required **Request Body**: + ```json { "thresholds": { @@ -203,6 +223,7 @@ POST /api/admin/security-monitoring ### CSP Monitoring #### CSP Violation Reporting + ```http POST /api/csp-report ``` @@ -210,9 +231,11 @@ POST /api/csp-report **Authorization**: None (public endpoint) **Headers**: -- `Content-Type`: `application/csp-report` or `application/json` + +- `Content-Type`: `application/csp-report` or `application/json` **Request Body** (automatic from browser): + ```json { "csp-report": { @@ -230,6 +253,7 @@ POST /api/csp-report **Response**: `204 No Content` #### Get CSP Metrics + ```http GET /api/csp-metrics ``` @@ -237,12 +261,14 @@ GET /api/csp-metrics **Authorization**: Admin role required **Query Parameters**: -- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d) -- `format` (string, optional): Response format (json, csv) -- `groupBy` (string, optional): Group by field (hour, directive, etc.) -- `includeDetails` (boolean, optional): Include violation details + +- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d) +- `format` (string, optional): Response format (json, csv) +- `groupBy` (string, optional): Group by field (hour, directive, etc.) +- `includeDetails` (boolean, optional): Include violation details **Response**: + ```json { "success": true, @@ -264,6 +290,7 @@ GET /api/csp-metrics ### Batch Monitoring #### Get Batch Monitoring Data + ```http GET /api/admin/batch-monitoring ``` @@ -271,14 +298,16 @@ GET /api/admin/batch-monitoring **Authorization**: ADMIN role required **Query Parameters**: -- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d) -- `status` (string, optional): Filter by status (pending, completed, failed) -- `jobType` (string, optional): Filter by job type -- `includeDetails` (boolean, optional): Include detailed job information -- `page` (number, optional): Page number -- `limit` (number, optional): Records per page + +- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d) +- `status` (string, optional): Filter by status (pending, completed, failed) +- `jobType` (string, optional): Filter by job type +- `includeDetails` (boolean, optional): Include detailed job information +- `page` (number, optional): Page number +- `limit` (number, optional): Records per page **Response**: + ```json { "success": true, @@ -297,6 +326,7 @@ GET /api/admin/batch-monitoring ``` #### Retry Batch Job + ```http POST /api/admin/batch-monitoring/{jobId}/retry ``` @@ -304,6 +334,7 @@ POST /api/admin/batch-monitoring/{jobId}/retry **Authorization**: ADMIN role required **Response**: + ```json { "success": true, @@ -318,6 +349,7 @@ POST /api/admin/batch-monitoring/{jobId}/retry ### CSRF Token #### Get CSRF Token + ```http GET /api/csrf-token ``` @@ -325,6 +357,7 @@ GET /api/csrf-token **Authorization**: None **Response**: + ```json { "csrfToken": "abc123..." @@ -332,11 +365,13 @@ GET /api/csrf-token ``` **Headers Set**: -- `Set-Cookie`: HTTP-only CSRF token cookie + +- `Set-Cookie`: HTTP-only CSRF token cookie ### Authentication #### User Registration + ```http POST /api/register ``` @@ -344,9 +379,11 @@ POST /api/register **Authorization**: None **Headers Required**: -- `X-CSRF-Token`: CSRF token + +- `X-CSRF-Token`: CSRF token **Request Body**: + ```json { "email": "user@example.com", @@ -359,6 +396,7 @@ POST /api/register **Rate Limit**: 3 attempts per hour per IP **Response**: + ```json { "success": true, @@ -368,6 +406,7 @@ POST /api/register ``` #### Password Reset Request + ```http POST /api/forgot-password ``` @@ -375,9 +414,11 @@ POST /api/forgot-password **Authorization**: None **Headers Required**: -- `X-CSRF-Token`: CSRF token + +- `X-CSRF-Token`: CSRF token **Request Body**: + ```json { "email": "user@example.com" @@ -387,6 +428,7 @@ POST /api/forgot-password **Rate Limit**: 5 attempts per 15 minutes per IP **Response**: + ```json { "success": true, @@ -395,6 +437,7 @@ POST /api/forgot-password ``` #### Password Reset Completion + ```http POST /api/reset-password ``` @@ -402,9 +445,11 @@ POST /api/reset-password **Authorization**: None **Headers Required**: -- `X-CSRF-Token`: CSRF token + +- `X-CSRF-Token`: CSRF token **Request Body**: + ```json { "token": "reset-token-123", @@ -413,6 +458,7 @@ POST /api/reset-password ``` **Response**: + ```json { "success": true, @@ -464,18 +510,21 @@ POST /api/reset-password ## Rate Limiting ### Authentication Endpoints -- **Login**: 5 attempts per 15 minutes per IP -- **Registration**: 3 attempts per hour per IP -- **Password Reset**: 5 attempts per 15 minutes per IP + +- **Login**: 5 attempts per 15 minutes per IP +- **Registration**: 3 attempts per hour per IP +- **Password Reset**: 5 attempts per 15 minutes per IP ### Security Endpoints -- **CSP Reports**: 10 reports per minute per IP -- **Admin Endpoints**: 60 requests per minute per user -- **Security Monitoring**: 30 requests per minute per user + +- **CSP Reports**: 10 reports per minute per IP +- **Admin Endpoints**: 60 requests per minute per user +- **Security Monitoring**: 30 requests per minute per user ### General API -- **Dashboard Endpoints**: 120 requests per minute per user -- **Platform Management**: 60 requests per minute per user + +- **Dashboard Endpoints**: 120 requests per minute per user +- **Platform Management**: 60 requests per minute per user ## Security Headers @@ -492,14 +541,17 @@ Content-Security-Policy: [CSP directives] ## CORS Configuration ### Allowed Origins -- Development: `http://localhost:3000` -- Production: `https://your-domain.com` + +- Development: `http://localhost:3000` +- Production: `https://your-domain.com` ### Allowed Methods -- `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS` + +- `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS` ### Allowed Headers -- `Content-Type`, `Authorization`, `X-CSRF-Token`, `X-Requested-With` + +- `Content-Type`, `Authorization`, `X-CSRF-Token`, `X-Requested-With` ## Pagination @@ -520,25 +572,29 @@ Content-Security-Policy: [CSP directives] ``` ### Pagination Parameters -- `page`: Page number (1-based, default: 1) -- `limit`: Records per page (default: 50, max: 100) + +- `page`: Page number (1-based, default: 1) +- `limit`: Records per page (default: 50, max: 100) ## Filtering and Sorting ### Common Filter Parameters -- `startDate` / `endDate`: Date range filtering (ISO 8601) -- `status`: Status filtering -- `userId` / `companyId`: Entity filtering -- `eventType`: Event type filtering -- `severity`: Severity level filtering + +- `startDate` / `endDate`: Date range filtering (ISO 8601) +- `status`: Status filtering +- `userId` / `companyId`: Entity filtering +- `eventType`: Event type filtering +- `severity`: Severity level filtering ### Sorting Parameters -- `sortBy`: Field to sort by -- `sortOrder`: `asc` or `desc` (default: `desc`) + +- `sortBy`: Field to sort by +- `sortOrder`: `asc` or `desc` (default: `desc`) ## Response Caching ### Cache Headers + ```http Cache-Control: no-cache, no-store, must-revalidate Pragma: no-cache @@ -546,20 +602,23 @@ Expires: 0 ``` ### Cache Strategy -- **Security data**: Never cached -- **Static data**: Browser cache for 5 minutes -- **User data**: No cache for security + +- **Security data**: Never cached +- **Static data**: Browser cache for 5 minutes +- **User data**: No cache for security ## API Versioning ### Current Version -- Version: `v1` (implied, no version prefix required) -- Introduced: January 2025 + +- Version: `v1` (implied, no version prefix required) +- Introduced: January 2025 ### Future Versioning -- Breaking changes will introduce new versions -- Format: `/api/v2/endpoint` -- Backward compatibility maintained for 12 months + +- Breaking changes will introduce new versions +- Format: `/api/v2/endpoint` +- Backward compatibility maintained for 12 months ## SDK and Client Libraries @@ -639,10 +698,10 @@ describe('Admin Audit Logs API', () => { ## Related Documentation -- [Admin Audit Logs API](./admin-audit-logs-api.md) -- [CSP Metrics API](./csp-metrics-api.md) -- [Security Monitoring](./security-monitoring.md) -- [CSRF Protection](./CSRF_PROTECTION.md) -- [Batch Monitoring Dashboard](./batch-monitoring-dashboard.md) +- [Admin Audit Logs API](./admin-audit-logs-api.md) +- [CSP Metrics API](./csp-metrics-api.md) +- [Security Monitoring](./security-monitoring.md) +- [CSRF Protection](./CSRF_PROTECTION.md) +- [Batch Monitoring Dashboard](./batch-monitoring-dashboard.md) -This API reference provides comprehensive documentation for all endpoints in the LiveDash-Node application. For specific implementation details, refer to the individual documentation files for each feature area. \ No newline at end of file +This API reference provides comprehensive documentation for all endpoints in the LiveDash-Node application. For specific implementation details, refer to the individual documentation files for each feature area. diff --git a/docs/batch-monitoring-dashboard.md b/docs/batch-monitoring-dashboard.md index 924dda7..e8629a7 100644 --- a/docs/batch-monitoring-dashboard.md +++ b/docs/batch-monitoring-dashboard.md @@ -9,22 +9,25 @@ The Batch Monitoring Dashboard provides real-time visibility into the OpenAI Bat ## Features ### Real-time Monitoring -- **Job Status Tracking**: Monitor batch jobs from creation to completion -- **Queue Management**: View pending, running, and completed batch queues -- **Processing Metrics**: Track throughput, success rates, and error patterns -- **Cost Analysis**: Monitor API costs and savings compared to individual requests + +- **Job Status Tracking**: Monitor batch jobs from creation to completion +- **Queue Management**: View pending, running, and completed batch queues +- **Processing Metrics**: Track throughput, success rates, and error patterns +- **Cost Analysis**: Monitor API costs and savings compared to individual requests ### Performance Analytics -- **Batch Efficiency**: Analyze batch size optimization and processing times -- **Success Rates**: Track completion and failure rates across different job types -- **Resource Utilization**: Monitor API quota usage and rate limiting -- **Historical Trends**: View processing patterns over time + +- **Batch Efficiency**: Analyze batch size optimization and processing times +- **Success Rates**: Track completion and failure rates across different job types +- **Resource Utilization**: Monitor API quota usage and rate limiting +- **Historical Trends**: View processing patterns over time ### Administrative Controls -- **Manual Intervention**: Pause, resume, or cancel batch operations -- **Priority Management**: Adjust processing priorities for urgent requests -- **Error Handling**: Review and retry failed batch operations -- **Configuration Management**: Adjust batch parameters and thresholds + +- **Manual Intervention**: Pause, resume, or cancel batch operations +- **Priority Management**: Adjust processing priorities for urgent requests +- **Error Handling**: Review and retry failed batch operations +- **Configuration Management**: Adjust batch parameters and thresholds ## API Endpoints @@ -132,6 +135,7 @@ const data = await response.json(); The main dashboard component (`components/admin/BatchMonitoringDashboard.tsx`) provides: #### Key Metrics Cards + ```tsx // Real-time overview cards 2000) { ## Related Documentation -- [Enhanced CSP Implementation](./security/enhanced-csp.md) -- [Security Monitoring](./security-monitoring.md) -- [Security Headers](./security-headers.md) -- [Rate Limiting](../lib/rateLimiter.ts) +- [Enhanced CSP Implementation](./security/enhanced-csp.md) +- [Security Monitoring](./security-monitoring.md) +- [Security Headers](./security-headers.md) +- [Rate Limiting](../lib/rateLimiter.ts) ## API Reference Summary diff --git a/docs/dashboard-components.md b/docs/dashboard-components.md index 0aa0337..c56bf9f 100644 --- a/docs/dashboard-components.md +++ b/docs/dashboard-components.md @@ -12,10 +12,10 @@ The WordCloud component visualizes categories or topics based on their frequency **Features:** -- Dynamic sizing based on frequency -- Colorful display with a pleasing color palette -- Responsive design -- Interactive hover effects +- Dynamic sizing based on frequency +- Colorful display with a pleasing color palette +- Responsive design +- Interactive hover effects ### 2. GeographicMap @@ -25,10 +25,10 @@ This component displays a world map with circles representing the number of sess **Features:** -- Interactive map using React Leaflet -- Circle sizes scaled by session count -- Tooltips showing country names and session counts -- Responsive design +- Interactive map using React Leaflet +- Circle sizes scaled by session count +- Tooltips showing country names and session counts +- Responsive design ### 3. MetricCard @@ -38,10 +38,10 @@ A modern, visually appealing card for displaying key metrics. **Features:** -- Multiple design variants (default, primary, success, warning, danger) -- Support for trend indicators -- Icons and descriptions -- Clean, modern styling +- Multiple design variants (default, primary, success, warning, danger) +- Support for trend indicators +- Icons and descriptions +- Clean, modern styling ### 4. DonutChart @@ -51,10 +51,10 @@ An enhanced donut chart with better styling and a central text display capabilit **Features:** -- Customizable colors -- Center text area for displaying summaries -- Interactive tooltips with percentages -- Well-balanced legend display +- Customizable colors +- Center text area for displaying summaries +- Interactive tooltips with percentages +- Well-balanced legend display ### 5. ResponseTimeDistribution @@ -64,10 +64,10 @@ Visualizes the distribution of response times as a histogram. **Features:** -- Color-coded bars (green for fast, yellow for medium, red for slow) -- Target time indicator -- Automatic binning of response times -- Clear labeling and scales +- Color-coded bars (green for fast, yellow for medium, red for slow) +- Target time indicator +- Automatic binning of response times +- Clear labeling and scales ## Dashboard Enhancements @@ -85,7 +85,7 @@ The dashboard has been enhanced with: ## Usage Notes -- The geographic map and response time distribution use simulated data where actual data is not available -- All components are responsive and will adjust to different screen sizes -- The dashboard automatically refreshes data when using the refresh button -- Admin users have access to additional controls at the bottom of the dashboard +- The geographic map and response time distribution use simulated data where actual data is not available +- All components are responsive and will adjust to different screen sizes +- The dashboard automatically refreshes data when using the refresh button +- Admin users have access to additional controls at the bottom of the dashboard diff --git a/docs/database-connection-pooling.md b/docs/database-connection-pooling.md index 378f691..54fe105 100644 --- a/docs/database-connection-pooling.md +++ b/docs/database-connection-pooling.md @@ -6,8 +6,8 @@ This document explains how to optimize database connection pooling for better pe The application now supports two connection pooling modes: -1. **Standard Pooling**: Default Prisma client connection pooling -2. **Enhanced Pooling**: Advanced PostgreSQL connection pooling with custom configuration +1. **Standard Pooling**: Default Prisma client connection pooling +2. **Enhanced Pooling**: Advanced PostgreSQL connection pooling with custom configuration ## Configuration @@ -29,42 +29,42 @@ DATABASE_URL="postgresql://user:pass@host:5432/db?connection_limit=20&pool_timeo #### Standard Pooling (Default) -- Uses Prisma's built-in connection pooling -- Simpler configuration -- Good for development and small-scale deployments +- Uses Prisma's built-in connection pooling +- Simpler configuration +- Good for development and small-scale deployments #### Enhanced Pooling (Recommended for Production) -- Uses PostgreSQL native connection pooling with `@prisma/adapter-pg` -- Advanced monitoring and health checks -- Better resource management -- Detailed connection metrics +- Uses PostgreSQL native connection pooling with `@prisma/adapter-pg` +- Advanced monitoring and health checks +- Better resource management +- Detailed connection metrics ## Implementation Details ### Fixed Issues -1. **Multiple PrismaClient Instances**: - - ❌ Before: Each scheduler created its own PrismaClient - - ✅ After: All modules use singleton pattern from `lib/prisma.ts` +1. **Multiple PrismaClient Instances**: + - ❌ Before: Each scheduler created its own PrismaClient + - ✅ After: All modules use singleton pattern from `lib/prisma.ts` -2. **No Connection Management**: - - ❌ Before: No graceful shutdown or connection cleanup - - ✅ After: Proper cleanup on process termination +2. **No Connection Management**: + - ❌ Before: No graceful shutdown or connection cleanup + - ✅ After: Proper cleanup on process termination -3. **No Monitoring**: - - ❌ Before: No visibility into connection usage - - ✅ After: Health check endpoint and connection metrics +3. **No Monitoring**: + - ❌ Before: No visibility into connection usage + - ✅ After: Health check endpoint and connection metrics ### Key Files Modified -- `lib/prisma.ts` - Enhanced singleton with pooling options -- `lib/database-pool.ts` - Advanced pooling configuration -- `lib/processingScheduler.ts` - Fixed to use singleton -- `lib/importProcessor.ts` - Fixed to use singleton -- `lib/processingStatusManager.ts` - Fixed to use singleton -- `lib/schedulers.ts` - Added graceful shutdown -- `app/api/admin/database-health/route.ts` - Monitoring endpoint +- `lib/prisma.ts` - Enhanced singleton with pooling options +- `lib/database-pool.ts` - Advanced pooling configuration +- `lib/processingScheduler.ts` - Fixed to use singleton +- `lib/importProcessor.ts` - Fixed to use singleton +- `lib/processingStatusManager.ts` - Fixed to use singleton +- `lib/schedulers.ts` - Added graceful shutdown +- `app/api/admin/database-health/route.ts` - Monitoring endpoint ## Monitoring @@ -79,36 +79,36 @@ curl -H "Authorization: Bearer your-token" \ Response includes: -- Connection status -- Pool statistics (if enhanced pooling enabled) -- Basic metrics (session counts, etc.) -- Configuration details +- Connection status +- Pool statistics (if enhanced pooling enabled) +- Basic metrics (session counts, etc.) +- Configuration details ### Connection Metrics With enhanced pooling enabled, you'll see console logs for: -- Connection acquisitions/releases -- Pool size changes -- Error events -- Health check results +- Connection acquisitions/releases +- Pool size changes +- Error events +- Health check results ## Performance Benefits ### Before Optimization -- Multiple connection pools (one per scheduler) -- Potential connection exhaustion under load -- No connection monitoring -- Resource waste from idle connections +- Multiple connection pools (one per scheduler) +- Potential connection exhaustion under load +- No connection monitoring +- Resource waste from idle connections ### After Optimization -- Single shared connection pool -- Configurable pool size and timeouts -- Connection health monitoring -- Graceful shutdown and cleanup -- Better resource utilization +- Single shared connection pool +- Configurable pool size and timeouts +- Connection health monitoring +- Graceful shutdown and cleanup +- Better resource utilization ## Recommended Settings @@ -142,40 +142,40 @@ USE_ENHANCED_POOLING=true If you see "too many connections" errors: -1. Increase `DATABASE_CONNECTION_LIMIT` -2. Check for connection leaks in application code -3. Monitor the health endpoint for pool statistics +1. Increase `DATABASE_CONNECTION_LIMIT` +2. Check for connection leaks in application code +3. Monitor the health endpoint for pool statistics ### Slow Database Queries If queries are timing out: -1. Decrease `DATABASE_POOL_TIMEOUT` -2. Check database query performance -3. Consider connection pooling at the infrastructure level (PgBouncer) +1. Decrease `DATABASE_POOL_TIMEOUT` +2. Check database query performance +3. Consider connection pooling at the infrastructure level (PgBouncer) ### Memory Usage If memory usage is high: -1. Decrease `DATABASE_CONNECTION_LIMIT` -2. Enable enhanced pooling for better resource management -3. Monitor idle connection cleanup +1. Decrease `DATABASE_CONNECTION_LIMIT` +2. Enable enhanced pooling for better resource management +3. Monitor idle connection cleanup ## Best Practices -1. **Always use the singleton**: Import `prisma` from `lib/prisma.ts` -2. **Monitor connection usage**: Use the health endpoint regularly -3. **Set appropriate limits**: Don't over-provision connections -4. **Enable enhanced pooling in production**: Better resource management -5. **Implement graceful shutdown**: Ensure connections are properly closed -6. **Log connection events**: Monitor for issues and optimize accordingly +1. **Always use the singleton**: Import `prisma` from `lib/prisma.ts` +2. **Monitor connection usage**: Use the health endpoint regularly +3. **Set appropriate limits**: Don't over-provision connections +4. **Enable enhanced pooling in production**: Better resource management +5. **Implement graceful shutdown**: Ensure connections are properly closed +6. **Log connection events**: Monitor for issues and optimize accordingly ## Next Steps Consider implementing: -1. **Connection pooling middleware**: PgBouncer or similar -2. **Read replicas**: For read-heavy workloads -3. **Connection retry logic**: For handling temporary failures -4. **Metrics collection**: Prometheus/Grafana for detailed monitoring +1. **Connection pooling middleware**: PgBouncer or similar +2. **Read replicas**: For read-heavy workloads +3. **Connection retry logic**: For handling temporary failures +4. **Metrics collection**: Prometheus/Grafana for detailed monitoring diff --git a/docs/database-performance-optimizations.md b/docs/database-performance-optimizations.md index d220f21..84b1eb6 100644 --- a/docs/database-performance-optimizations.md +++ b/docs/database-performance-optimizations.md @@ -6,10 +6,10 @@ This document outlines the comprehensive database performance optimizations impl The optimization focuses on the most frequently queried patterns in the application, particularly around: -- AI processing request tracking and batching -- Session analytics and filtering -- Security audit log analysis -- Multi-tenant data isolation performance +- AI processing request tracking and batching +- Session analytics and filtering +- Security audit log analysis +- Multi-tenant data isolation performance ## Applied Optimizations @@ -31,9 +31,9 @@ INCLUDE ("processingStatus", "batchId", "requestedAt", "sessionId"); **Impact**: -- ~70% faster batch job queries -- Reduced I/O for cost analysis reports -- Improved scheduler performance +- ~70% faster batch job queries +- Reduced I/O for cost analysis reports +- Improved scheduler performance ### 2. Session Analytics Optimizations @@ -54,9 +54,9 @@ INCLUDE ("startTime", "messagesSent"); **Impact**: -- ~85% faster dashboard load times -- Efficient date range filtering -- Optimized sentiment analysis queries +- ~85% faster dashboard load times +- Efficient date range filtering +- Optimized sentiment analysis queries ### 3. Security Audit Log Optimizations @@ -77,9 +77,9 @@ INCLUDE ("eventType", "severity", "userId", "companyId"); **Impact**: -- ~90% faster security monitoring -- Efficient threat detection -- Improved compliance reporting +- ~90% faster security monitoring +- Efficient threat detection +- Improved compliance reporting ### 4. Message Processing Optimizations @@ -95,8 +95,8 @@ INCLUDE ("content"); **Impact**: -- ~60% faster conversation loading -- Reduced memory usage for message queries +- ~60% faster conversation loading +- Reduced memory usage for message queries ### 5. Processing Pipeline Optimizations @@ -118,29 +118,29 @@ INCLUDE ("sessionId", "errorMessage", "retryCount", "startedAt"); **Impact**: -- ~75% faster processing monitoring -- Efficient error tracking -- Improved retry logic performance +- ~75% faster processing monitoring +- Efficient error tracking +- Improved retry logic performance ## Index Strategy Principles ### 1. Composite Index Design -- **Leading column**: Most selective filter (usually companyId for multi-tenancy) -- **Secondary columns**: Common WHERE clause filters -- **Covering columns**: SELECT list columns via INCLUDE +- **Leading column**: Most selective filter (usually companyId for multi-tenancy) +- **Secondary columns**: Common WHERE clause filters +- **Covering columns**: SELECT list columns via INCLUDE ### 2. Partial Indexes -- Used for error analysis and specific status filtering -- Reduces index size and maintenance overhead -- Improves write performance +- Used for error analysis and specific status filtering +- Reduces index size and maintenance overhead +- Improves write performance ### 3. Covering Indexes -- Include frequently accessed columns to avoid table lookups -- Reduces I/O for read-heavy operations -- Particularly effective for dashboard queries +- Include frequently accessed columns to avoid table lookups +- Reduces I/O for read-heavy operations +- Particularly effective for dashboard queries ## Query Pattern Analysis @@ -166,29 +166,29 @@ INCLUDE ("sessionId", "errorMessage", "retryCount", "startedAt"); ### Index Monitoring -- Monitor index usage with `pg_stat_user_indexes` -- Track bloat with `pg_stat_user_tables` -- Regular ANALYZE after bulk operations +- Monitor index usage with `pg_stat_user_indexes` +- Track bloat with `pg_stat_user_tables` +- Regular ANALYZE after bulk operations ### Write Performance Impact -- Composite indexes add ~15% write overhead -- Offset by dramatic read performance gains -- Monitored via slow query logs +- Composite indexes add ~15% write overhead +- Offset by dramatic read performance gains +- Monitored via slow query logs ### Storage Impact -- Indexes add ~25% to total storage -- Covering indexes reduce need for table scans -- Partial indexes minimize storage overhead +- Indexes add ~25% to total storage +- Covering indexes reduce need for table scans +- Partial indexes minimize storage overhead ## Migration Safety ### CONCURRENTLY Operations -- All indexes created with `CREATE INDEX CONCURRENTLY` -- No table locks during creation -- Production-safe deployment +- All indexes created with `CREATE INDEX CONCURRENTLY` +- No table locks during creation +- Production-safe deployment ### Rollback Strategy @@ -238,18 +238,18 @@ LIMIT 10; ### Monitoring Strategy -- Set up automated index usage monitoring -- Track slow query evolution -- Monitor storage growth patterns -- Implement performance alerting +- Set up automated index usage monitoring +- Track slow query evolution +- Monitor storage growth patterns +- Implement performance alerting ## Conclusion These database optimizations provide: -- **70-90% improvement** in query performance -- **Reduced server load** through efficient indexing -- **Better user experience** with faster dashboards -- **Scalable foundation** for future growth +- **70-90% improvement** in query performance +- **Reduced server load** through efficient indexing +- **Better user experience** with faster dashboards +- **Scalable foundation** for future growth The optimizations are designed to be production-safe and monitoring-friendly, ensuring both immediate performance gains and long-term maintainability. diff --git a/docs/neon-database-optimization.md b/docs/neon-database-optimization.md index 76067de..3c072ba 100644 --- a/docs/neon-database-optimization.md +++ b/docs/neon-database-optimization.md @@ -15,21 +15,21 @@ Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.ne ### 1. Neon Connection Limits -- **Free Tier**: 20 concurrent connections -- **Pro Tier**: 100 concurrent connections -- **Multiple schedulers** can quickly exhaust connections +- **Free Tier**: 20 concurrent connections +- **Pro Tier**: 100 concurrent connections +- **Multiple schedulers** can quickly exhaust connections ### 2. Connection Pooling Issues -- Each scheduler was creating separate PrismaClient instances -- No connection reuse between operations -- No retry logic for temporary failures +- Each scheduler was creating separate PrismaClient instances +- No connection reuse between operations +- No retry logic for temporary failures ### 3. Neon-Specific Challenges -- **Auto-pause**: Databases pause after inactivity -- **Cold starts**: First connection after pause takes longer -- **Regional latency**: eu-central-1 may have variable latency +- **Auto-pause**: Databases pause after inactivity +- **Cold starts**: First connection after pause takes longer +- **Regional latency**: eu-central-1 may have variable latency ## Solutions Implemented @@ -106,9 +106,9 @@ curl -H "Authorization: Bearer YOUR_API_TOKEN" \ ### 2. Neon Dashboard Monitoring -- Monitor "Active connections" in Neon dashboard -- Check for connection spikes during scheduler runs -- Review query performance and slow queries +- Monitor "Active connections" in Neon dashboard +- Check for connection spikes during scheduler runs +- Review query performance and slow queries ### 3. Application Logs @@ -158,9 +158,9 @@ const prisma = new PrismaClient({ **Causes:** -- Neon database auto-paused -- Connection limit exceeded -- Network issues +- Neon database auto-paused +- Connection limit exceeded +- Network issues **Solutions:** @@ -173,9 +173,9 @@ const prisma = new PrismaClient({ **Causes:** -- Idle connection timeout -- Neon maintenance -- Long-running transactions +- Idle connection timeout +- Neon maintenance +- Long-running transactions **Solutions:** @@ -187,9 +187,9 @@ const prisma = new PrismaClient({ **Causes:** -- Blocking database operations -- Scheduler overlap -- High CPU usage +- Blocking database operations +- Scheduler overlap +- High CPU usage **Solutions:** @@ -230,10 +230,10 @@ SESSION_PROCESSING_INTERVAL="0 */2 * * *" ## Monitoring Checklist -- [ ] Check Neon dashboard for connection spikes -- [ ] Monitor scheduler execution times -- [ ] Review error logs for connection patterns -- [ ] Test health endpoint regularly -- [ ] Set up alerts for connection failures +- [ ] Check Neon dashboard for connection spikes +- [ ] Monitor scheduler execution times +- [ ] Review error logs for connection patterns +- [ ] Test health endpoint regularly +- [ ] Set up alerts for connection failures With these optimizations, your Neon database connections should be much more stable and efficient! diff --git a/docs/postgresql-migration.md b/docs/postgresql-migration.md index 67c0e1a..2f36d93 100644 --- a/docs/postgresql-migration.md +++ b/docs/postgresql-migration.md @@ -17,48 +17,48 @@ Successfully migrated the livedash-node application from SQLite to PostgreSQL us #### Production/Development -- **Provider**: PostgreSQL (Neon) -- **Environment Variable**: `DATABASE_URL` -- **Connection**: Neon PostgreSQL cluster +- **Provider**: PostgreSQL (Neon) +- **Environment Variable**: `DATABASE_URL` +- **Connection**: Neon PostgreSQL cluster #### Testing -- **Provider**: PostgreSQL (Neon - separate database) -- **Environment Variable**: `DATABASE_URL_TEST` -- **Test Setup**: Automatically switches to test database during test runs +- **Provider**: PostgreSQL (Neon - separate database) +- **Environment Variable**: `DATABASE_URL_TEST` +- **Test Setup**: Automatically switches to test database during test runs ### Files Modified 1. **`prisma/schema.prisma`** -- Changed provider from `sqlite` to `postgresql` -- Updated URL to use `env("DATABASE_URL")` +- Changed provider from `sqlite` to `postgresql` +- Updated URL to use `env("DATABASE_URL")` 2. **`tests/setup.ts`** -- Added logic to use `DATABASE_URL_TEST` when available -- Ensures test isolation with separate database +- Added logic to use `DATABASE_URL_TEST` when available +- Ensures test isolation with separate database 3. **`.env`** (created) -- Contains `DATABASE_URL` for Prisma CLI operations +- Contains `DATABASE_URL` for Prisma CLI operations 4. **`.env.local`** (existing) -- Contains both `DATABASE_URL` and `DATABASE_URL_TEST` +- Contains both `DATABASE_URL` and `DATABASE_URL_TEST` ### Database Schema All existing models and relationships were preserved: -- **Company**: Multi-tenant root entity -- **User**: Authentication and authorization -- **Session**: Processed session data -- **SessionImport**: Raw CSV import data -- **Message**: Individual conversation messages -- **Question**: Normalized question storage -- **SessionQuestion**: Session-question relationships -- **AIProcessingRequest**: AI cost tracking +- **Company**: Multi-tenant root entity +- **User**: Authentication and authorization +- **Session**: Processed session data +- **SessionImport**: Raw CSV import data +- **Message**: Individual conversation messages +- **Question**: Normalized question storage +- **SessionQuestion**: Session-question relationships +- **AIProcessingRequest**: AI cost tracking ### Migration Process @@ -103,11 +103,11 @@ if (process.env.DATABASE_URL_TEST) { All tests pass successfully: -- ✅ Environment configuration tests -- ✅ Transcript fetcher tests -- ✅ Database connection tests -- ✅ Schema validation tests -- ✅ CRUD operation tests +- ✅ Environment configuration tests +- ✅ Transcript fetcher tests +- ✅ Database connection tests +- ✅ Schema validation tests +- ✅ CRUD operation tests ### Next Steps diff --git a/docs/processing-system-refactor.md b/docs/processing-system-refactor.md index 224d41c..c57497c 100644 --- a/docs/processing-system-refactor.md +++ b/docs/processing-system-refactor.md @@ -7,21 +7,24 @@ Successfully refactored the session processing pipeline from a simple status-bas ## Problems Solved ### Original Issues -1. **Inconsistent Status Tracking**: The old system used a simple enum on SessionImport that didn't properly track the multi-stage processing pipeline -2. **Poor Error Visibility**: Error messages were buried in the SessionImport table and not easily accessible -3. **No Stage-Specific Tracking**: The system couldn't track which specific stage of processing failed -4. **Difficult Recovery**: Failed sessions were hard to identify and retry -5. **Linting Errors**: Multiple TypeScript files referencing removed database fields + +1. **Inconsistent Status Tracking**: The old system used a simple enum on SessionImport that didn't properly track the multi-stage processing pipeline +2. **Poor Error Visibility**: Error messages were buried in the SessionImport table and not easily accessible +3. **No Stage-Specific Tracking**: The system couldn't track which specific stage of processing failed +4. **Difficult Recovery**: Failed sessions were hard to identify and retry +5. **Linting Errors**: Multiple TypeScript files referencing removed database fields ### Schema Changes Made -- **Removed** old `status`, `errorMsg`, and `processedAt` columns from SessionImport -- **Removed** `processed` field from Session -- **Added** new `SessionProcessingStatus` table with granular stage tracking -- **Added** `ProcessingStage` and `ProcessingStatus` enums + +- **Removed** old `status`, `errorMsg`, and `processedAt` columns from SessionImport +- **Removed** `processed` field from Session +- **Added** new `SessionProcessingStatus` table with granular stage tracking +- **Added** `ProcessingStage` and `ProcessingStatus` enums ## New Processing Pipeline ### Processing Stages + ```typescript enum ProcessingStage { CSV_IMPORT // SessionImport created @@ -39,95 +42,107 @@ enum ProcessingStatus { ### Key Components #### 1. ProcessingStatusManager + Centralized class for managing processing status with methods: -- `initializeSession()` - Set up processing status for new sessions -- `startStage()`, `completeStage()`, `failStage()`, `skipStage()` - Stage management -- `getSessionsNeedingProcessing()` - Query sessions by stage and status -- `getPipelineStatus()` - Get overview of entire pipeline -- `getFailedSessions()` - Find sessions needing retry -- `resetStageForRetry()` - Reset failed stages + +- `initializeSession()` - Set up processing status for new sessions +- `startStage()`, `completeStage()`, `failStage()`, `skipStage()` - Stage management +- `getSessionsNeedingProcessing()` - Query sessions by stage and status +- `getPipelineStatus()` - Get overview of entire pipeline +- `getFailedSessions()` - Find sessions needing retry +- `resetStageForRetry()` - Reset failed stages #### 2. Updated Processing Scheduler -- Integrated with new `ProcessingStatusManager` -- Tracks AI analysis and question extraction stages -- Records detailed processing metadata -- Proper error handling and retry capabilities + +- Integrated with new `ProcessingStatusManager` +- Tracks AI analysis and question extraction stages +- Records detailed processing metadata +- Proper error handling and retry capabilities #### 3. Migration System -- Successfully migrated all 109 existing sessions -- Determined current state based on existing data -- Preserved all existing functionality + +- Successfully migrated all 109 existing sessions +- Determined current state based on existing data +- Preserved all existing functionality ## Current Pipeline Status After migration and refactoring: -- **CSV_IMPORT**: 109 completed -- **TRANSCRIPT_FETCH**: 109 completed -- **SESSION_CREATION**: 109 completed -- **AI_ANALYSIS**: 16 completed, 93 pending -- **QUESTION_EXTRACTION**: 11 completed, 98 pending + +- **CSV_IMPORT**: 109 completed +- **TRANSCRIPT_FETCH**: 109 completed +- **SESSION_CREATION**: 109 completed +- **AI_ANALYSIS**: 16 completed, 93 pending +- **QUESTION_EXTRACTION**: 11 completed, 98 pending ## Files Updated/Created ### New Files -- `lib/processingStatusManager.ts` - Core processing status management -- `check-refactored-pipeline-status.ts` - New pipeline status checker -- `migrate-to-refactored-system.ts` - Migration script -- `docs/processing-system-refactor.md` - This documentation + +- `lib/processingStatusManager.ts` - Core processing status management +- `check-refactored-pipeline-status.ts` - New pipeline status checker +- `migrate-to-refactored-system.ts` - Migration script +- `docs/processing-system-refactor.md` - This documentation ### Updated Files -- `prisma/schema.prisma` - Added new processing status tables -- `lib/processingScheduler.ts` - Integrated with new status system -- `debug-import-status.ts` - Updated to use new system -- `fix-import-status.ts` - Updated to use new system + +- `prisma/schema.prisma` - Added new processing status tables +- `lib/processingScheduler.ts` - Integrated with new status system +- `debug-import-status.ts` - Updated to use new system +- `fix-import-status.ts` - Updated to use new system ### Removed Files -- `check-pipeline-status.ts` - Replaced by refactored version + +- `check-pipeline-status.ts` - Replaced by refactored version ## Benefits Achieved -1. **Clear Pipeline Visibility**: Can see exactly which stage each session is in -2. **Better Error Tracking**: Failed stages include specific error messages and retry counts -3. **Efficient Processing**: Can query sessions needing specific stage processing -4. **Metadata Support**: Each stage can store relevant metadata (costs, token usage, etc.) -5. **Easy Recovery**: Failed sessions can be easily identified and retried -6. **Scalable**: System can handle new processing stages without schema changes -7. **No Linting Errors**: All TypeScript compilation issues resolved +1. **Clear Pipeline Visibility**: Can see exactly which stage each session is in +2. **Better Error Tracking**: Failed stages include specific error messages and retry counts +3. **Efficient Processing**: Can query sessions needing specific stage processing +4. **Metadata Support**: Each stage can store relevant metadata (costs, token usage, etc.) +5. **Easy Recovery**: Failed sessions can be easily identified and retried +6. **Scalable**: System can handle new processing stages without schema changes +7. **No Linting Errors**: All TypeScript compilation issues resolved ## Usage Examples ### Check Pipeline Status + ```bash npx tsx check-refactored-pipeline-status.ts ``` ### Debug Processing Issues + ```bash npx tsx debug-import-status.ts ``` ### Fix/Retry Failed Sessions + ```bash npx tsx fix-import-status.ts ``` ### Process Sessions + ```bash npx tsx test-ai-processing.ts ``` ## Next Steps -1. **Test AI Processing**: Run AI processing on pending sessions -2. **Monitor Performance**: Watch for any issues with the new system -3. **Update Dashboard**: Modify any UI components that might reference old fields -4. **Documentation**: Update any API documentation that references the old system +1. **Test AI Processing**: Run AI processing on pending sessions +2. **Monitor Performance**: Watch for any issues with the new system +3. **Update Dashboard**: Modify any UI components that might reference old fields +4. **Documentation**: Update any API documentation that references the old system ## Migration Notes -- All existing data preserved -- No data loss during migration -- Backward compatibility maintained where possible -- System ready for production use +- All existing data preserved +- No data loss during migration +- Backward compatibility maintained where possible +- System ready for production use The refactored system provides much better visibility into the processing pipeline and makes it easy to identify and resolve any issues that arise during session processing. diff --git a/docs/scheduler-architecture.md b/docs/scheduler-architecture.md index bba17dc..243079e 100644 --- a/docs/scheduler-architecture.md +++ b/docs/scheduler-architecture.md @@ -6,11 +6,11 @@ This document describes the extracted scheduler architecture that enables horizo The scheduler system has been refactored from a monolithic approach to a service-oriented architecture that supports: -- **Individual Scheduler Services** - Each scheduler runs as a separate service -- **Horizontal Scaling** - Multiple instances of the same scheduler can run across different machines -- **Health Monitoring** - Built-in health checks for load balancers and orchestrators -- **Graceful Shutdown** - Proper handling of shutdown signals for zero-downtime deployments -- **Centralized Management** - Optional scheduler manager for coordinated operations +- **Individual Scheduler Services** - Each scheduler runs as a separate service +- **Horizontal Scaling** - Multiple instances of the same scheduler can run across different machines +- **Health Monitoring** - Built-in health checks for load balancers and orchestrators +- **Graceful Shutdown** - Proper handling of shutdown signals for zero-downtime deployments +- **Centralized Management** - Optional scheduler manager for coordinated operations ## Components @@ -34,11 +34,11 @@ export abstract class BaseSchedulerService extends EventEmitter { **Features:** -- Status management (STOPPED, STARTING, RUNNING, PAUSED, ERROR) -- Metrics collection (run counts, timing, success/failure rates) -- Event emission for monitoring -- Configurable intervals and timeouts -- Automatic retry handling +- Status management (STOPPED, STARTING, RUNNING, PAUSED, ERROR) +- Metrics collection (run counts, timing, success/failure rates) +- Event emission for monitoring +- Configurable intervals and timeouts +- Automatic retry handling ### 2. Individual Scheduler Services @@ -57,16 +57,16 @@ const csvScheduler = new CsvImportSchedulerService({ **Features:** -- Batch processing with configurable concurrency -- Duplicate detection -- Company-specific error handling -- Progress monitoring +- Batch processing with configurable concurrency +- Duplicate detection +- Company-specific error handling +- Progress monitoring #### Additional Schedulers (To Be Implemented) -- `ImportProcessingSchedulerService` - Process imported CSV data into sessions -- `SessionProcessingSchedulerService` - AI analysis and categorization -- `BatchProcessingSchedulerService` - OpenAI Batch API integration +- `ImportProcessingSchedulerService` - Process imported CSV data into sessions +- `SessionProcessingSchedulerService` - AI analysis and categorization +- `BatchProcessingSchedulerService` - OpenAI Batch API integration ### 3. SchedulerManager @@ -88,10 +88,10 @@ await manager.startAll(); **Features:** -- Automatic restart of failed critical schedulers -- Health monitoring across all schedulers -- Coordinated start/stop operations -- Event aggregation and logging +- Automatic restart of failed critical schedulers +- Health monitoring across all schedulers +- Coordinated start/stop operations +- Event aggregation and logging ### 4. Standalone Scheduler Runner @@ -107,10 +107,10 @@ npx tsx lib/services/schedulers/StandaloneSchedulerRunner.ts --list **Features:** -- Independent process execution -- Environment variable configuration -- Graceful shutdown handling -- Health reporting for monitoring +- Independent process execution +- Environment variable configuration +- Graceful shutdown handling +- Health reporting for monitoring ## Deployment Patterns @@ -127,15 +127,15 @@ await initializeSchedulers(); **Pros:** -- Simple deployment -- Lower resource usage -- Easy local development +- Simple deployment +- Lower resource usage +- Easy local development **Cons:** -- Limited scalability -- Single point of failure -- Resource contention +- Limited scalability +- Single point of failure +- Resource contention ### 2. Separate Processes @@ -154,15 +154,15 @@ npm run scheduler:session-processing **Pros:** -- Independent scaling -- Fault isolation -- Resource optimization per scheduler +- Independent scaling +- Fault isolation +- Resource optimization per scheduler **Cons:** -- More complex deployment -- Higher resource overhead -- Inter-process coordination needed +- More complex deployment +- Higher resource overhead +- Inter-process coordination needed ### 3. Container Orchestration (Recommended for Production) @@ -193,16 +193,16 @@ services: **Pros:** -- Full horizontal scaling -- Independent resource allocation -- Health monitoring integration -- Zero-downtime deployments +- Full horizontal scaling +- Independent resource allocation +- Health monitoring integration +- Zero-downtime deployments **Cons:** -- Complex orchestration setup -- Network latency considerations -- Distributed system challenges +- Complex orchestration setup +- Network latency considerations +- Distributed system challenges ## Configuration @@ -379,46 +379,46 @@ csv-import-scheduler-eu: ### From Current Architecture -1. **Phase 1: Extract Schedulers** - - ✅ Create BaseSchedulerService - - ✅ Implement CsvImportSchedulerService - - ✅ Create SchedulerManager - - ⏳ Implement remaining scheduler services +1. **Phase 1: Extract Schedulers** + - ✅ Create BaseSchedulerService + - ✅ Implement CsvImportSchedulerService + - ✅ Create SchedulerManager + - ⏳ Implement remaining scheduler services -2. **Phase 2: Deployment Options** - - ✅ Add ServerSchedulerIntegration for backwards compatibility - - ✅ Create StandaloneSchedulerRunner - - ✅ Add health check endpoints +2. **Phase 2: Deployment Options** + - ✅ Add ServerSchedulerIntegration for backwards compatibility + - ✅ Create StandaloneSchedulerRunner + - ✅ Add health check endpoints -3. **Phase 3: Container Support** - - ⏳ Create Dockerfile for scheduler containers - - ⏳ Add Kubernetes manifests - - ⏳ Implement distributed coordination +3. **Phase 3: Container Support** + - ⏳ Create Dockerfile for scheduler containers + - ⏳ Add Kubernetes manifests + - ⏳ Implement distributed coordination -4. **Phase 4: Production Migration** - - ⏳ Deploy separate scheduler containers - - ⏳ Monitor performance and stability - - ⏳ Gradually increase horizontal scaling +4. **Phase 4: Production Migration** + - ⏳ Deploy separate scheduler containers + - ⏳ Monitor performance and stability + - ⏳ Gradually increase horizontal scaling ### Breaking Changes -- Scheduler initialization moved from `server.ts` to `ServerSchedulerIntegration` -- Individual scheduler functions replaced with service classes -- Configuration moved to environment variables +- Scheduler initialization moved from `server.ts` to `ServerSchedulerIntegration` +- Individual scheduler functions replaced with service classes +- Configuration moved to environment variables ## Benefits -1. **Scalability**: Independent scaling of different scheduler types -2. **Reliability**: Fault isolation prevents cascading failures -3. **Performance**: Optimized resource allocation per scheduler -4. **Monitoring**: Granular health checks and metrics -5. **Deployment**: Zero-downtime updates and rollbacks -6. **Development**: Easier testing and debugging of individual schedulers +1. **Scalability**: Independent scaling of different scheduler types +2. **Reliability**: Fault isolation prevents cascading failures +3. **Performance**: Optimized resource allocation per scheduler +4. **Monitoring**: Granular health checks and metrics +5. **Deployment**: Zero-downtime updates and rollbacks +6. **Development**: Easier testing and debugging of individual schedulers ## Next Steps -1. Implement remaining scheduler services (ImportProcessing, SessionProcessing, BatchProcessing) -2. Add distributed coordination for multi-instance schedulers -3. Create Kubernetes operators for automatic scaling -4. Implement scheduler-specific metrics and dashboards -5. Add scheduler performance optimization tools +1. Implement remaining scheduler services (ImportProcessing, SessionProcessing, BatchProcessing) +2. Add distributed coordination for multi-instance schedulers +3. Create Kubernetes operators for automatic scaling +4. Implement scheduler-specific metrics and dashboards +5. Add scheduler performance optimization tools diff --git a/docs/scheduler-fixes.md b/docs/scheduler-fixes.md index 0765625..aa628ef 100644 --- a/docs/scheduler-fixes.md +++ b/docs/scheduler-fixes.md @@ -8,8 +8,8 @@ **Solution**: -- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs -- Removed the invalid company record from the database using `fix_companies.js` +- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs +- Removed the invalid company record from the database using `fix_companies.js` ### 2. Transcript Fetching Errors @@ -17,10 +17,10 @@ **Solution**: -- Improved error handling in `fetchTranscriptContent()` function -- Added probabilistic logging (only ~10% of errors logged) to prevent log spam -- Added timeout (10 seconds) for transcript fetching -- Made transcript fetching failures non-blocking (sessions are still created without transcript content) +- Improved error handling in `fetchTranscriptContent()` function +- Added probabilistic logging (only ~10% of errors logged) to prevent log spam +- Added timeout (10 seconds) for transcript fetching +- Made transcript fetching failures non-blocking (sessions are still created without transcript content) ### 3. CSV Fetching Errors @@ -28,8 +28,8 @@ **Solution**: -- Added URL validation to skip companies with `example.com` URLs -- Improved error logging to be more descriptive +- Added URL validation to skip companies with `example.com` URLs +- Improved error logging to be more descriptive ## Current Status @@ -42,23 +42,23 @@ After cleanup, only valid companies remain: -- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`) - - CSV URL: `https://proto.notso.ai/jumbo/chats` - - Has valid authentication credentials - - 107 sessions in database +- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`) + - CSV URL: `https://proto.notso.ai/jumbo/chats` + - Has valid authentication credentials + - 107 sessions in database ## Files Modified 1. **lib/csvFetcher.js** -- Added company URL validation -- Improved transcript fetching error handling -- Reduced error log verbosity +- Added company URL validation +- Improved transcript fetching error handling +- Reduced error log verbosity 2. **fix_companies.js** (cleanup script) -- Removes invalid company records -- Can be run again if needed +- Removes invalid company records +- Can be run again if needed ## Monitoring diff --git a/docs/scheduler-workflow.md b/docs/scheduler-workflow.md index cc270b6..e8ca5ae 100644 --- a/docs/scheduler-workflow.md +++ b/docs/scheduler-workflow.md @@ -99,12 +99,14 @@ node scripts/manual-triggers.js both ``` 2. **If "Sessions with transcript" is 0:** - - Sessions exist but transcripts haven't been fetched yet - - Run session refresh: `node scripts/manual-triggers.js refresh` + +- Sessions exist but transcripts haven't been fetched yet +- Run session refresh: `node scripts/manual-triggers.js refresh` 3. **If "Ready for processing" is 0 but "Sessions with transcript" > 0:** - - All sessions with transcripts have already been processed - - Check if `OPENAI_API_KEY` is set in environment + +- All sessions with transcripts have already been processed +- Check if `OPENAI_API_KEY` is set in environment ### Common Issues diff --git a/docs/security-audit-logging.md b/docs/security-audit-logging.md index 372a914..bae56c9 100644 --- a/docs/security-audit-logging.md +++ b/docs/security-audit-logging.md @@ -12,59 +12,59 @@ The security audit logging system provides comprehensive tracking of security-cr The system logs the following event types: -- **Authentication Events**: Login attempts, password changes, session management -- **Authorization Events**: Permission checks, access denied events -- **User Management**: User creation, modification, deletion, invitations -- **Company Management**: Company suspension, settings changes -- **Rate Limiting**: Abuse prevention and rate limit violations -- **CSRF Protection**: Cross-site request forgery protection events -- **Security Headers**: Security header violations -- **Password Reset**: Password reset flows and token validation -- **Platform Admin**: Administrative activities by platform users -- **Data Privacy**: Data export and privacy-related events -- **System Configuration**: System setting changes -- **API Security**: API-related security events +- **Authentication Events**: Login attempts, password changes, session management +- **Authorization Events**: Permission checks, access denied events +- **User Management**: User creation, modification, deletion, invitations +- **Company Management**: Company suspension, settings changes +- **Rate Limiting**: Abuse prevention and rate limit violations +- **CSRF Protection**: Cross-site request forgery protection events +- **Security Headers**: Security header violations +- **Password Reset**: Password reset flows and token validation +- **Platform Admin**: Administrative activities by platform users +- **Data Privacy**: Data export and privacy-related events +- **System Configuration**: System setting changes +- **API Security**: API-related security events ### 2. Structured Logging Each audit log entry includes: -- **Event Type**: Categorizes the security event -- **Action**: Specific action performed -- **Outcome**: Success, failure, blocked, rate limited, or suspicious -- **Severity**: Info, low, medium, high, or critical -- **Context**: User ID, company ID, platform user ID, IP address, user agent -- **Metadata**: Structured additional information -- **Timestamp**: Immutable timestamp for chronological ordering +- **Event Type**: Categorizes the security event +- **Action**: Specific action performed +- **Outcome**: Success, failure, blocked, rate limited, or suspicious +- **Severity**: Info, low, medium, high, or critical +- **Context**: User ID, company ID, platform user ID, IP address, user agent +- **Metadata**: Structured additional information +- **Timestamp**: Immutable timestamp for chronological ordering ### 3. Multi-Tenant Security -- Company-scoped audit logs ensure data isolation -- Platform admin actions tracked separately -- Role-based access controls for audit log viewing +- Company-scoped audit logs ensure data isolation +- Platform admin actions tracked separately +- Role-based access controls for audit log viewing ### 4. Log Retention and Management -- **Configurable Retention Policies**: Different retention periods based on event type and severity -- **Automatic Archival**: Critical and high-severity events archived before deletion -- **Scheduled Cleanup**: Weekly automated retention policy execution -- **Manual Controls**: Admin interface for manual retention execution +- **Configurable Retention Policies**: Different retention periods based on event type and severity +- **Automatic Archival**: Critical and high-severity events archived before deletion +- **Scheduled Cleanup**: Weekly automated retention policy execution +- **Manual Controls**: Admin interface for manual retention execution ### 5. Administrative Interface -- **Audit Log Viewer**: Comprehensive filtering and search capabilities -- **Retention Management**: View statistics and execute retention policies -- **Real-time Monitoring**: Track security events as they occur +- **Audit Log Viewer**: Comprehensive filtering and search capabilities +- **Retention Management**: View statistics and execute retention policies +- **Real-time Monitoring**: Track security events as they occur ## Architecture ### Core Components -1. **SecurityAuditLogger** (`lib/securityAuditLogger.ts`): Centralized logging service -2. **AuditLogRetentionManager** (`lib/auditLogRetention.ts`): Retention policy management -3. **AuditLogScheduler** (`lib/auditLogScheduler.ts`): Scheduled retention execution -4. **Admin API** (`app/api/admin/audit-logs/`): REST API for audit log access -5. **Admin UI** (`app/dashboard/audit-logs/`): Administrative interface +1. **SecurityAuditLogger** (`lib/securityAuditLogger.ts`): Centralized logging service +2. **AuditLogRetentionManager** (`lib/auditLogRetention.ts`): Retention policy management +3. **AuditLogScheduler** (`lib/auditLogScheduler.ts`): Scheduled retention execution +4. **Admin API** (`app/api/admin/audit-logs/`): REST API for audit log access +5. **Admin UI** (`app/dashboard/audit-logs/`): Administrative interface ### Database Schema @@ -126,18 +126,18 @@ await securityAuditLogger.logAuthorization( Administrators can access audit logs through: -1. **Dashboard UI**: Navigate to "Audit Logs" in the sidebar -2. **API Access**: GET `/api/admin/audit-logs` with filtering parameters -3. **Retention Management**: GET/POST `/api/admin/audit-logs/retention` +1. **Dashboard UI**: Navigate to "Audit Logs" in the sidebar +2. **API Access**: GET `/api/admin/audit-logs` with filtering parameters +3. **Retention Management**: GET/POST `/api/admin/audit-logs/retention` ### Filtering Options -- Event type (authentication, authorization, etc.) -- Outcome (success, failure, blocked, etc.) -- Severity level (info, low, medium, high, critical) -- Date range -- User ID -- Pagination support +- Event type (authentication, authorization, etc.) +- Outcome (success, failure, blocked, etc.) +- Severity level (info, low, medium, high, critical) +- Date range +- User ID +- Pagination support ## Configuration @@ -159,105 +159,105 @@ AUDIT_LOG_RETENTION_DRY_RUN=false ### Default Retention Policies -1. **Critical Events**: 7 years retention with archival -2. **High Severity Events**: 3 years retention with archival -3. **Authentication Events**: 2 years retention with archival -4. **Platform Admin Events**: 3 years retention with archival -5. **User Management Events**: 2 years retention with archival -6. **General Events**: 1 year retention without archival +1. **Critical Events**: 7 years retention with archival +2. **High Severity Events**: 3 years retention with archival +3. **Authentication Events**: 2 years retention with archival +4. **Platform Admin Events**: 3 years retention with archival +5. **User Management Events**: 2 years retention with archival +6. **General Events**: 1 year retention without archival ## Security Considerations ### Data Protection -- **IP Address Storage**: Client IP addresses stored for geographic analysis -- **Sensitive Data Redaction**: Passwords, tokens, and emails marked as `[REDACTED]` -- **Metadata Sanitization**: Complex objects sanitized to prevent data leakage +- **IP Address Storage**: Client IP addresses stored for geographic analysis +- **Sensitive Data Redaction**: Passwords, tokens, and emails marked as `[REDACTED]` +- **Metadata Sanitization**: Complex objects sanitized to prevent data leakage ### Access Controls -- **Admin-Only Access**: Only users with `ADMIN` role can view audit logs -- **Company Isolation**: Users can only view logs for their own company -- **Platform Separation**: Platform admin logs tracked separately +- **Admin-Only Access**: Only users with `ADMIN` role can view audit logs +- **Company Isolation**: Users can only view logs for their own company +- **Platform Separation**: Platform admin logs tracked separately ### Performance -- **Async Logging**: All logging operations are asynchronous to avoid blocking -- **Error Handling**: Logging failures don't affect application functionality -- **Indexed Queries**: Database indexes optimize common query patterns -- **Batch Operations**: Retention policies use batch operations for efficiency +- **Async Logging**: All logging operations are asynchronous to avoid blocking +- **Error Handling**: Logging failures don't affect application functionality +- **Indexed Queries**: Database indexes optimize common query patterns +- **Batch Operations**: Retention policies use batch operations for efficiency ## Compliance Features ### Audit Standards -- **Immutable Records**: Audit logs cannot be modified after creation -- **Chronological Ordering**: Precise timestamps for event sequencing -- **Non-Repudiation**: User actions clearly attributed and timestamped -- **Comprehensive Coverage**: All security-relevant events logged +- **Immutable Records**: Audit logs cannot be modified after creation +- **Chronological Ordering**: Precise timestamps for event sequencing +- **Non-Repudiation**: User actions clearly attributed and timestamped +- **Comprehensive Coverage**: All security-relevant events logged ### Reporting -- **Event Statistics**: Summary statistics by event type, severity, and time period -- **Export Capabilities**: Structured data export for compliance reporting -- **Retention Tracking**: Detailed logging of retention policy execution +- **Event Statistics**: Summary statistics by event type, severity, and time period +- **Export Capabilities**: Structured data export for compliance reporting +- **Retention Tracking**: Detailed logging of retention policy execution ## Monitoring and Alerting ### System Health -- **Scheduler Status**: Monitor retention scheduler health -- **Error Tracking**: Log retention and audit logging errors -- **Performance Metrics**: Track logging performance and database impact +- **Scheduler Status**: Monitor retention scheduler health +- **Error Tracking**: Log retention and audit logging errors +- **Performance Metrics**: Track logging performance and database impact ### Security Monitoring -- **Failed Authentication Patterns**: Track repeated login failures -- **Privilege Escalation**: Monitor administrative action patterns -- **Suspicious Activity**: Identify unusual access patterns +- **Failed Authentication Patterns**: Track repeated login failures +- **Privilege Escalation**: Monitor administrative action patterns +- **Suspicious Activity**: Identify unusual access patterns ## Troubleshooting ### Common Issues -1. **Audit Logging Disabled**: Check `AUDIT_LOGGING_ENABLED` environment variable -2. **Retention Not Running**: Verify `AUDIT_LOG_RETENTION_ENABLED` and scheduler status -3. **Access Denied**: Ensure user has `ADMIN` role for audit log access -4. **Performance Issues**: Review retention policies and database indexes +1. **Audit Logging Disabled**: Check `AUDIT_LOGGING_ENABLED` environment variable +2. **Retention Not Running**: Verify `AUDIT_LOG_RETENTION_ENABLED` and scheduler status +3. **Access Denied**: Ensure user has `ADMIN` role for audit log access +4. **Performance Issues**: Review retention policies and database indexes ### Debug Information -- Check application logs for scheduler startup messages -- Monitor database query performance for audit log operations -- Review retention policy validation warnings +- Check application logs for scheduler startup messages +- Monitor database query performance for audit log operations +- Review retention policy validation warnings ## Best Practices ### Implementation -1. **Always use the centralized logger**: Don't bypass the `securityAuditLogger` -2. **Include relevant context**: Provide user, company, and IP information -3. **Use appropriate severity levels**: Follow the severity assignment guidelines -4. **Sanitize sensitive data**: Use `createAuditMetadata()` for safe metadata +1. **Always use the centralized logger**: Don't bypass the `securityAuditLogger` +2. **Include relevant context**: Provide user, company, and IP information +3. **Use appropriate severity levels**: Follow the severity assignment guidelines +4. **Sanitize sensitive data**: Use `createAuditMetadata()` for safe metadata ### Operations -1. **Regular retention review**: Monitor retention statistics and adjust policies -2. **Archive critical data**: Ensure important logs are archived before deletion -3. **Monitor storage usage**: Track audit log database growth -4. **Test restoration**: Verify archived data can be restored when needed +1. **Regular retention review**: Monitor retention statistics and adjust policies +2. **Archive critical data**: Ensure important logs are archived before deletion +3. **Monitor storage usage**: Track audit log database growth +4. **Test restoration**: Verify archived data can be restored when needed ## Future Enhancements ### Planned Features -- **Real-time Alerting**: Immediate notifications for critical security events -- **Advanced Analytics**: ML-based anomaly detection and pattern recognition -- **Export Formats**: Additional export formats for compliance reporting -- **External Integration**: SIEM and security tool integrations +- **Real-time Alerting**: Immediate notifications for critical security events +- **Advanced Analytics**: ML-based anomaly detection and pattern recognition +- **Export Formats**: Additional export formats for compliance reporting +- **External Integration**: SIEM and security tool integrations ### Performance Optimizations -- **Log Partitioning**: Database partitioning for improved query performance -- **Compression**: Log compression for storage efficiency -- **Streaming**: Real-time log streaming for external systems +- **Log Partitioning**: Database partitioning for improved query performance +- **Compression**: Log compression for storage efficiency +- **Streaming**: Real-time log streaming for external systems diff --git a/docs/security-headers.md b/docs/security-headers.md index 787d8f5..cd8cde0 100644 --- a/docs/security-headers.md +++ b/docs/security-headers.md @@ -12,33 +12,33 @@ The application implements multiple layers of HTTP security headers to provide d #### X-Content-Type-Options: nosniff -- **Purpose**: Prevents MIME type sniffing attacks -- **Protection**: Stops browsers from interpreting files as different MIME types than declared -- **Value**: `nosniff` +- **Purpose**: Prevents MIME type sniffing attacks +- **Protection**: Stops browsers from interpreting files as different MIME types than declared +- **Value**: `nosniff` #### X-Frame-Options: DENY -- **Purpose**: Prevents clickjacking attacks -- **Protection**: Blocks embedding the site in frames/iframes -- **Value**: `DENY` +- **Purpose**: Prevents clickjacking attacks +- **Protection**: Blocks embedding the site in frames/iframes +- **Value**: `DENY` #### X-XSS-Protection: 1; mode=block -- **Purpose**: Enables XSS protection in legacy browsers -- **Protection**: Activates built-in XSS filtering (primarily for older browsers) -- **Value**: `1; mode=block` +- **Purpose**: Enables XSS protection in legacy browsers +- **Protection**: Activates built-in XSS filtering (primarily for older browsers) +- **Value**: `1; mode=block` #### Referrer-Policy: strict-origin-when-cross-origin -- **Purpose**: Controls referrer information leakage -- **Protection**: Limits referrer data sent to external sites -- **Value**: `strict-origin-when-cross-origin` +- **Purpose**: Controls referrer information leakage +- **Protection**: Limits referrer data sent to external sites +- **Value**: `strict-origin-when-cross-origin` #### X-DNS-Prefetch-Control: off -- **Purpose**: Prevents DNS rebinding attacks -- **Protection**: Disables DNS prefetching to reduce attack surface -- **Value**: `off` +- **Purpose**: Prevents DNS rebinding attacks +- **Protection**: Disables DNS prefetching to reduce attack surface +- **Value**: `off` ### Content Security Policy (CSP) @@ -48,15 +48,15 @@ Comprehensive CSP implementation with the following directives: Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self' https:; frame-ancestors 'none'; base-uri 'self'; form-action 'self'; object-src 'none'; upgrade-insecure-requests ``` -#### Key CSP Directives: +#### Key CSP Directives -- **default-src 'self'**: Restrictive default for all resource types -- **script-src 'self' 'unsafe-eval' 'unsafe-inline'**: Allows Next.js dev tools and React functionality -- **style-src 'self' 'unsafe-inline'**: Enables TailwindCSS and component styles -- **img-src 'self' data: https:**: Allows secure image sources -- **frame-ancestors 'none'**: Prevents embedding (reinforces X-Frame-Options) -- **object-src 'none'**: Blocks dangerous plugins and embeds -- **upgrade-insecure-requests**: Automatically upgrades HTTP to HTTPS +- **default-src 'self'**: Restrictive default for all resource types +- **script-src 'self' 'unsafe-eval' 'unsafe-inline'**: Allows Next.js dev tools and React functionality +- **style-src 'self' 'unsafe-inline'**: Enables TailwindCSS and component styles +- **img-src 'self' data: https:**: Allows secure image sources +- **frame-ancestors 'none'**: Prevents embedding (reinforces X-Frame-Options) +- **object-src 'none'**: Blocks dangerous plugins and embeds +- **upgrade-insecure-requests**: Automatically upgrades HTTP to HTTPS ### Permissions Policy @@ -66,19 +66,19 @@ Controls browser feature access: Permissions-Policy: camera=(), microphone=(), geolocation=(), interest-cohort=(), browsing-topics=() ``` -- **camera=()**: Disables camera access -- **microphone=()**: Disables microphone access -- **geolocation=()**: Disables location tracking -- **interest-cohort=()**: Blocks FLoC (privacy protection) -- **browsing-topics=()**: Blocks Topics API (privacy protection) +- **camera=()**: Disables camera access +- **microphone=()**: Disables microphone access +- **geolocation=()**: Disables location tracking +- **interest-cohort=()**: Blocks FLoC (privacy protection) +- **browsing-topics=()**: Blocks Topics API (privacy protection) ### Strict Transport Security (HSTS) **Production Only**: `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload` -- **max-age=31536000**: 1 year HSTS policy -- **includeSubDomains**: Applies to all subdomains -- **preload**: Ready for HSTS preload list inclusion +- **max-age=31536000**: 1 year HSTS policy +- **includeSubDomains**: Applies to all subdomains +- **preload**: Ready for HSTS preload list inclusion ## Configuration @@ -110,8 +110,8 @@ headers: async () => { ### Environment-Specific Behavior -- **Development**: All headers except HSTS -- **Production**: All headers including HSTS +- **Development**: All headers except HSTS +- **Production**: All headers including HSTS ## Testing @@ -121,11 +121,11 @@ Location: `tests/unit/http-security-headers.test.ts` Tests cover: -- Individual header validation -- CSP directive verification -- Permissions Policy validation -- Environment-specific configuration -- Next.js compatibility checks +- Individual header validation +- CSP directive verification +- Permissions Policy validation +- Environment-specific configuration +- Next.js compatibility checks ### Integration Tests @@ -133,9 +133,9 @@ Location: `tests/integration/security-headers-basic.test.ts` Tests cover: -- Next.js configuration validation -- Header generation verification -- Environment-based header differences +- Next.js configuration validation +- Header generation verification +- Environment-based header differences ### Manual Testing @@ -153,41 +153,41 @@ pnpm test:security-headers https://your-domain.com ### Protection Against OWASP Top 10 -1. **A03:2021 - Injection**: CSP prevents script injection -2. **A05:2021 - Security Misconfiguration**: Comprehensive headers reduce attack surface -3. **A06:2021 - Vulnerable Components**: CSP limits execution context -4. **A07:2021 - Identification and Authentication Failures**: HSTS prevents downgrade attacks +1. **A03:2021 - Injection**: CSP prevents script injection +2. **A05:2021 - Security Misconfiguration**: Comprehensive headers reduce attack surface +3. **A06:2021 - Vulnerable Components**: CSP limits execution context +4. **A07:2021 - Identification and Authentication Failures**: HSTS prevents downgrade attacks ### Additional Security Benefits -- **Clickjacking Protection**: X-Frame-Options + CSP frame-ancestors -- **MIME Sniffing Prevention**: X-Content-Type-Options -- **Information Leakage Reduction**: Referrer-Policy -- **Privacy Protection**: Permissions Policy restrictions -- **Transport Security**: HSTS enforcement +- **Clickjacking Protection**: X-Frame-Options + CSP frame-ancestors +- **MIME Sniffing Prevention**: X-Content-Type-Options +- **Information Leakage Reduction**: Referrer-Policy +- **Privacy Protection**: Permissions Policy restrictions +- **Transport Security**: HSTS enforcement ## Maintenance ### Regular Reviews -1. **Quarterly CSP Review**: Analyze CSP violations and tighten policies -2. **Annual Header Audit**: Review new security headers and standards -3. **Dependency Updates**: Ensure compatibility with framework updates +1. **Quarterly CSP Review**: Analyze CSP violations and tighten policies +2. **Annual Header Audit**: Review new security headers and standards +3. **Dependency Updates**: Ensure compatibility with framework updates ### Monitoring -- Monitor CSP violation reports (when implemented) -- Use online tools like securityheaders.com for validation -- Include security header tests in CI/CD pipeline +- Monitor CSP violation reports (when implemented) +- Use online tools like securityheaders.com for validation +- Include security header tests in CI/CD pipeline ### Future Enhancements Planned improvements: -1. CSP violation reporting endpoint -2. Nonce-based CSP for inline scripts -3. Additional Permissions Policy restrictions -4. Content-Type validation middleware +1. CSP violation reporting endpoint +2. Nonce-based CSP for inline scripts +3. Additional Permissions Policy restrictions +4. Content-Type validation middleware ## Compatibility @@ -195,37 +195,37 @@ Planned improvements: Headers are configured to be compatible with: -- Next.js 15+ App Router -- React 19 development tools -- TailwindCSS 4 styling system -- Development hot reload functionality +- Next.js 15+ App Router +- React 19 development tools +- TailwindCSS 4 styling system +- Development hot reload functionality ### Browser Support Security headers are supported by: -- All modern browsers (Chrome 60+, Firefox 60+, Safari 12+) -- Graceful degradation for older browsers -- Progressive enhancement approach +- All modern browsers (Chrome 60+, Firefox 60+, Safari 12+) +- Graceful degradation for older browsers +- Progressive enhancement approach ## Troubleshooting ### Common Issues -1. **CSP Violations**: Check browser console for CSP errors -2. **Styling Issues**: Verify style-src allows 'unsafe-inline' -3. **Script Errors**: Ensure script-src permits necessary scripts -4. **Development Issues**: Use `pnpm dev:next-only` to isolate Next.js +1. **CSP Violations**: Check browser console for CSP errors +2. **Styling Issues**: Verify style-src allows 'unsafe-inline' +3. **Script Errors**: Ensure script-src permits necessary scripts +4. **Development Issues**: Use `pnpm dev:next-only` to isolate Next.js ### Debug Tools -- Browser DevTools Security tab -- CSP Evaluator: https://csp-evaluator.withgoogle.com/ -- Security Headers Scanner: https://securityheaders.com/ +- Browser DevTools Security tab +- CSP Evaluator: +- Security Headers Scanner: ## References -- [OWASP Secure Headers Project](https://owasp.org/www-project-secure-headers/) -- [MDN Security Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#security) -- [Next.js Security Headers](https://nextjs.org/docs/app/api-reference/config/headers) -- [Content Security Policy Reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) +- [OWASP Secure Headers Project](https://owasp.org/www-project-secure-headers/) +- [MDN Security Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#security) +- [Next.js Security Headers](https://nextjs.org/docs/app/api-reference/config/headers) +- [Content Security Policy Reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) diff --git a/docs/security-monitoring.md b/docs/security-monitoring.md index 9724530..51fbcba 100644 --- a/docs/security-monitoring.md +++ b/docs/security-monitoring.md @@ -8,39 +8,39 @@ The Security Monitoring and Alerting System provides comprehensive real-time sec ### Core Components -1. **Security Monitoring Service** (`lib/securityMonitoring.ts`) - - Real-time event processing - - Anomaly detection algorithms - - Alert generation and management - - Security score calculation - - Threat level assessment +1. **Security Monitoring Service** (`lib/securityMonitoring.ts`) + - Real-time event processing + - Anomaly detection algorithms + - Alert generation and management + - Security score calculation + - Threat level assessment -2. **Enhanced Security Logging** (`enhancedSecurityLog`) - - Integrates with existing audit logger - - Processes events through monitoring system - - Triggers immediate threat detection +2. **Enhanced Security Logging** (`enhancedSecurityLog`) + - Integrates with existing audit logger + - Processes events through monitoring system + - Triggers immediate threat detection -3. **API Endpoints** (`app/api/admin/security-monitoring/`) - - `/api/admin/security-monitoring` - Main metrics and configuration - - `/api/admin/security-monitoring/alerts` - Alert management - - `/api/admin/security-monitoring/export` - Data export - - `/api/admin/security-monitoring/threat-analysis` - Threat analysis +3. **API Endpoints** (`app/api/admin/security-monitoring/`) + - `/api/admin/security-monitoring` - Main metrics and configuration + - `/api/admin/security-monitoring/alerts` - Alert management + - `/api/admin/security-monitoring/export` - Data export + - `/api/admin/security-monitoring/threat-analysis` - Threat analysis -4. **Dashboard UI** (`app/platform/security/page.tsx`) - - Real-time security metrics - - Active alerts management - - Threat analysis visualization - - Configuration management +4. **Dashboard UI** (`app/platform/security/page.tsx`) + - Real-time security metrics + - Active alerts management + - Threat analysis visualization + - Configuration management ## Features ### Real-time Monitoring -- **Authentication Events**: Login attempts, failures, brute force attacks -- **Rate Limiting**: Excessive request patterns, API abuse -- **Admin Activity**: Unusual administrative actions -- **Geographic Anomalies**: Logins from unusual locations -- **Temporal Anomalies**: Activity spikes outside normal patterns +- **Authentication Events**: Login attempts, failures, brute force attacks +- **Rate Limiting**: Excessive request patterns, API abuse +- **Admin Activity**: Unusual administrative actions +- **Geographic Anomalies**: Logins from unusual locations +- **Temporal Anomalies**: Activity spikes outside normal patterns ### Alert Types @@ -68,30 +68,30 @@ enum AlertType { The system implements several anomaly detection algorithms: -1. **Geographic Anomaly Detection** - - Detects logins from unusual countries - - Compares against historical user patterns - - Confidence scoring based on deviation +1. **Geographic Anomaly Detection** + - Detects logins from unusual countries + - Compares against historical user patterns + - Confidence scoring based on deviation -2. **Temporal Anomaly Detection** - - Identifies activity spikes during unusual hours - - Compares current activity to historical averages - - Configurable thresholds for different event types +2. **Temporal Anomaly Detection** + - Identifies activity spikes during unusual hours + - Compares current activity to historical averages + - Configurable thresholds for different event types -3. **Behavioral Anomaly Detection** - - Multiple failed login attempts - - Rapid succession of actions - - Pattern deviation analysis +3. **Behavioral Anomaly Detection** + - Multiple failed login attempts + - Rapid succession of actions + - Pattern deviation analysis ### Security Scoring The system calculates a real-time security score (0-100) based on: -- Critical security events (weight: 25) -- Active unresolved alerts (weight: 30) -- High-severity threats (weight: 20) -- Overall event volume (weight: 15) -- System stability factors (weight: 10) +- Critical security events (weight: 25) +- Active unresolved alerts (weight: 30) +- High-severity threats (weight: 20) +- Overall event volume (weight: 15) +- System stability factors (weight: 10) ### Threat Levels @@ -255,117 +255,117 @@ await enhancedSecurityLog( ### Security Overview -- Real-time security score (0-100) -- Current threat level indicator -- Active alerts count -- Security events summary +- Real-time security score (0-100) +- Current threat level indicator +- Active alerts count +- Security events summary ### Alert Management -- View active and historical alerts -- Filter by severity and type -- Acknowledge alerts with tracking -- Detailed alert context and metadata +- View active and historical alerts +- Filter by severity and type +- Acknowledge alerts with tracking +- Detailed alert context and metadata ### Threat Analysis -- Geographic distribution of events -- Top threat types and patterns -- User risk scoring -- IP threat level analysis +- Geographic distribution of events +- Top threat types and patterns +- User risk scoring +- IP threat level analysis ### Configuration Management -- Adjust detection thresholds -- Configure alerting channels -- Set data retention policies -- Export capabilities +- Adjust detection thresholds +- Configure alerting channels +- Set data retention policies +- Export capabilities ## Performance Considerations ### Memory Management -- Event buffer limited to 1 hour of data -- Automatic cleanup of old alerts (configurable) -- Efficient in-memory storage for real-time analysis +- Event buffer limited to 1 hour of data +- Automatic cleanup of old alerts (configurable) +- Efficient in-memory storage for real-time analysis ### Database Impact -- Leverages existing audit log indexes -- Optimized queries for time-range filtering -- Background processing to avoid blocking operations +- Leverages existing audit log indexes +- Optimized queries for time-range filtering +- Background processing to avoid blocking operations ### Scalability -- Stateless architecture (except for buffering) -- Horizontal scaling support -- Configurable processing intervals +- Stateless architecture (except for buffering) +- Horizontal scaling support +- Configurable processing intervals ## Security Considerations ### Access Control -- Platform admin authentication required -- Role-based access to security endpoints -- Audit logging of all monitoring activities +- Platform admin authentication required +- Role-based access to security endpoints +- Audit logging of all monitoring activities ### Data Privacy -- Sensitive data redaction in logs -- IP address anonymization options -- Configurable data retention periods +- Sensitive data redaction in logs +- IP address anonymization options +- Configurable data retention periods ### Alert Suppression -- Duplicate alert suppression (configurable window) -- Rate limiting on alert generation -- Escalation policies for critical threats +- Duplicate alert suppression (configurable window) +- Rate limiting on alert generation +- Escalation policies for critical threats ## Monitoring and Maintenance ### Health Checks -- Monitor service availability -- Check alert generation pipeline -- Verify data export functionality +- Monitor service availability +- Check alert generation pipeline +- Verify data export functionality ### Regular Tasks -- Review and adjust thresholds quarterly -- Analyze false positive rates -- Update threat detection patterns -- Clean up old alert data +- Review and adjust thresholds quarterly +- Analyze false positive rates +- Update threat detection patterns +- Clean up old alert data ### Performance Metrics -- Alert response time -- False positive/negative rates -- System resource usage -- User engagement with alerts +- Alert response time +- False positive/negative rates +- System resource usage +- User engagement with alerts ## Future Enhancements ### Planned Features -1. **Machine Learning Integration** - - Behavioral pattern recognition - - Adaptive threshold adjustment - - Predictive threat modeling +1. **Machine Learning Integration** + - Behavioral pattern recognition + - Adaptive threshold adjustment + - Predictive threat modeling -2. **Advanced Analytics** - - Threat intelligence integration - - Cross-correlation analysis - - Risk trend analysis +2. **Advanced Analytics** + - Threat intelligence integration + - Cross-correlation analysis + - Risk trend analysis -3. **Integration Enhancements** - - SIEM system connectors - - Webhook customization - - Mobile app notifications +3. **Integration Enhancements** + - SIEM system connectors + - Webhook customization + - Mobile app notifications -4. **Automated Response** - - IP blocking automation - - Account suspension workflows - - Incident response orchestration +4. **Automated Response** + - IP blocking automation + - Account suspension workflows + - Incident response orchestration ## Troubleshooting @@ -373,27 +373,27 @@ await enhancedSecurityLog( **High False Positive Rate** -- Review and adjust detection thresholds -- Analyze user behavior patterns -- Consider geographical variations +- Review and adjust detection thresholds +- Analyze user behavior patterns +- Consider geographical variations **Missing Alerts** -- Check service configuration -- Verify audit log integration -- Review threshold settings +- Check service configuration +- Verify audit log integration +- Review threshold settings **Performance Issues** -- Monitor memory usage -- Adjust cleanup intervals -- Optimize database queries +- Monitor memory usage +- Adjust cleanup intervals +- Optimize database queries **Export Failures** -- Check file permissions -- Verify date range validity -- Monitor server resources +- Check file permissions +- Verify date range validity +- Monitor server resources ### Debugging @@ -419,24 +419,24 @@ console.log("Active alerts:", alerts.length); ### Unit Tests -- Alert generation logic -- Anomaly detection algorithms -- Configuration management -- Data export functionality +- Alert generation logic +- Anomaly detection algorithms +- Configuration management +- Data export functionality ### Integration Tests -- API endpoint security -- Database integration -- Real-time event processing -- Alert acknowledgment flow +- API endpoint security +- Database integration +- Real-time event processing +- Alert acknowledgment flow ### Load Testing -- High-volume event processing -- Concurrent alert generation -- Database performance under load -- Memory usage patterns +- High-volume event processing +- Concurrent alert generation +- Database performance under load +- Memory usage patterns Run tests: diff --git a/docs/security/enhanced-csp.md b/docs/security/enhanced-csp.md index b28e65a..673a098 100644 --- a/docs/security/enhanced-csp.md +++ b/docs/security/enhanced-csp.md @@ -8,54 +8,54 @@ This document outlines the comprehensive Content Security Policy implementation The enhanced CSP implementation provides: -- **Nonce-based script execution** for maximum security in production -- **Strict mode policies** with configurable external domain allowlists -- **Environment-specific configurations** for development vs production -- **CSP violation reporting and monitoring** system with real-time analysis -- **Advanced bypass detection and alerting** capabilities with risk assessment -- **Comprehensive testing framework** with automated validation -- **Performance metrics and policy recommendations** -- **Framework compatibility** with Next.js, TailwindCSS, and Leaflet maps +- **Nonce-based script execution** for maximum security in production +- **Strict mode policies** with configurable external domain allowlists +- **Environment-specific configurations** for development vs production +- **CSP violation reporting and monitoring** system with real-time analysis +- **Advanced bypass detection and alerting** capabilities with risk assessment +- **Comprehensive testing framework** with automated validation +- **Performance metrics and policy recommendations** +- **Framework compatibility** with Next.js, TailwindCSS, and Leaflet maps ## Architecture ### Core Components -1. **CSP Utility Library** (`lib/csp.ts`) - - Nonce generation with cryptographic security - - Dynamic CSP building based on environment - - Violation parsing and bypass detection - - Policy validation and testing +1. **CSP Utility Library** (`lib/csp.ts`) + - Nonce generation with cryptographic security + - Dynamic CSP building based on environment + - Violation parsing and bypass detection + - Policy validation and testing -2. **Middleware Implementation** (`middleware.ts`) - - Automatic nonce generation per request - - Environment-aware policy application - - Enhanced security headers - - Route-based CSP filtering +2. **Middleware Implementation** (`middleware.ts`) + - Automatic nonce generation per request + - Environment-aware policy application + - Enhanced security headers + - Route-based CSP filtering -3. **Violation Reporting** (`app/api/csp-report/route.ts`) - - Real-time violation monitoring with intelligent analysis - - Rate-limited endpoint protection (10 reports/minute per IP) - - Advanced bypass attempt detection with risk assessment - - Automated alerting for critical violations with recommendations +3. **Violation Reporting** (`app/api/csp-report/route.ts`) + - Real-time violation monitoring with intelligent analysis + - Rate-limited endpoint protection (10 reports/minute per IP) + - Advanced bypass attempt detection with risk assessment + - Automated alerting for critical violations with recommendations -4. **Monitoring Service** (`lib/csp-monitoring.ts`) - - Violation tracking and metrics collection - - Policy recommendation engine based on violation patterns - - Export capabilities for external analysis (JSON/CSV) - - Automatic cleanup of old violation data +4. **Monitoring Service** (`lib/csp-monitoring.ts`) + - Violation tracking and metrics collection + - Policy recommendation engine based on violation patterns + - Export capabilities for external analysis (JSON/CSV) + - Automatic cleanup of old violation data -5. **Metrics API** (`app/api/csp-metrics/route.ts`) - - Real-time CSP violation metrics (1h, 6h, 24h, 7d, 30d ranges) - - Top violated directives and blocked URIs analysis - - Violation trend tracking and visualization data - - Policy optimization recommendations +5. **Metrics API** (`app/api/csp-metrics/route.ts`) + - Real-time CSP violation metrics (1h, 6h, 24h, 7d, 30d ranges) + - Top violated directives and blocked URIs analysis + - Violation trend tracking and visualization data + - Policy optimization recommendations -6. **Testing Framework** - - Comprehensive unit and integration tests - - Enhanced CSP validation tools with security scoring - - Automated compliance verification - - Real-world scenario testing for application compatibility +6. **Testing Framework** + - Comprehensive unit and integration tests + - Enhanced CSP validation tools with security scoring + - Automated compliance verification + - Real-world scenario testing for application compatibility ## CSP Policies @@ -118,9 +118,9 @@ const developmentCSP = { ### 1. Nonce-Based Script Execution -- **128-bit cryptographically secure nonces** generated per request -- **Strict-dynamic policy** prevents inline script execution -- **Automatic nonce injection** into layout components +- **128-bit cryptographically secure nonces** generated per request +- **Strict-dynamic policy** prevents inline script execution +- **Automatic nonce injection** into layout components ```tsx // Layout with nonce support @@ -149,27 +149,32 @@ export default async function RootLayout({ children }: { children: ReactNode }) ### 2. Content Source Restrictions #### Script Sources -- **Production**: Only `'self'` and nonce-approved scripts -- **Development**: Additional `'unsafe-eval'` for dev tools -- **Blocked**: All external CDNs, inline scripts without nonce + +- **Production**: Only `'self'` and nonce-approved scripts +- **Development**: Additional `'unsafe-eval'` for dev tools +- **Blocked**: All external CDNs, inline scripts without nonce #### Style Sources -- **Production**: Nonce-based inline styles preferred -- **Fallback**: `'unsafe-inline'` for TailwindCSS compatibility -- **External**: Only self-hosted stylesheets + +- **Production**: Nonce-based inline styles preferred +- **Fallback**: `'unsafe-inline'` for TailwindCSS compatibility +- **External**: Only self-hosted stylesheets #### Image Sources -- **Allowed**: Self, data URIs, schema.org, application domain -- **Blocked**: All other external domains + +- **Allowed**: Self, data URIs, schema.org, application domain +- **Blocked**: All other external domains #### Connection Sources -- **Production**: Self, OpenAI API, application domain -- **Development**: Additional WebSocket for HMR -- **Blocked**: All other external connections + +- **Production**: Self, OpenAI API, application domain +- **Development**: Additional WebSocket for HMR +- **Blocked**: All other external connections ### 3. XSS Protection Mechanisms #### Inline Script Prevention + ```javascript // Blocked by CSP @@ -179,18 +184,21 @@ export default async function RootLayout({ children }: { children: ReactNode }) ``` #### Object Injection Prevention + ```javascript // Completely blocked object-src 'none' ``` #### Base Tag Injection Prevention + ```javascript // Restricted to same origin base-uri 'self' ``` #### Clickjacking Protection + ```javascript // No framing allowed frame-ancestors 'none' @@ -230,20 +238,21 @@ CSP violations are automatically reported to `/api/csp-report`: ### Violation Processing -1. **Rate Limiting**: 10 reports per minute per IP -2. **Parsing**: Extract violation details and context -3. **Risk Assessment**: Classify as low/medium/high risk -4. **Bypass Detection**: Check for known attack patterns -5. **Alerting**: Immediate notifications for critical violations +1. **Rate Limiting**: 10 reports per minute per IP +2. **Parsing**: Extract violation details and context +3. **Risk Assessment**: Classify as low/medium/high risk +4. **Bypass Detection**: Check for known attack patterns +5. **Alerting**: Immediate notifications for critical violations ### Monitoring Dashboard Violations are logged with: -- Timestamp and source IP -- User agent and referer -- Violation type and blocked content -- Risk level and bypass indicators -- Response actions taken + +- Timestamp and source IP +- User agent and referer +- Violation type and blocked content +- Risk level and bypass indicators +- Response actions taken ## Testing and Validation @@ -262,19 +271,20 @@ pnpm test:csp:full ### Manual Testing -1. **Nonce Validation**: Verify unique nonces per request -2. **Policy Compliance**: Check all required directives -3. **Bypass Resistance**: Test common XSS techniques -4. **Framework Compatibility**: Ensure Next.js/TailwindCSS work -5. **Performance Impact**: Measure overhead +1. **Nonce Validation**: Verify unique nonces per request +2. **Policy Compliance**: Check all required directives +3. **Bypass Resistance**: Test common XSS techniques +4. **Framework Compatibility**: Ensure Next.js/TailwindCSS work +5. **Performance Impact**: Measure overhead ### Security Scoring The validation framework provides a security score: -- **90-100%**: Excellent implementation -- **80-89%**: Good with minor improvements needed -- **70-79%**: Needs attention -- **<70%**: Serious security issues + +- **90-100%**: Excellent implementation +- **80-89%**: Good with minor improvements needed +- **70-79%**: Needs attention +- **<70%**: Serious security issues ## Deployment Considerations @@ -288,74 +298,74 @@ NODE_ENV=development # Enables permissive CSP ### Performance Impact -- **Nonce generation**: ~0.1ms per request -- **Header processing**: ~0.05ms per request -- **Total overhead**: <1ms per request +- **Nonce generation**: ~0.1ms per request +- **Header processing**: ~0.05ms per request +- **Total overhead**: <1ms per request ### Browser Compatibility -- **Modern browsers**: Full CSP Level 3 support -- **Legacy browsers**: Graceful degradation with X-XSS-Protection -- **Reporting**: Supported in all major browsers +- **Modern browsers**: Full CSP Level 3 support +- **Legacy browsers**: Graceful degradation with X-XSS-Protection +- **Reporting**: Supported in all major browsers ## Maintenance ### Regular Reviews -1. **Monthly**: Review violation reports and patterns -2. **Quarterly**: Update content source restrictions -3. **Per release**: Validate CSP with new features -4. **Annually**: Security audit and penetration testing +1. **Monthly**: Review violation reports and patterns +2. **Quarterly**: Update content source restrictions +3. **Per release**: Validate CSP with new features +4. **Annually**: Security audit and penetration testing ### Updates and Modifications When adding new content sources: -1. Update `buildCSP()` function in `lib/csp.ts` -2. Add tests for new directives -3. Validate security impact -4. Update documentation +1. Update `buildCSP()` function in `lib/csp.ts` +2. Add tests for new directives +3. Validate security impact +4. Update documentation ### Incident Response For CSP violations: -1. **High-risk violations**: Immediate investigation -2. **Bypass attempts**: Security team notification -3. **Mass violations**: Check for policy issues -4. **False positives**: Adjust policies as needed +1. **High-risk violations**: Immediate investigation +2. **Bypass attempts**: Security team notification +3. **Mass violations**: Check for policy issues +4. **False positives**: Adjust policies as needed ## Best Practices ### Development -- Always test CSP changes in development first -- Use nonce provider for new inline scripts -- Validate external resources before adding -- Monitor console for CSP violations +- Always test CSP changes in development first +- Use nonce provider for new inline scripts +- Validate external resources before adding +- Monitor console for CSP violations ### Production -- Never disable CSP in production -- Monitor violation rates and patterns -- Keep nonce generation entropy high -- Regular security audits +- Never disable CSP in production +- Monitor violation rates and patterns +- Keep nonce generation entropy high +- Regular security audits ### Code Review -- Check all inline scripts have nonce -- Verify external resources are approved -- Ensure CSP tests pass -- Document any policy changes +- Check all inline scripts have nonce +- Verify external resources are approved +- Ensure CSP tests pass +- Document any policy changes ## Troubleshooting ### Common Issues -1. **Inline styles blocked**: Use nonce or move to external CSS -2. **Third-party scripts blocked**: Add to approved sources -3. **Dev tools not working**: Ensure development CSP allows unsafe-eval -4. **Images not loading**: Check image source restrictions +1. **Inline styles blocked**: Use nonce or move to external CSS +2. **Third-party scripts blocked**: Add to approved sources +3. **Dev tools not working**: Ensure development CSP allows unsafe-eval +4. **Images not loading**: Check image source restrictions ### Debug Tools @@ -374,19 +384,19 @@ curl -X POST /api/csp-report -d '{"csp-report": {...}}' If CSP breaks production: -1. Check violation reports for patterns -2. Identify blocking directive -3. Test fix in staging environment -4. Deploy emergency policy update -5. Monitor for resolved issues +1. Check violation reports for patterns +2. Identify blocking directive +3. Test fix in staging environment +4. Deploy emergency policy update +5. Monitor for resolved issues ## Compliance This CSP implementation addresses: -- **OWASP Top 10**: XSS prevention -- **CSP Level 3**: Modern security standards -- **GDPR**: Privacy-preserving monitoring -- **SOC 2**: Security controls documentation +- **OWASP Top 10**: XSS prevention +- **CSP Level 3**: Modern security standards +- **GDPR**: Privacy-preserving monitoring +- **SOC 2**: Security controls documentation -The enhanced CSP provides defense-in-depth against XSS attacks while maintaining application functionality and performance. \ No newline at end of file +The enhanced CSP provides defense-in-depth against XSS attacks while maintaining application functionality and performance. diff --git a/docs/session-processing.md b/docs/session-processing.md index ccfe2ee..abb1a31 100644 --- a/docs/session-processing.md +++ b/docs/session-processing.md @@ -15,22 +15,22 @@ The system now includes an automated process for analyzing chat session transcri ### Session Fetching -- The system fetches session data from configured CSV URLs for each company -- Unlike the previous implementation, it now only adds sessions that don't already exist in the database -- This prevents duplicate sessions and allows for incremental updates +- The system fetches session data from configured CSV URLs for each company +- Unlike the previous implementation, it now only adds sessions that don't already exist in the database +- This prevents duplicate sessions and allows for incremental updates ### Transcript Processing -- For sessions with transcript content that haven't been processed yet, the system calls OpenAI's API -- The API analyzes the transcript and extracts the following information: - - Primary language used (ISO 639-1 code) - - Number of messages sent by the user - - Overall sentiment (positive, neutral, negative) - - Whether the conversation was escalated - - Whether HR contact was mentioned or provided - - Best-fitting category for the conversation - - Up to 5 paraphrased questions asked by the user - - A brief summary of the conversation +- For sessions with transcript content that haven't been processed yet, the system calls OpenAI's API +- The API analyzes the transcript and extracts the following information: + - Primary language used (ISO 639-1 code) + - Number of messages sent by the user + - Overall sentiment (positive, neutral, negative) + - Whether the conversation was escalated + - Whether HR contact was mentioned or provided + - Best-fitting category for the conversation + - Up to 5 paraphrased questions asked by the user + - A brief summary of the conversation ### Scheduling @@ -43,10 +43,10 @@ The system includes two schedulers: The Session model has been updated with new fields to store the processed data: -- `processed`: Boolean flag indicating whether the session has been processed -- `sentimentCategory`: String value ("positive", "neutral", "negative") from OpenAI -- `questions`: JSON array of questions asked by the user -- `summary`: Brief summary of the conversation +- `processed`: Boolean flag indicating whether the session has been processed +- `sentimentCategory`: String value ("positive", "neutral", "negative") from OpenAI +- `questions`: JSON array of questions asked by the user +- `summary`: Brief summary of the conversation ## Configuration @@ -62,9 +62,9 @@ OPENAI_API_KEY=your_api_key_here To run the application with schedulers enabled: -- Development: `npm run dev` -- Development (with schedulers disabled): `npm run dev:no-schedulers` -- Production: `npm run start` +- Development: `npm run dev` +- Development (with schedulers disabled): `npm run dev:no-schedulers` +- Production: `npm run start` Note: These commands will start a custom Next.js server with the schedulers enabled. You'll need to have an OpenAI API key set in your `.env.local` file for the session processing to work. @@ -82,5 +82,5 @@ This will process all unprocessed sessions that have transcript content. The processing logic can be customized by modifying: -- `lib/processingScheduler.ts`: Contains the OpenAI processing logic -- `scripts/process_sessions.ts`: Standalone script for manual processing +- `lib/processingScheduler.ts`: Contains the OpenAI processing logic +- `scripts/process_sessions.ts`: Standalone script for manual processing diff --git a/docs/troubleshooting-fixes.md b/docs/troubleshooting-fixes.md index ad3138b..aebe306 100644 --- a/docs/troubleshooting-fixes.md +++ b/docs/troubleshooting-fixes.md @@ -5,9 +5,11 @@ This document outlines the fixes applied to resolve TypeScript compilation error ## Issues Resolved ### 1. Missing Type Imports + **Problem:** `lib/api/index.ts` was missing required type imports **Error:** `Cannot find name 'APIHandler'`, `Cannot find name 'Permission'` **Fix:** Added proper imports at the top of the file + ```typescript import type { APIContext, APIHandler, APIHandlerOptions } from "./handler"; import { createAPIHandler } from "./handler"; @@ -15,9 +17,11 @@ import { Permission, createPermissionChecker } from "./authorization"; ``` ### 2. Zod API Breaking Change + **Problem:** Zod error property name changed from `errors` to `issues` **Error:** `Property 'errors' does not exist on type 'ZodError'` **Fix:** Updated all references to use `error.issues` instead of `error.errors` + ```typescript // Before error.errors.map((e) => `${e.path.join(".")}: ${e.message}`) @@ -26,17 +30,21 @@ error.issues.map((e) => `${e.path.join(".")}: ${e.message}`) ``` ### 3. Missing LRU Cache Dependency + **Problem:** `lru-cache` package was missing from dependencies **Error:** `Cannot find module 'lru-cache'` **Fix:** Installed the missing dependency + ```bash pnpm add lru-cache ``` ### 4. LRU Cache Generic Type Constraints + **Problem:** TypeScript generic constraints not satisfied **Error:** `Type 'K' does not satisfy the constraint '{}'` **Fix:** Added proper generic type constraints + ```typescript // Before @@ -45,9 +53,11 @@ pnpm add lru-cache ``` ### 5. Map Iteration ES5 Compatibility + **Problem:** Map iteration requires downlevel iteration flag **Error:** `can only be iterated through when using the '--downlevelIteration' flag` **Fix:** Used `Array.from()` pattern for compatibility + ```typescript // Before for (const [key, value] of map) { ... } @@ -56,9 +66,11 @@ for (const [key, value] of Array.from(map.entries())) { ... } ``` ### 6. Redis Configuration Issues + **Problem:** Invalid Redis socket options **Error:** Redis connection failed with unsupported options **Fix:** Simplified Redis configuration to only include supported options + ```typescript this.client = createClient({ url: env.REDIS_URL, @@ -69,9 +81,11 @@ this.client = createClient({ ``` ### 7. Prisma Relationship Naming Mismatches + **Problem:** Code referenced non-existent Prisma relationships **Error:** `securityAuditLogs` and `sessionImport` don't exist **Fix:** Used correct relationship names + ```typescript // Before user.securityAuditLogs @@ -82,17 +96,21 @@ session.import ``` ### 8. Missing Schema Fields + **Problem:** Code referenced fields that don't exist in the database schema **Error:** `Property 'userId' does not exist on type` **Fix:** Applied type casting where schema fields were missing + ```typescript userId: (session as any).userId || null ``` ### 9. Deprecated Package Dependencies + **Problem:** `critters` package is deprecated and caused build failures **Error:** `Cannot find module 'critters'` **Fix:** Disabled CSS optimization feature that required critters + ```javascript experimental: { optimizeCss: false, // Disabled due to critters dependency @@ -100,9 +118,11 @@ experimental: { ``` ### 10. ESLint vs Biome Conflict + **Problem:** ESLint warnings treated as build errors **Error:** Build failed due to linting warnings **Fix:** Disabled ESLint during build since Biome is used for linting + ```javascript eslint: { ignoreDuringBuilds: true, @@ -112,6 +132,7 @@ eslint: { ## Schema Enhancements ### Enhanced User Management + Added comprehensive user management fields to the User model: ```prisma @@ -137,38 +158,45 @@ model User { ``` ### Updated Repository Methods + Enhanced UserRepository with new methods: -- `updateLastLogin()` - Tracks user login times -- `incrementFailedLoginAttempts()` - Security feature for account locking -- `verifyEmail()` - Email verification management -- `deactivateUser()` - Account management -- `unlockUser()` - Security administration -- `updatePreferences()` - User settings management -- `findInactiveUsers()` - Now uses `lastLoginAt` instead of `createdAt` + +- `updateLastLogin()` - Tracks user login times +- `incrementFailedLoginAttempts()` - Security feature for account locking +- `verifyEmail()` - Email verification management +- `deactivateUser()` - Account management +- `unlockUser()` - Security administration +- `updatePreferences()` - User settings management +- `findInactiveUsers()` - Now uses `lastLoginAt` instead of `createdAt` ## Prevention Measures ### 1. Regular Dependency Updates -- Monitor for breaking changes in dependencies like Zod -- Use `pnpm outdated` to check for deprecated packages -- Test builds after dependency updates + +- Monitor for breaking changes in dependencies like Zod +- Use `pnpm outdated` to check for deprecated packages +- Test builds after dependency updates ### 2. TypeScript Strict Checking -- Enable strict TypeScript checking to catch type errors early -- Use proper type imports and exports -- Avoid `any` types where possible + +- Enable strict TypeScript checking to catch type errors early +- Use proper type imports and exports +- Avoid `any` types where possible ### 3. Build Pipeline Validation -- Run `pnpm build` before committing -- Include type checking in CI/CD pipeline -- Separate linting from build process + +- Run `pnpm build` before committing +- Include type checking in CI/CD pipeline +- Separate linting from build process ### 4. Schema Management -- Regenerate Prisma client after schema changes: `pnpm prisma:generate` -- Validate schema changes with database migrations -- Use proper TypeScript types for database operations + +- Regenerate Prisma client after schema changes: `pnpm prisma:generate` +- Validate schema changes with database migrations +- Use proper TypeScript types for database operations ### 5. Development Workflow + ```bash # Recommended development workflow pnpm prisma:generate # After schema changes @@ -206,4 +234,4 @@ pnpm install --- *Last updated: 2025-07-12* -*Build Status: ✅ Success (47/47 pages generated)* \ No newline at end of file +*Build Status: ✅ Success (47/47 pages generated)* diff --git a/lib/api/errors.ts b/lib/api/errors.ts index 1bfd9a4..9931c0c 100644 --- a/lib/api/errors.ts +++ b/lib/api/errors.ts @@ -17,7 +17,11 @@ export class APIError extends Error { message: string, public readonly statusCode: number = 500, public readonly code: string = "INTERNAL_ERROR", - public readonly details?: any, + public readonly details?: + | Record + | string[] + | string + | number, public readonly logLevel: "info" | "warn" | "error" = "error" ) { super(message); @@ -100,7 +104,10 @@ export class ConflictError extends APIError { * Database Error - for database operation failures */ export class DatabaseError extends APIError { - constructor(message = "Database operation failed", details?: any) { + constructor( + message = "Database operation failed", + details?: Record | string + ) { super(message, 500, "DATABASE_ERROR", details, "error"); } } @@ -112,7 +119,7 @@ export class ExternalServiceError extends APIError { constructor( service: string, message = "External service error", - details?: any + details?: Record ) { super( `${service} service error: ${message}`, @@ -138,7 +145,11 @@ function shouldExposeError(error: unknown): boolean { /** * Log error with appropriate level */ -function logError(error: unknown, requestId: string, context?: any): void { +function logError( + error: unknown, + requestId: string, + context?: Record +): void { const logData = { requestId, error: error instanceof Error ? error.message : String(error), @@ -170,7 +181,7 @@ function logError(error: unknown, requestId: string, context?: any): void { export function handleAPIError( error: unknown, requestId?: string, - context?: any + context?: Record ): NextResponse { const id = requestId || crypto.randomUUID(); @@ -220,7 +231,7 @@ export function handleAPIError( /** * Async error handler for promise chains */ -export function asyncErrorHandler( +export function asyncErrorHandler( fn: (...args: T) => Promise ) { return async (...args: T): Promise => { @@ -237,7 +248,7 @@ export function asyncErrorHandler( /** * Error boundary for API route handlers */ -export function withErrorHandling( +export function withErrorHandling( handler: (...args: T) => Promise | NextResponse ) { return async (...args: T): Promise => { diff --git a/lib/api/handler.ts b/lib/api/handler.ts index 932e059..b3aefe8 100644 --- a/lib/api/handler.ts +++ b/lib/api/handler.ts @@ -91,10 +91,10 @@ export interface APIHandlerOptions { /** * API handler function type */ -export type APIHandler = ( +export type APIHandler = ( context: APIContext, - validatedData?: any, - validatedQuery?: any + validatedData?: unknown, + validatedQuery?: unknown ) => Promise; /** @@ -226,7 +226,7 @@ async function validateInput( if (error instanceof SyntaxError) { throw new ValidationError(["Invalid JSON in request body"]); } - throw new ValidationError(error as any); + throw new ValidationError(error as z.ZodError); } } @@ -239,7 +239,7 @@ function validateQuery(request: NextRequest, schema: z.ZodSchema): T { const query = Object.fromEntries(searchParams.entries()); return schema.parse(query); } catch (error) { - throw new ValidationError(error as any); + throw new ValidationError(error as z.ZodError); } } @@ -285,7 +285,7 @@ function addCORSHeaders( /** * Main API handler factory */ -export function createAPIHandler( +export function createAPIHandler( handler: APIHandler, options: APIHandlerOptions = {} ) { @@ -368,7 +368,7 @@ export function createAPIHandler( /** * Utility function for GET endpoints */ -export function createGETHandler( +export function createGETHandler( handler: APIHandler, options: Omit = {} ) { @@ -381,7 +381,7 @@ export function createGETHandler( /** * Utility function for POST endpoints */ -export function createPOSTHandler( +export function createPOSTHandler( handler: APIHandler, options: APIHandlerOptions = {} ) { @@ -394,7 +394,7 @@ export function createPOSTHandler( /** * Utility function for authenticated endpoints */ -export function createAuthenticatedHandler( +export function createAuthenticatedHandler( handler: APIHandler, options: APIHandlerOptions = {} ) { @@ -408,7 +408,7 @@ export function createAuthenticatedHandler( /** * Utility function for admin endpoints */ -export function createAdminHandler( +export function createAdminHandler( handler: APIHandler, options: APIHandlerOptions = {} ) { diff --git a/lib/api/index.ts b/lib/api/index.ts index b02625e..63c52d0 100644 --- a/lib/api/index.ts +++ b/lib/api/index.ts @@ -58,10 +58,11 @@ export { UserRole, } from "./handler"; +import { createPermissionChecker, type Permission } from "./authorization"; // Re-import types for use in functions below import type { APIContext, APIHandler, APIHandlerOptions } from "./handler"; import { createAPIHandler } from "./handler"; -import { Permission, createPermissionChecker } from "./authorization"; + // Response utilities export { type APIResponse, diff --git a/lib/api/response.ts b/lib/api/response.ts index dc38e5f..e9af48b 100644 --- a/lib/api/response.ts +++ b/lib/api/response.ts @@ -19,7 +19,7 @@ export interface APIResponseMeta { version?: string; } -export interface APIResponse { +export interface APIResponse { success: boolean; data?: T; error?: string; diff --git a/lib/performance/cache.ts b/lib/performance/cache.ts index cf84c50..68182a0 100644 --- a/lib/performance/cache.ts +++ b/lib/performance/cache.ts @@ -47,7 +47,7 @@ export interface CacheStats { /** * High-performance memory cache with advanced features */ -export class PerformanceCache { +export class PerformanceCache { private cache: LRUCache>; private stats: { hits: number; @@ -245,7 +245,7 @@ class CacheManager { /** * Create or get a named cache instance */ - getCache( + getCache( name: string, options: CacheOptions = {} ): PerformanceCache { diff --git a/lib/performance/deduplication.ts b/lib/performance/deduplication.ts index 908c7fb..87c4e1e 100644 --- a/lib/performance/deduplication.ts +++ b/lib/performance/deduplication.ts @@ -13,7 +13,7 @@ import { TIME } from "../constants"; export interface DeduplicationOptions { ttl?: number; // How long to keep results cached maxPending?: number; // Maximum pending requests per key - keyGenerator?: (...args: any[]) => string; + keyGenerator?: (...args: unknown[]) => string; timeout?: number; // Request timeout } @@ -94,7 +94,7 @@ export class RequestDeduplicator { /** * Memoize a function with deduplication */ - memoize( + memoize( fn: (...args: Args) => Promise, options: DeduplicationOptions = {} ) { @@ -212,7 +212,7 @@ export class RequestDeduplicator { /** * Generate a key from function arguments */ - private generateKey(...args: any[]): string { + private generateKey(...args: unknown[]): string { try { return JSON.stringify(args); } catch { @@ -351,7 +351,9 @@ class DeduplicationManager { ReturnType > = {}; - for (const [name, deduplicator] of Array.from(this.deduplicators.entries())) { + for (const [name, deduplicator] of Array.from( + this.deduplicators.entries() + )) { stats[name] = deduplicator.getStats(); } @@ -427,7 +429,7 @@ export class DeduplicationUtils { /** * Create a deduplicated version of an async function */ - static deduplicate( + static deduplicate( fn: (...args: T) => Promise, deduplicatorName = "default", options: DeduplicationOptions = {} @@ -464,7 +466,7 @@ export class DeduplicationUtils { options ); - descriptor.value = function (...args: any[]) { + descriptor.value = function (...args: unknown[]) { const key = `${target.constructor.name}.${propertyKey}:${JSON.stringify(args)}`; return deduplicator.execute( key, diff --git a/lib/performance/integration.ts b/lib/performance/integration.ts index 8f61e72..977e477 100644 --- a/lib/performance/integration.ts +++ b/lib/performance/integration.ts @@ -5,10 +5,10 @@ * caching, and deduplication into existing services and API endpoints. */ -import { PerformanceUtils, performanceMonitor } from "./monitor"; -import { caches, CacheUtils } from "./cache"; -import { deduplicators, DeduplicationUtils } from "./deduplication"; import type { NextRequest, NextResponse } from "next/server"; +import { CacheUtils, caches } from "./cache"; +import { DeduplicationUtils, deduplicators } from "./deduplication"; +import { PerformanceUtils, performanceMonitor } from "./monitor"; /** * Performance integration options @@ -235,8 +235,8 @@ export function enhanceAPIRoute( export function PerformanceEnhanced( options: PerformanceIntegrationOptions = {} ) { - return function {}>(constructor: T) { - return class extends constructor { + return {}>(constructor: T) => + class extends constructor { constructor(...args: any[]) { super(...args); @@ -259,7 +259,6 @@ export function PerformanceEnhanced( }); } }; - }; } /** @@ -268,11 +267,11 @@ export function PerformanceEnhanced( export function PerformanceOptimized( options: PerformanceIntegrationOptions = {} ) { - return function ( + return ( target: unknown, propertyKey: string, descriptor: PropertyDescriptor - ) { + ) => { const originalMethod = descriptor.value; if (typeof originalMethod !== "function") { @@ -280,7 +279,7 @@ export function PerformanceOptimized( } descriptor.value = enhanceServiceMethod( - `${(target as any).constructor.name}.${propertyKey}`, + `${(target as { constructor: { name: string } }).constructor.name}.${propertyKey}`, originalMethod, options ); @@ -293,15 +292,15 @@ export function PerformanceOptimized( * Simple caching decorator */ export function Cached( - cacheName: string = "default", + cacheName = "default", ttl: number = 5 * 60 * 1000, keyGenerator?: (...args: unknown[]) => string ) { - return function ( + return ( target: unknown, propertyKey: string, descriptor: PropertyDescriptor - ) { + ) => { const originalMethod = descriptor.value; if (typeof originalMethod !== "function") { @@ -309,14 +308,14 @@ export function Cached( } descriptor.value = CacheUtils.cached( - `${(target as any).constructor.name}.${propertyKey}`, + `${(target as { constructor: { name: string } }).constructor.name}.${propertyKey}`, originalMethod, { ttl, keyGenerator: keyGenerator || ((...args) => - `${(target as any).constructor.name}.${propertyKey}:${JSON.stringify(args)}`), + `${(target as { constructor: { name: string } }).constructor.name}.${propertyKey}:${JSON.stringify(args)}`), } ); @@ -328,7 +327,7 @@ export function Cached( * Simple deduplication decorator */ export function Deduplicated( - deduplicatorName: string = "default", + deduplicatorName = "default", ttl: number = 2 * 60 * 1000 ) { return DeduplicationUtils.deduplicatedMethod(deduplicatorName, { ttl }); @@ -349,15 +348,18 @@ function mergeOptions( overrides: PerformanceIntegrationOptions ): PerformanceIntegrationOptions { return { - cache: defaults.cache && overrides.cache - ? { ...defaults.cache, ...overrides.cache } - : defaults.cache || overrides.cache, - deduplication: defaults.deduplication && overrides.deduplication - ? { ...defaults.deduplication, ...overrides.deduplication } - : defaults.deduplication || overrides.deduplication, - monitoring: defaults.monitoring && overrides.monitoring - ? { ...defaults.monitoring, ...overrides.monitoring } - : defaults.monitoring || overrides.monitoring, + cache: + defaults.cache && overrides.cache + ? { ...defaults.cache, ...overrides.cache } + : defaults.cache || overrides.cache, + deduplication: + defaults.deduplication && overrides.deduplication + ? { ...defaults.deduplication, ...overrides.deduplication } + : defaults.deduplication || overrides.deduplication, + monitoring: + defaults.monitoring && overrides.monitoring + ? { ...defaults.monitoring, ...overrides.monitoring } + : defaults.monitoring || overrides.monitoring, }; } @@ -367,7 +369,9 @@ function mergeOptions( export function createEnhancedService( ServiceClass: new (...args: unknown[]) => T, options: PerformanceIntegrationOptions = {} -): new (...args: unknown[]) => T { +): new ( + ...args: unknown[] +) => T { return PerformanceEnhanced(options)(ServiceClass as never); } @@ -436,10 +440,7 @@ export function getPerformanceIntegrationStatus() { * Initialize performance systems */ export function initializePerformanceSystems( - options: { - monitoring?: boolean; - monitoringInterval?: number; - } = {} + options: { monitoring?: boolean; monitoringInterval?: number } = {} ) { if (options.monitoring !== false) { const interval = options.monitoringInterval || 30000; diff --git a/lib/performance/monitor.ts b/lib/performance/monitor.ts index 67acf98..9cb5dba 100644 --- a/lib/performance/monitor.ts +++ b/lib/performance/monitor.ts @@ -777,7 +777,7 @@ export class PerformanceUtils { throw new Error("Measured decorator can only be applied to methods"); } - descriptor.value = async function (...args: any[]) { + descriptor.value = async function (...args: unknown[]) { const { result, duration } = await PerformanceUtils.measureAsync( metricName, () => originalMethod.apply(this, args) diff --git a/lib/performance/optimizer.ts b/lib/performance/optimizer.ts index a87b9a8..1111f97 100644 --- a/lib/performance/optimizer.ts +++ b/lib/performance/optimizer.ts @@ -5,14 +5,14 @@ * to improve system performance based on real-time metrics. */ -import { - performanceMonitor, - type PerformanceMetrics, - type Bottleneck, -} from "./monitor"; -import { cacheManager, type CacheStats } from "./cache"; -import { deduplicationManager } from "./deduplication"; import { TIME } from "../constants"; +import { type CacheStats, cacheManager } from "./cache"; +import { deduplicationManager } from "./deduplication"; +import { + type Bottleneck, + type PerformanceMetrics, + performanceMonitor, +} from "./monitor"; /** * Optimization action types @@ -40,8 +40,8 @@ export interface OptimizationResult { success: boolean; message: string; metrics?: { - before: any; - after: any; + before: Record; + after: Record; improvement: number; // Percentage }; }; @@ -408,18 +408,17 @@ export class PerformanceOptimizer { }, timestamp: new Date(), }; - } else { - return { - action: OptimizationAction.TRIGGER_GARBAGE_COLLECTION, - target: "system", - applied: false, - result: { - success: false, - message: "Garbage collection not available (run with --expose-gc)", - }, - timestamp: new Date(), - }; } + return { + action: OptimizationAction.TRIGGER_GARBAGE_COLLECTION, + target: "system", + applied: false, + result: { + success: false, + message: "Garbage collection not available (run with --expose-gc)", + }, + timestamp: new Date(), + }; } catch (error) { return { action: OptimizationAction.TRIGGER_GARBAGE_COLLECTION, diff --git a/lib/repositories/SecurityAuditLogRepository.ts b/lib/repositories/SecurityAuditLogRepository.ts index f6bd861..164e3fb 100644 --- a/lib/repositories/SecurityAuditLogRepository.ts +++ b/lib/repositories/SecurityAuditLogRepository.ts @@ -346,7 +346,7 @@ export class SecurityAuditLogRepository if (!acc[key]) { acc[key] = { userId: event.userId!, - email: event.user?.email || 'Unknown', + email: event.user?.email || "Unknown", count: 0, }; } diff --git a/lib/repositories/UserRepository.ts b/lib/repositories/UserRepository.ts index ac51481..95c4a41 100644 --- a/lib/repositories/UserRepository.ts +++ b/lib/repositories/UserRepository.ts @@ -231,10 +231,10 @@ export class UserRepository implements BaseRepository { try { return await prisma.user.update({ where: { id }, - data: { + data: { lastLoginAt: new Date(), failedLoginAttempts: 0, // Reset failed attempts on successful login - lockedAt: null // Unlock account if it was locked + lockedAt: null, // Unlock account if it was locked }, }); } catch (error) { @@ -330,7 +330,9 @@ export class UserRepository implements BaseRepository { ).length; const lastActivity = events.length > 0 ? events[0].timestamp : null; const countriesAccessed = Array.from( - new Set(events.map((e) => e.country).filter((c): c is string => c !== null)) + new Set( + events.map((e) => e.country).filter((c): c is string => c !== null) + ) ); return { @@ -406,7 +408,10 @@ export class UserRepository implements BaseRepository { /** * Increment failed login attempts and lock account if threshold exceeded */ - async incrementFailedLoginAttempts(email: string, maxAttempts = 5): Promise { + async incrementFailedLoginAttempts( + email: string, + maxAttempts = 5 + ): Promise { try { const user = await prisma.user.findUnique({ where: { email }, @@ -458,7 +463,11 @@ export class UserRepository implements BaseRepository { /** * Set email verification token */ - async setEmailVerificationToken(id: string, token: string, expiryHours = 24): Promise { + async setEmailVerificationToken( + id: string, + token: string, + expiryHours = 24 + ): Promise { try { const expiry = new Date(Date.now() + expiryHours * 60 * 60 * 1000); return await prisma.user.update({ @@ -519,7 +528,10 @@ export class UserRepository implements BaseRepository { /** * Update user preferences */ - async updatePreferences(id: string, preferences: Record): Promise { + async updatePreferences( + id: string, + preferences: Record + ): Promise { try { return await prisma.user.update({ where: { id }, diff --git a/lib/securityMonitoring.ts b/lib/securityMonitoring.ts index 5689f1a..3893a49 100644 --- a/lib/securityMonitoring.ts +++ b/lib/securityMonitoring.ts @@ -242,7 +242,10 @@ class SecurityMonitoringService { * Configure monitoring thresholds */ updateConfig(config: DeepPartial): void { - this.config = this.deepMerge(this.config as any, config as any) as unknown as MonitoringConfig; + this.config = this.deepMerge( + this.config as any, + config as any + ) as unknown as MonitoringConfig; } /** @@ -260,7 +263,10 @@ class SecurityMonitoringService { typeof source[key] === "object" && !Array.isArray(source[key]) ) { - result[key] = this.deepMerge(target[key] || {} as any, source[key] as any); + result[key] = this.deepMerge( + target[key] || ({} as any), + source[key] as any + ); } else { result[key] = source[key]; } diff --git a/lib/services/EnhancedSecurityService.ts b/lib/services/EnhancedSecurityService.ts index e8cb1ff..b1a8462 100644 --- a/lib/services/EnhancedSecurityService.ts +++ b/lib/services/EnhancedSecurityService.ts @@ -6,19 +6,19 @@ */ import { - PerformanceEnhanced, - PerformanceOptimized, Cached, Deduplicated, Monitored, + PerformanceEnhanced, + PerformanceOptimized, } from "../performance/integration"; +import { AuditOutcome, AuditSeverity } from "../securityAuditLogger"; +import { AlertChannel, type MonitoringConfig } from "../securityMonitoring"; +import type { Alert, SecurityEvent } from "../types/security"; +import { ThreatLevel } from "../types/security"; +import { AlertManagementService } from "./AlertManagementService"; import { SecurityEventProcessor } from "./SecurityEventProcessor"; import { ThreatDetectionService } from "./ThreatDetectionService"; -import { AlertManagementService } from "./AlertManagementService"; -import { AlertChannel, type MonitoringConfig } from "../securityMonitoring"; -import { AuditOutcome, AuditSeverity } from "../securityAuditLogger"; -import { ThreatLevel } from "../types/security"; -import type { SecurityEvent, Alert } from "../types/security"; /** * Configuration for enhanced security service @@ -161,26 +161,31 @@ export class EnhancedSecurityService { { metadata: event.metadata }, // Cast to AuditLogContext event.metadata ); - + // Return threat level based on detected threats if (result.threats.length === 0) { return ThreatLevel.LOW; } - + // Find the highest severity threat const highestSeverity = result.threats.reduce((max, threat) => { const severityOrder = { LOW: 1, MEDIUM: 2, HIGH: 3, CRITICAL: 4 }; - const current = severityOrder[threat.severity as keyof typeof severityOrder] || 1; + const current = + severityOrder[threat.severity as keyof typeof severityOrder] || 1; const maxVal = severityOrder[max as keyof typeof severityOrder] || 1; return current > maxVal ? threat.severity : max; }, "LOW" as any); - + // Map AlertSeverity to ThreatLevel switch (highestSeverity) { - case "CRITICAL": return ThreatLevel.CRITICAL; - case "HIGH": return ThreatLevel.HIGH; - case "MEDIUM": return ThreatLevel.MEDIUM; - default: return ThreatLevel.LOW; + case "CRITICAL": + return ThreatLevel.CRITICAL; + case "HIGH": + return ThreatLevel.HIGH; + case "MEDIUM": + return ThreatLevel.MEDIUM; + default: + return ThreatLevel.LOW; } } @@ -349,7 +354,7 @@ export class EnhancedSecurityService { // cache: { // enabled: true, // ttl: 10 * 60 * 1000, // 10 minutes - // keyGenerator: (query: any) => `search:${JSON.stringify(query)}`, + // keyGenerator: (query: Record) => `search:${JSON.stringify(query)}`, // }, // deduplication: { // enabled: true, @@ -394,11 +399,11 @@ export class EnhancedSecurityService { private calculateThreatDistribution( events: SecurityEvent[] ): Record { - return { - [ThreatLevel.LOW]: 0, - [ThreatLevel.MEDIUM]: 0, - [ThreatLevel.HIGH]: 0, - [ThreatLevel.CRITICAL]: 0 + return { + [ThreatLevel.LOW]: 0, + [ThreatLevel.MEDIUM]: 0, + [ThreatLevel.HIGH]: 0, + [ThreatLevel.CRITICAL]: 0, }; } @@ -441,7 +446,9 @@ export class EnhancedSecurityService { }; } - private async performSearch(query: any): Promise { + private async performSearch( + query: Record + ): Promise { // Mock search implementation return []; } diff --git a/lib/services/SecurityMetricsService.ts b/lib/services/SecurityMetricsService.ts index 67af543..e90da33 100644 --- a/lib/services/SecurityMetricsService.ts +++ b/lib/services/SecurityMetricsService.ts @@ -68,7 +68,7 @@ export class SecurityMetricsService { .slice(0, 5); // User risk scores - transform data to match expected format - const transformedEvents = events.map(event => ({ + const transformedEvents = events.map((event) => ({ userId: event.userId || undefined, user: event.user ? { email: event.user.email } : undefined, eventType: event.eventType as SecurityEventType, @@ -76,7 +76,8 @@ export class SecurityMetricsService { severity: event.severity as AuditSeverity, country: event.country || undefined, })); - const userRiskScores = await this.calculateUserRiskScores(transformedEvents); + const userRiskScores = + await this.calculateUserRiskScores(transformedEvents); // Calculate overall security score const securityScore = this.calculateSecurityScore({ @@ -122,7 +123,9 @@ export class SecurityMetricsService { country?: string; }> ): Promise> { - const userEvents = events.filter((e) => e.userId) as Array; + const userEvents = events.filter((e) => e.userId) as Array< + (typeof events)[0] & { userId: string } + >; const userScores = new Map< string, { email: string; score: number; events: typeof userEvents } diff --git a/lib/services/ThreatDetectionService.ts b/lib/services/ThreatDetectionService.ts index 295aa46..e4af051 100644 --- a/lib/services/ThreatDetectionService.ts +++ b/lib/services/ThreatDetectionService.ts @@ -139,7 +139,7 @@ export class ThreatDetectionService { // Check for geographical anomalies if (context.country && context.userId) { // Transform historical events to match expected type - const transformedEvents = historicalEvents.map(event => ({ + const transformedEvents = historicalEvents.map((event) => ({ userId: event.userId || undefined, country: event.country || undefined, })); diff --git a/lib/trpc.ts b/lib/trpc.ts index af98e47..ce8cf82 100644 --- a/lib/trpc.ts +++ b/lib/trpc.ts @@ -9,8 +9,8 @@ import { initTRPC, TRPCError } from "@trpc/server"; import type { FetchCreateContextFnOptions } from "@trpc/server/adapters/fetch"; -import { getServerSession } from "next-auth/next"; import type { NextRequest } from "next/server"; +import { getServerSession } from "next-auth/next"; import superjson from "superjson"; import type { z } from "zod"; import { authOptions } from "./auth"; diff --git a/lib/types/security.ts b/lib/types/security.ts index 76b6703..43a40ad 100644 --- a/lib/types/security.ts +++ b/lib/types/security.ts @@ -26,4 +26,4 @@ export interface Alert { timestamp: Date; resolved: boolean; metadata?: Record; -} \ No newline at end of file +} diff --git a/lib/utils/BoundedBuffer.ts b/lib/utils/BoundedBuffer.ts index a8b48d1..324b294 100644 --- a/lib/utils/BoundedBuffer.ts +++ b/lib/utils/BoundedBuffer.ts @@ -26,7 +26,6 @@ export class BoundedBuffer { * Add item to buffer with automatic cleanup */ push(item: T): void { - this.buffer.push(item); // Trigger cleanup if threshold reached diff --git a/package.json b/package.json index 4634bef..57fc868 100644 --- a/package.json +++ b/package.json @@ -55,7 +55,8 @@ "migration:test-trpc": "pnpm exec tsx scripts/migration/trpc-endpoint-tests.ts", "migration:test-batch": "pnpm exec tsx scripts/migration/batch-processing-tests.ts", "migration:test-all": "pnpm migration:test-trpc && pnpm migration:test-batch && pnpm migration:health-check", - "migration:full": "pnpm migration:pre-check && pnpm migration:backup && pnpm migration:deploy && pnpm migration:health-check" + "migration:full": "pnpm migration:pre-check && pnpm migration:backup && pnpm migration:deploy && pnpm migration:health-check", + "prepare": "husky" }, "dependencies": { "@prisma/adapter-pg": "^6.11.1", @@ -152,8 +153,10 @@ "eslint-config-next": "^15.3.5", "eslint-plugin-prettier": "^5.5.1", "eslint-plugin-react-hooks": "^5.2.0", + "husky": "^9.1.7", "jest-axe": "^10.0.0", "jsdom": "^26.1.0", + "lint-staged": "^16.1.2", "markdownlint-cli2": "^0.18.1", "node-mocks-http": "^1.17.2", "postcss": "^8.5.6", @@ -198,19 +201,17 @@ }, "markdownlint-cli2": { "config": { - "MD007": { - "indent": 4, - "start_indented": false, - "start_indent": 4 - }, + "MD007": false, "MD013": false, - "MD030": { - "ul_single": 3, - "ol_single": 2, - "ul_multi": 3, - "ol_multi": 2 - }, - "MD033": false + "MD024": false, + "MD029": false, + "MD030": false, + "MD032": false, + "MD033": false, + "MD036": false, + "MD040": false, + "MD041": false, + "MD046": false }, "ignores": [ "node_modules", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 40c4737..27b6917 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -266,10 +266,10 @@ importers: version: 8.36.0(eslint@9.31.0(jiti@2.4.2))(typescript@5.8.3) '@vitejs/plugin-react': specifier: ^4.6.0 - version: 4.6.0(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)) + version: 4.6.0(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0)) '@vitest/coverage-v8': specifier: ^3.2.4 - version: 3.2.4(vitest@3.2.4(@types/debug@4.1.12)(@types/node@24.0.13)(jiti@2.4.2)(jsdom@26.1.0)(lightningcss@1.30.1)(tsx@4.20.3)) + version: 3.2.4(vitest@3.2.4(@types/debug@4.1.12)(@types/node@24.0.13)(jiti@2.4.2)(jsdom@26.1.0)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0)) concurrently: specifier: ^9.2.0 version: 9.2.0 @@ -285,12 +285,18 @@ importers: eslint-plugin-react-hooks: specifier: ^5.2.0 version: 5.2.0(eslint@9.31.0(jiti@2.4.2)) + husky: + specifier: ^9.1.7 + version: 9.1.7 jest-axe: specifier: ^10.0.0 version: 10.0.0 jsdom: specifier: ^26.1.0 version: 26.1.0 + lint-staged: + specifier: ^16.1.2 + version: 16.1.2 markdownlint-cli2: specifier: ^0.18.1 version: 0.18.1 @@ -326,10 +332,10 @@ importers: version: 5.8.3 vite-tsconfig-paths: specifier: ^5.1.4 - version: 5.1.4(typescript@5.8.3)(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)) + version: 5.1.4(typescript@5.8.3)(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0)) vitest: specifier: ^3.2.4 - version: 3.2.4(@types/debug@4.1.12)(@types/node@24.0.13)(jiti@2.4.2)(jsdom@26.1.0)(lightningcss@1.30.1)(tsx@4.20.3) + version: 3.2.4(@types/debug@4.1.12)(@types/node@24.0.13)(jiti@2.4.2)(jsdom@26.1.0)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0) webpack-bundle-analyzer: specifier: ^4.10.2 version: 4.10.2 @@ -2257,6 +2263,10 @@ packages: ajv@6.12.6: resolution: {integrity: sha512-j3fVLgvTo527anyYyJOGTYJbG+vnnQYvE0m5mmkc1TK+nxAppkCLMIL0aZ4dblVCNoGShhm+kzE4ZUykBoMg4g==} + ansi-escapes@7.0.0: + resolution: {integrity: sha512-GdYO7a61mR0fOlAsvC9/rIHf7L96sBc6dEWzeOu+KAea5bZyQRPIpojrVoI4AXGJS/ycu/fBTdLrUkA4ODrvjw==} + engines: {node: '>=18'} + ansi-regex@5.0.1: resolution: {integrity: sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==} engines: {node: '>=8'} @@ -2429,6 +2439,10 @@ packages: resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==} engines: {node: '>=10'} + chalk@5.4.1: + resolution: {integrity: sha512-zgVZuo2WcZgfUEmsn6eO3kINexW8RAE4maiQ8QNs8CtpPCSyMiYsULR3HQYkm3w8FIA3SberyMJMSldGsW+U3w==} + engines: {node: ^12.17.0 || ^14.13 || >=16.0.0} + character-entities-html4@2.1.0: resolution: {integrity: sha512-1v7fgQRj6hnSwFpq1Eu0ynr/CDEw0rXo2B61qXrLNdHZmPKgb7fqS1a2JwF0rISo9q77jDI8VMEHoApn8qDoZA==} @@ -2452,6 +2466,14 @@ packages: class-variance-authority@0.7.1: resolution: {integrity: sha512-Ka+9Trutv7G8M6WT6SeiRWz792K5qEqIGEGzXKhAE6xOWAY6pPH8U+9IY3oCMv6kqTmLsv7Xh/2w2RigkePMsg==} + cli-cursor@5.0.0: + resolution: {integrity: sha512-aCj4O5wKyszjMmDT4tZj93kxyydN/K5zPWSCe6/0AV/AA1pqe5ZBIw0a2ZfPQV7lL5/yb5HsUreJ6UFAF1tEQw==} + engines: {node: '>=18'} + + cli-truncate@4.0.0: + resolution: {integrity: sha512-nPdaFdQ0h/GEigbPClz11D0v/ZJEwxmeVZGeMo3Z5StPtUTkA9o1lD6QwoirYiSDzbcwn2XcjwmCp68W1IS4TA==} + engines: {node: '>=18'} + client-only@0.0.1: resolution: {integrity: sha512-IV3Ou0jSMzZrd3pZ48nLkT9DA7Ag1pnPzaiQhpW7c3RbcqqzvzzVu+L8gfqMp/8IM2MQtSiqaCxrrcfu8I8rMA==} @@ -2481,6 +2503,9 @@ packages: resolution: {integrity: sha512-1rXeuUUiGGrykh+CeBdu5Ie7OJwinCgQY0bc7GCRxy5xVHy+moaqkpL/jqQq0MtQOeYcrqEz4abc5f0KtU7W4A==} engines: {node: '>=12.5.0'} + colorette@2.0.20: + resolution: {integrity: sha512-IfEDxwoWIjkeXL1eXcDiow4UbKjhLdq6/EuSVR9GMN7KVH3r9gQ83e73hsz1Nd1T3ijd5xv1wcWRYO+D6kCI2w==} + combined-stream@1.0.8: resolution: {integrity: sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==} engines: {node: '>= 0.8'} @@ -2812,6 +2837,9 @@ packages: electron-to-chromium@1.5.182: resolution: {integrity: sha512-Lv65Btwv9W4J9pyODI6EWpdnhfvrve/us5h1WspW8B2Fb0366REPtY3hX7ounk1CkV/TBjWCEvCBBbYbmV0qCA==} + emoji-regex@10.4.0: + resolution: {integrity: sha512-EC+0oUMY1Rqm4O6LLrgjtYDvcVYTy7chDnM4Q7030tP4Kwj3u/pR6gP9ygnp2CJMK5Gq+9Q2oqmrFJAz01DXjw==} + emoji-regex@8.0.0: resolution: {integrity: sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==} @@ -2830,6 +2858,10 @@ packages: resolution: {integrity: sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==} engines: {node: '>=0.12'} + environment@1.1.0: + resolution: {integrity: sha512-xUtoPkMggbz0MPyPiIWr1Kp4aeWJjDZ6SMvURhimjdZgsRuDplF5/s9hcgGhyXMhs+6vpnuoiZ2kFiu3FMnS8Q==} + engines: {node: '>=18'} + es-abstract@1.24.0: resolution: {integrity: sha512-WSzPgsdLtTcQwm4CROfS5ju2Wa1QQcVeT37jFjYzdFz1r9ahadC8B8/a4qxJxM+09F18iumCdRmlr96ZYkQvEg==} engines: {node: '>= 0.4'} @@ -3143,6 +3175,10 @@ packages: resolution: {integrity: sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==} engines: {node: 6.* || 8.* || >= 10.*} + get-east-asian-width@1.3.0: + resolution: {integrity: sha512-vpeMIQKxczTD/0s2CdEWHcb0eeJe6TFjxb+J5xgX7hScxqrGuyjmv4c1D4A/gelKfyox0gJJwIHF+fLjeaM8kQ==} + engines: {node: '>=18'} + get-intrinsic@1.3.0: resolution: {integrity: sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==} engines: {node: '>= 0.4'} @@ -3272,6 +3308,11 @@ packages: resolution: {integrity: sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==} engines: {node: '>= 14'} + husky@9.1.7: + resolution: {integrity: sha512-5gs5ytaNjBrh5Ow3zrvdUUY+0VxIuWVL4i9irt6friV+BqdCfmV11CQTWMiBYWHbXhco+J1kHfTOUkePhCDvMA==} + engines: {node: '>=18'} + hasBin: true + i18n-iso-countries@7.14.0: resolution: {integrity: sha512-nXHJZYtNrfsi1UQbyRqm3Gou431elgLjKl//CYlnBGt5aTWdRPH1PiS2T/p/n8Q8LnqYqzQJik3Q7mkwvLokeg==} engines: {node: '>= 12'} @@ -3373,6 +3414,14 @@ packages: resolution: {integrity: sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==} engines: {node: '>=8'} + is-fullwidth-code-point@4.0.0: + resolution: {integrity: sha512-O4L094N2/dZ7xqVdrXhh9r1KODPJpFms8B5sGdJLPy664AgvXsreZUyCQQNItZRDlYug4xStLjNp/sz3HvBowQ==} + engines: {node: '>=12'} + + is-fullwidth-code-point@5.0.0: + resolution: {integrity: sha512-OVa3u9kkBbw7b8Xw5F9P+D/T9X+Z4+JruYVNapTjPYZYUznQ5YfWeFkOj606XYYW8yugTfC8Pj0hYqvi4ryAhA==} + engines: {node: '>=18'} + is-generator-function@1.1.0: resolution: {integrity: sha512-nPUB5km40q9e8UfN/Zc24eLlzdSf9OfKByBw9CIdw4H1giPMeA0OIJvbchsCu4npfI2QcMVBsGEBHKZ7wLTWmQ==} engines: {node: '>= 0.4'} @@ -3641,12 +3690,25 @@ packages: resolution: {integrity: sha512-xi6IyHML+c9+Q3W0S4fCQJOym42pyurFiJUHEcEyHS0CeKzia4yZDEsLlqOFykxOdHpNy0NmvVO31vcSqAxJCg==} engines: {node: '>= 12.0.0'} + lilconfig@3.1.3: + resolution: {integrity: sha512-/vlFKAoH5Cgt3Ie+JLhRbwOsCQePABiU3tJ1egGvyQ+33R/vcwM2Zl2QR/LzjsBeItPt3oSVXapn+m4nQDvpzw==} + engines: {node: '>=14'} + lineclip@1.1.5: resolution: {integrity: sha512-KlA/wRSjpKl7tS9iRUdlG72oQ7qZ1IlVbVgHwoO10TBR/4gQ86uhKow6nlzMAJJhjCWKto8OeoAzzIzKSmN25A==} linkify-it@5.0.0: resolution: {integrity: sha512-5aHCbzQRADcdP+ATqnDuhhJ/MRIqDkZX5pyjFHRRysS8vZ5AbqGEoFIb6pYHPZ+L/OC2Lc+xT8uHVVR5CAK/wQ==} + lint-staged@16.1.2: + resolution: {integrity: sha512-sQKw2Si2g9KUZNY3XNvRuDq4UJqpHwF0/FQzZR2M7I5MvtpWvibikCjUVJzZdGE0ByurEl3KQNvsGetd1ty1/Q==} + engines: {node: '>=20.17'} + hasBin: true + + listr2@8.3.3: + resolution: {integrity: sha512-LWzX2KsqcB1wqQ4AHgYb4RsDXauQiqhjLk+6hjbaeHG4zpjjVAB6wC/gz6X0l+Du1cN3pUB5ZlrvTbhGSNnUQQ==} + engines: {node: '>=18.0.0'} + locate-path@6.0.0: resolution: {integrity: sha512-iPZK6eYjbxRu3uB4/WZ3EsEIMJFMqAoopl3R+zuq0UjcAm/MO6KCweDgPfP3elTztoKP3KtnVHxTn2NHBSDVUw==} engines: {node: '>=10'} @@ -3657,6 +3719,10 @@ packages: lodash@4.17.21: resolution: {integrity: sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==} + log-update@6.1.0: + resolution: {integrity: sha512-9ie8ItPR6tjY5uYJh8K/Zrv/RMZ5VOlOWvtZdEHYSTFKZfIBPQa9tOAEeAWhd+AnIneLJ22w5fjOYtoutpWq5w==} + engines: {node: '>=18'} + longest-streak@3.1.0: resolution: {integrity: sha512-9Ri+o0JYgehTaVBBDoMqIl8GXtbWg711O3srftcHhZ0dqnETqLaoIK0x17fUw9rFSlK/0NlsKe0Ahhyl5pXE2g==} @@ -3862,6 +3928,10 @@ packages: engines: {node: '>=4'} hasBin: true + mimic-function@5.0.1: + resolution: {integrity: sha512-VP79XUPxV2CigYP3jWwAUFSku2aKqBH7uTAapFWCBqutsbmDo96KY5o8uh6U+/YSIn5OxJnXp73beVkpqMIGhA==} + engines: {node: '>=18'} + min-indent@1.0.1: resolution: {integrity: sha512-I9jwMn07Sy/IwOj3zVkVik2JTvgpaykDZEigL6Rx6N9LbMywwUSMtxET+7lVoDLLd3O3IXwJwvuuns8UB/HeAg==} engines: {node: '>=4'} @@ -3916,6 +3986,10 @@ packages: ms@2.1.3: resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==} + nano-spawn@1.0.2: + resolution: {integrity: sha512-21t+ozMQDAL/UGgQVBbZ/xXvNO10++ZPuTmKRO8k9V3AClVRht49ahtDjfY8l1q6nSHOrE5ASfthzH3ol6R/hg==} + engines: {node: '>=20.17'} + nanoid@3.3.11: resolution: {integrity: sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==} engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} @@ -4048,6 +4122,10 @@ packages: resolution: {integrity: sha512-y0W+X7Ppo7oZX6eovsRkuzcSM40Bicg2JEJkDJ4irIt1wsYAP5MLSNv+QAogO8xivMffw/9OvV3um1pxXgt1uA==} engines: {node: ^10.13.0 || >=12.0.0} + onetime@7.0.0: + resolution: {integrity: sha512-VXJjc87FScF88uafS3JllDgvAm+c/Slfz06lorj2uAY34rlUu0Nt+v8wreiImcrgAjjIHp1rXpTDlLOGw29WwQ==} + engines: {node: '>=18'} + opener@1.5.2: resolution: {integrity: sha512-ur5UIdyw5Y7yEj9wLzhqXiy6GZ3Mwx0yGI+5sMn2r0N0v3cKJvUmFH5yPP+WXh9e0xfyzyJX95D8l088DNFj7A==} hasBin: true @@ -4159,6 +4237,11 @@ packages: resolution: {integrity: sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==} engines: {node: '>=12'} + pidtree@0.6.0: + resolution: {integrity: sha512-eG2dWTVw5bzqGRztnHExczNxt5VGsE6OwTeCG3fdUf9KBsZzO3R5OIIIzWR+iZA0NtZ+RDVdaoE2dK1cn6jH4g==} + engines: {node: '>=0.10'} + hasBin: true + playwright-core@1.54.1: resolution: {integrity: sha512-Nbjs2zjj0htNhzgiy5wu+3w09YetDx5pkrpI/kZotDlDUaYk0HVA5xrBVPdow4SAUIlhgKcJeJg4GRKW6xHusA==} engines: {node: '>=18'} @@ -4429,10 +4512,17 @@ packages: resolution: {integrity: sha512-U7WjGVG9sH8tvjW5SmGbQuui75FiyjAX72HX15DwBBwF9dNiQZRQAg9nnPhYy+TUnE0+VcrttuvNI8oSxZcocA==} hasBin: true + restore-cursor@5.1.0: + resolution: {integrity: sha512-oMA2dcrw6u0YfxJQXm342bFKX/E4sG9rbTzO9ptUcR/e8A33cHuvStiYOwH7fszkZlZ1z/ta9AAoPk2F4qIOHA==} + engines: {node: '>=18'} + reusify@1.1.0: resolution: {integrity: sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw==} engines: {iojs: '>=1.0.0', node: '>=0.10.0'} + rfdc@1.4.1: + resolution: {integrity: sha512-q1b3N5QkRUWUl7iyylaaj3kOpIT0N2i9MqIEQXP73GVsN9cw3fdx8X63cEmWhJGi2PPCF23Ijp7ktmd39rawIA==} + rndm@1.2.0: resolution: {integrity: sha512-fJhQQI5tLrQvYIYFpOnFinzv9dwmR7hRnUz1XqP3OJ1jIweTNOd6aTO4jwQSgcBSFUB+/KHJxuGneime+FdzOw==} @@ -4552,6 +4642,14 @@ packages: resolution: {integrity: sha512-ZA6oR3T/pEyuqwMgAKT0/hAv8oAXckzbkmR0UkUosQ+Mc4RxGoJkRmwHgHufaenlyAgE1Mxgpdcrf75y6XcnDg==} engines: {node: '>=14.16'} + slice-ansi@5.0.0: + resolution: {integrity: sha512-FC+lgizVPfie0kkhqUScwRu1O/lF6NOgJmlCgK+/LYxDCTk8sGelYaHDhFcDN+Sn3Cv+3VSa4Byeo+IMCzpMgQ==} + engines: {node: '>=12'} + + slice-ansi@7.1.0: + resolution: {integrity: sha512-bSiSngZ/jWeX93BqeIAbImyTbEihizcwNjFoRUIY/T1wWQsfsm2Vw1agPKylXvQTU7iASGdHhyqRlqQzfz+Htg==} + engines: {node: '>=18'} + sonner@2.0.6: resolution: {integrity: sha512-yHFhk8T/DK3YxjFQXIrcHT1rGEeTLliVzWbO0xN8GberVun2RiBnxAjXAYpZrqwEVHBG9asI/Li8TAAhN9m59Q==} peerDependencies: @@ -4586,6 +4684,10 @@ packages: resolution: {integrity: sha512-Mcc5wHehp9aXz1ax6bZUyY5afg9u2rv5cqQI3mRrYkGC8rW2hM02jWuwjtL++LS5qinSyhj2QfLyNsuc+VsExg==} engines: {node: '>=10.0.0'} + string-argv@0.3.2: + resolution: {integrity: sha512-aqD2Q0144Z+/RqG52NeHEkZauTAUWJO8c6yTftGJKO3Tja5tUgIfmIl6kExvhtxSDP7fXB6DvzkfMpCd/F3G+Q==} + engines: {node: '>=0.6.19'} + string-width@4.2.3: resolution: {integrity: sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==} engines: {node: '>=8'} @@ -4594,6 +4696,10 @@ packages: resolution: {integrity: sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA==} engines: {node: '>=12'} + string-width@7.2.0: + resolution: {integrity: sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ==} + engines: {node: '>=18'} + string.prototype.includes@2.0.1: resolution: {integrity: sha512-o7+c9bW6zpAdJHTtujeePODAhkuicdAryFsfVKwA+wGw89wJ4GTY484WTucM9hLtDEOpOvI+aHnzqnC5lHp4Rg==} engines: {node: '>= 0.4'} @@ -5095,6 +5201,10 @@ packages: resolution: {integrity: sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ==} engines: {node: '>=12'} + wrap-ansi@9.0.0: + resolution: {integrity: sha512-G8ura3S+3Z2G+mkgNRq8dqaFZAuxfsxpBB8OCTGRTCtp+l/v9nbFNmCUP1BZMts3G1142MsZfn6eeUKrr4PD1Q==} + engines: {node: '>=18'} + ws@7.5.10: resolution: {integrity: sha512-+dbF1tHwZpXcbOJdVOkzLDxZP1ailvSxM6ZweXTegylPny803bFhA+vqBYw4s31NSAk4S2Qz+AKXK9a4wkdjcQ==} engines: {node: '>=8.3.0'} @@ -5144,6 +5254,11 @@ packages: resolution: {integrity: sha512-YgvUTfwqyc7UXVMrB+SImsVYSmTS8X/tSrtdNZMImM+n7+QTriRXyXim0mBrTXNeqzVF0KWGgHPeiyViFFrNDw==} engines: {node: '>=18'} + yaml@2.8.0: + resolution: {integrity: sha512-4lLa/EcQCB0cJkyts+FpIRx5G/llPxfP6VQU5KByHEhLxY3IJCH0f0Hy1MHI8sClTvsIb8qwRJ6R/ZdlDJ/leQ==} + engines: {node: '>= 14.6'} + hasBin: true + yargs-parser@21.1.1: resolution: {integrity: sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==} engines: {node: '>=12'} @@ -6884,7 +6999,7 @@ snapshots: '@unrs/resolver-binding-win32-x64-msvc@1.11.1': optional: true - '@vitejs/plugin-react@4.6.0(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3))': + '@vitejs/plugin-react@4.6.0(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0))': dependencies: '@babel/core': 7.28.0 '@babel/plugin-transform-react-jsx-self': 7.27.1(@babel/core@7.28.0) @@ -6892,11 +7007,11 @@ snapshots: '@rolldown/pluginutils': 1.0.0-beta.19 '@types/babel__core': 7.20.5 react-refresh: 0.17.0 - vite: 7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3) + vite: 7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0) transitivePeerDependencies: - supports-color - '@vitest/coverage-v8@3.2.4(vitest@3.2.4(@types/debug@4.1.12)(@types/node@24.0.13)(jiti@2.4.2)(jsdom@26.1.0)(lightningcss@1.30.1)(tsx@4.20.3))': + '@vitest/coverage-v8@3.2.4(vitest@3.2.4(@types/debug@4.1.12)(@types/node@24.0.13)(jiti@2.4.2)(jsdom@26.1.0)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0))': dependencies: '@ampproject/remapping': 2.3.0 '@bcoe/v8-coverage': 1.0.2 @@ -6911,7 +7026,7 @@ snapshots: std-env: 3.9.0 test-exclude: 7.0.1 tinyrainbow: 2.0.0 - vitest: 3.2.4(@types/debug@4.1.12)(@types/node@24.0.13)(jiti@2.4.2)(jsdom@26.1.0)(lightningcss@1.30.1)(tsx@4.20.3) + vitest: 3.2.4(@types/debug@4.1.12)(@types/node@24.0.13)(jiti@2.4.2)(jsdom@26.1.0)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0) transitivePeerDependencies: - supports-color @@ -6923,13 +7038,13 @@ snapshots: chai: 5.2.1 tinyrainbow: 2.0.0 - '@vitest/mocker@3.2.4(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3))': + '@vitest/mocker@3.2.4(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0))': dependencies: '@vitest/spy': 3.2.4 estree-walker: 3.0.3 magic-string: 0.30.17 optionalDependencies: - vite: 7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3) + vite: 7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0) '@vitest/pretty-format@3.2.4': dependencies: @@ -6981,6 +7096,10 @@ snapshots: json-schema-traverse: 0.4.1 uri-js: 4.4.1 + ansi-escapes@7.0.0: + dependencies: + environment: 1.1.0 + ansi-regex@5.0.1: {} ansi-regex@6.1.0: {} @@ -7173,6 +7292,8 @@ snapshots: ansi-styles: 4.3.0 supports-color: 7.2.0 + chalk@5.4.1: {} + character-entities-html4@2.1.0: {} character-entities-legacy@3.0.0: {} @@ -7189,6 +7310,15 @@ snapshots: dependencies: clsx: 2.1.1 + cli-cursor@5.0.0: + dependencies: + restore-cursor: 5.1.0 + + cli-truncate@4.0.0: + dependencies: + slice-ansi: 5.0.0 + string-width: 7.2.0 + client-only@0.0.1: {} cliui@8.0.1: @@ -7219,6 +7349,8 @@ snapshots: color-string: 1.9.1 optional: true + colorette@2.0.20: {} + combined-stream@1.0.8: dependencies: delayed-stream: 1.0.0 @@ -7547,6 +7679,8 @@ snapshots: electron-to-chromium@1.5.182: {} + emoji-regex@10.4.0: {} + emoji-regex@8.0.0: {} emoji-regex@9.2.2: {} @@ -7560,6 +7694,8 @@ snapshots: entities@6.0.1: {} + environment@1.1.0: {} + es-abstract@1.24.0: dependencies: array-buffer-byte-length: 1.0.2 @@ -8026,6 +8162,8 @@ snapshots: get-caller-file@2.0.5: {} + get-east-asian-width@1.3.0: {} + get-intrinsic@1.3.0: dependencies: call-bind-apply-helpers: 1.0.2 @@ -8220,6 +8358,8 @@ snapshots: transitivePeerDependencies: - supports-color + husky@9.1.7: {} + i18n-iso-countries@7.14.0: dependencies: diacritics: 1.3.0 @@ -8317,6 +8457,12 @@ snapshots: is-fullwidth-code-point@3.0.0: {} + is-fullwidth-code-point@4.0.0: {} + + is-fullwidth-code-point@5.0.0: + dependencies: + get-east-asian-width: 1.3.0 + is-generator-function@1.1.0: dependencies: call-bound: 1.0.4 @@ -8581,12 +8727,38 @@ snapshots: lightningcss-win32-arm64-msvc: 1.30.1 lightningcss-win32-x64-msvc: 1.30.1 + lilconfig@3.1.3: {} + lineclip@1.1.5: {} linkify-it@5.0.0: dependencies: uc.micro: 2.1.0 + lint-staged@16.1.2: + dependencies: + chalk: 5.4.1 + commander: 14.0.0 + debug: 4.4.1 + lilconfig: 3.1.3 + listr2: 8.3.3 + micromatch: 4.0.8 + nano-spawn: 1.0.2 + pidtree: 0.6.0 + string-argv: 0.3.2 + yaml: 2.8.0 + transitivePeerDependencies: + - supports-color + + listr2@8.3.3: + dependencies: + cli-truncate: 4.0.0 + colorette: 2.0.20 + eventemitter3: 5.0.1 + log-update: 6.1.0 + rfdc: 1.4.1 + wrap-ansi: 9.0.0 + locate-path@6.0.0: dependencies: p-locate: 5.0.0 @@ -8595,6 +8767,14 @@ snapshots: lodash@4.17.21: {} + log-update@6.1.0: + dependencies: + ansi-escapes: 7.0.0 + cli-cursor: 5.0.0 + slice-ansi: 7.1.0 + strip-ansi: 7.1.0 + wrap-ansi: 9.0.0 + longest-streak@3.1.0: {} loose-envify@1.4.0: @@ -8968,6 +9148,8 @@ snapshots: mime@1.6.0: {} + mimic-function@5.0.1: {} + min-indent@1.0.1: {} minimatch@3.1.2: @@ -9006,6 +9188,8 @@ snapshots: ms@2.1.3: {} + nano-spawn@1.0.2: {} + nanoid@3.3.11: {} napi-postinstall@0.3.0: {} @@ -9137,6 +9321,10 @@ snapshots: oidc-token-hash@5.1.0: {} + onetime@7.0.0: + dependencies: + mimic-function: 5.0.1 + opener@1.5.2: {} openid-client@5.7.1: @@ -9249,6 +9437,8 @@ snapshots: picomatch@4.0.2: {} + pidtree@0.6.0: {} + playwright-core@1.54.1: {} playwright@1.54.1: @@ -9534,8 +9724,15 @@ snapshots: path-parse: 1.0.7 supports-preserve-symlinks-flag: 1.0.0 + restore-cursor@5.1.0: + dependencies: + onetime: 7.0.0 + signal-exit: 4.1.0 + reusify@1.1.0: {} + rfdc@1.4.1: {} + rndm@1.2.0: {} robust-predicates@3.0.2: {} @@ -9716,6 +9913,16 @@ snapshots: slash@5.1.0: {} + slice-ansi@5.0.0: + dependencies: + ansi-styles: 6.2.1 + is-fullwidth-code-point: 4.0.0 + + slice-ansi@7.1.0: + dependencies: + ansi-styles: 6.2.1 + is-fullwidth-code-point: 5.0.0 + sonner@2.0.6(react-dom@19.1.0(react@19.1.0))(react@19.1.0): dependencies: react: 19.1.0 @@ -9740,6 +9947,8 @@ snapshots: streamsearch@1.1.0: {} + string-argv@0.3.2: {} + string-width@4.2.3: dependencies: emoji-regex: 8.0.0 @@ -9752,6 +9961,12 @@ snapshots: emoji-regex: 9.2.2 strip-ansi: 7.1.0 + string-width@7.2.0: + dependencies: + emoji-regex: 10.4.0 + get-east-asian-width: 1.3.0 + strip-ansi: 7.1.0 + string.prototype.includes@2.0.1: dependencies: call-bind: 1.0.8 @@ -10164,13 +10379,13 @@ snapshots: d3-time: 3.1.0 d3-timer: 3.0.1 - vite-node@3.2.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3): + vite-node@3.2.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0): dependencies: cac: 6.7.14 debug: 4.4.1 es-module-lexer: 1.7.0 pathe: 2.0.3 - vite: 7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3) + vite: 7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0) transitivePeerDependencies: - '@types/node' - jiti @@ -10185,18 +10400,18 @@ snapshots: - tsx - yaml - vite-tsconfig-paths@5.1.4(typescript@5.8.3)(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)): + vite-tsconfig-paths@5.1.4(typescript@5.8.3)(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0)): dependencies: debug: 4.4.1 globrex: 0.1.2 tsconfck: 3.1.6(typescript@5.8.3) optionalDependencies: - vite: 7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3) + vite: 7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0) transitivePeerDependencies: - supports-color - typescript - vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3): + vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0): dependencies: esbuild: 0.25.6 fdir: 6.4.6(picomatch@4.0.2) @@ -10210,12 +10425,13 @@ snapshots: jiti: 2.4.2 lightningcss: 1.30.1 tsx: 4.20.3 + yaml: 2.8.0 - vitest@3.2.4(@types/debug@4.1.12)(@types/node@24.0.13)(jiti@2.4.2)(jsdom@26.1.0)(lightningcss@1.30.1)(tsx@4.20.3): + vitest@3.2.4(@types/debug@4.1.12)(@types/node@24.0.13)(jiti@2.4.2)(jsdom@26.1.0)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0): dependencies: '@types/chai': 5.2.2 '@vitest/expect': 3.2.4 - '@vitest/mocker': 3.2.4(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)) + '@vitest/mocker': 3.2.4(vite@7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0)) '@vitest/pretty-format': 3.2.4 '@vitest/runner': 3.2.4 '@vitest/snapshot': 3.2.4 @@ -10233,8 +10449,8 @@ snapshots: tinyglobby: 0.2.14 tinypool: 1.1.1 tinyrainbow: 2.0.0 - vite: 7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3) - vite-node: 3.2.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3) + vite: 7.0.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0) + vite-node: 3.2.4(@types/node@24.0.13)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.20.3)(yaml@2.8.0) why-is-node-running: 2.3.0 optionalDependencies: '@types/debug': 4.1.12 @@ -10381,6 +10597,12 @@ snapshots: string-width: 5.1.2 strip-ansi: 7.1.0 + wrap-ansi@9.0.0: + dependencies: + ansi-styles: 6.2.1 + string-width: 7.2.0 + strip-ansi: 7.1.0 + ws@7.5.10: {} ws@8.18.3: {} @@ -10399,6 +10621,8 @@ snapshots: yallist@5.0.0: {} + yaml@2.8.0: {} + yargs-parser@21.1.1: {} yargs@17.7.2: diff --git a/refactor/implement-tRPC.md b/refactor/implement-tRPC.md index 95d27ef..7c8e9f6 100644 --- a/refactor/implement-tRPC.md +++ b/refactor/implement-tRPC.md @@ -15,10 +15,10 @@ The migration will be performed incrementally to minimize disruption. We will st ### Why tRPC? -- **End-to-End Type Safety:** Eliminates a class of runtime errors by ensuring the client and server conform to the same data contracts. TypeScript errors will appear at build time if the client and server are out of sync. -- **Improved Developer Experience:** Provides autocompletion for API procedures and their data types directly in the editor. -- **Simplified Data Fetching:** Replaces manual `fetch` calls and `useEffect` hooks with clean, declarative tRPC hooks (`useQuery`, `useMutation`). -- **No Code Generation:** Leverages TypeScript inference, avoiding a separate schema definition or code generation step. +- **End-to-End Type Safety:** Eliminates a class of runtime errors by ensuring the client and server conform to the same data contracts. TypeScript errors will appear at build time if the client and server are out of sync. +- **Improved Developer Experience:** Provides autocompletion for API procedures and their data types directly in the editor. +- **Simplified Data Fetching:** Replaces manual `fetch` calls and `useEffect` hooks with clean, declarative tRPC hooks (`useQuery`, `useMutation`). +- **No Code Generation:** Leverages TypeScript inference, avoiding a separate schema definition or code generation step. ### Integration Strategy: Gradual Adoption @@ -240,11 +240,11 @@ export default function UsersPage() { ## 4. Next Steps & Future Enhancements -- **Authentication & Context:** Implement a `createContext` function to pass session data (e.g., from NextAuth.js) to your tRPC procedures. This will allow for protected procedures. -- **Input Validation:** Extensively use `zod` in the `.input()` part of procedures to validate all incoming data. -- **Error Handling:** Implement robust error handling on both the client and server. -- **Mutations:** Begin migrating `POST`, `PUT`, and `DELETE` endpoints to tRPC mutations. -- **Optimistic UI:** For mutations, implement optimistic updates to provide a faster user experience. +- **Authentication & Context:** Implement a `createContext` function to pass session data (e.g., from NextAuth.js) to your tRPC procedures. This will allow for protected procedures. +- **Input Validation:** Extensively use `zod` in the `.input()` part of procedures to validate all incoming data. +- **Error Handling:** Implement robust error handling on both the client and server. +- **Mutations:** Begin migrating `POST`, `PUT`, and `DELETE` endpoints to tRPC mutations. +- **Optimistic UI:** For mutations, implement optimistic updates to provide a faster user experience. --- diff --git a/scripts/migration/README.md b/scripts/migration/README.md index b760880..ae6eb3c 100644 --- a/scripts/migration/README.md +++ b/scripts/migration/README.md @@ -6,58 +6,58 @@ This directory contains comprehensive migration scripts for deploying the new ar ### 1. Database Migrations -- `01-schema-migrations.sql` - Prisma database schema migrations -- `02-data-migrations.sql` - Data transformation scripts -- `validate-database.ts` - Database validation and health checks +- `01-schema-migrations.sql` - Prisma database schema migrations +- `02-data-migrations.sql` - Data transformation scripts +- `validate-database.ts` - Database validation and health checks ### 2. Environment Configuration -- `environment-migration.ts` - Environment variable migration guide -- `config-validator.ts` - Configuration validation scripts +- `environment-migration.ts` - Environment variable migration guide +- `config-validator.ts` - Configuration validation scripts ### 3. Deployment Scripts -- `deploy.ts` - Main deployment orchestrator -- `pre-deployment-checks.ts` - Pre-deployment validation -- `post-deployment-validation.ts` - Post-deployment verification -- `rollback.ts` - Rollback procedures +- `deploy.ts` - Main deployment orchestrator +- `pre-deployment-checks.ts` - Pre-deployment validation +- `post-deployment-validation.ts` - Post-deployment verification +- `rollback.ts` - Rollback procedures ### 4. Health Checks -- `health-checks.ts` - Comprehensive system health validation -- `trpc-endpoint-tests.ts` - tRPC endpoint validation -- `batch-processing-tests.ts` - Batch processing system tests +- `health-checks.ts` - Comprehensive system health validation +- `trpc-endpoint-tests.ts` - tRPC endpoint validation +- `batch-processing-tests.ts` - Batch processing system tests ### 5. Migration Utilities -- `backup-database.ts` - Database backup procedures -- `restore-database.ts` - Database restore procedures -- `migration-logger.ts` - Migration logging utilities +- `backup-database.ts` - Database backup procedures +- `restore-database.ts` - Database restore procedures +- `migration-logger.ts` - Migration logging utilities ## Usage ### Pre-Migration -1. Run database backup: `pnpm migration:backup` -2. Validate environment: `pnpm migration:validate-env` -3. Run pre-deployment checks: `pnpm migration:pre-check` +1. Run database backup: `pnpm migration:backup` +2. Validate environment: `pnpm migration:validate-env` +3. Run pre-deployment checks: `pnpm migration:pre-check` ### Migration -1. Run schema migrations: `pnpm migration:schema` -2. Run data migrations: `pnpm migration:data` -3. Deploy application: `pnpm migration:deploy` +1. Run schema migrations: `pnpm migration:schema` +2. Run data migrations: `pnpm migration:data` +3. Deploy application: `pnpm migration:deploy` ### Post-Migration -1. Validate deployment: `pnpm migration:validate` -2. Run health checks: `pnpm migration:health-check` -3. Test critical paths: `pnpm migration:test` +1. Validate deployment: `pnpm migration:validate` +2. Run health checks: `pnpm migration:health-check` +3. Test critical paths: `pnpm migration:test` ### Rollback (if needed) -1. Rollback deployment: `pnpm migration:rollback` -2. Restore database: `pnpm migration:restore` +1. Rollback deployment: `pnpm migration:rollback` +2. Restore database: `pnpm migration:restore` ## Environment Variables @@ -86,17 +86,17 @@ MIGRATION_ROLLBACK_ENABLED=true The migration implements a blue-green deployment strategy: -1. **Phase 1**: Deploy new code with feature flags disabled -2. **Phase 2**: Run database migrations -3. **Phase 3**: Enable tRPC endpoints progressively -4. **Phase 4**: Enable batch processing system -5. **Phase 5**: Full activation and old system decommission +1. **Phase 1**: Deploy new code with feature flags disabled +2. **Phase 2**: Run database migrations +3. **Phase 3**: Enable tRPC endpoints progressively +4. **Phase 4**: Enable batch processing system +5. **Phase 5**: Full activation and old system decommission ## Safety Features -- Automatic database backups before migration -- Rollback scripts for quick recovery -- Health checks at each stage -- Progressive feature enablement -- Comprehensive logging and monitoring -- Backwards compatibility maintained during migration +- Automatic database backups before migration +- Rollback scripts for quick recovery +- Health checks at each stage +- Progressive feature enablement +- Comprehensive logging and monitoring +- Backwards compatibility maintained during migration