feat: complete development environment setup and code quality improvements

- Set up pre-commit hooks with husky and lint-staged for automated code quality
- Improved TypeScript type safety by replacing 'any' types with proper generics
- Fixed markdown linting violations (MD030 spacing) across all documentation
- Fixed compound adjective hyphenation in technical documentation
- Fixed invalid JSON union syntax in API documentation examples
- Automated code formatting and linting on commit
- Enhanced error handling with better type constraints
- Configured biome and markdownlint for consistent code style
- All changes verified with successful production build
This commit is contained in:
2025-07-13 14:44:05 +02:00
parent 1d4e695e41
commit e2301725a3
54 changed files with 2335 additions and 1863 deletions

View File

@ -6,10 +6,10 @@ This document describes the comprehensive CSRF (Cross-Site Request Forgery) prot
CSRF protection has been implemented to prevent cross-site request forgery attacks on state-changing operations. The implementation follows industry best practices and provides protection at multiple layers:
- **Middleware Level**: Automatic CSRF validation for protected endpoints
- **tRPC Level**: CSRF protection for all state-changing tRPC procedures
- **Client Level**: Automatic token management and inclusion in requests
- **Component Level**: React components and hooks for easy integration
- **Middleware Level**: Automatic CSRF validation for protected endpoints
- **tRPC Level**: CSRF protection for all state-changing tRPC procedures
- **Client Level**: Automatic token management and inclusion in requests
- **Component Level**: React components and hooks for easy integration
## Implementation Components
@ -17,52 +17,58 @@ CSRF protection has been implemented to prevent cross-site request forgery attac
The core CSRF functionality includes:
- **Token Generation**: Cryptographically secure token generation using the `csrf` library
- **Token Verification**: Server-side token validation
- **Request Parsing**: Support for tokens in headers, JSON bodies, and form data
- **Client Utilities**: Browser-side token management and request enhancement
- **Token Generation**: Cryptographically secure token generation using the `csrf` library
- **Token Verification**: Server-side token validation
- **Request Parsing**: Support for tokens in headers, JSON bodies, and form data
- **Client Utilities**: Browser-side token management and request enhancement
**Key Functions:**
- `generateCSRFToken()` - Creates new CSRF tokens
- `verifyCSRFToken()` - Validates tokens server-side
- `CSRFProtection.validateRequest()` - Request validation middleware
- `CSRFClient.*` - Client-side utilities
- `generateCSRFToken()` - Creates new CSRF tokens
- `verifyCSRFToken()` - Validates tokens server-side
- `CSRFProtection.validateRequest()` - Request validation middleware
- `CSRFClient.*` - Client-side utilities
### 2. Middleware Protection (`middleware/csrfProtection.ts`)
Provides automatic CSRF protection for API endpoints:
**Protected Endpoints:**
- `/api/auth/*` - Authentication endpoints
- `/api/register` - User registration
- `/api/forgot-password` - Password reset requests
- `/api/reset-password` - Password reset completion
- `/api/dashboard/*` - Dashboard API endpoints
- `/api/platform/*` - Platform admin endpoints
- `/api/trpc/*` - All tRPC endpoints
- `/api/auth/*` - Authentication endpoints
- `/api/register` - User registration
- `/api/forgot-password` - Password reset requests
- `/api/reset-password` - Password reset completion
- `/api/dashboard/*` - Dashboard API endpoints
- `/api/platform/*` - Platform admin endpoints
- `/api/trpc/*` - All tRPC endpoints
**Protected Methods:**
- `POST` - Create operations
- `PUT` - Update operations
- `DELETE` - Delete operations
- `PATCH` - Partial update operations
- `POST` - Create operations
- `PUT` - Update operations
- `DELETE` - Delete operations
- `PATCH` - Partial update operations
**Safe Methods (Not Protected):**
- `GET` - Read operations
- `HEAD` - Metadata requests
- `OPTIONS` - CORS preflight requests
- `GET` - Read operations
- `HEAD` - Metadata requests
- `OPTIONS` - CORS preflight requests
### 3. tRPC Integration (`lib/trpc.ts`)
CSRF protection integrated into tRPC procedures:
**New Procedure Types:**
- `csrfProtectedProcedure` - Basic CSRF protection
- `csrfProtectedAuthProcedure` - CSRF + authentication protection
- `csrfProtectedCompanyProcedure` - CSRF + company access protection
- `csrfProtectedAdminProcedure` - CSRF + admin access protection
- `csrfProtectedProcedure` - Basic CSRF protection
- `csrfProtectedAuthProcedure` - CSRF + authentication protection
- `csrfProtectedCompanyProcedure` - CSRF + company access protection
- `csrfProtectedAdminProcedure` - CSRF + admin access protection
**Updated Router Example:**
```typescript
// Before
register: rateLimitedProcedure
@ -78,30 +84,35 @@ register: csrfProtectedProcedure
### 4. Client-Side Integration
#### tRPC Client (`lib/trpc-client.ts`)
- Automatic CSRF token inclusion in tRPC requests
- Token extracted from cookies and added to request headers
- Automatic CSRF token inclusion in tRPC requests
- Token extracted from cookies and added to request headers
#### React Hooks (`lib/hooks/useCSRF.ts`)
- `useCSRF()` - Basic token management
- `useCSRFFetch()` - Enhanced fetch with automatic CSRF tokens
- `useCSRFForm()` - Form submission with CSRF protection
- `useCSRF()` - Basic token management
- `useCSRFFetch()` - Enhanced fetch with automatic CSRF tokens
- `useCSRFForm()` - Form submission with CSRF protection
#### Provider Component (`components/providers/CSRFProvider.tsx`)
- Application-wide CSRF token management
- Automatic token fetching and refresh
- Context-based token sharing
- Application-wide CSRF token management
- Automatic token fetching and refresh
- Context-based token sharing
#### Protected Form Component (`components/forms/CSRFProtectedForm.tsx`)
- Ready-to-use form component with CSRF protection
- Automatic token inclusion in form submissions
- Graceful fallback for non-JavaScript environments
- Ready-to-use form component with CSRF protection
- Automatic token inclusion in form submissions
- Graceful fallback for non-JavaScript environments
### 5. API Endpoint (`app/api/csrf-token/route.ts`)
Provides CSRF tokens to client applications:
- `GET /api/csrf-token` - Returns new CSRF token
- Sets HTTP-only cookie for automatic inclusion
- Used by client-side hooks and components
- `GET /api/csrf-token` - Returns new CSRF token
- Sets HTTP-only cookie for automatic inclusion
- Used by client-side hooks and components
## Configuration
@ -206,33 +217,38 @@ const dataWithToken = CSRFClient.addTokenToObject({ data: 'example' });
## Security Features
### 1. Token Properties
- **Cryptographically Secure**: Uses the `csrf` library with secure random generation
- **Short-Lived**: 24-hour expiration by default
- **HTTP-Only Cookies**: Prevents XSS-based token theft
- **SameSite Protection**: Reduces CSRF attack surface
- **Cryptographically Secure**: Uses the `csrf` library with secure random generation
- **Short-Lived**: 24-hour expiration by default
- **HTTP-Only Cookies**: Prevents XSS-based token theft
- **SameSite Protection**: Reduces CSRF attack surface
### 2. Validation Process
1. Extract token from request (header, form data, or JSON body)
2. Retrieve stored token from HTTP-only cookie
3. Verify tokens match
4. Validate token cryptographic integrity
5. Allow or reject request based on validation
1. Extract token from request (header, form data, or JSON body)
2. Retrieve stored token from HTTP-only cookie
3. Verify tokens match
4. Validate token cryptographic integrity
5. Allow or reject request based on validation
### 3. Error Handling
- **Graceful Degradation**: Form fallbacks for JavaScript-disabled browsers
- **Clear Error Messages**: Specific error codes for debugging
- **Rate Limiting Integration**: Works with existing auth rate limiting
- **Logging**: Comprehensive logging for security monitoring
- **Graceful Degradation**: Form fallbacks for JavaScript-disabled browsers
- **Clear Error Messages**: Specific error codes for debugging
- **Rate Limiting Integration**: Works with existing auth rate limiting
- **Logging**: Comprehensive logging for security monitoring
## Testing
### Test Coverage
- **Unit Tests**: Token generation, validation, and client utilities
- **Integration Tests**: Middleware behavior and endpoint protection
- **Component Tests**: React hooks and form components
- **End-to-End**: Full request/response cycle testing
- **Unit Tests**: Token generation, validation, and client utilities
- **Integration Tests**: Middleware behavior and endpoint protection
- **Component Tests**: React hooks and form components
- **End-to-End**: Full request/response cycle testing
### Running Tests
```bash
# Run all CSRF tests
pnpm test:vitest tests/unit/csrf*.test.ts tests/integration/csrf*.test.ts
@ -246,32 +262,36 @@ pnpm test:vitest tests/unit/csrf-hooks.test.tsx
## Monitoring and Debugging
### CSRF Validation Logs
Failed CSRF validations are logged with details:
```
CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from request
```
### Common Issues and Solutions
1. **Token Missing from Request**
- Ensure CSRFProvider is wrapping your app
- Check that hooks are being used correctly
- Verify network requests include credentials
1. **Token Missing from Request**
- Ensure CSRFProvider is wrapping your app
- Check that hooks are being used correctly
- Verify network requests include credentials
2. **Token Mismatch**
- Clear browser cookies and refresh
- Check for multiple token sources conflicting
- Verify server and client time synchronization
2. **Token Mismatch**
- Clear browser cookies and refresh
- Check for multiple token sources conflicting
- Verify server and client time synchronization
3. **Integration Issues**
- Ensure middleware is properly configured
- Check tRPC client configuration
- Verify protected procedures are using correct types
3. **Integration Issues**
- Ensure middleware is properly configured
- Check tRPC client configuration
- Verify protected procedures are using correct types
## Migration Guide
### For Existing Endpoints
1. Update tRPC procedures to use CSRF-protected variants:
1. Update tRPC procedures to use CSRF-protected variants:
```typescript
// Old
someAction: protectedProcedure.mutation(...)
@ -280,7 +300,8 @@ CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from
someAction: csrfProtectedAuthProcedure.mutation(...)
```
2. Update client components to use CSRF hooks:
2. Update client components to use CSRF hooks:
```tsx
// Old
const { data, mutate } = trpc.user.update.useMutation();
@ -289,7 +310,8 @@ CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from
const { data, mutate } = trpc.user.update.useMutation();
```
3. Update manual API calls to include CSRF tokens:
3. Update manual API calls to include CSRF tokens:
```typescript
// Old
fetch('/api/endpoint', { method: 'POST', ... });
@ -301,21 +323,21 @@ CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from
## Performance Considerations
- **Minimal Overhead**: Token validation adds ~1ms per request
- **Efficient Caching**: Tokens cached in memory and cookies
- **Selective Protection**: Only state-changing operations protected
- **Optimized Parsing**: Smart content-type detection for token extraction
- **Minimal Overhead**: Token validation adds ~1ms per request
- **Efficient Caching**: Tokens cached in memory and cookies
- **Selective Protection**: Only state-changing operations protected
- **Optimized Parsing**: Smart content-type detection for token extraction
## Security Best Practices
1. **Always use HTTPS in production** - CSRF tokens should never be transmitted over HTTP
2. **Monitor CSRF failures** - Implement alerting for unusual CSRF failure patterns
3. **Regular secret rotation** - Consider rotating CSRF secrets periodically
4. **Validate referrer headers** - Additional protection layer (not implemented but recommended)
5. **Content Security Policy** - Use CSP headers to prevent XSS attacks that could steal tokens
1. **Always use HTTPS in production** - CSRF tokens should never be transmitted over HTTP
2. **Monitor CSRF failures** - Implement alerting for unusual CSRF failure patterns
3. **Regular secret rotation** - Consider rotating CSRF secrets periodically
4. **Validate referrer headers** - Additional protection layer (not implemented but recommended)
5. **Content Security Policy** - Use CSP headers to prevent XSS attacks that could steal tokens
## Conclusion
The CSRF protection implementation provides comprehensive defense against cross-site request forgery attacks while maintaining ease of use for developers. The multi-layer approach ensures protection at the middleware, application, and component levels, with automatic token management reducing the risk of developer error.
For questions or issues related to CSRF protection, refer to the test files for examples and the security documentation for additional context.
For questions or issues related to CSRF protection, refer to the test files for examples and the security documentation for additional context.

View File

@ -8,10 +8,10 @@ The Admin Audit Logs API provides secure access to security audit trails for adm
## Authentication & Authorization
- **Authentication**: NextAuth.js session required
- **Authorization**: ADMIN role required for all endpoints
- **Rate Limiting**: Integrated with existing auth rate limiting system
- **Audit Trail**: All API access is logged for security monitoring
- **Authentication**: NextAuth.js session required
- **Authorization**: ADMIN role required for all endpoints
- **Rate-Limiting**: Integrated with existing authentication rate-limiting system
- **Audit Trail**: All API access is logged for security monitoring
## API Endpoints
@ -128,12 +128,14 @@ POST /api/admin/audit-logs/retention
```json
{
"action": "cleanup" | "configure" | "status",
"action": "cleanup",
"retentionDays": 90,
"dryRun": true
}
```
**Note**: `action` field accepts one of: `"cleanup"`, `"configure"`, or `"status"`
#### Parameters
| Parameter | Type | Required | Description |
@ -145,6 +147,7 @@ POST /api/admin/audit-logs/retention
#### Example Requests
**Check retention status:**
```javascript
const response = await fetch('/api/admin/audit-logs/retention', {
method: 'POST',
@ -154,6 +157,7 @@ const response = await fetch('/api/admin/audit-logs/retention', {
```
**Configure retention policy:**
```javascript
const response = await fetch('/api/admin/audit-logs/retention', {
method: 'POST',
@ -166,6 +170,7 @@ const response = await fetch('/api/admin/audit-logs/retention', {
```
**Cleanup old logs (dry run):**
```javascript
const response = await fetch('/api/admin/audit-logs/retention', {
method: 'POST',
@ -180,19 +185,22 @@ const response = await fetch('/api/admin/audit-logs/retention', {
## Security Features
### Access Control
- **Role-based Access**: Only ADMIN users can access audit logs
- **Company Isolation**: Users only see logs for their company
- **Session Validation**: Active NextAuth session required
- **Role-based Access**: Only ADMIN users can access audit logs
- **Company Isolation**: Users only see logs for their company
- **Session Validation**: Active NextAuth session required
### Audit Trail
- **Access Logging**: All audit log access is recorded
- **Metadata Tracking**: Request parameters and results are logged
- **IP Tracking**: Client IP addresses are recorded for all requests
- **Access Logging**: All audit log access is recorded
- **Metadata Tracking**: Request parameters and results are logged
- **IP Tracking**: Client IP addresses are recorded for all requests
### Rate Limiting
- **Integrated Protection**: Uses existing authentication rate limiting
- **Abuse Prevention**: Protects against excessive API usage
- **Error Tracking**: Failed attempts are monitored
- **Integrated Protection**: Uses existing authentication rate-limiting
- **Abuse Prevention**: Protects against excessive API usage
- **Error Tracking**: Failed attempts are monitored
## Event Types
@ -294,19 +302,22 @@ async function getUserActivity(userId, days = 7) {
## Performance Considerations
### Database Optimization
- **Indexed Queries**: All filter columns are properly indexed
- **Pagination**: Efficient offset-based pagination with limits
- **Time Range Filtering**: Optimized for date range queries
- **Indexed Queries**: All filter columns are properly indexed
- **Pagination**: Efficient offset-based pagination with limits
- **Time Range Filtering**: Optimized for date range queries
### Memory Usage
- **Limited Results**: Maximum 100 records per request
- **Streaming**: Large exports use streaming for memory efficiency
- **Connection Pooling**: Database connections are pooled
- **Limited Results**: Maximum 100 records per request
- **Streaming**: Large exports use streaming for memory efficiency
- **Connection Pooling**: Database connections are pooled
### Caching Considerations
- **No Caching**: Audit logs are never cached for security reasons
- **Fresh Data**: All queries hit the database for real-time results
- **Read Replicas**: Consider using read replicas for heavy reporting
- **No Caching**: Audit logs are never cached for security reasons
- **Fresh Data**: All queries hit the database for real-time results
- **Read Replicas**: Consider using read replicas for heavy reporting
## Error Handling
@ -335,7 +346,7 @@ try {
}
```
### Rate Limiting Handling
### Rate-Limiting Handling
```javascript
async function fetchWithRetry(url, options = {}) {
@ -354,40 +365,45 @@ async function fetchWithRetry(url, options = {}) {
## Monitoring and Alerting
### Key Metrics to Monitor
- **Request Volume**: Track API usage patterns
- **Error Rates**: Monitor authentication and authorization failures
- **Query Performance**: Track slow queries and optimize
- **Data Growth**: Monitor audit log size and plan retention
- **Request Volume**: Track API usage patterns
- **Error Rates**: Monitor authentication and authorization failures
- **Query Performance**: Track slow queries and optimize
- **Data Growth**: Monitor audit log size and plan retention
### Alert Conditions
- **High Error Rates**: >5% of requests failing
- **Unusual Access Patterns**: Off-hours access, high volume
- **Performance Degradation**: Query times >2 seconds
- **Security Events**: Multiple failed admin access attempts
- **High Error Rates**: >5% of requests failing
- **Unusual Access Patterns**: Off-hours access, high-volume usage
- **Performance Degradation**: Query times >2 seconds
- **Security Events**: Multiple failed admin access attempts
## Best Practices
### Security
- Always validate user permissions before displaying UI
- Log all administrative access to audit logs
- Use HTTPS in production environments
- Implement proper error handling to avoid information leakage
- Always validate user permissions before displaying UI
- Log all administrative access to audit logs
- Use HTTPS in production environments
- Implement proper error handling to avoid information leakage
### Performance
- Use appropriate page sizes (25-50 records typical)
- Implement client-side pagination for better UX
- Cache results only in memory, never persist
- Use date range filters to limit query scope
- Use appropriate page sizes (25-50 records typical)
- Implement client-side pagination for better UX
- Cache results only in memory, never persist
- Use date range filters to limit query scope
### User Experience
- Provide clear filtering options in the UI
- Show loading states for long-running queries
- Implement export functionality for reports
- Provide search and sort capabilities
- Provide clear filtering options in the UI
- Show loading states for long-running queries
- Implement export functionality for reports
- Provide search and sort capabilities
## Related Documentation
- [Security Audit Logging](./security-audit-logging.md)
- [Security Monitoring](./security-monitoring.md)
- [CSRF Protection](./CSRF_PROTECTION.md)
- [Authentication System](../lib/auth.ts)
- [Security Audit Logging](./security-audit-logging.md)
- [Security Monitoring](./security-monitoring.md)
- [CSRF Protection](./CSRF_PROTECTION.md)
- [Authentication System](../lib/auth.ts)

View File

@ -28,6 +28,7 @@ X-CSRF-Token: <csrf-token>
```
Get CSRF token:
```http
GET /api/csrf-token
```
@ -35,55 +36,64 @@ GET /api/csrf-token
## API Endpoints Overview
### Public Endpoints
- `POST /api/csp-report` - CSP violation reporting (no auth required)
- `OPTIONS /api/csp-report` - CORS preflight
- `POST /api/csp-report` - CSP violation reporting (no auth required)
- `OPTIONS /api/csp-report` - CORS preflight
### Authentication Endpoints
- `POST /api/auth/[...nextauth]` - NextAuth.js authentication
- `GET /api/csrf-token` - Get CSRF token
- `POST /api/register` - User registration
- `POST /api/forgot-password` - Password reset request
- `POST /api/reset-password` - Password reset completion
- `POST /api/auth/[...nextauth]` - NextAuth.js authentication
- `GET /api/csrf-token` - Get CSRF token
- `POST /api/register` - User registration
- `POST /api/forgot-password` - Password reset request
- `POST /api/reset-password` - Password reset completion
### Admin Endpoints (ADMIN role required)
- `GET /api/admin/audit-logs` - Retrieve audit logs
- `POST /api/admin/audit-logs/retention` - Manage audit log retention
- `GET /api/admin/batch-monitoring` - Batch processing monitoring
- `POST /api/admin/batch-monitoring/{id}/retry` - Retry failed batch job
- `GET /api/admin/audit-logs` - Retrieve audit logs
- `POST /api/admin/audit-logs/retention` - Manage audit log retention
- `GET /api/admin/batch-monitoring` - Batch processing monitoring
- `POST /api/admin/batch-monitoring/{id}/retry` - Retry failed batch job
### Platform Admin Endpoints (Platform admin only)
- `GET /api/admin/security-monitoring` - Security monitoring metrics
- `POST /api/admin/security-monitoring` - Update security configuration
- `GET /api/admin/security-monitoring/alerts` - Alert management
- `POST /api/admin/security-monitoring/alerts` - Acknowledge alerts
- `GET /api/admin/security-monitoring/export` - Export security data
- `POST /api/admin/security-monitoring/threat-analysis` - Threat analysis
- `GET /api/admin/security-monitoring` - Security monitoring metrics
- `POST /api/admin/security-monitoring` - Update security configuration
- `GET /api/admin/security-monitoring/alerts` - Alert management
- `POST /api/admin/security-monitoring/alerts` - Acknowledge alerts
- `GET /api/admin/security-monitoring/export` - Export security data
- `POST /api/admin/security-monitoring/threat-analysis` - Threat analysis
### Security Monitoring Endpoints
- `GET /api/csp-metrics` - CSP violation metrics
- `POST /api/csp-report` - CSP violation reporting
- `GET /api/csp-metrics` - CSP violation metrics
- `POST /api/csp-report` - CSP violation reporting
### Dashboard Endpoints
- `GET /api/dashboard/sessions` - Session data
- `GET /api/dashboard/session/{id}` - Individual session details
- `GET /api/dashboard/metrics` - Dashboard metrics
- `GET /api/dashboard/config` - Dashboard configuration
- `GET /api/dashboard/sessions` - Session data
- `GET /api/dashboard/session/{id}` - Individual session details
- `GET /api/dashboard/metrics` - Dashboard metrics
- `GET /api/dashboard/config` - Dashboard configuration
### Platform Management
- `GET /api/platform/companies` - Company management
- `POST /api/platform/companies` - Create company
- `GET /api/platform/companies/{id}` - Company details
- `GET /api/platform/companies/{id}/users` - Company users
- `POST /api/platform/companies/{id}/users` - Add company user
- `GET /api/platform/companies` - Company management
- `POST /api/platform/companies` - Create company
- `GET /api/platform/companies/{id}` - Company details
- `GET /api/platform/companies/{id}/users` - Company users
- `POST /api/platform/companies/{id}/users` - Add company user
### tRPC Endpoints
- `POST /api/trpc/[trpc]` - tRPC procedure calls
- `POST /api/trpc/[trpc]` - tRPC procedure calls
## Detailed Endpoint Documentation
### Admin Audit Logs
#### Get Audit Logs
```http
GET /api/admin/audit-logs
```
@ -91,16 +101,18 @@ GET /api/admin/audit-logs
**Authorization**: ADMIN role required
**Query Parameters**:
- `page` (number, optional): Page number (default: 1)
- `limit` (number, optional): Records per page, max 100 (default: 50)
- `eventType` (string, optional): Filter by event type
- `outcome` (string, optional): Filter by outcome (SUCCESS, FAILURE, BLOCKED, etc.)
- `severity` (string, optional): Filter by severity (LOW, MEDIUM, HIGH, CRITICAL)
- `userId` (string, optional): Filter by user ID
- `startDate` (string, optional): Start date (ISO 8601)
- `endDate` (string, optional): End date (ISO 8601)
- `page` (number, optional): Page number (default: 1)
- `limit` (number, optional): Records per page, max 100 (default: 50)
- `eventType` (string, optional): Filter by event type
- `outcome` (string, optional): Filter by outcome (SUCCESS, FAILURE, BLOCKED, etc.)
- `severity` (string, optional): Filter by severity (LOW, MEDIUM, HIGH, CRITICAL)
- `userId` (string, optional): Filter by user ID
- `startDate` (string, optional): Start date (ISO 8601)
- `endDate` (string, optional): End date (ISO 8601)
**Response**:
```json
{
"success": true,
@ -121,6 +133,7 @@ GET /api/admin/audit-logs
**Rate Limit**: Inherits from auth rate limiting
#### Manage Audit Log Retention
```http
POST /api/admin/audit-logs/retention
```
@ -128,6 +141,7 @@ POST /api/admin/audit-logs/retention
**Authorization**: ADMIN role required
**Request Body**:
```json
{
"action": "cleanup" | "configure" | "status",
@ -137,6 +151,7 @@ POST /api/admin/audit-logs/retention
```
**Response**:
```json
{
"success": true,
@ -152,6 +167,7 @@ POST /api/admin/audit-logs/retention
### Security Monitoring
#### Get Security Metrics
```http
GET /api/admin/security-monitoring
```
@ -159,12 +175,14 @@ GET /api/admin/security-monitoring
**Authorization**: Platform admin required
**Query Parameters**:
- `startDate` (string, optional): Start date (ISO 8601)
- `endDate` (string, optional): End date (ISO 8601)
- `companyId` (string, optional): Filter by company
- `severity` (string, optional): Filter by severity
- `startDate` (string, optional): Start date (ISO 8601)
- `endDate` (string, optional): End date (ISO 8601)
- `companyId` (string, optional): Filter by company
- `severity` (string, optional): Filter by severity
**Response**:
```json
{
"metrics": {
@ -180,6 +198,7 @@ GET /api/admin/security-monitoring
```
#### Update Security Configuration
```http
POST /api/admin/security-monitoring
```
@ -187,6 +206,7 @@ POST /api/admin/security-monitoring
**Authorization**: Platform admin required
**Request Body**:
```json
{
"thresholds": {
@ -203,6 +223,7 @@ POST /api/admin/security-monitoring
### CSP Monitoring
#### CSP Violation Reporting
```http
POST /api/csp-report
```
@ -210,9 +231,11 @@ POST /api/csp-report
**Authorization**: None (public endpoint)
**Headers**:
- `Content-Type`: `application/csp-report` or `application/json`
- `Content-Type`: `application/csp-report` or `application/json`
**Request Body** (automatic from browser):
```json
{
"csp-report": {
@ -230,6 +253,7 @@ POST /api/csp-report
**Response**: `204 No Content`
#### Get CSP Metrics
```http
GET /api/csp-metrics
```
@ -237,12 +261,14 @@ GET /api/csp-metrics
**Authorization**: Admin role required
**Query Parameters**:
- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d)
- `format` (string, optional): Response format (json, csv)
- `groupBy` (string, optional): Group by field (hour, directive, etc.)
- `includeDetails` (boolean, optional): Include violation details
- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d)
- `format` (string, optional): Response format (json, csv)
- `groupBy` (string, optional): Group by field (hour, directive, etc.)
- `includeDetails` (boolean, optional): Include violation details
**Response**:
```json
{
"success": true,
@ -264,6 +290,7 @@ GET /api/csp-metrics
### Batch Monitoring
#### Get Batch Monitoring Data
```http
GET /api/admin/batch-monitoring
```
@ -271,14 +298,16 @@ GET /api/admin/batch-monitoring
**Authorization**: ADMIN role required
**Query Parameters**:
- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d)
- `status` (string, optional): Filter by status (pending, completed, failed)
- `jobType` (string, optional): Filter by job type
- `includeDetails` (boolean, optional): Include detailed job information
- `page` (number, optional): Page number
- `limit` (number, optional): Records per page
- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d)
- `status` (string, optional): Filter by status (pending, completed, failed)
- `jobType` (string, optional): Filter by job type
- `includeDetails` (boolean, optional): Include detailed job information
- `page` (number, optional): Page number
- `limit` (number, optional): Records per page
**Response**:
```json
{
"success": true,
@ -297,6 +326,7 @@ GET /api/admin/batch-monitoring
```
#### Retry Batch Job
```http
POST /api/admin/batch-monitoring/{jobId}/retry
```
@ -304,6 +334,7 @@ POST /api/admin/batch-monitoring/{jobId}/retry
**Authorization**: ADMIN role required
**Response**:
```json
{
"success": true,
@ -318,6 +349,7 @@ POST /api/admin/batch-monitoring/{jobId}/retry
### CSRF Token
#### Get CSRF Token
```http
GET /api/csrf-token
```
@ -325,6 +357,7 @@ GET /api/csrf-token
**Authorization**: None
**Response**:
```json
{
"csrfToken": "abc123..."
@ -332,11 +365,13 @@ GET /api/csrf-token
```
**Headers Set**:
- `Set-Cookie`: HTTP-only CSRF token cookie
- `Set-Cookie`: HTTP-only CSRF token cookie
### Authentication
#### User Registration
```http
POST /api/register
```
@ -344,9 +379,11 @@ POST /api/register
**Authorization**: None
**Headers Required**:
- `X-CSRF-Token`: CSRF token
- `X-CSRF-Token`: CSRF token
**Request Body**:
```json
{
"email": "user@example.com",
@ -359,6 +396,7 @@ POST /api/register
**Rate Limit**: 3 attempts per hour per IP
**Response**:
```json
{
"success": true,
@ -368,6 +406,7 @@ POST /api/register
```
#### Password Reset Request
```http
POST /api/forgot-password
```
@ -375,9 +414,11 @@ POST /api/forgot-password
**Authorization**: None
**Headers Required**:
- `X-CSRF-Token`: CSRF token
- `X-CSRF-Token`: CSRF token
**Request Body**:
```json
{
"email": "user@example.com"
@ -387,6 +428,7 @@ POST /api/forgot-password
**Rate Limit**: 5 attempts per 15 minutes per IP
**Response**:
```json
{
"success": true,
@ -395,6 +437,7 @@ POST /api/forgot-password
```
#### Password Reset Completion
```http
POST /api/reset-password
```
@ -402,9 +445,11 @@ POST /api/reset-password
**Authorization**: None
**Headers Required**:
- `X-CSRF-Token`: CSRF token
- `X-CSRF-Token`: CSRF token
**Request Body**:
```json
{
"token": "reset-token-123",
@ -413,6 +458,7 @@ POST /api/reset-password
```
**Response**:
```json
{
"success": true,
@ -464,18 +510,21 @@ POST /api/reset-password
## Rate Limiting
### Authentication Endpoints
- **Login**: 5 attempts per 15 minutes per IP
- **Registration**: 3 attempts per hour per IP
- **Password Reset**: 5 attempts per 15 minutes per IP
- **Login**: 5 attempts per 15 minutes per IP
- **Registration**: 3 attempts per hour per IP
- **Password Reset**: 5 attempts per 15 minutes per IP
### Security Endpoints
- **CSP Reports**: 10 reports per minute per IP
- **Admin Endpoints**: 60 requests per minute per user
- **Security Monitoring**: 30 requests per minute per user
- **CSP Reports**: 10 reports per minute per IP
- **Admin Endpoints**: 60 requests per minute per user
- **Security Monitoring**: 30 requests per minute per user
### General API
- **Dashboard Endpoints**: 120 requests per minute per user
- **Platform Management**: 60 requests per minute per user
- **Dashboard Endpoints**: 120 requests per minute per user
- **Platform Management**: 60 requests per minute per user
## Security Headers
@ -492,14 +541,17 @@ Content-Security-Policy: [CSP directives]
## CORS Configuration
### Allowed Origins
- Development: `http://localhost:3000`
- Production: `https://your-domain.com`
- Development: `http://localhost:3000`
- Production: `https://your-domain.com`
### Allowed Methods
- `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS`
- `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS`
### Allowed Headers
- `Content-Type`, `Authorization`, `X-CSRF-Token`, `X-Requested-With`
- `Content-Type`, `Authorization`, `X-CSRF-Token`, `X-Requested-With`
## Pagination
@ -520,25 +572,29 @@ Content-Security-Policy: [CSP directives]
```
### Pagination Parameters
- `page`: Page number (1-based, default: 1)
- `limit`: Records per page (default: 50, max: 100)
- `page`: Page number (1-based, default: 1)
- `limit`: Records per page (default: 50, max: 100)
## Filtering and Sorting
### Common Filter Parameters
- `startDate` / `endDate`: Date range filtering (ISO 8601)
- `status`: Status filtering
- `userId` / `companyId`: Entity filtering
- `eventType`: Event type filtering
- `severity`: Severity level filtering
- `startDate` / `endDate`: Date range filtering (ISO 8601)
- `status`: Status filtering
- `userId` / `companyId`: Entity filtering
- `eventType`: Event type filtering
- `severity`: Severity level filtering
### Sorting Parameters
- `sortBy`: Field to sort by
- `sortOrder`: `asc` or `desc` (default: `desc`)
- `sortBy`: Field to sort by
- `sortOrder`: `asc` or `desc` (default: `desc`)
## Response Caching
### Cache Headers
```http
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
@ -546,20 +602,23 @@ Expires: 0
```
### Cache Strategy
- **Security data**: Never cached
- **Static data**: Browser cache for 5 minutes
- **User data**: No cache for security
- **Security data**: Never cached
- **Static data**: Browser cache for 5 minutes
- **User data**: No cache for security
## API Versioning
### Current Version
- Version: `v1` (implied, no version prefix required)
- Introduced: January 2025
- Version: `v1` (implied, no version prefix required)
- Introduced: January 2025
### Future Versioning
- Breaking changes will introduce new versions
- Format: `/api/v2/endpoint`
- Backward compatibility maintained for 12 months
- Breaking changes will introduce new versions
- Format: `/api/v2/endpoint`
- Backward compatibility maintained for 12 months
## SDK and Client Libraries
@ -639,10 +698,10 @@ describe('Admin Audit Logs API', () => {
## Related Documentation
- [Admin Audit Logs API](./admin-audit-logs-api.md)
- [CSP Metrics API](./csp-metrics-api.md)
- [Security Monitoring](./security-monitoring.md)
- [CSRF Protection](./CSRF_PROTECTION.md)
- [Batch Monitoring Dashboard](./batch-monitoring-dashboard.md)
- [Admin Audit Logs API](./admin-audit-logs-api.md)
- [CSP Metrics API](./csp-metrics-api.md)
- [Security Monitoring](./security-monitoring.md)
- [CSRF Protection](./CSRF_PROTECTION.md)
- [Batch Monitoring Dashboard](./batch-monitoring-dashboard.md)
This API reference provides comprehensive documentation for all endpoints in the LiveDash-Node application. For specific implementation details, refer to the individual documentation files for each feature area.
This API reference provides comprehensive documentation for all endpoints in the LiveDash-Node application. For specific implementation details, refer to the individual documentation files for each feature area.

View File

@ -9,22 +9,25 @@ The Batch Monitoring Dashboard provides real-time visibility into the OpenAI Bat
## Features
### Real-time Monitoring
- **Job Status Tracking**: Monitor batch jobs from creation to completion
- **Queue Management**: View pending, running, and completed batch queues
- **Processing Metrics**: Track throughput, success rates, and error patterns
- **Cost Analysis**: Monitor API costs and savings compared to individual requests
- **Job Status Tracking**: Monitor batch jobs from creation to completion
- **Queue Management**: View pending, running, and completed batch queues
- **Processing Metrics**: Track throughput, success rates, and error patterns
- **Cost Analysis**: Monitor API costs and savings compared to individual requests
### Performance Analytics
- **Batch Efficiency**: Analyze batch size optimization and processing times
- **Success Rates**: Track completion and failure rates across different job types
- **Resource Utilization**: Monitor API quota usage and rate limiting
- **Historical Trends**: View processing patterns over time
- **Batch Efficiency**: Analyze batch size optimization and processing times
- **Success Rates**: Track completion and failure rates across different job types
- **Resource Utilization**: Monitor API quota usage and rate limiting
- **Historical Trends**: View processing patterns over time
### Administrative Controls
- **Manual Intervention**: Pause, resume, or cancel batch operations
- **Priority Management**: Adjust processing priorities for urgent requests
- **Error Handling**: Review and retry failed batch operations
- **Configuration Management**: Adjust batch parameters and thresholds
- **Manual Intervention**: Pause, resume, or cancel batch operations
- **Priority Management**: Adjust processing priorities for urgent requests
- **Error Handling**: Review and retry failed batch operations
- **Configuration Management**: Adjust batch parameters and thresholds
## API Endpoints
@ -132,6 +135,7 @@ const data = await response.json();
The main dashboard component (`components/admin/BatchMonitoringDashboard.tsx`) provides:
#### Key Metrics Cards
```tsx
// Real-time overview cards
<MetricCard
@ -157,6 +161,7 @@ The main dashboard component (`components/admin/BatchMonitoringDashboard.tsx`) p
```
#### Queue Status Visualization
```tsx
// Visual representation of batch job queues
<QueueStatusChart
@ -168,6 +173,7 @@ The main dashboard component (`components/admin/BatchMonitoringDashboard.tsx`) p
```
#### Performance Charts
```tsx
// Processing throughput over time
<ThroughputChart
@ -183,6 +189,7 @@ The main dashboard component (`components/admin/BatchMonitoringDashboard.tsx`) p
```
#### Job Management Table
```tsx
// Detailed job listing with actions
<BatchJobTable
@ -400,6 +407,7 @@ async function configureAlerts(alertConfig) {
### Common Issues
#### High Error Rates
```javascript
// Investigate high error rates
async function investigateErrors() {
@ -424,6 +432,7 @@ async function investigateErrors() {
```
#### Slow Processing
```javascript
// Analyze processing bottlenecks
async function analyzePerformance() {
@ -457,6 +466,7 @@ async function analyzePerformance() {
### Performance Optimization
#### Batch Size Optimization
```javascript
// Analyze optimal batch sizes
async function optimizeBatchSizes() {
@ -497,6 +507,7 @@ async function optimizeBatchSizes() {
## Integration with Existing Systems
### Security Audit Integration
All batch monitoring activities are logged through the security audit system:
```javascript
@ -510,6 +521,7 @@ await securityAuditLogger.logPlatformAdmin(
```
### Rate Limiting Integration
Monitoring API endpoints use the existing rate limiting system:
```javascript
@ -523,9 +535,9 @@ const rateLimitResult = await rateLimiter.check(
## Related Documentation
- [Batch Processing Optimizations](./batch-processing-optimizations.md)
- [Security Monitoring](./security-monitoring.md)
- [Admin Audit Logs API](./admin-audit-logs-api.md)
- [OpenAI Batch API Integration](../lib/batchProcessor.ts)
- [Batch Processing Optimizations](./batch-processing-optimizations.md)
- [Security Monitoring](./security-monitoring.md)
- [Admin Audit Logs API](./admin-audit-logs-api.md)
- [OpenAI Batch API Integration](../lib/batchProcessor.ts)
The batch monitoring dashboard provides comprehensive visibility into the AI processing pipeline, enabling administrators to optimize performance, monitor costs, and ensure reliable operation of the batch processing system.
The batch monitoring dashboard provides comprehensive visibility into the AI processing pipeline, enabling administrators to optimize performance, monitor costs, and ensure reliable operation of the batch processing system.

View File

@ -6,11 +6,11 @@ This document outlines the database query optimizations implemented to improve t
The batch processing system was optimized to reduce database load and improve response times through several key strategies:
1. **Database Index Optimization**
2. **Query Pattern Improvements**
3. **Company Caching**
4. **Batch Operations**
5. **Integration Layer with Fallback**
1. **Database Index Optimization**
2. **Query Pattern Improvements**
3. **Company Caching**
4. **Batch Operations**
5. **Integration Layer with Fallback**
## Database Index Improvements
@ -32,15 +32,17 @@ The following composite indexes were added to the `AIProcessingRequest` table in
### Query Performance Impact
These indexes specifically optimize:
- Finding pending requests by status and creation time
- Batch-related lookups by batch ID
- Combined status and batch filtering operations
- Finding pending requests by status and creation time
- Batch-related lookups by batch ID
- Combined status and batch filtering operations
## Query Optimization Strategies
### 1. Selective Data Fetching
**Before:**
```typescript
// Loaded full session with all messages
include: {
@ -55,6 +57,7 @@ include: {
```
**After:**
```typescript
// Only essential data with message count
include: {
@ -86,6 +89,7 @@ class CompanyCache {
### 3. Batch Operations
**Before:** N+1 queries for each company
```typescript
// Sequential processing per company
for (const company of companies) {
@ -95,6 +99,7 @@ for (const company of companies) {
```
**After:** Single query for all companies
```typescript
// Batch query for all companies at once
const allRequests = await prisma.aIProcessingRequest.findMany({
@ -114,10 +119,10 @@ const requestsByCompany = groupByCompany(allRequests);
### Query Count Reduction
- **Company lookups:** Reduced from 4 separate queries per scheduler run to 1 cached lookup
- **Pending requests:** Reduced from N queries (one per company) to 1 batch query
- **Status checks:** Reduced from N queries to 1 batch query
- **Failed requests:** Reduced from N queries to 1 batch query
- **Company lookups:** Reduced from 4 separate queries per scheduler run to 1 cached lookup
- **Pending requests:** Reduced from N queries (one per company) to 1 batch query
- **Status checks:** Reduced from N queries to 1 batch query
- **Failed requests:** Reduced from N queries to 1 batch query
### Parallel Processing
@ -133,9 +138,9 @@ const SCHEDULER_CONFIG = {
### Memory Optimization
- Eliminated loading unnecessary message content
- Used `select` instead of `include` where possible
- Implemented automatic cache cleanup
- Eliminated loading unnecessary message content
- Used `select` instead of `include` where possible
- Implemented automatic cache cleanup
## Integration Layer
@ -169,45 +174,47 @@ class PerformanceTracker {
## Files Modified
### New Files
- `lib/batchProcessorOptimized.ts` - Optimized query implementations
- `lib/batchSchedulerOptimized.ts` - Optimized scheduler
- `lib/batchProcessorIntegration.ts` - Integration layer with fallback
- `lib/batchProcessorOptimized.ts` - Optimized query implementations
- `lib/batchSchedulerOptimized.ts` - Optimized scheduler
- `lib/batchProcessorIntegration.ts` - Integration layer with fallback
### Modified Files
- `prisma/schema.prisma` - Added composite indexes
- `server.ts` - Updated to use integration layer
- `app/api/admin/batch-monitoring/route.ts` - Updated import
- `prisma/schema.prisma` - Added composite indexes
- `server.ts` - Updated to use integration layer
- `app/api/admin/batch-monitoring/route.ts` - Updated import
## Monitoring
The optimizations include comprehensive logging and monitoring:
- Performance metrics for each operation type
- Cache hit/miss statistics
- Fallback events tracking
- Query execution time monitoring
- Performance metrics for each operation type
- Cache hit/miss statistics
- Fallback events tracking
- Query execution time monitoring
## Rollback Strategy
The integration layer allows for easy rollback:
1. Set `ENABLE_BATCH_OPTIMIZATION=false`
2. System automatically uses original implementation
3. No database schema changes needed for rollback
4. Indexes remain beneficial for manual queries
1. Set `ENABLE_BATCH_OPTIMIZATION=false`
2. System automatically uses original implementation
3. No database schema changes needed for rollback
4. Indexes remain beneficial for manual queries
## Expected Performance Gains
- **Database Query Count:** 60-80% reduction in scheduler operations
- **Memory Usage:** 40-60% reduction from selective data loading
- **Response Time:** 30-50% improvement for batch operations
- **Cache Hit Rate:** 95%+ for company lookups after warmup
- **Database Query Count:** 60-80% reduction in scheduler operations
- **Memory Usage:** 40-60% reduction from selective data loading
- **Response Time:** 30-50% improvement for batch operations
- **Cache Hit Rate:** 95%+ for company lookups after warmup
## Testing
Performance improvements can be validated by:
1. Monitoring the batch monitoring dashboard
2. Checking performance metrics in logs
3. Comparing execution times before/after optimization
4. Load testing with multiple companies and large batches
1. Monitoring the batch monitoring dashboard
2. Checking performance metrics in logs
3. Comparing execution times before/after optimization
4. Load testing with multiple companies and large batches

View File

@ -6,11 +6,11 @@ This document describes the Content Security Policy (CSP) metrics and violation
The CSP Metrics API provides comprehensive monitoring of Content Security Policy violations, including:
- Real-time violation tracking and metrics
- Bypass attempt detection and risk assessment
- Policy optimization recommendations
- Historical trend analysis
- Export capabilities for security analysis
- Real-time violation tracking and metrics
- Bypass attempt detection and risk assessment
- Policy optimization recommendations
- Historical trend analysis
- Export capabilities for security analysis
## API Endpoints
@ -24,7 +24,7 @@ POST /api/csp-report
#### Request Headers
- `Content-Type`: `application/csp-report` or `application/json`
- `Content-Type`: `application/csp-report` or `application/json`
#### Request Body (Automatic from Browser)
@ -43,10 +43,10 @@ POST /api/csp-report
#### Features
- **Rate Limiting**: 10 reports per minute per IP
- **Risk Assessment**: Automatic classification of violation severity
- **Bypass Detection**: Identifies potential CSP bypass attempts
- **Real-time Processing**: Immediate analysis and alerting
- **Rate Limiting**: 10 reports per minute per IP
- **Risk Assessment**: Automatic classification of violation severity
- **Bypass Detection**: Identifies potential CSP bypass attempts
- **Real-time Processing**: Immediate analysis and alerting
### CSP Metrics API
@ -67,11 +67,11 @@ GET /api/csp-metrics
#### Time Range Options
- `1h` - Last 1 hour
- `6h` - Last 6 hours
- `24h` - Last 24 hours (default)
- `7d` - Last 7 days
- `30d` - Last 30 days
- `1h` - Last 1 hour
- `6h` - Last 6 hours
- `24h` - Last 24 hours (default)
- `7d` - Last 7 days
- `30d` - Last 30 days
#### Example Request
@ -165,11 +165,11 @@ console.log(result.recommendations); // array of suggestions
The service automatically assesses violation risk based on:
- **Directive Type**: Script violations are higher risk than style violations
- **Source Pattern**: External domains vs inline vs data URIs
- **Bypass Indicators**: Known CSP bypass techniques
- **Frequency**: Repeated violations from same source
- **Geographic Factors**: Unusual source locations
- **Directive Type**: Script violations are higher risk than style violations
- **Source Pattern**: External domains vs inline vs data URIs
- **Bypass Indicators**: Known CSP bypass techniques
- **Frequency**: Repeated violations from same source
- **Geographic Factors**: Unusual source locations
#### 3. Bypass Detection
@ -191,10 +191,10 @@ const bypassPatterns = [
Based on violation patterns, the service provides actionable recommendations:
- **Tighten Policies**: Suggest removing broad allowlists
- **Add Domains**: Recommend allowing legitimate external resources
- **Implement Nonces**: Suggest nonce-based policies for inline content
- **Upgrade Directives**: Recommend modern CSP features
- **Tighten Policies**: Suggest removing broad allowlists
- **Add Domains**: Recommend allowing legitimate external resources
- **Implement Nonces**: Suggest nonce-based policies for inline content
- **Upgrade Directives**: Recommend modern CSP features
## Violation Analysis
@ -405,48 +405,50 @@ CSP_ALERT_THRESHOLD=5 # violations per 10 minutes
### Rate Limiting
- **10 reports per minute per IP** prevents spam attacks
- **Exponential backoff** for repeated violations from same source
- **Memory cleanup** removes old violations automatically
- **10 reports per minute per IP** prevents spam attacks
- **Exponential backoff** for repeated violations from same source
- **Memory cleanup** removes old violations automatically
### Memory Management
- **Violation buffer** limited to 1 hour of data in memory
- **Automatic cleanup** runs every 100 requests (1% probability)
- **Efficient storage** using Map data structures
- **Violation buffer** limited to 1 hour of data in memory
- **Automatic cleanup** runs every 100 requests (1% probability)
- **Efficient storage** using Map data structures
### Database Impact
- **No persistent storage** for real-time metrics (memory only)
- **Optional logging** to database for long-term analysis
- **Indexed queries** for historical data retrieval
- **No persistent storage** for real-time metrics (memory only)
- **Optional logging** to database for long-term analysis
- **Indexed queries** for historical data retrieval
## Security Considerations
### Privacy Protection
**⚠️ Data Collection Notice:**
- **IP addresses** are collected and stored in memory for security monitoring
- **User agent strings** are stored for browser compatibility analysis
- **Legal basis**: Legitimate interest for security incident detection and prevention
- **Retention**: In-memory storage only, automatically purged after 7 days or application restart
- **Data minimization**: Only violation-related metadata is retained, not page content
- **IP addresses** are collected and stored in memory for security monitoring
- **User agent strings** are stored for browser compatibility analysis
- **Legal basis**: Legitimate interest for security incident detection and prevention
- **Retention**: In-memory storage only, automatically purged after 7 days or application restart
- **Data minimization**: Only violation-related metadata is retained, not page content
**Planned Privacy Enhancements:**
- IP anonymization options for GDPR compliance (roadmap)
- User agent sanitization to remove sensitive information (roadmap)
### Rate Limiting Protection
- IP anonymization options for GDPR compliance (roadmap)
- User agent sanitization to remove sensitive information (roadmap)
- **Per-IP limits** prevent DoS attacks on reporting endpoint
- **Content-type validation** ensures proper report format
- **Request size limits** prevent memory exhaustion
### Rate-Limiting Protection
- **Per-IP limits** prevent DoS attacks on reporting endpoint
- **Content-type validation** ensures proper report format
- **Request size limits** prevent memory exhaustion
### False Positive Handling
- **Learning mode** for new deployments
- **Whitelist support** for known legitimate violations
- **Risk score adjustment** based on historical patterns
- **Learning mode** for new deployments
- **Whitelist support** for known legitimate violations
- **Risk score adjustment** based on historical patterns
## Troubleshooting
@ -495,10 +497,10 @@ if (duration > 2000) {
## Related Documentation
- [Enhanced CSP Implementation](./security/enhanced-csp.md)
- [Security Monitoring](./security-monitoring.md)
- [Security Headers](./security-headers.md)
- [Rate Limiting](../lib/rateLimiter.ts)
- [Enhanced CSP Implementation](./security/enhanced-csp.md)
- [Security Monitoring](./security-monitoring.md)
- [Security Headers](./security-headers.md)
- [Rate Limiting](../lib/rateLimiter.ts)
## API Reference Summary

View File

@ -12,10 +12,10 @@ The WordCloud component visualizes categories or topics based on their frequency
**Features:**
- Dynamic sizing based on frequency
- Colorful display with a pleasing color palette
- Responsive design
- Interactive hover effects
- Dynamic sizing based on frequency
- Colorful display with a pleasing color palette
- Responsive design
- Interactive hover effects
### 2. GeographicMap
@ -25,10 +25,10 @@ This component displays a world map with circles representing the number of sess
**Features:**
- Interactive map using React Leaflet
- Circle sizes scaled by session count
- Tooltips showing country names and session counts
- Responsive design
- Interactive map using React Leaflet
- Circle sizes scaled by session count
- Tooltips showing country names and session counts
- Responsive design
### 3. MetricCard
@ -38,10 +38,10 @@ A modern, visually appealing card for displaying key metrics.
**Features:**
- Multiple design variants (default, primary, success, warning, danger)
- Support for trend indicators
- Icons and descriptions
- Clean, modern styling
- Multiple design variants (default, primary, success, warning, danger)
- Support for trend indicators
- Icons and descriptions
- Clean, modern styling
### 4. DonutChart
@ -51,10 +51,10 @@ An enhanced donut chart with better styling and a central text display capabilit
**Features:**
- Customizable colors
- Center text area for displaying summaries
- Interactive tooltips with percentages
- Well-balanced legend display
- Customizable colors
- Center text area for displaying summaries
- Interactive tooltips with percentages
- Well-balanced legend display
### 5. ResponseTimeDistribution
@ -64,10 +64,10 @@ Visualizes the distribution of response times as a histogram.
**Features:**
- Color-coded bars (green for fast, yellow for medium, red for slow)
- Target time indicator
- Automatic binning of response times
- Clear labeling and scales
- Color-coded bars (green for fast, yellow for medium, red for slow)
- Target time indicator
- Automatic binning of response times
- Clear labeling and scales
## Dashboard Enhancements
@ -85,7 +85,7 @@ The dashboard has been enhanced with:
## Usage Notes
- The geographic map and response time distribution use simulated data where actual data is not available
- All components are responsive and will adjust to different screen sizes
- The dashboard automatically refreshes data when using the refresh button
- Admin users have access to additional controls at the bottom of the dashboard
- The geographic map and response time distribution use simulated data where actual data is not available
- All components are responsive and will adjust to different screen sizes
- The dashboard automatically refreshes data when using the refresh button
- Admin users have access to additional controls at the bottom of the dashboard

View File

@ -6,8 +6,8 @@ This document explains how to optimize database connection pooling for better pe
The application now supports two connection pooling modes:
1. **Standard Pooling**: Default Prisma client connection pooling
2. **Enhanced Pooling**: Advanced PostgreSQL connection pooling with custom configuration
1. **Standard Pooling**: Default Prisma client connection pooling
2. **Enhanced Pooling**: Advanced PostgreSQL connection pooling with custom configuration
## Configuration
@ -29,42 +29,42 @@ DATABASE_URL="postgresql://user:pass@host:5432/db?connection_limit=20&pool_timeo
#### Standard Pooling (Default)
- Uses Prisma's built-in connection pooling
- Simpler configuration
- Good for development and small-scale deployments
- Uses Prisma's built-in connection pooling
- Simpler configuration
- Good for development and small-scale deployments
#### Enhanced Pooling (Recommended for Production)
- Uses PostgreSQL native connection pooling with `@prisma/adapter-pg`
- Advanced monitoring and health checks
- Better resource management
- Detailed connection metrics
- Uses PostgreSQL native connection pooling with `@prisma/adapter-pg`
- Advanced monitoring and health checks
- Better resource management
- Detailed connection metrics
## Implementation Details
### Fixed Issues
1. **Multiple PrismaClient Instances**:
- ❌ Before: Each scheduler created its own PrismaClient
- ✅ After: All modules use singleton pattern from `lib/prisma.ts`
1. **Multiple PrismaClient Instances**:
- ❌ Before: Each scheduler created its own PrismaClient
- ✅ After: All modules use singleton pattern from `lib/prisma.ts`
2. **No Connection Management**:
- ❌ Before: No graceful shutdown or connection cleanup
- ✅ After: Proper cleanup on process termination
2. **No Connection Management**:
- ❌ Before: No graceful shutdown or connection cleanup
- ✅ After: Proper cleanup on process termination
3. **No Monitoring**:
- ❌ Before: No visibility into connection usage
- ✅ After: Health check endpoint and connection metrics
3. **No Monitoring**:
- ❌ Before: No visibility into connection usage
- ✅ After: Health check endpoint and connection metrics
### Key Files Modified
- `lib/prisma.ts` - Enhanced singleton with pooling options
- `lib/database-pool.ts` - Advanced pooling configuration
- `lib/processingScheduler.ts` - Fixed to use singleton
- `lib/importProcessor.ts` - Fixed to use singleton
- `lib/processingStatusManager.ts` - Fixed to use singleton
- `lib/schedulers.ts` - Added graceful shutdown
- `app/api/admin/database-health/route.ts` - Monitoring endpoint
- `lib/prisma.ts` - Enhanced singleton with pooling options
- `lib/database-pool.ts` - Advanced pooling configuration
- `lib/processingScheduler.ts` - Fixed to use singleton
- `lib/importProcessor.ts` - Fixed to use singleton
- `lib/processingStatusManager.ts` - Fixed to use singleton
- `lib/schedulers.ts` - Added graceful shutdown
- `app/api/admin/database-health/route.ts` - Monitoring endpoint
## Monitoring
@ -79,36 +79,36 @@ curl -H "Authorization: Bearer your-token" \
Response includes:
- Connection status
- Pool statistics (if enhanced pooling enabled)
- Basic metrics (session counts, etc.)
- Configuration details
- Connection status
- Pool statistics (if enhanced pooling enabled)
- Basic metrics (session counts, etc.)
- Configuration details
### Connection Metrics
With enhanced pooling enabled, you'll see console logs for:
- Connection acquisitions/releases
- Pool size changes
- Error events
- Health check results
- Connection acquisitions/releases
- Pool size changes
- Error events
- Health check results
## Performance Benefits
### Before Optimization
- Multiple connection pools (one per scheduler)
- Potential connection exhaustion under load
- No connection monitoring
- Resource waste from idle connections
- Multiple connection pools (one per scheduler)
- Potential connection exhaustion under load
- No connection monitoring
- Resource waste from idle connections
### After Optimization
- Single shared connection pool
- Configurable pool size and timeouts
- Connection health monitoring
- Graceful shutdown and cleanup
- Better resource utilization
- Single shared connection pool
- Configurable pool size and timeouts
- Connection health monitoring
- Graceful shutdown and cleanup
- Better resource utilization
## Recommended Settings
@ -142,40 +142,40 @@ USE_ENHANCED_POOLING=true
If you see "too many connections" errors:
1. Increase `DATABASE_CONNECTION_LIMIT`
2. Check for connection leaks in application code
3. Monitor the health endpoint for pool statistics
1. Increase `DATABASE_CONNECTION_LIMIT`
2. Check for connection leaks in application code
3. Monitor the health endpoint for pool statistics
### Slow Database Queries
If queries are timing out:
1. Decrease `DATABASE_POOL_TIMEOUT`
2. Check database query performance
3. Consider connection pooling at the infrastructure level (PgBouncer)
1. Decrease `DATABASE_POOL_TIMEOUT`
2. Check database query performance
3. Consider connection pooling at the infrastructure level (PgBouncer)
### Memory Usage
If memory usage is high:
1. Decrease `DATABASE_CONNECTION_LIMIT`
2. Enable enhanced pooling for better resource management
3. Monitor idle connection cleanup
1. Decrease `DATABASE_CONNECTION_LIMIT`
2. Enable enhanced pooling for better resource management
3. Monitor idle connection cleanup
## Best Practices
1. **Always use the singleton**: Import `prisma` from `lib/prisma.ts`
2. **Monitor connection usage**: Use the health endpoint regularly
3. **Set appropriate limits**: Don't over-provision connections
4. **Enable enhanced pooling in production**: Better resource management
5. **Implement graceful shutdown**: Ensure connections are properly closed
6. **Log connection events**: Monitor for issues and optimize accordingly
1. **Always use the singleton**: Import `prisma` from `lib/prisma.ts`
2. **Monitor connection usage**: Use the health endpoint regularly
3. **Set appropriate limits**: Don't over-provision connections
4. **Enable enhanced pooling in production**: Better resource management
5. **Implement graceful shutdown**: Ensure connections are properly closed
6. **Log connection events**: Monitor for issues and optimize accordingly
## Next Steps
Consider implementing:
1. **Connection pooling middleware**: PgBouncer or similar
2. **Read replicas**: For read-heavy workloads
3. **Connection retry logic**: For handling temporary failures
4. **Metrics collection**: Prometheus/Grafana for detailed monitoring
1. **Connection pooling middleware**: PgBouncer or similar
2. **Read replicas**: For read-heavy workloads
3. **Connection retry logic**: For handling temporary failures
4. **Metrics collection**: Prometheus/Grafana for detailed monitoring

View File

@ -6,10 +6,10 @@ This document outlines the comprehensive database performance optimizations impl
The optimization focuses on the most frequently queried patterns in the application, particularly around:
- AI processing request tracking and batching
- Session analytics and filtering
- Security audit log analysis
- Multi-tenant data isolation performance
- AI processing request tracking and batching
- Session analytics and filtering
- Security audit log analysis
- Multi-tenant data isolation performance
## Applied Optimizations
@ -31,9 +31,9 @@ INCLUDE ("processingStatus", "batchId", "requestedAt", "sessionId");
**Impact**:
- ~70% faster batch job queries
- Reduced I/O for cost analysis reports
- Improved scheduler performance
- ~70% faster batch job queries
- Reduced I/O for cost analysis reports
- Improved scheduler performance
### 2. Session Analytics Optimizations
@ -54,9 +54,9 @@ INCLUDE ("startTime", "messagesSent");
**Impact**:
- ~85% faster dashboard load times
- Efficient date range filtering
- Optimized sentiment analysis queries
- ~85% faster dashboard load times
- Efficient date range filtering
- Optimized sentiment analysis queries
### 3. Security Audit Log Optimizations
@ -77,9 +77,9 @@ INCLUDE ("eventType", "severity", "userId", "companyId");
**Impact**:
- ~90% faster security monitoring
- Efficient threat detection
- Improved compliance reporting
- ~90% faster security monitoring
- Efficient threat detection
- Improved compliance reporting
### 4. Message Processing Optimizations
@ -95,8 +95,8 @@ INCLUDE ("content");
**Impact**:
- ~60% faster conversation loading
- Reduced memory usage for message queries
- ~60% faster conversation loading
- Reduced memory usage for message queries
### 5. Processing Pipeline Optimizations
@ -118,29 +118,29 @@ INCLUDE ("sessionId", "errorMessage", "retryCount", "startedAt");
**Impact**:
- ~75% faster processing monitoring
- Efficient error tracking
- Improved retry logic performance
- ~75% faster processing monitoring
- Efficient error tracking
- Improved retry logic performance
## Index Strategy Principles
### 1. Composite Index Design
- **Leading column**: Most selective filter (usually companyId for multi-tenancy)
- **Secondary columns**: Common WHERE clause filters
- **Covering columns**: SELECT list columns via INCLUDE
- **Leading column**: Most selective filter (usually companyId for multi-tenancy)
- **Secondary columns**: Common WHERE clause filters
- **Covering columns**: SELECT list columns via INCLUDE
### 2. Partial Indexes
- Used for error analysis and specific status filtering
- Reduces index size and maintenance overhead
- Improves write performance
- Used for error analysis and specific status filtering
- Reduces index size and maintenance overhead
- Improves write performance
### 3. Covering Indexes
- Include frequently accessed columns to avoid table lookups
- Reduces I/O for read-heavy operations
- Particularly effective for dashboard queries
- Include frequently accessed columns to avoid table lookups
- Reduces I/O for read-heavy operations
- Particularly effective for dashboard queries
## Query Pattern Analysis
@ -166,29 +166,29 @@ INCLUDE ("sessionId", "errorMessage", "retryCount", "startedAt");
### Index Monitoring
- Monitor index usage with `pg_stat_user_indexes`
- Track bloat with `pg_stat_user_tables`
- Regular ANALYZE after bulk operations
- Monitor index usage with `pg_stat_user_indexes`
- Track bloat with `pg_stat_user_tables`
- Regular ANALYZE after bulk operations
### Write Performance Impact
- Composite indexes add ~15% write overhead
- Offset by dramatic read performance gains
- Monitored via slow query logs
- Composite indexes add ~15% write overhead
- Offset by dramatic read performance gains
- Monitored via slow query logs
### Storage Impact
- Indexes add ~25% to total storage
- Covering indexes reduce need for table scans
- Partial indexes minimize storage overhead
- Indexes add ~25% to total storage
- Covering indexes reduce need for table scans
- Partial indexes minimize storage overhead
## Migration Safety
### CONCURRENTLY Operations
- All indexes created with `CREATE INDEX CONCURRENTLY`
- No table locks during creation
- Production-safe deployment
- All indexes created with `CREATE INDEX CONCURRENTLY`
- No table locks during creation
- Production-safe deployment
### Rollback Strategy
@ -238,18 +238,18 @@ LIMIT 10;
### Monitoring Strategy
- Set up automated index usage monitoring
- Track slow query evolution
- Monitor storage growth patterns
- Implement performance alerting
- Set up automated index usage monitoring
- Track slow query evolution
- Monitor storage growth patterns
- Implement performance alerting
## Conclusion
These database optimizations provide:
- **70-90% improvement** in query performance
- **Reduced server load** through efficient indexing
- **Better user experience** with faster dashboards
- **Scalable foundation** for future growth
- **70-90% improvement** in query performance
- **Reduced server load** through efficient indexing
- **Better user experience** with faster dashboards
- **Scalable foundation** for future growth
The optimizations are designed to be production-safe and monitoring-friendly, ensuring both immediate performance gains and long-term maintainability.

View File

@ -15,21 +15,21 @@ Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.ne
### 1. Neon Connection Limits
- **Free Tier**: 20 concurrent connections
- **Pro Tier**: 100 concurrent connections
- **Multiple schedulers** can quickly exhaust connections
- **Free Tier**: 20 concurrent connections
- **Pro Tier**: 100 concurrent connections
- **Multiple schedulers** can quickly exhaust connections
### 2. Connection Pooling Issues
- Each scheduler was creating separate PrismaClient instances
- No connection reuse between operations
- No retry logic for temporary failures
- Each scheduler was creating separate PrismaClient instances
- No connection reuse between operations
- No retry logic for temporary failures
### 3. Neon-Specific Challenges
- **Auto-pause**: Databases pause after inactivity
- **Cold starts**: First connection after pause takes longer
- **Regional latency**: eu-central-1 may have variable latency
- **Auto-pause**: Databases pause after inactivity
- **Cold starts**: First connection after pause takes longer
- **Regional latency**: eu-central-1 may have variable latency
## Solutions Implemented
@ -106,9 +106,9 @@ curl -H "Authorization: Bearer YOUR_API_TOKEN" \
### 2. Neon Dashboard Monitoring
- Monitor "Active connections" in Neon dashboard
- Check for connection spikes during scheduler runs
- Review query performance and slow queries
- Monitor "Active connections" in Neon dashboard
- Check for connection spikes during scheduler runs
- Review query performance and slow queries
### 3. Application Logs
@ -158,9 +158,9 @@ const prisma = new PrismaClient({
**Causes:**
- Neon database auto-paused
- Connection limit exceeded
- Network issues
- Neon database auto-paused
- Connection limit exceeded
- Network issues
**Solutions:**
@ -173,9 +173,9 @@ const prisma = new PrismaClient({
**Causes:**
- Idle connection timeout
- Neon maintenance
- Long-running transactions
- Idle connection timeout
- Neon maintenance
- Long-running transactions
**Solutions:**
@ -187,9 +187,9 @@ const prisma = new PrismaClient({
**Causes:**
- Blocking database operations
- Scheduler overlap
- High CPU usage
- Blocking database operations
- Scheduler overlap
- High CPU usage
**Solutions:**
@ -230,10 +230,10 @@ SESSION_PROCESSING_INTERVAL="0 */2 * * *"
## Monitoring Checklist
- [ ] Check Neon dashboard for connection spikes
- [ ] Monitor scheduler execution times
- [ ] Review error logs for connection patterns
- [ ] Test health endpoint regularly
- [ ] Set up alerts for connection failures
- [ ] Check Neon dashboard for connection spikes
- [ ] Monitor scheduler execution times
- [ ] Review error logs for connection patterns
- [ ] Test health endpoint regularly
- [ ] Set up alerts for connection failures
With these optimizations, your Neon database connections should be much more stable and efficient!

View File

@ -17,48 +17,48 @@ Successfully migrated the livedash-node application from SQLite to PostgreSQL us
#### Production/Development
- **Provider**: PostgreSQL (Neon)
- **Environment Variable**: `DATABASE_URL`
- **Connection**: Neon PostgreSQL cluster
- **Provider**: PostgreSQL (Neon)
- **Environment Variable**: `DATABASE_URL`
- **Connection**: Neon PostgreSQL cluster
#### Testing
- **Provider**: PostgreSQL (Neon - separate database)
- **Environment Variable**: `DATABASE_URL_TEST`
- **Test Setup**: Automatically switches to test database during test runs
- **Provider**: PostgreSQL (Neon - separate database)
- **Environment Variable**: `DATABASE_URL_TEST`
- **Test Setup**: Automatically switches to test database during test runs
### Files Modified
1. **`prisma/schema.prisma`**
- Changed provider from `sqlite` to `postgresql`
- Updated URL to use `env("DATABASE_URL")`
- Changed provider from `sqlite` to `postgresql`
- Updated URL to use `env("DATABASE_URL")`
2. **`tests/setup.ts`**
- Added logic to use `DATABASE_URL_TEST` when available
- Ensures test isolation with separate database
- Added logic to use `DATABASE_URL_TEST` when available
- Ensures test isolation with separate database
3. **`.env`** (created)
- Contains `DATABASE_URL` for Prisma CLI operations
- Contains `DATABASE_URL` for Prisma CLI operations
4. **`.env.local`** (existing)
- Contains both `DATABASE_URL` and `DATABASE_URL_TEST`
- Contains both `DATABASE_URL` and `DATABASE_URL_TEST`
### Database Schema
All existing models and relationships were preserved:
- **Company**: Multi-tenant root entity
- **User**: Authentication and authorization
- **Session**: Processed session data
- **SessionImport**: Raw CSV import data
- **Message**: Individual conversation messages
- **Question**: Normalized question storage
- **SessionQuestion**: Session-question relationships
- **AIProcessingRequest**: AI cost tracking
- **Company**: Multi-tenant root entity
- **User**: Authentication and authorization
- **Session**: Processed session data
- **SessionImport**: Raw CSV import data
- **Message**: Individual conversation messages
- **Question**: Normalized question storage
- **SessionQuestion**: Session-question relationships
- **AIProcessingRequest**: AI cost tracking
### Migration Process
@ -103,11 +103,11 @@ if (process.env.DATABASE_URL_TEST) {
All tests pass successfully:
- ✅ Environment configuration tests
- ✅ Transcript fetcher tests
- ✅ Database connection tests
- ✅ Schema validation tests
- ✅ CRUD operation tests
- ✅ Environment configuration tests
- ✅ Transcript fetcher tests
- ✅ Database connection tests
- ✅ Schema validation tests
- ✅ CRUD operation tests
### Next Steps

View File

@ -7,21 +7,24 @@ Successfully refactored the session processing pipeline from a simple status-bas
## Problems Solved
### Original Issues
1. **Inconsistent Status Tracking**: The old system used a simple enum on SessionImport that didn't properly track the multi-stage processing pipeline
2. **Poor Error Visibility**: Error messages were buried in the SessionImport table and not easily accessible
3. **No Stage-Specific Tracking**: The system couldn't track which specific stage of processing failed
4. **Difficult Recovery**: Failed sessions were hard to identify and retry
5. **Linting Errors**: Multiple TypeScript files referencing removed database fields
1. **Inconsistent Status Tracking**: The old system used a simple enum on SessionImport that didn't properly track the multi-stage processing pipeline
2. **Poor Error Visibility**: Error messages were buried in the SessionImport table and not easily accessible
3. **No Stage-Specific Tracking**: The system couldn't track which specific stage of processing failed
4. **Difficult Recovery**: Failed sessions were hard to identify and retry
5. **Linting Errors**: Multiple TypeScript files referencing removed database fields
### Schema Changes Made
- **Removed** old `status`, `errorMsg`, and `processedAt` columns from SessionImport
- **Removed** `processed` field from Session
- **Added** new `SessionProcessingStatus` table with granular stage tracking
- **Added** `ProcessingStage` and `ProcessingStatus` enums
- **Removed** old `status`, `errorMsg`, and `processedAt` columns from SessionImport
- **Removed** `processed` field from Session
- **Added** new `SessionProcessingStatus` table with granular stage tracking
- **Added** `ProcessingStage` and `ProcessingStatus` enums
## New Processing Pipeline
### Processing Stages
```typescript
enum ProcessingStage {
CSV_IMPORT // SessionImport created
@ -39,95 +42,107 @@ enum ProcessingStatus {
### Key Components
#### 1. ProcessingStatusManager
Centralized class for managing processing status with methods:
- `initializeSession()` - Set up processing status for new sessions
- `startStage()`, `completeStage()`, `failStage()`, `skipStage()` - Stage management
- `getSessionsNeedingProcessing()` - Query sessions by stage and status
- `getPipelineStatus()` - Get overview of entire pipeline
- `getFailedSessions()` - Find sessions needing retry
- `resetStageForRetry()` - Reset failed stages
- `initializeSession()` - Set up processing status for new sessions
- `startStage()`, `completeStage()`, `failStage()`, `skipStage()` - Stage management
- `getSessionsNeedingProcessing()` - Query sessions by stage and status
- `getPipelineStatus()` - Get overview of entire pipeline
- `getFailedSessions()` - Find sessions needing retry
- `resetStageForRetry()` - Reset failed stages
#### 2. Updated Processing Scheduler
- Integrated with new `ProcessingStatusManager`
- Tracks AI analysis and question extraction stages
- Records detailed processing metadata
- Proper error handling and retry capabilities
- Integrated with new `ProcessingStatusManager`
- Tracks AI analysis and question extraction stages
- Records detailed processing metadata
- Proper error handling and retry capabilities
#### 3. Migration System
- Successfully migrated all 109 existing sessions
- Determined current state based on existing data
- Preserved all existing functionality
- Successfully migrated all 109 existing sessions
- Determined current state based on existing data
- Preserved all existing functionality
## Current Pipeline Status
After migration and refactoring:
- **CSV_IMPORT**: 109 completed
- **TRANSCRIPT_FETCH**: 109 completed
- **SESSION_CREATION**: 109 completed
- **AI_ANALYSIS**: 16 completed, 93 pending
- **QUESTION_EXTRACTION**: 11 completed, 98 pending
- **CSV_IMPORT**: 109 completed
- **TRANSCRIPT_FETCH**: 109 completed
- **SESSION_CREATION**: 109 completed
- **AI_ANALYSIS**: 16 completed, 93 pending
- **QUESTION_EXTRACTION**: 11 completed, 98 pending
## Files Updated/Created
### New Files
- `lib/processingStatusManager.ts` - Core processing status management
- `check-refactored-pipeline-status.ts` - New pipeline status checker
- `migrate-to-refactored-system.ts` - Migration script
- `docs/processing-system-refactor.md` - This documentation
- `lib/processingStatusManager.ts` - Core processing status management
- `check-refactored-pipeline-status.ts` - New pipeline status checker
- `migrate-to-refactored-system.ts` - Migration script
- `docs/processing-system-refactor.md` - This documentation
### Updated Files
- `prisma/schema.prisma` - Added new processing status tables
- `lib/processingScheduler.ts` - Integrated with new status system
- `debug-import-status.ts` - Updated to use new system
- `fix-import-status.ts` - Updated to use new system
- `prisma/schema.prisma` - Added new processing status tables
- `lib/processingScheduler.ts` - Integrated with new status system
- `debug-import-status.ts` - Updated to use new system
- `fix-import-status.ts` - Updated to use new system
### Removed Files
- `check-pipeline-status.ts` - Replaced by refactored version
- `check-pipeline-status.ts` - Replaced by refactored version
## Benefits Achieved
1. **Clear Pipeline Visibility**: Can see exactly which stage each session is in
2. **Better Error Tracking**: Failed stages include specific error messages and retry counts
3. **Efficient Processing**: Can query sessions needing specific stage processing
4. **Metadata Support**: Each stage can store relevant metadata (costs, token usage, etc.)
5. **Easy Recovery**: Failed sessions can be easily identified and retried
6. **Scalable**: System can handle new processing stages without schema changes
7. **No Linting Errors**: All TypeScript compilation issues resolved
1. **Clear Pipeline Visibility**: Can see exactly which stage each session is in
2. **Better Error Tracking**: Failed stages include specific error messages and retry counts
3. **Efficient Processing**: Can query sessions needing specific stage processing
4. **Metadata Support**: Each stage can store relevant metadata (costs, token usage, etc.)
5. **Easy Recovery**: Failed sessions can be easily identified and retried
6. **Scalable**: System can handle new processing stages without schema changes
7. **No Linting Errors**: All TypeScript compilation issues resolved
## Usage Examples
### Check Pipeline Status
```bash
npx tsx check-refactored-pipeline-status.ts
```
### Debug Processing Issues
```bash
npx tsx debug-import-status.ts
```
### Fix/Retry Failed Sessions
```bash
npx tsx fix-import-status.ts
```
### Process Sessions
```bash
npx tsx test-ai-processing.ts
```
## Next Steps
1. **Test AI Processing**: Run AI processing on pending sessions
2. **Monitor Performance**: Watch for any issues with the new system
3. **Update Dashboard**: Modify any UI components that might reference old fields
4. **Documentation**: Update any API documentation that references the old system
1. **Test AI Processing**: Run AI processing on pending sessions
2. **Monitor Performance**: Watch for any issues with the new system
3. **Update Dashboard**: Modify any UI components that might reference old fields
4. **Documentation**: Update any API documentation that references the old system
## Migration Notes
- All existing data preserved
- No data loss during migration
- Backward compatibility maintained where possible
- System ready for production use
- All existing data preserved
- No data loss during migration
- Backward compatibility maintained where possible
- System ready for production use
The refactored system provides much better visibility into the processing pipeline and makes it easy to identify and resolve any issues that arise during session processing.

View File

@ -6,11 +6,11 @@ This document describes the extracted scheduler architecture that enables horizo
The scheduler system has been refactored from a monolithic approach to a service-oriented architecture that supports:
- **Individual Scheduler Services** - Each scheduler runs as a separate service
- **Horizontal Scaling** - Multiple instances of the same scheduler can run across different machines
- **Health Monitoring** - Built-in health checks for load balancers and orchestrators
- **Graceful Shutdown** - Proper handling of shutdown signals for zero-downtime deployments
- **Centralized Management** - Optional scheduler manager for coordinated operations
- **Individual Scheduler Services** - Each scheduler runs as a separate service
- **Horizontal Scaling** - Multiple instances of the same scheduler can run across different machines
- **Health Monitoring** - Built-in health checks for load balancers and orchestrators
- **Graceful Shutdown** - Proper handling of shutdown signals for zero-downtime deployments
- **Centralized Management** - Optional scheduler manager for coordinated operations
## Components
@ -34,11 +34,11 @@ export abstract class BaseSchedulerService extends EventEmitter {
**Features:**
- Status management (STOPPED, STARTING, RUNNING, PAUSED, ERROR)
- Metrics collection (run counts, timing, success/failure rates)
- Event emission for monitoring
- Configurable intervals and timeouts
- Automatic retry handling
- Status management (STOPPED, STARTING, RUNNING, PAUSED, ERROR)
- Metrics collection (run counts, timing, success/failure rates)
- Event emission for monitoring
- Configurable intervals and timeouts
- Automatic retry handling
### 2. Individual Scheduler Services
@ -57,16 +57,16 @@ const csvScheduler = new CsvImportSchedulerService({
**Features:**
- Batch processing with configurable concurrency
- Duplicate detection
- Company-specific error handling
- Progress monitoring
- Batch processing with configurable concurrency
- Duplicate detection
- Company-specific error handling
- Progress monitoring
#### Additional Schedulers (To Be Implemented)
- `ImportProcessingSchedulerService` - Process imported CSV data into sessions
- `SessionProcessingSchedulerService` - AI analysis and categorization
- `BatchProcessingSchedulerService` - OpenAI Batch API integration
- `ImportProcessingSchedulerService` - Process imported CSV data into sessions
- `SessionProcessingSchedulerService` - AI analysis and categorization
- `BatchProcessingSchedulerService` - OpenAI Batch API integration
### 3. SchedulerManager
@ -88,10 +88,10 @@ await manager.startAll();
**Features:**
- Automatic restart of failed critical schedulers
- Health monitoring across all schedulers
- Coordinated start/stop operations
- Event aggregation and logging
- Automatic restart of failed critical schedulers
- Health monitoring across all schedulers
- Coordinated start/stop operations
- Event aggregation and logging
### 4. Standalone Scheduler Runner
@ -107,10 +107,10 @@ npx tsx lib/services/schedulers/StandaloneSchedulerRunner.ts --list
**Features:**
- Independent process execution
- Environment variable configuration
- Graceful shutdown handling
- Health reporting for monitoring
- Independent process execution
- Environment variable configuration
- Graceful shutdown handling
- Health reporting for monitoring
## Deployment Patterns
@ -127,15 +127,15 @@ await initializeSchedulers();
**Pros:**
- Simple deployment
- Lower resource usage
- Easy local development
- Simple deployment
- Lower resource usage
- Easy local development
**Cons:**
- Limited scalability
- Single point of failure
- Resource contention
- Limited scalability
- Single point of failure
- Resource contention
### 2. Separate Processes
@ -154,15 +154,15 @@ npm run scheduler:session-processing
**Pros:**
- Independent scaling
- Fault isolation
- Resource optimization per scheduler
- Independent scaling
- Fault isolation
- Resource optimization per scheduler
**Cons:**
- More complex deployment
- Higher resource overhead
- Inter-process coordination needed
- More complex deployment
- Higher resource overhead
- Inter-process coordination needed
### 3. Container Orchestration (Recommended for Production)
@ -193,16 +193,16 @@ services:
**Pros:**
- Full horizontal scaling
- Independent resource allocation
- Health monitoring integration
- Zero-downtime deployments
- Full horizontal scaling
- Independent resource allocation
- Health monitoring integration
- Zero-downtime deployments
**Cons:**
- Complex orchestration setup
- Network latency considerations
- Distributed system challenges
- Complex orchestration setup
- Network latency considerations
- Distributed system challenges
## Configuration
@ -379,46 +379,46 @@ csv-import-scheduler-eu:
### From Current Architecture
1. **Phase 1: Extract Schedulers**
- ✅ Create BaseSchedulerService
- ✅ Implement CsvImportSchedulerService
- ✅ Create SchedulerManager
- ⏳ Implement remaining scheduler services
1. **Phase 1: Extract Schedulers**
- ✅ Create BaseSchedulerService
- ✅ Implement CsvImportSchedulerService
- ✅ Create SchedulerManager
- ⏳ Implement remaining scheduler services
2. **Phase 2: Deployment Options**
- ✅ Add ServerSchedulerIntegration for backwards compatibility
- ✅ Create StandaloneSchedulerRunner
- ✅ Add health check endpoints
2. **Phase 2: Deployment Options**
- ✅ Add ServerSchedulerIntegration for backwards compatibility
- ✅ Create StandaloneSchedulerRunner
- ✅ Add health check endpoints
3. **Phase 3: Container Support**
- ⏳ Create Dockerfile for scheduler containers
- ⏳ Add Kubernetes manifests
- ⏳ Implement distributed coordination
3. **Phase 3: Container Support**
- ⏳ Create Dockerfile for scheduler containers
- ⏳ Add Kubernetes manifests
- ⏳ Implement distributed coordination
4. **Phase 4: Production Migration**
- ⏳ Deploy separate scheduler containers
- ⏳ Monitor performance and stability
- ⏳ Gradually increase horizontal scaling
4. **Phase 4: Production Migration**
- ⏳ Deploy separate scheduler containers
- ⏳ Monitor performance and stability
- ⏳ Gradually increase horizontal scaling
### Breaking Changes
- Scheduler initialization moved from `server.ts` to `ServerSchedulerIntegration`
- Individual scheduler functions replaced with service classes
- Configuration moved to environment variables
- Scheduler initialization moved from `server.ts` to `ServerSchedulerIntegration`
- Individual scheduler functions replaced with service classes
- Configuration moved to environment variables
## Benefits
1. **Scalability**: Independent scaling of different scheduler types
2. **Reliability**: Fault isolation prevents cascading failures
3. **Performance**: Optimized resource allocation per scheduler
4. **Monitoring**: Granular health checks and metrics
5. **Deployment**: Zero-downtime updates and rollbacks
6. **Development**: Easier testing and debugging of individual schedulers
1. **Scalability**: Independent scaling of different scheduler types
2. **Reliability**: Fault isolation prevents cascading failures
3. **Performance**: Optimized resource allocation per scheduler
4. **Monitoring**: Granular health checks and metrics
5. **Deployment**: Zero-downtime updates and rollbacks
6. **Development**: Easier testing and debugging of individual schedulers
## Next Steps
1. Implement remaining scheduler services (ImportProcessing, SessionProcessing, BatchProcessing)
2. Add distributed coordination for multi-instance schedulers
3. Create Kubernetes operators for automatic scaling
4. Implement scheduler-specific metrics and dashboards
5. Add scheduler performance optimization tools
1. Implement remaining scheduler services (ImportProcessing, SessionProcessing, BatchProcessing)
2. Add distributed coordination for multi-instance schedulers
3. Create Kubernetes operators for automatic scaling
4. Implement scheduler-specific metrics and dashboards
5. Add scheduler performance optimization tools

View File

@ -8,8 +8,8 @@
**Solution**:
- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs
- Removed the invalid company record from the database using `fix_companies.js`
- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs
- Removed the invalid company record from the database using `fix_companies.js`
### 2. Transcript Fetching Errors
@ -17,10 +17,10 @@
**Solution**:
- Improved error handling in `fetchTranscriptContent()` function
- Added probabilistic logging (only ~10% of errors logged) to prevent log spam
- Added timeout (10 seconds) for transcript fetching
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
- Improved error handling in `fetchTranscriptContent()` function
- Added probabilistic logging (only ~10% of errors logged) to prevent log spam
- Added timeout (10 seconds) for transcript fetching
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
### 3. CSV Fetching Errors
@ -28,8 +28,8 @@
**Solution**:
- Added URL validation to skip companies with `example.com` URLs
- Improved error logging to be more descriptive
- Added URL validation to skip companies with `example.com` URLs
- Improved error logging to be more descriptive
## Current Status
@ -42,23 +42,23 @@
After cleanup, only valid companies remain:
- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`)
- CSV URL: `https://proto.notso.ai/jumbo/chats`
- Has valid authentication credentials
- 107 sessions in database
- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`)
- CSV URL: `https://proto.notso.ai/jumbo/chats`
- Has valid authentication credentials
- 107 sessions in database
## Files Modified
1. **lib/csvFetcher.js**
- Added company URL validation
- Improved transcript fetching error handling
- Reduced error log verbosity
- Added company URL validation
- Improved transcript fetching error handling
- Reduced error log verbosity
2. **fix_companies.js** (cleanup script)
- Removes invalid company records
- Can be run again if needed
- Removes invalid company records
- Can be run again if needed
## Monitoring

View File

@ -99,12 +99,14 @@ node scripts/manual-triggers.js both
```
2. **If "Sessions with transcript" is 0:**
- Sessions exist but transcripts haven't been fetched yet
- Run session refresh: `node scripts/manual-triggers.js refresh`
- Sessions exist but transcripts haven't been fetched yet
- Run session refresh: `node scripts/manual-triggers.js refresh`
3. **If "Ready for processing" is 0 but "Sessions with transcript" > 0:**
- All sessions with transcripts have already been processed
- Check if `OPENAI_API_KEY` is set in environment
- All sessions with transcripts have already been processed
- Check if `OPENAI_API_KEY` is set in environment
### Common Issues

View File

@ -12,59 +12,59 @@ The security audit logging system provides comprehensive tracking of security-cr
The system logs the following event types:
- **Authentication Events**: Login attempts, password changes, session management
- **Authorization Events**: Permission checks, access denied events
- **User Management**: User creation, modification, deletion, invitations
- **Company Management**: Company suspension, settings changes
- **Rate Limiting**: Abuse prevention and rate limit violations
- **CSRF Protection**: Cross-site request forgery protection events
- **Security Headers**: Security header violations
- **Password Reset**: Password reset flows and token validation
- **Platform Admin**: Administrative activities by platform users
- **Data Privacy**: Data export and privacy-related events
- **System Configuration**: System setting changes
- **API Security**: API-related security events
- **Authentication Events**: Login attempts, password changes, session management
- **Authorization Events**: Permission checks, access denied events
- **User Management**: User creation, modification, deletion, invitations
- **Company Management**: Company suspension, settings changes
- **Rate Limiting**: Abuse prevention and rate limit violations
- **CSRF Protection**: Cross-site request forgery protection events
- **Security Headers**: Security header violations
- **Password Reset**: Password reset flows and token validation
- **Platform Admin**: Administrative activities by platform users
- **Data Privacy**: Data export and privacy-related events
- **System Configuration**: System setting changes
- **API Security**: API-related security events
### 2. Structured Logging
Each audit log entry includes:
- **Event Type**: Categorizes the security event
- **Action**: Specific action performed
- **Outcome**: Success, failure, blocked, rate limited, or suspicious
- **Severity**: Info, low, medium, high, or critical
- **Context**: User ID, company ID, platform user ID, IP address, user agent
- **Metadata**: Structured additional information
- **Timestamp**: Immutable timestamp for chronological ordering
- **Event Type**: Categorizes the security event
- **Action**: Specific action performed
- **Outcome**: Success, failure, blocked, rate limited, or suspicious
- **Severity**: Info, low, medium, high, or critical
- **Context**: User ID, company ID, platform user ID, IP address, user agent
- **Metadata**: Structured additional information
- **Timestamp**: Immutable timestamp for chronological ordering
### 3. Multi-Tenant Security
- Company-scoped audit logs ensure data isolation
- Platform admin actions tracked separately
- Role-based access controls for audit log viewing
- Company-scoped audit logs ensure data isolation
- Platform admin actions tracked separately
- Role-based access controls for audit log viewing
### 4. Log Retention and Management
- **Configurable Retention Policies**: Different retention periods based on event type and severity
- **Automatic Archival**: Critical and high-severity events archived before deletion
- **Scheduled Cleanup**: Weekly automated retention policy execution
- **Manual Controls**: Admin interface for manual retention execution
- **Configurable Retention Policies**: Different retention periods based on event type and severity
- **Automatic Archival**: Critical and high-severity events archived before deletion
- **Scheduled Cleanup**: Weekly automated retention policy execution
- **Manual Controls**: Admin interface for manual retention execution
### 5. Administrative Interface
- **Audit Log Viewer**: Comprehensive filtering and search capabilities
- **Retention Management**: View statistics and execute retention policies
- **Real-time Monitoring**: Track security events as they occur
- **Audit Log Viewer**: Comprehensive filtering and search capabilities
- **Retention Management**: View statistics and execute retention policies
- **Real-time Monitoring**: Track security events as they occur
## Architecture
### Core Components
1. **SecurityAuditLogger** (`lib/securityAuditLogger.ts`): Centralized logging service
2. **AuditLogRetentionManager** (`lib/auditLogRetention.ts`): Retention policy management
3. **AuditLogScheduler** (`lib/auditLogScheduler.ts`): Scheduled retention execution
4. **Admin API** (`app/api/admin/audit-logs/`): REST API for audit log access
5. **Admin UI** (`app/dashboard/audit-logs/`): Administrative interface
1. **SecurityAuditLogger** (`lib/securityAuditLogger.ts`): Centralized logging service
2. **AuditLogRetentionManager** (`lib/auditLogRetention.ts`): Retention policy management
3. **AuditLogScheduler** (`lib/auditLogScheduler.ts`): Scheduled retention execution
4. **Admin API** (`app/api/admin/audit-logs/`): REST API for audit log access
5. **Admin UI** (`app/dashboard/audit-logs/`): Administrative interface
### Database Schema
@ -126,18 +126,18 @@ await securityAuditLogger.logAuthorization(
Administrators can access audit logs through:
1. **Dashboard UI**: Navigate to "Audit Logs" in the sidebar
2. **API Access**: GET `/api/admin/audit-logs` with filtering parameters
3. **Retention Management**: GET/POST `/api/admin/audit-logs/retention`
1. **Dashboard UI**: Navigate to "Audit Logs" in the sidebar
2. **API Access**: GET `/api/admin/audit-logs` with filtering parameters
3. **Retention Management**: GET/POST `/api/admin/audit-logs/retention`
### Filtering Options
- Event type (authentication, authorization, etc.)
- Outcome (success, failure, blocked, etc.)
- Severity level (info, low, medium, high, critical)
- Date range
- User ID
- Pagination support
- Event type (authentication, authorization, etc.)
- Outcome (success, failure, blocked, etc.)
- Severity level (info, low, medium, high, critical)
- Date range
- User ID
- Pagination support
## Configuration
@ -159,105 +159,105 @@ AUDIT_LOG_RETENTION_DRY_RUN=false
### Default Retention Policies
1. **Critical Events**: 7 years retention with archival
2. **High Severity Events**: 3 years retention with archival
3. **Authentication Events**: 2 years retention with archival
4. **Platform Admin Events**: 3 years retention with archival
5. **User Management Events**: 2 years retention with archival
6. **General Events**: 1 year retention without archival
1. **Critical Events**: 7 years retention with archival
2. **High Severity Events**: 3 years retention with archival
3. **Authentication Events**: 2 years retention with archival
4. **Platform Admin Events**: 3 years retention with archival
5. **User Management Events**: 2 years retention with archival
6. **General Events**: 1 year retention without archival
## Security Considerations
### Data Protection
- **IP Address Storage**: Client IP addresses stored for geographic analysis
- **Sensitive Data Redaction**: Passwords, tokens, and emails marked as `[REDACTED]`
- **Metadata Sanitization**: Complex objects sanitized to prevent data leakage
- **IP Address Storage**: Client IP addresses stored for geographic analysis
- **Sensitive Data Redaction**: Passwords, tokens, and emails marked as `[REDACTED]`
- **Metadata Sanitization**: Complex objects sanitized to prevent data leakage
### Access Controls
- **Admin-Only Access**: Only users with `ADMIN` role can view audit logs
- **Company Isolation**: Users can only view logs for their own company
- **Platform Separation**: Platform admin logs tracked separately
- **Admin-Only Access**: Only users with `ADMIN` role can view audit logs
- **Company Isolation**: Users can only view logs for their own company
- **Platform Separation**: Platform admin logs tracked separately
### Performance
- **Async Logging**: All logging operations are asynchronous to avoid blocking
- **Error Handling**: Logging failures don't affect application functionality
- **Indexed Queries**: Database indexes optimize common query patterns
- **Batch Operations**: Retention policies use batch operations for efficiency
- **Async Logging**: All logging operations are asynchronous to avoid blocking
- **Error Handling**: Logging failures don't affect application functionality
- **Indexed Queries**: Database indexes optimize common query patterns
- **Batch Operations**: Retention policies use batch operations for efficiency
## Compliance Features
### Audit Standards
- **Immutable Records**: Audit logs cannot be modified after creation
- **Chronological Ordering**: Precise timestamps for event sequencing
- **Non-Repudiation**: User actions clearly attributed and timestamped
- **Comprehensive Coverage**: All security-relevant events logged
- **Immutable Records**: Audit logs cannot be modified after creation
- **Chronological Ordering**: Precise timestamps for event sequencing
- **Non-Repudiation**: User actions clearly attributed and timestamped
- **Comprehensive Coverage**: All security-relevant events logged
### Reporting
- **Event Statistics**: Summary statistics by event type, severity, and time period
- **Export Capabilities**: Structured data export for compliance reporting
- **Retention Tracking**: Detailed logging of retention policy execution
- **Event Statistics**: Summary statistics by event type, severity, and time period
- **Export Capabilities**: Structured data export for compliance reporting
- **Retention Tracking**: Detailed logging of retention policy execution
## Monitoring and Alerting
### System Health
- **Scheduler Status**: Monitor retention scheduler health
- **Error Tracking**: Log retention and audit logging errors
- **Performance Metrics**: Track logging performance and database impact
- **Scheduler Status**: Monitor retention scheduler health
- **Error Tracking**: Log retention and audit logging errors
- **Performance Metrics**: Track logging performance and database impact
### Security Monitoring
- **Failed Authentication Patterns**: Track repeated login failures
- **Privilege Escalation**: Monitor administrative action patterns
- **Suspicious Activity**: Identify unusual access patterns
- **Failed Authentication Patterns**: Track repeated login failures
- **Privilege Escalation**: Monitor administrative action patterns
- **Suspicious Activity**: Identify unusual access patterns
## Troubleshooting
### Common Issues
1. **Audit Logging Disabled**: Check `AUDIT_LOGGING_ENABLED` environment variable
2. **Retention Not Running**: Verify `AUDIT_LOG_RETENTION_ENABLED` and scheduler status
3. **Access Denied**: Ensure user has `ADMIN` role for audit log access
4. **Performance Issues**: Review retention policies and database indexes
1. **Audit Logging Disabled**: Check `AUDIT_LOGGING_ENABLED` environment variable
2. **Retention Not Running**: Verify `AUDIT_LOG_RETENTION_ENABLED` and scheduler status
3. **Access Denied**: Ensure user has `ADMIN` role for audit log access
4. **Performance Issues**: Review retention policies and database indexes
### Debug Information
- Check application logs for scheduler startup messages
- Monitor database query performance for audit log operations
- Review retention policy validation warnings
- Check application logs for scheduler startup messages
- Monitor database query performance for audit log operations
- Review retention policy validation warnings
## Best Practices
### Implementation
1. **Always use the centralized logger**: Don't bypass the `securityAuditLogger`
2. **Include relevant context**: Provide user, company, and IP information
3. **Use appropriate severity levels**: Follow the severity assignment guidelines
4. **Sanitize sensitive data**: Use `createAuditMetadata()` for safe metadata
1. **Always use the centralized logger**: Don't bypass the `securityAuditLogger`
2. **Include relevant context**: Provide user, company, and IP information
3. **Use appropriate severity levels**: Follow the severity assignment guidelines
4. **Sanitize sensitive data**: Use `createAuditMetadata()` for safe metadata
### Operations
1. **Regular retention review**: Monitor retention statistics and adjust policies
2. **Archive critical data**: Ensure important logs are archived before deletion
3. **Monitor storage usage**: Track audit log database growth
4. **Test restoration**: Verify archived data can be restored when needed
1. **Regular retention review**: Monitor retention statistics and adjust policies
2. **Archive critical data**: Ensure important logs are archived before deletion
3. **Monitor storage usage**: Track audit log database growth
4. **Test restoration**: Verify archived data can be restored when needed
## Future Enhancements
### Planned Features
- **Real-time Alerting**: Immediate notifications for critical security events
- **Advanced Analytics**: ML-based anomaly detection and pattern recognition
- **Export Formats**: Additional export formats for compliance reporting
- **External Integration**: SIEM and security tool integrations
- **Real-time Alerting**: Immediate notifications for critical security events
- **Advanced Analytics**: ML-based anomaly detection and pattern recognition
- **Export Formats**: Additional export formats for compliance reporting
- **External Integration**: SIEM and security tool integrations
### Performance Optimizations
- **Log Partitioning**: Database partitioning for improved query performance
- **Compression**: Log compression for storage efficiency
- **Streaming**: Real-time log streaming for external systems
- **Log Partitioning**: Database partitioning for improved query performance
- **Compression**: Log compression for storage efficiency
- **Streaming**: Real-time log streaming for external systems

View File

@ -12,33 +12,33 @@ The application implements multiple layers of HTTP security headers to provide d
#### X-Content-Type-Options: nosniff
- **Purpose**: Prevents MIME type sniffing attacks
- **Protection**: Stops browsers from interpreting files as different MIME types than declared
- **Value**: `nosniff`
- **Purpose**: Prevents MIME type sniffing attacks
- **Protection**: Stops browsers from interpreting files as different MIME types than declared
- **Value**: `nosniff`
#### X-Frame-Options: DENY
- **Purpose**: Prevents clickjacking attacks
- **Protection**: Blocks embedding the site in frames/iframes
- **Value**: `DENY`
- **Purpose**: Prevents clickjacking attacks
- **Protection**: Blocks embedding the site in frames/iframes
- **Value**: `DENY`
#### X-XSS-Protection: 1; mode=block
- **Purpose**: Enables XSS protection in legacy browsers
- **Protection**: Activates built-in XSS filtering (primarily for older browsers)
- **Value**: `1; mode=block`
- **Purpose**: Enables XSS protection in legacy browsers
- **Protection**: Activates built-in XSS filtering (primarily for older browsers)
- **Value**: `1; mode=block`
#### Referrer-Policy: strict-origin-when-cross-origin
- **Purpose**: Controls referrer information leakage
- **Protection**: Limits referrer data sent to external sites
- **Value**: `strict-origin-when-cross-origin`
- **Purpose**: Controls referrer information leakage
- **Protection**: Limits referrer data sent to external sites
- **Value**: `strict-origin-when-cross-origin`
#### X-DNS-Prefetch-Control: off
- **Purpose**: Prevents DNS rebinding attacks
- **Protection**: Disables DNS prefetching to reduce attack surface
- **Value**: `off`
- **Purpose**: Prevents DNS rebinding attacks
- **Protection**: Disables DNS prefetching to reduce attack surface
- **Value**: `off`
### Content Security Policy (CSP)
@ -48,15 +48,15 @@ Comprehensive CSP implementation with the following directives:
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self' https:; frame-ancestors 'none'; base-uri 'self'; form-action 'self'; object-src 'none'; upgrade-insecure-requests
```
#### Key CSP Directives:
#### Key CSP Directives
- **default-src 'self'**: Restrictive default for all resource types
- **script-src 'self' 'unsafe-eval' 'unsafe-inline'**: Allows Next.js dev tools and React functionality
- **style-src 'self' 'unsafe-inline'**: Enables TailwindCSS and component styles
- **img-src 'self' data: https:**: Allows secure image sources
- **frame-ancestors 'none'**: Prevents embedding (reinforces X-Frame-Options)
- **object-src 'none'**: Blocks dangerous plugins and embeds
- **upgrade-insecure-requests**: Automatically upgrades HTTP to HTTPS
- **default-src 'self'**: Restrictive default for all resource types
- **script-src 'self' 'unsafe-eval' 'unsafe-inline'**: Allows Next.js dev tools and React functionality
- **style-src 'self' 'unsafe-inline'**: Enables TailwindCSS and component styles
- **img-src 'self' data: https:**: Allows secure image sources
- **frame-ancestors 'none'**: Prevents embedding (reinforces X-Frame-Options)
- **object-src 'none'**: Blocks dangerous plugins and embeds
- **upgrade-insecure-requests**: Automatically upgrades HTTP to HTTPS
### Permissions Policy
@ -66,19 +66,19 @@ Controls browser feature access:
Permissions-Policy: camera=(), microphone=(), geolocation=(), interest-cohort=(), browsing-topics=()
```
- **camera=()**: Disables camera access
- **microphone=()**: Disables microphone access
- **geolocation=()**: Disables location tracking
- **interest-cohort=()**: Blocks FLoC (privacy protection)
- **browsing-topics=()**: Blocks Topics API (privacy protection)
- **camera=()**: Disables camera access
- **microphone=()**: Disables microphone access
- **geolocation=()**: Disables location tracking
- **interest-cohort=()**: Blocks FLoC (privacy protection)
- **browsing-topics=()**: Blocks Topics API (privacy protection)
### Strict Transport Security (HSTS)
**Production Only**: `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload`
- **max-age=31536000**: 1 year HSTS policy
- **includeSubDomains**: Applies to all subdomains
- **preload**: Ready for HSTS preload list inclusion
- **max-age=31536000**: 1 year HSTS policy
- **includeSubDomains**: Applies to all subdomains
- **preload**: Ready for HSTS preload list inclusion
## Configuration
@ -110,8 +110,8 @@ headers: async () => {
### Environment-Specific Behavior
- **Development**: All headers except HSTS
- **Production**: All headers including HSTS
- **Development**: All headers except HSTS
- **Production**: All headers including HSTS
## Testing
@ -121,11 +121,11 @@ Location: `tests/unit/http-security-headers.test.ts`
Tests cover:
- Individual header validation
- CSP directive verification
- Permissions Policy validation
- Environment-specific configuration
- Next.js compatibility checks
- Individual header validation
- CSP directive verification
- Permissions Policy validation
- Environment-specific configuration
- Next.js compatibility checks
### Integration Tests
@ -133,9 +133,9 @@ Location: `tests/integration/security-headers-basic.test.ts`
Tests cover:
- Next.js configuration validation
- Header generation verification
- Environment-based header differences
- Next.js configuration validation
- Header generation verification
- Environment-based header differences
### Manual Testing
@ -153,41 +153,41 @@ pnpm test:security-headers https://your-domain.com
### Protection Against OWASP Top 10
1. **A03:2021 - Injection**: CSP prevents script injection
2. **A05:2021 - Security Misconfiguration**: Comprehensive headers reduce attack surface
3. **A06:2021 - Vulnerable Components**: CSP limits execution context
4. **A07:2021 - Identification and Authentication Failures**: HSTS prevents downgrade attacks
1. **A03:2021 - Injection**: CSP prevents script injection
2. **A05:2021 - Security Misconfiguration**: Comprehensive headers reduce attack surface
3. **A06:2021 - Vulnerable Components**: CSP limits execution context
4. **A07:2021 - Identification and Authentication Failures**: HSTS prevents downgrade attacks
### Additional Security Benefits
- **Clickjacking Protection**: X-Frame-Options + CSP frame-ancestors
- **MIME Sniffing Prevention**: X-Content-Type-Options
- **Information Leakage Reduction**: Referrer-Policy
- **Privacy Protection**: Permissions Policy restrictions
- **Transport Security**: HSTS enforcement
- **Clickjacking Protection**: X-Frame-Options + CSP frame-ancestors
- **MIME Sniffing Prevention**: X-Content-Type-Options
- **Information Leakage Reduction**: Referrer-Policy
- **Privacy Protection**: Permissions Policy restrictions
- **Transport Security**: HSTS enforcement
## Maintenance
### Regular Reviews
1. **Quarterly CSP Review**: Analyze CSP violations and tighten policies
2. **Annual Header Audit**: Review new security headers and standards
3. **Dependency Updates**: Ensure compatibility with framework updates
1. **Quarterly CSP Review**: Analyze CSP violations and tighten policies
2. **Annual Header Audit**: Review new security headers and standards
3. **Dependency Updates**: Ensure compatibility with framework updates
### Monitoring
- Monitor CSP violation reports (when implemented)
- Use online tools like securityheaders.com for validation
- Include security header tests in CI/CD pipeline
- Monitor CSP violation reports (when implemented)
- Use online tools like securityheaders.com for validation
- Include security header tests in CI/CD pipeline
### Future Enhancements
Planned improvements:
1. CSP violation reporting endpoint
2. Nonce-based CSP for inline scripts
3. Additional Permissions Policy restrictions
4. Content-Type validation middleware
1. CSP violation reporting endpoint
2. Nonce-based CSP for inline scripts
3. Additional Permissions Policy restrictions
4. Content-Type validation middleware
## Compatibility
@ -195,37 +195,37 @@ Planned improvements:
Headers are configured to be compatible with:
- Next.js 15+ App Router
- React 19 development tools
- TailwindCSS 4 styling system
- Development hot reload functionality
- Next.js 15+ App Router
- React 19 development tools
- TailwindCSS 4 styling system
- Development hot reload functionality
### Browser Support
Security headers are supported by:
- All modern browsers (Chrome 60+, Firefox 60+, Safari 12+)
- Graceful degradation for older browsers
- Progressive enhancement approach
- All modern browsers (Chrome 60+, Firefox 60+, Safari 12+)
- Graceful degradation for older browsers
- Progressive enhancement approach
## Troubleshooting
### Common Issues
1. **CSP Violations**: Check browser console for CSP errors
2. **Styling Issues**: Verify style-src allows 'unsafe-inline'
3. **Script Errors**: Ensure script-src permits necessary scripts
4. **Development Issues**: Use `pnpm dev:next-only` to isolate Next.js
1. **CSP Violations**: Check browser console for CSP errors
2. **Styling Issues**: Verify style-src allows 'unsafe-inline'
3. **Script Errors**: Ensure script-src permits necessary scripts
4. **Development Issues**: Use `pnpm dev:next-only` to isolate Next.js
### Debug Tools
- Browser DevTools Security tab
- CSP Evaluator: https://csp-evaluator.withgoogle.com/
- Security Headers Scanner: https://securityheaders.com/
- Browser DevTools Security tab
- CSP Evaluator: <https://csp-evaluator.withgoogle.com/>
- Security Headers Scanner: <https://securityheaders.com/>
## References
- [OWASP Secure Headers Project](https://owasp.org/www-project-secure-headers/)
- [MDN Security Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#security)
- [Next.js Security Headers](https://nextjs.org/docs/app/api-reference/config/headers)
- [Content Security Policy Reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)
- [OWASP Secure Headers Project](https://owasp.org/www-project-secure-headers/)
- [MDN Security Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#security)
- [Next.js Security Headers](https://nextjs.org/docs/app/api-reference/config/headers)
- [Content Security Policy Reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)

View File

@ -8,39 +8,39 @@ The Security Monitoring and Alerting System provides comprehensive real-time sec
### Core Components
1. **Security Monitoring Service** (`lib/securityMonitoring.ts`)
- Real-time event processing
- Anomaly detection algorithms
- Alert generation and management
- Security score calculation
- Threat level assessment
1. **Security Monitoring Service** (`lib/securityMonitoring.ts`)
- Real-time event processing
- Anomaly detection algorithms
- Alert generation and management
- Security score calculation
- Threat level assessment
2. **Enhanced Security Logging** (`enhancedSecurityLog`)
- Integrates with existing audit logger
- Processes events through monitoring system
- Triggers immediate threat detection
2. **Enhanced Security Logging** (`enhancedSecurityLog`)
- Integrates with existing audit logger
- Processes events through monitoring system
- Triggers immediate threat detection
3. **API Endpoints** (`app/api/admin/security-monitoring/`)
- `/api/admin/security-monitoring` - Main metrics and configuration
- `/api/admin/security-monitoring/alerts` - Alert management
- `/api/admin/security-monitoring/export` - Data export
- `/api/admin/security-monitoring/threat-analysis` - Threat analysis
3. **API Endpoints** (`app/api/admin/security-monitoring/`)
- `/api/admin/security-monitoring` - Main metrics and configuration
- `/api/admin/security-monitoring/alerts` - Alert management
- `/api/admin/security-monitoring/export` - Data export
- `/api/admin/security-monitoring/threat-analysis` - Threat analysis
4. **Dashboard UI** (`app/platform/security/page.tsx`)
- Real-time security metrics
- Active alerts management
- Threat analysis visualization
- Configuration management
4. **Dashboard UI** (`app/platform/security/page.tsx`)
- Real-time security metrics
- Active alerts management
- Threat analysis visualization
- Configuration management
## Features
### Real-time Monitoring
- **Authentication Events**: Login attempts, failures, brute force attacks
- **Rate Limiting**: Excessive request patterns, API abuse
- **Admin Activity**: Unusual administrative actions
- **Geographic Anomalies**: Logins from unusual locations
- **Temporal Anomalies**: Activity spikes outside normal patterns
- **Authentication Events**: Login attempts, failures, brute force attacks
- **Rate Limiting**: Excessive request patterns, API abuse
- **Admin Activity**: Unusual administrative actions
- **Geographic Anomalies**: Logins from unusual locations
- **Temporal Anomalies**: Activity spikes outside normal patterns
### Alert Types
@ -68,30 +68,30 @@ enum AlertType {
The system implements several anomaly detection algorithms:
1. **Geographic Anomaly Detection**
- Detects logins from unusual countries
- Compares against historical user patterns
- Confidence scoring based on deviation
1. **Geographic Anomaly Detection**
- Detects logins from unusual countries
- Compares against historical user patterns
- Confidence scoring based on deviation
2. **Temporal Anomaly Detection**
- Identifies activity spikes during unusual hours
- Compares current activity to historical averages
- Configurable thresholds for different event types
2. **Temporal Anomaly Detection**
- Identifies activity spikes during unusual hours
- Compares current activity to historical averages
- Configurable thresholds for different event types
3. **Behavioral Anomaly Detection**
- Multiple failed login attempts
- Rapid succession of actions
- Pattern deviation analysis
3. **Behavioral Anomaly Detection**
- Multiple failed login attempts
- Rapid succession of actions
- Pattern deviation analysis
### Security Scoring
The system calculates a real-time security score (0-100) based on:
- Critical security events (weight: 25)
- Active unresolved alerts (weight: 30)
- High-severity threats (weight: 20)
- Overall event volume (weight: 15)
- System stability factors (weight: 10)
- Critical security events (weight: 25)
- Active unresolved alerts (weight: 30)
- High-severity threats (weight: 20)
- Overall event volume (weight: 15)
- System stability factors (weight: 10)
### Threat Levels
@ -255,117 +255,117 @@ await enhancedSecurityLog(
### Security Overview
- Real-time security score (0-100)
- Current threat level indicator
- Active alerts count
- Security events summary
- Real-time security score (0-100)
- Current threat level indicator
- Active alerts count
- Security events summary
### Alert Management
- View active and historical alerts
- Filter by severity and type
- Acknowledge alerts with tracking
- Detailed alert context and metadata
- View active and historical alerts
- Filter by severity and type
- Acknowledge alerts with tracking
- Detailed alert context and metadata
### Threat Analysis
- Geographic distribution of events
- Top threat types and patterns
- User risk scoring
- IP threat level analysis
- Geographic distribution of events
- Top threat types and patterns
- User risk scoring
- IP threat level analysis
### Configuration Management
- Adjust detection thresholds
- Configure alerting channels
- Set data retention policies
- Export capabilities
- Adjust detection thresholds
- Configure alerting channels
- Set data retention policies
- Export capabilities
## Performance Considerations
### Memory Management
- Event buffer limited to 1 hour of data
- Automatic cleanup of old alerts (configurable)
- Efficient in-memory storage for real-time analysis
- Event buffer limited to 1 hour of data
- Automatic cleanup of old alerts (configurable)
- Efficient in-memory storage for real-time analysis
### Database Impact
- Leverages existing audit log indexes
- Optimized queries for time-range filtering
- Background processing to avoid blocking operations
- Leverages existing audit log indexes
- Optimized queries for time-range filtering
- Background processing to avoid blocking operations
### Scalability
- Stateless architecture (except for buffering)
- Horizontal scaling support
- Configurable processing intervals
- Stateless architecture (except for buffering)
- Horizontal scaling support
- Configurable processing intervals
## Security Considerations
### Access Control
- Platform admin authentication required
- Role-based access to security endpoints
- Audit logging of all monitoring activities
- Platform admin authentication required
- Role-based access to security endpoints
- Audit logging of all monitoring activities
### Data Privacy
- Sensitive data redaction in logs
- IP address anonymization options
- Configurable data retention periods
- Sensitive data redaction in logs
- IP address anonymization options
- Configurable data retention periods
### Alert Suppression
- Duplicate alert suppression (configurable window)
- Rate limiting on alert generation
- Escalation policies for critical threats
- Duplicate alert suppression (configurable window)
- Rate limiting on alert generation
- Escalation policies for critical threats
## Monitoring and Maintenance
### Health Checks
- Monitor service availability
- Check alert generation pipeline
- Verify data export functionality
- Monitor service availability
- Check alert generation pipeline
- Verify data export functionality
### Regular Tasks
- Review and adjust thresholds quarterly
- Analyze false positive rates
- Update threat detection patterns
- Clean up old alert data
- Review and adjust thresholds quarterly
- Analyze false positive rates
- Update threat detection patterns
- Clean up old alert data
### Performance Metrics
- Alert response time
- False positive/negative rates
- System resource usage
- User engagement with alerts
- Alert response time
- False positive/negative rates
- System resource usage
- User engagement with alerts
## Future Enhancements
### Planned Features
1. **Machine Learning Integration**
- Behavioral pattern recognition
- Adaptive threshold adjustment
- Predictive threat modeling
1. **Machine Learning Integration**
- Behavioral pattern recognition
- Adaptive threshold adjustment
- Predictive threat modeling
2. **Advanced Analytics**
- Threat intelligence integration
- Cross-correlation analysis
- Risk trend analysis
2. **Advanced Analytics**
- Threat intelligence integration
- Cross-correlation analysis
- Risk trend analysis
3. **Integration Enhancements**
- SIEM system connectors
- Webhook customization
- Mobile app notifications
3. **Integration Enhancements**
- SIEM system connectors
- Webhook customization
- Mobile app notifications
4. **Automated Response**
- IP blocking automation
- Account suspension workflows
- Incident response orchestration
4. **Automated Response**
- IP blocking automation
- Account suspension workflows
- Incident response orchestration
## Troubleshooting
@ -373,27 +373,27 @@ await enhancedSecurityLog(
**High False Positive Rate**
- Review and adjust detection thresholds
- Analyze user behavior patterns
- Consider geographical variations
- Review and adjust detection thresholds
- Analyze user behavior patterns
- Consider geographical variations
**Missing Alerts**
- Check service configuration
- Verify audit log integration
- Review threshold settings
- Check service configuration
- Verify audit log integration
- Review threshold settings
**Performance Issues**
- Monitor memory usage
- Adjust cleanup intervals
- Optimize database queries
- Monitor memory usage
- Adjust cleanup intervals
- Optimize database queries
**Export Failures**
- Check file permissions
- Verify date range validity
- Monitor server resources
- Check file permissions
- Verify date range validity
- Monitor server resources
### Debugging
@ -419,24 +419,24 @@ console.log("Active alerts:", alerts.length);
### Unit Tests
- Alert generation logic
- Anomaly detection algorithms
- Configuration management
- Data export functionality
- Alert generation logic
- Anomaly detection algorithms
- Configuration management
- Data export functionality
### Integration Tests
- API endpoint security
- Database integration
- Real-time event processing
- Alert acknowledgment flow
- API endpoint security
- Database integration
- Real-time event processing
- Alert acknowledgment flow
### Load Testing
- High-volume event processing
- Concurrent alert generation
- Database performance under load
- Memory usage patterns
- High-volume event processing
- Concurrent alert generation
- Database performance under load
- Memory usage patterns
Run tests:

View File

@ -8,54 +8,54 @@ This document outlines the comprehensive Content Security Policy implementation
The enhanced CSP implementation provides:
- **Nonce-based script execution** for maximum security in production
- **Strict mode policies** with configurable external domain allowlists
- **Environment-specific configurations** for development vs production
- **CSP violation reporting and monitoring** system with real-time analysis
- **Advanced bypass detection and alerting** capabilities with risk assessment
- **Comprehensive testing framework** with automated validation
- **Performance metrics and policy recommendations**
- **Framework compatibility** with Next.js, TailwindCSS, and Leaflet maps
- **Nonce-based script execution** for maximum security in production
- **Strict mode policies** with configurable external domain allowlists
- **Environment-specific configurations** for development vs production
- **CSP violation reporting and monitoring** system with real-time analysis
- **Advanced bypass detection and alerting** capabilities with risk assessment
- **Comprehensive testing framework** with automated validation
- **Performance metrics and policy recommendations**
- **Framework compatibility** with Next.js, TailwindCSS, and Leaflet maps
## Architecture
### Core Components
1. **CSP Utility Library** (`lib/csp.ts`)
- Nonce generation with cryptographic security
- Dynamic CSP building based on environment
- Violation parsing and bypass detection
- Policy validation and testing
1. **CSP Utility Library** (`lib/csp.ts`)
- Nonce generation with cryptographic security
- Dynamic CSP building based on environment
- Violation parsing and bypass detection
- Policy validation and testing
2. **Middleware Implementation** (`middleware.ts`)
- Automatic nonce generation per request
- Environment-aware policy application
- Enhanced security headers
- Route-based CSP filtering
2. **Middleware Implementation** (`middleware.ts`)
- Automatic nonce generation per request
- Environment-aware policy application
- Enhanced security headers
- Route-based CSP filtering
3. **Violation Reporting** (`app/api/csp-report/route.ts`)
- Real-time violation monitoring with intelligent analysis
- Rate-limited endpoint protection (10 reports/minute per IP)
- Advanced bypass attempt detection with risk assessment
- Automated alerting for critical violations with recommendations
3. **Violation Reporting** (`app/api/csp-report/route.ts`)
- Real-time violation monitoring with intelligent analysis
- Rate-limited endpoint protection (10 reports/minute per IP)
- Advanced bypass attempt detection with risk assessment
- Automated alerting for critical violations with recommendations
4. **Monitoring Service** (`lib/csp-monitoring.ts`)
- Violation tracking and metrics collection
- Policy recommendation engine based on violation patterns
- Export capabilities for external analysis (JSON/CSV)
- Automatic cleanup of old violation data
4. **Monitoring Service** (`lib/csp-monitoring.ts`)
- Violation tracking and metrics collection
- Policy recommendation engine based on violation patterns
- Export capabilities for external analysis (JSON/CSV)
- Automatic cleanup of old violation data
5. **Metrics API** (`app/api/csp-metrics/route.ts`)
- Real-time CSP violation metrics (1h, 6h, 24h, 7d, 30d ranges)
- Top violated directives and blocked URIs analysis
- Violation trend tracking and visualization data
- Policy optimization recommendations
5. **Metrics API** (`app/api/csp-metrics/route.ts`)
- Real-time CSP violation metrics (1h, 6h, 24h, 7d, 30d ranges)
- Top violated directives and blocked URIs analysis
- Violation trend tracking and visualization data
- Policy optimization recommendations
6. **Testing Framework**
- Comprehensive unit and integration tests
- Enhanced CSP validation tools with security scoring
- Automated compliance verification
- Real-world scenario testing for application compatibility
6. **Testing Framework**
- Comprehensive unit and integration tests
- Enhanced CSP validation tools with security scoring
- Automated compliance verification
- Real-world scenario testing for application compatibility
## CSP Policies
@ -118,9 +118,9 @@ const developmentCSP = {
### 1. Nonce-Based Script Execution
- **128-bit cryptographically secure nonces** generated per request
- **Strict-dynamic policy** prevents inline script execution
- **Automatic nonce injection** into layout components
- **128-bit cryptographically secure nonces** generated per request
- **Strict-dynamic policy** prevents inline script execution
- **Automatic nonce injection** into layout components
```tsx
// Layout with nonce support
@ -149,27 +149,32 @@ export default async function RootLayout({ children }: { children: ReactNode })
### 2. Content Source Restrictions
#### Script Sources
- **Production**: Only `'self'` and nonce-approved scripts
- **Development**: Additional `'unsafe-eval'` for dev tools
- **Blocked**: All external CDNs, inline scripts without nonce
- **Production**: Only `'self'` and nonce-approved scripts
- **Development**: Additional `'unsafe-eval'` for dev tools
- **Blocked**: All external CDNs, inline scripts without nonce
#### Style Sources
- **Production**: Nonce-based inline styles preferred
- **Fallback**: `'unsafe-inline'` for TailwindCSS compatibility
- **External**: Only self-hosted stylesheets
- **Production**: Nonce-based inline styles preferred
- **Fallback**: `'unsafe-inline'` for TailwindCSS compatibility
- **External**: Only self-hosted stylesheets
#### Image Sources
- **Allowed**: Self, data URIs, schema.org, application domain
- **Blocked**: All other external domains
- **Allowed**: Self, data URIs, schema.org, application domain
- **Blocked**: All other external domains
#### Connection Sources
- **Production**: Self, OpenAI API, application domain
- **Development**: Additional WebSocket for HMR
- **Blocked**: All other external connections
- **Production**: Self, OpenAI API, application domain
- **Development**: Additional WebSocket for HMR
- **Blocked**: All other external connections
### 3. XSS Protection Mechanisms
#### Inline Script Prevention
```javascript
// Blocked by CSP
<script>alert('xss')</script>
@ -179,18 +184,21 @@ export default async function RootLayout({ children }: { children: ReactNode })
```
#### Object Injection Prevention
```javascript
// Completely blocked
object-src 'none'
```
#### Base Tag Injection Prevention
```javascript
// Restricted to same origin
base-uri 'self'
```
#### Clickjacking Protection
```javascript
// No framing allowed
frame-ancestors 'none'
@ -230,20 +238,21 @@ CSP violations are automatically reported to `/api/csp-report`:
### Violation Processing
1. **Rate Limiting**: 10 reports per minute per IP
2. **Parsing**: Extract violation details and context
3. **Risk Assessment**: Classify as low/medium/high risk
4. **Bypass Detection**: Check for known attack patterns
5. **Alerting**: Immediate notifications for critical violations
1. **Rate Limiting**: 10 reports per minute per IP
2. **Parsing**: Extract violation details and context
3. **Risk Assessment**: Classify as low/medium/high risk
4. **Bypass Detection**: Check for known attack patterns
5. **Alerting**: Immediate notifications for critical violations
### Monitoring Dashboard
Violations are logged with:
- Timestamp and source IP
- User agent and referer
- Violation type and blocked content
- Risk level and bypass indicators
- Response actions taken
- Timestamp and source IP
- User agent and referer
- Violation type and blocked content
- Risk level and bypass indicators
- Response actions taken
## Testing and Validation
@ -262,19 +271,20 @@ pnpm test:csp:full
### Manual Testing
1. **Nonce Validation**: Verify unique nonces per request
2. **Policy Compliance**: Check all required directives
3. **Bypass Resistance**: Test common XSS techniques
4. **Framework Compatibility**: Ensure Next.js/TailwindCSS work
5. **Performance Impact**: Measure overhead
1. **Nonce Validation**: Verify unique nonces per request
2. **Policy Compliance**: Check all required directives
3. **Bypass Resistance**: Test common XSS techniques
4. **Framework Compatibility**: Ensure Next.js/TailwindCSS work
5. **Performance Impact**: Measure overhead
### Security Scoring
The validation framework provides a security score:
- **90-100%**: Excellent implementation
- **80-89%**: Good with minor improvements needed
- **70-79%**: Needs attention
- **<70%**: Serious security issues
- **90-100%**: Excellent implementation
- **80-89%**: Good with minor improvements needed
- **70-79%**: Needs attention
- **<70%**: Serious security issues
## Deployment Considerations
@ -288,74 +298,74 @@ NODE_ENV=development # Enables permissive CSP
### Performance Impact
- **Nonce generation**: ~0.1ms per request
- **Header processing**: ~0.05ms per request
- **Total overhead**: <1ms per request
- **Nonce generation**: ~0.1ms per request
- **Header processing**: ~0.05ms per request
- **Total overhead**: <1ms per request
### Browser Compatibility
- **Modern browsers**: Full CSP Level 3 support
- **Legacy browsers**: Graceful degradation with X-XSS-Protection
- **Reporting**: Supported in all major browsers
- **Modern browsers**: Full CSP Level 3 support
- **Legacy browsers**: Graceful degradation with X-XSS-Protection
- **Reporting**: Supported in all major browsers
## Maintenance
### Regular Reviews
1. **Monthly**: Review violation reports and patterns
2. **Quarterly**: Update content source restrictions
3. **Per release**: Validate CSP with new features
4. **Annually**: Security audit and penetration testing
1. **Monthly**: Review violation reports and patterns
2. **Quarterly**: Update content source restrictions
3. **Per release**: Validate CSP with new features
4. **Annually**: Security audit and penetration testing
### Updates and Modifications
When adding new content sources:
1. Update `buildCSP()` function in `lib/csp.ts`
2. Add tests for new directives
3. Validate security impact
4. Update documentation
1. Update `buildCSP()` function in `lib/csp.ts`
2. Add tests for new directives
3. Validate security impact
4. Update documentation
### Incident Response
For CSP violations:
1. **High-risk violations**: Immediate investigation
2. **Bypass attempts**: Security team notification
3. **Mass violations**: Check for policy issues
4. **False positives**: Adjust policies as needed
1. **High-risk violations**: Immediate investigation
2. **Bypass attempts**: Security team notification
3. **Mass violations**: Check for policy issues
4. **False positives**: Adjust policies as needed
## Best Practices
### Development
- Always test CSP changes in development first
- Use nonce provider for new inline scripts
- Validate external resources before adding
- Monitor console for CSP violations
- Always test CSP changes in development first
- Use nonce provider for new inline scripts
- Validate external resources before adding
- Monitor console for CSP violations
### Production
- Never disable CSP in production
- Monitor violation rates and patterns
- Keep nonce generation entropy high
- Regular security audits
- Never disable CSP in production
- Monitor violation rates and patterns
- Keep nonce generation entropy high
- Regular security audits
### Code Review
- Check all inline scripts have nonce
- Verify external resources are approved
- Ensure CSP tests pass
- Document any policy changes
- Check all inline scripts have nonce
- Verify external resources are approved
- Ensure CSP tests pass
- Document any policy changes
## Troubleshooting
### Common Issues
1. **Inline styles blocked**: Use nonce or move to external CSS
2. **Third-party scripts blocked**: Add to approved sources
3. **Dev tools not working**: Ensure development CSP allows unsafe-eval
4. **Images not loading**: Check image source restrictions
1. **Inline styles blocked**: Use nonce or move to external CSS
2. **Third-party scripts blocked**: Add to approved sources
3. **Dev tools not working**: Ensure development CSP allows unsafe-eval
4. **Images not loading**: Check image source restrictions
### Debug Tools
@ -374,19 +384,19 @@ curl -X POST /api/csp-report -d '{"csp-report": {...}}'
If CSP breaks production:
1. Check violation reports for patterns
2. Identify blocking directive
3. Test fix in staging environment
4. Deploy emergency policy update
5. Monitor for resolved issues
1. Check violation reports for patterns
2. Identify blocking directive
3. Test fix in staging environment
4. Deploy emergency policy update
5. Monitor for resolved issues
## Compliance
This CSP implementation addresses:
- **OWASP Top 10**: XSS prevention
- **CSP Level 3**: Modern security standards
- **GDPR**: Privacy-preserving monitoring
- **SOC 2**: Security controls documentation
- **OWASP Top 10**: XSS prevention
- **CSP Level 3**: Modern security standards
- **GDPR**: Privacy-preserving monitoring
- **SOC 2**: Security controls documentation
The enhanced CSP provides defense-in-depth against XSS attacks while maintaining application functionality and performance.
The enhanced CSP provides defense-in-depth against XSS attacks while maintaining application functionality and performance.

View File

@ -15,22 +15,22 @@ The system now includes an automated process for analyzing chat session transcri
### Session Fetching
- The system fetches session data from configured CSV URLs for each company
- Unlike the previous implementation, it now only adds sessions that don't already exist in the database
- This prevents duplicate sessions and allows for incremental updates
- The system fetches session data from configured CSV URLs for each company
- Unlike the previous implementation, it now only adds sessions that don't already exist in the database
- This prevents duplicate sessions and allows for incremental updates
### Transcript Processing
- For sessions with transcript content that haven't been processed yet, the system calls OpenAI's API
- The API analyzes the transcript and extracts the following information:
- Primary language used (ISO 639-1 code)
- Number of messages sent by the user
- Overall sentiment (positive, neutral, negative)
- Whether the conversation was escalated
- Whether HR contact was mentioned or provided
- Best-fitting category for the conversation
- Up to 5 paraphrased questions asked by the user
- A brief summary of the conversation
- For sessions with transcript content that haven't been processed yet, the system calls OpenAI's API
- The API analyzes the transcript and extracts the following information:
- Primary language used (ISO 639-1 code)
- Number of messages sent by the user
- Overall sentiment (positive, neutral, negative)
- Whether the conversation was escalated
- Whether HR contact was mentioned or provided
- Best-fitting category for the conversation
- Up to 5 paraphrased questions asked by the user
- A brief summary of the conversation
### Scheduling
@ -43,10 +43,10 @@ The system includes two schedulers:
The Session model has been updated with new fields to store the processed data:
- `processed`: Boolean flag indicating whether the session has been processed
- `sentimentCategory`: String value ("positive", "neutral", "negative") from OpenAI
- `questions`: JSON array of questions asked by the user
- `summary`: Brief summary of the conversation
- `processed`: Boolean flag indicating whether the session has been processed
- `sentimentCategory`: String value ("positive", "neutral", "negative") from OpenAI
- `questions`: JSON array of questions asked by the user
- `summary`: Brief summary of the conversation
## Configuration
@ -62,9 +62,9 @@ OPENAI_API_KEY=your_api_key_here
To run the application with schedulers enabled:
- Development: `npm run dev`
- Development (with schedulers disabled): `npm run dev:no-schedulers`
- Production: `npm run start`
- Development: `npm run dev`
- Development (with schedulers disabled): `npm run dev:no-schedulers`
- Production: `npm run start`
Note: These commands will start a custom Next.js server with the schedulers enabled. You'll need to have an OpenAI API key set in your `.env.local` file for the session processing to work.
@ -82,5 +82,5 @@ This will process all unprocessed sessions that have transcript content.
The processing logic can be customized by modifying:
- `lib/processingScheduler.ts`: Contains the OpenAI processing logic
- `scripts/process_sessions.ts`: Standalone script for manual processing
- `lib/processingScheduler.ts`: Contains the OpenAI processing logic
- `scripts/process_sessions.ts`: Standalone script for manual processing

View File

@ -5,9 +5,11 @@ This document outlines the fixes applied to resolve TypeScript compilation error
## Issues Resolved
### 1. Missing Type Imports
**Problem:** `lib/api/index.ts` was missing required type imports
**Error:** `Cannot find name 'APIHandler'`, `Cannot find name 'Permission'`
**Fix:** Added proper imports at the top of the file
```typescript
import type { APIContext, APIHandler, APIHandlerOptions } from "./handler";
import { createAPIHandler } from "./handler";
@ -15,9 +17,11 @@ import { Permission, createPermissionChecker } from "./authorization";
```
### 2. Zod API Breaking Change
**Problem:** Zod error property name changed from `errors` to `issues`
**Error:** `Property 'errors' does not exist on type 'ZodError'`
**Fix:** Updated all references to use `error.issues` instead of `error.errors`
```typescript
// Before
error.errors.map((e) => `${e.path.join(".")}: ${e.message}`)
@ -26,17 +30,21 @@ error.issues.map((e) => `${e.path.join(".")}: ${e.message}`)
```
### 3. Missing LRU Cache Dependency
**Problem:** `lru-cache` package was missing from dependencies
**Error:** `Cannot find module 'lru-cache'`
**Fix:** Installed the missing dependency
```bash
pnpm add lru-cache
```
### 4. LRU Cache Generic Type Constraints
**Problem:** TypeScript generic constraints not satisfied
**Error:** `Type 'K' does not satisfy the constraint '{}'`
**Fix:** Added proper generic type constraints
```typescript
// Before
<K = string, V = any>
@ -45,9 +53,11 @@ pnpm add lru-cache
```
### 5. Map Iteration ES5 Compatibility
**Problem:** Map iteration requires downlevel iteration flag
**Error:** `can only be iterated through when using the '--downlevelIteration' flag`
**Fix:** Used `Array.from()` pattern for compatibility
```typescript
// Before
for (const [key, value] of map) { ... }
@ -56,9 +66,11 @@ for (const [key, value] of Array.from(map.entries())) { ... }
```
### 6. Redis Configuration Issues
**Problem:** Invalid Redis socket options
**Error:** Redis connection failed with unsupported options
**Fix:** Simplified Redis configuration to only include supported options
```typescript
this.client = createClient({
url: env.REDIS_URL,
@ -69,9 +81,11 @@ this.client = createClient({
```
### 7. Prisma Relationship Naming Mismatches
**Problem:** Code referenced non-existent Prisma relationships
**Error:** `securityAuditLogs` and `sessionImport` don't exist
**Fix:** Used correct relationship names
```typescript
// Before
user.securityAuditLogs
@ -82,17 +96,21 @@ session.import
```
### 8. Missing Schema Fields
**Problem:** Code referenced fields that don't exist in the database schema
**Error:** `Property 'userId' does not exist on type`
**Fix:** Applied type casting where schema fields were missing
```typescript
userId: (session as any).userId || null
```
### 9. Deprecated Package Dependencies
**Problem:** `critters` package is deprecated and caused build failures
**Error:** `Cannot find module 'critters'`
**Fix:** Disabled CSS optimization feature that required critters
```javascript
experimental: {
optimizeCss: false, // Disabled due to critters dependency
@ -100,9 +118,11 @@ experimental: {
```
### 10. ESLint vs Biome Conflict
**Problem:** ESLint warnings treated as build errors
**Error:** Build failed due to linting warnings
**Fix:** Disabled ESLint during build since Biome is used for linting
```javascript
eslint: {
ignoreDuringBuilds: true,
@ -112,6 +132,7 @@ eslint: {
## Schema Enhancements
### Enhanced User Management
Added comprehensive user management fields to the User model:
```prisma
@ -137,38 +158,45 @@ model User {
```
### Updated Repository Methods
Enhanced UserRepository with new methods:
- `updateLastLogin()` - Tracks user login times
- `incrementFailedLoginAttempts()` - Security feature for account locking
- `verifyEmail()` - Email verification management
- `deactivateUser()` - Account management
- `unlockUser()` - Security administration
- `updatePreferences()` - User settings management
- `findInactiveUsers()` - Now uses `lastLoginAt` instead of `createdAt`
- `updateLastLogin()` - Tracks user login times
- `incrementFailedLoginAttempts()` - Security feature for account locking
- `verifyEmail()` - Email verification management
- `deactivateUser()` - Account management
- `unlockUser()` - Security administration
- `updatePreferences()` - User settings management
- `findInactiveUsers()` - Now uses `lastLoginAt` instead of `createdAt`
## Prevention Measures
### 1. Regular Dependency Updates
- Monitor for breaking changes in dependencies like Zod
- Use `pnpm outdated` to check for deprecated packages
- Test builds after dependency updates
- Monitor for breaking changes in dependencies like Zod
- Use `pnpm outdated` to check for deprecated packages
- Test builds after dependency updates
### 2. TypeScript Strict Checking
- Enable strict TypeScript checking to catch type errors early
- Use proper type imports and exports
- Avoid `any` types where possible
- Enable strict TypeScript checking to catch type errors early
- Use proper type imports and exports
- Avoid `any` types where possible
### 3. Build Pipeline Validation
- Run `pnpm build` before committing
- Include type checking in CI/CD pipeline
- Separate linting from build process
- Run `pnpm build` before committing
- Include type checking in CI/CD pipeline
- Separate linting from build process
### 4. Schema Management
- Regenerate Prisma client after schema changes: `pnpm prisma:generate`
- Validate schema changes with database migrations
- Use proper TypeScript types for database operations
- Regenerate Prisma client after schema changes: `pnpm prisma:generate`
- Validate schema changes with database migrations
- Use proper TypeScript types for database operations
### 5. Development Workflow
```bash
# Recommended development workflow
pnpm prisma:generate # After schema changes
@ -206,4 +234,4 @@ pnpm install
---
*Last updated: 2025-07-12*
*Build Status: ✅ Success (47/47 pages generated)*
*Build Status: ✅ Success (47/47 pages generated)*