mirror of
https://github.com/kjanat/livedash-node.git
synced 2026-01-16 12:32:10 +01:00
feat: complete development environment setup and code quality improvements
- Set up pre-commit hooks with husky and lint-staged for automated code quality - Improved TypeScript type safety by replacing 'any' types with proper generics - Fixed markdown linting violations (MD030 spacing) across all documentation - Fixed compound adjective hyphenation in technical documentation - Fixed invalid JSON union syntax in API documentation examples - Automated code formatting and linting on commit - Enhanced error handling with better type constraints - Configured biome and markdownlint for consistent code style - All changes verified with successful production build
This commit is contained in:
@ -6,10 +6,10 @@ This document describes the comprehensive CSRF (Cross-Site Request Forgery) prot
|
||||
|
||||
CSRF protection has been implemented to prevent cross-site request forgery attacks on state-changing operations. The implementation follows industry best practices and provides protection at multiple layers:
|
||||
|
||||
- **Middleware Level**: Automatic CSRF validation for protected endpoints
|
||||
- **tRPC Level**: CSRF protection for all state-changing tRPC procedures
|
||||
- **Client Level**: Automatic token management and inclusion in requests
|
||||
- **Component Level**: React components and hooks for easy integration
|
||||
- **Middleware Level**: Automatic CSRF validation for protected endpoints
|
||||
- **tRPC Level**: CSRF protection for all state-changing tRPC procedures
|
||||
- **Client Level**: Automatic token management and inclusion in requests
|
||||
- **Component Level**: React components and hooks for easy integration
|
||||
|
||||
## Implementation Components
|
||||
|
||||
@ -17,52 +17,58 @@ CSRF protection has been implemented to prevent cross-site request forgery attac
|
||||
|
||||
The core CSRF functionality includes:
|
||||
|
||||
- **Token Generation**: Cryptographically secure token generation using the `csrf` library
|
||||
- **Token Verification**: Server-side token validation
|
||||
- **Request Parsing**: Support for tokens in headers, JSON bodies, and form data
|
||||
- **Client Utilities**: Browser-side token management and request enhancement
|
||||
- **Token Generation**: Cryptographically secure token generation using the `csrf` library
|
||||
- **Token Verification**: Server-side token validation
|
||||
- **Request Parsing**: Support for tokens in headers, JSON bodies, and form data
|
||||
- **Client Utilities**: Browser-side token management and request enhancement
|
||||
|
||||
**Key Functions:**
|
||||
- `generateCSRFToken()` - Creates new CSRF tokens
|
||||
- `verifyCSRFToken()` - Validates tokens server-side
|
||||
- `CSRFProtection.validateRequest()` - Request validation middleware
|
||||
- `CSRFClient.*` - Client-side utilities
|
||||
|
||||
- `generateCSRFToken()` - Creates new CSRF tokens
|
||||
- `verifyCSRFToken()` - Validates tokens server-side
|
||||
- `CSRFProtection.validateRequest()` - Request validation middleware
|
||||
- `CSRFClient.*` - Client-side utilities
|
||||
|
||||
### 2. Middleware Protection (`middleware/csrfProtection.ts`)
|
||||
|
||||
Provides automatic CSRF protection for API endpoints:
|
||||
|
||||
**Protected Endpoints:**
|
||||
- `/api/auth/*` - Authentication endpoints
|
||||
- `/api/register` - User registration
|
||||
- `/api/forgot-password` - Password reset requests
|
||||
- `/api/reset-password` - Password reset completion
|
||||
- `/api/dashboard/*` - Dashboard API endpoints
|
||||
- `/api/platform/*` - Platform admin endpoints
|
||||
- `/api/trpc/*` - All tRPC endpoints
|
||||
|
||||
- `/api/auth/*` - Authentication endpoints
|
||||
- `/api/register` - User registration
|
||||
- `/api/forgot-password` - Password reset requests
|
||||
- `/api/reset-password` - Password reset completion
|
||||
- `/api/dashboard/*` - Dashboard API endpoints
|
||||
- `/api/platform/*` - Platform admin endpoints
|
||||
- `/api/trpc/*` - All tRPC endpoints
|
||||
|
||||
**Protected Methods:**
|
||||
- `POST` - Create operations
|
||||
- `PUT` - Update operations
|
||||
- `DELETE` - Delete operations
|
||||
- `PATCH` - Partial update operations
|
||||
|
||||
- `POST` - Create operations
|
||||
- `PUT` - Update operations
|
||||
- `DELETE` - Delete operations
|
||||
- `PATCH` - Partial update operations
|
||||
|
||||
**Safe Methods (Not Protected):**
|
||||
- `GET` - Read operations
|
||||
- `HEAD` - Metadata requests
|
||||
- `OPTIONS` - CORS preflight requests
|
||||
|
||||
- `GET` - Read operations
|
||||
- `HEAD` - Metadata requests
|
||||
- `OPTIONS` - CORS preflight requests
|
||||
|
||||
### 3. tRPC Integration (`lib/trpc.ts`)
|
||||
|
||||
CSRF protection integrated into tRPC procedures:
|
||||
|
||||
**New Procedure Types:**
|
||||
- `csrfProtectedProcedure` - Basic CSRF protection
|
||||
- `csrfProtectedAuthProcedure` - CSRF + authentication protection
|
||||
- `csrfProtectedCompanyProcedure` - CSRF + company access protection
|
||||
- `csrfProtectedAdminProcedure` - CSRF + admin access protection
|
||||
|
||||
- `csrfProtectedProcedure` - Basic CSRF protection
|
||||
- `csrfProtectedAuthProcedure` - CSRF + authentication protection
|
||||
- `csrfProtectedCompanyProcedure` - CSRF + company access protection
|
||||
- `csrfProtectedAdminProcedure` - CSRF + admin access protection
|
||||
|
||||
**Updated Router Example:**
|
||||
|
||||
```typescript
|
||||
// Before
|
||||
register: rateLimitedProcedure
|
||||
@ -78,30 +84,35 @@ register: csrfProtectedProcedure
|
||||
### 4. Client-Side Integration
|
||||
|
||||
#### tRPC Client (`lib/trpc-client.ts`)
|
||||
- Automatic CSRF token inclusion in tRPC requests
|
||||
- Token extracted from cookies and added to request headers
|
||||
|
||||
- Automatic CSRF token inclusion in tRPC requests
|
||||
- Token extracted from cookies and added to request headers
|
||||
|
||||
#### React Hooks (`lib/hooks/useCSRF.ts`)
|
||||
- `useCSRF()` - Basic token management
|
||||
- `useCSRFFetch()` - Enhanced fetch with automatic CSRF tokens
|
||||
- `useCSRFForm()` - Form submission with CSRF protection
|
||||
|
||||
- `useCSRF()` - Basic token management
|
||||
- `useCSRFFetch()` - Enhanced fetch with automatic CSRF tokens
|
||||
- `useCSRFForm()` - Form submission with CSRF protection
|
||||
|
||||
#### Provider Component (`components/providers/CSRFProvider.tsx`)
|
||||
- Application-wide CSRF token management
|
||||
- Automatic token fetching and refresh
|
||||
- Context-based token sharing
|
||||
|
||||
- Application-wide CSRF token management
|
||||
- Automatic token fetching and refresh
|
||||
- Context-based token sharing
|
||||
|
||||
#### Protected Form Component (`components/forms/CSRFProtectedForm.tsx`)
|
||||
- Ready-to-use form component with CSRF protection
|
||||
- Automatic token inclusion in form submissions
|
||||
- Graceful fallback for non-JavaScript environments
|
||||
|
||||
- Ready-to-use form component with CSRF protection
|
||||
- Automatic token inclusion in form submissions
|
||||
- Graceful fallback for non-JavaScript environments
|
||||
|
||||
### 5. API Endpoint (`app/api/csrf-token/route.ts`)
|
||||
|
||||
Provides CSRF tokens to client applications:
|
||||
- `GET /api/csrf-token` - Returns new CSRF token
|
||||
- Sets HTTP-only cookie for automatic inclusion
|
||||
- Used by client-side hooks and components
|
||||
|
||||
- `GET /api/csrf-token` - Returns new CSRF token
|
||||
- Sets HTTP-only cookie for automatic inclusion
|
||||
- Used by client-side hooks and components
|
||||
|
||||
## Configuration
|
||||
|
||||
@ -206,33 +217,38 @@ const dataWithToken = CSRFClient.addTokenToObject({ data: 'example' });
|
||||
## Security Features
|
||||
|
||||
### 1. Token Properties
|
||||
- **Cryptographically Secure**: Uses the `csrf` library with secure random generation
|
||||
- **Short-Lived**: 24-hour expiration by default
|
||||
- **HTTP-Only Cookies**: Prevents XSS-based token theft
|
||||
- **SameSite Protection**: Reduces CSRF attack surface
|
||||
|
||||
- **Cryptographically Secure**: Uses the `csrf` library with secure random generation
|
||||
- **Short-Lived**: 24-hour expiration by default
|
||||
- **HTTP-Only Cookies**: Prevents XSS-based token theft
|
||||
- **SameSite Protection**: Reduces CSRF attack surface
|
||||
|
||||
### 2. Validation Process
|
||||
1. Extract token from request (header, form data, or JSON body)
|
||||
2. Retrieve stored token from HTTP-only cookie
|
||||
3. Verify tokens match
|
||||
4. Validate token cryptographic integrity
|
||||
5. Allow or reject request based on validation
|
||||
|
||||
1. Extract token from request (header, form data, or JSON body)
|
||||
2. Retrieve stored token from HTTP-only cookie
|
||||
3. Verify tokens match
|
||||
4. Validate token cryptographic integrity
|
||||
5. Allow or reject request based on validation
|
||||
|
||||
### 3. Error Handling
|
||||
- **Graceful Degradation**: Form fallbacks for JavaScript-disabled browsers
|
||||
- **Clear Error Messages**: Specific error codes for debugging
|
||||
- **Rate Limiting Integration**: Works with existing auth rate limiting
|
||||
- **Logging**: Comprehensive logging for security monitoring
|
||||
|
||||
- **Graceful Degradation**: Form fallbacks for JavaScript-disabled browsers
|
||||
- **Clear Error Messages**: Specific error codes for debugging
|
||||
- **Rate Limiting Integration**: Works with existing auth rate limiting
|
||||
- **Logging**: Comprehensive logging for security monitoring
|
||||
|
||||
## Testing
|
||||
|
||||
### Test Coverage
|
||||
- **Unit Tests**: Token generation, validation, and client utilities
|
||||
- **Integration Tests**: Middleware behavior and endpoint protection
|
||||
- **Component Tests**: React hooks and form components
|
||||
- **End-to-End**: Full request/response cycle testing
|
||||
|
||||
- **Unit Tests**: Token generation, validation, and client utilities
|
||||
- **Integration Tests**: Middleware behavior and endpoint protection
|
||||
- **Component Tests**: React hooks and form components
|
||||
- **End-to-End**: Full request/response cycle testing
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all CSRF tests
|
||||
pnpm test:vitest tests/unit/csrf*.test.ts tests/integration/csrf*.test.ts
|
||||
@ -246,32 +262,36 @@ pnpm test:vitest tests/unit/csrf-hooks.test.tsx
|
||||
## Monitoring and Debugging
|
||||
|
||||
### CSRF Validation Logs
|
||||
|
||||
Failed CSRF validations are logged with details:
|
||||
|
||||
```
|
||||
CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from request
|
||||
```
|
||||
|
||||
### Common Issues and Solutions
|
||||
|
||||
1. **Token Missing from Request**
|
||||
- Ensure CSRFProvider is wrapping your app
|
||||
- Check that hooks are being used correctly
|
||||
- Verify network requests include credentials
|
||||
1. **Token Missing from Request**
|
||||
- Ensure CSRFProvider is wrapping your app
|
||||
- Check that hooks are being used correctly
|
||||
- Verify network requests include credentials
|
||||
|
||||
2. **Token Mismatch**
|
||||
- Clear browser cookies and refresh
|
||||
- Check for multiple token sources conflicting
|
||||
- Verify server and client time synchronization
|
||||
2. **Token Mismatch**
|
||||
- Clear browser cookies and refresh
|
||||
- Check for multiple token sources conflicting
|
||||
- Verify server and client time synchronization
|
||||
|
||||
3. **Integration Issues**
|
||||
- Ensure middleware is properly configured
|
||||
- Check tRPC client configuration
|
||||
- Verify protected procedures are using correct types
|
||||
3. **Integration Issues**
|
||||
- Ensure middleware is properly configured
|
||||
- Check tRPC client configuration
|
||||
- Verify protected procedures are using correct types
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### For Existing Endpoints
|
||||
1. Update tRPC procedures to use CSRF-protected variants:
|
||||
|
||||
1. Update tRPC procedures to use CSRF-protected variants:
|
||||
|
||||
```typescript
|
||||
// Old
|
||||
someAction: protectedProcedure.mutation(...)
|
||||
@ -280,7 +300,8 @@ CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from
|
||||
someAction: csrfProtectedAuthProcedure.mutation(...)
|
||||
```
|
||||
|
||||
2. Update client components to use CSRF hooks:
|
||||
2. Update client components to use CSRF hooks:
|
||||
|
||||
```tsx
|
||||
// Old
|
||||
const { data, mutate } = trpc.user.update.useMutation();
|
||||
@ -289,7 +310,8 @@ CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from
|
||||
const { data, mutate } = trpc.user.update.useMutation();
|
||||
```
|
||||
|
||||
3. Update manual API calls to include CSRF tokens:
|
||||
3. Update manual API calls to include CSRF tokens:
|
||||
|
||||
```typescript
|
||||
// Old
|
||||
fetch('/api/endpoint', { method: 'POST', ... });
|
||||
@ -301,21 +323,21 @@ CSRF validation failed for POST /api/dashboard/sessions: CSRF token missing from
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Minimal Overhead**: Token validation adds ~1ms per request
|
||||
- **Efficient Caching**: Tokens cached in memory and cookies
|
||||
- **Selective Protection**: Only state-changing operations protected
|
||||
- **Optimized Parsing**: Smart content-type detection for token extraction
|
||||
- **Minimal Overhead**: Token validation adds ~1ms per request
|
||||
- **Efficient Caching**: Tokens cached in memory and cookies
|
||||
- **Selective Protection**: Only state-changing operations protected
|
||||
- **Optimized Parsing**: Smart content-type detection for token extraction
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
1. **Always use HTTPS in production** - CSRF tokens should never be transmitted over HTTP
|
||||
2. **Monitor CSRF failures** - Implement alerting for unusual CSRF failure patterns
|
||||
3. **Regular secret rotation** - Consider rotating CSRF secrets periodically
|
||||
4. **Validate referrer headers** - Additional protection layer (not implemented but recommended)
|
||||
5. **Content Security Policy** - Use CSP headers to prevent XSS attacks that could steal tokens
|
||||
1. **Always use HTTPS in production** - CSRF tokens should never be transmitted over HTTP
|
||||
2. **Monitor CSRF failures** - Implement alerting for unusual CSRF failure patterns
|
||||
3. **Regular secret rotation** - Consider rotating CSRF secrets periodically
|
||||
4. **Validate referrer headers** - Additional protection layer (not implemented but recommended)
|
||||
5. **Content Security Policy** - Use CSP headers to prevent XSS attacks that could steal tokens
|
||||
|
||||
## Conclusion
|
||||
|
||||
The CSRF protection implementation provides comprehensive defense against cross-site request forgery attacks while maintaining ease of use for developers. The multi-layer approach ensures protection at the middleware, application, and component levels, with automatic token management reducing the risk of developer error.
|
||||
|
||||
For questions or issues related to CSRF protection, refer to the test files for examples and the security documentation for additional context.
|
||||
For questions or issues related to CSRF protection, refer to the test files for examples and the security documentation for additional context.
|
||||
|
||||
@ -8,10 +8,10 @@ The Admin Audit Logs API provides secure access to security audit trails for adm
|
||||
|
||||
## Authentication & Authorization
|
||||
|
||||
- **Authentication**: NextAuth.js session required
|
||||
- **Authorization**: ADMIN role required for all endpoints
|
||||
- **Rate Limiting**: Integrated with existing auth rate limiting system
|
||||
- **Audit Trail**: All API access is logged for security monitoring
|
||||
- **Authentication**: NextAuth.js session required
|
||||
- **Authorization**: ADMIN role required for all endpoints
|
||||
- **Rate-Limiting**: Integrated with existing authentication rate-limiting system
|
||||
- **Audit Trail**: All API access is logged for security monitoring
|
||||
|
||||
## API Endpoints
|
||||
|
||||
@ -128,12 +128,14 @@ POST /api/admin/audit-logs/retention
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "cleanup" | "configure" | "status",
|
||||
"action": "cleanup",
|
||||
"retentionDays": 90,
|
||||
"dryRun": true
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: `action` field accepts one of: `"cleanup"`, `"configure"`, or `"status"`
|
||||
|
||||
#### Parameters
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
@ -145,6 +147,7 @@ POST /api/admin/audit-logs/retention
|
||||
#### Example Requests
|
||||
|
||||
**Check retention status:**
|
||||
|
||||
```javascript
|
||||
const response = await fetch('/api/admin/audit-logs/retention', {
|
||||
method: 'POST',
|
||||
@ -154,6 +157,7 @@ const response = await fetch('/api/admin/audit-logs/retention', {
|
||||
```
|
||||
|
||||
**Configure retention policy:**
|
||||
|
||||
```javascript
|
||||
const response = await fetch('/api/admin/audit-logs/retention', {
|
||||
method: 'POST',
|
||||
@ -166,6 +170,7 @@ const response = await fetch('/api/admin/audit-logs/retention', {
|
||||
```
|
||||
|
||||
**Cleanup old logs (dry run):**
|
||||
|
||||
```javascript
|
||||
const response = await fetch('/api/admin/audit-logs/retention', {
|
||||
method: 'POST',
|
||||
@ -180,19 +185,22 @@ const response = await fetch('/api/admin/audit-logs/retention', {
|
||||
## Security Features
|
||||
|
||||
### Access Control
|
||||
- **Role-based Access**: Only ADMIN users can access audit logs
|
||||
- **Company Isolation**: Users only see logs for their company
|
||||
- **Session Validation**: Active NextAuth session required
|
||||
|
||||
- **Role-based Access**: Only ADMIN users can access audit logs
|
||||
- **Company Isolation**: Users only see logs for their company
|
||||
- **Session Validation**: Active NextAuth session required
|
||||
|
||||
### Audit Trail
|
||||
- **Access Logging**: All audit log access is recorded
|
||||
- **Metadata Tracking**: Request parameters and results are logged
|
||||
- **IP Tracking**: Client IP addresses are recorded for all requests
|
||||
|
||||
- **Access Logging**: All audit log access is recorded
|
||||
- **Metadata Tracking**: Request parameters and results are logged
|
||||
- **IP Tracking**: Client IP addresses are recorded for all requests
|
||||
|
||||
### Rate Limiting
|
||||
- **Integrated Protection**: Uses existing authentication rate limiting
|
||||
- **Abuse Prevention**: Protects against excessive API usage
|
||||
- **Error Tracking**: Failed attempts are monitored
|
||||
|
||||
- **Integrated Protection**: Uses existing authentication rate-limiting
|
||||
- **Abuse Prevention**: Protects against excessive API usage
|
||||
- **Error Tracking**: Failed attempts are monitored
|
||||
|
||||
## Event Types
|
||||
|
||||
@ -294,19 +302,22 @@ async function getUserActivity(userId, days = 7) {
|
||||
## Performance Considerations
|
||||
|
||||
### Database Optimization
|
||||
- **Indexed Queries**: All filter columns are properly indexed
|
||||
- **Pagination**: Efficient offset-based pagination with limits
|
||||
- **Time Range Filtering**: Optimized for date range queries
|
||||
|
||||
- **Indexed Queries**: All filter columns are properly indexed
|
||||
- **Pagination**: Efficient offset-based pagination with limits
|
||||
- **Time Range Filtering**: Optimized for date range queries
|
||||
|
||||
### Memory Usage
|
||||
- **Limited Results**: Maximum 100 records per request
|
||||
- **Streaming**: Large exports use streaming for memory efficiency
|
||||
- **Connection Pooling**: Database connections are pooled
|
||||
|
||||
- **Limited Results**: Maximum 100 records per request
|
||||
- **Streaming**: Large exports use streaming for memory efficiency
|
||||
- **Connection Pooling**: Database connections are pooled
|
||||
|
||||
### Caching Considerations
|
||||
- **No Caching**: Audit logs are never cached for security reasons
|
||||
- **Fresh Data**: All queries hit the database for real-time results
|
||||
- **Read Replicas**: Consider using read replicas for heavy reporting
|
||||
|
||||
- **No Caching**: Audit logs are never cached for security reasons
|
||||
- **Fresh Data**: All queries hit the database for real-time results
|
||||
- **Read Replicas**: Consider using read replicas for heavy reporting
|
||||
|
||||
## Error Handling
|
||||
|
||||
@ -335,7 +346,7 @@ try {
|
||||
}
|
||||
```
|
||||
|
||||
### Rate Limiting Handling
|
||||
### Rate-Limiting Handling
|
||||
|
||||
```javascript
|
||||
async function fetchWithRetry(url, options = {}) {
|
||||
@ -354,40 +365,45 @@ async function fetchWithRetry(url, options = {}) {
|
||||
## Monitoring and Alerting
|
||||
|
||||
### Key Metrics to Monitor
|
||||
- **Request Volume**: Track API usage patterns
|
||||
- **Error Rates**: Monitor authentication and authorization failures
|
||||
- **Query Performance**: Track slow queries and optimize
|
||||
- **Data Growth**: Monitor audit log size and plan retention
|
||||
|
||||
- **Request Volume**: Track API usage patterns
|
||||
- **Error Rates**: Monitor authentication and authorization failures
|
||||
- **Query Performance**: Track slow queries and optimize
|
||||
- **Data Growth**: Monitor audit log size and plan retention
|
||||
|
||||
### Alert Conditions
|
||||
- **High Error Rates**: >5% of requests failing
|
||||
- **Unusual Access Patterns**: Off-hours access, high volume
|
||||
- **Performance Degradation**: Query times >2 seconds
|
||||
- **Security Events**: Multiple failed admin access attempts
|
||||
|
||||
- **High Error Rates**: >5% of requests failing
|
||||
- **Unusual Access Patterns**: Off-hours access, high-volume usage
|
||||
- **Performance Degradation**: Query times >2 seconds
|
||||
- **Security Events**: Multiple failed admin access attempts
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Security
|
||||
- Always validate user permissions before displaying UI
|
||||
- Log all administrative access to audit logs
|
||||
- Use HTTPS in production environments
|
||||
- Implement proper error handling to avoid information leakage
|
||||
|
||||
- Always validate user permissions before displaying UI
|
||||
- Log all administrative access to audit logs
|
||||
- Use HTTPS in production environments
|
||||
- Implement proper error handling to avoid information leakage
|
||||
|
||||
### Performance
|
||||
- Use appropriate page sizes (25-50 records typical)
|
||||
- Implement client-side pagination for better UX
|
||||
- Cache results only in memory, never persist
|
||||
- Use date range filters to limit query scope
|
||||
|
||||
- Use appropriate page sizes (25-50 records typical)
|
||||
- Implement client-side pagination for better UX
|
||||
- Cache results only in memory, never persist
|
||||
- Use date range filters to limit query scope
|
||||
|
||||
### User Experience
|
||||
- Provide clear filtering options in the UI
|
||||
- Show loading states for long-running queries
|
||||
- Implement export functionality for reports
|
||||
- Provide search and sort capabilities
|
||||
|
||||
- Provide clear filtering options in the UI
|
||||
- Show loading states for long-running queries
|
||||
- Implement export functionality for reports
|
||||
- Provide search and sort capabilities
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Security Audit Logging](./security-audit-logging.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [CSRF Protection](./CSRF_PROTECTION.md)
|
||||
- [Authentication System](../lib/auth.ts)
|
||||
- [Security Audit Logging](./security-audit-logging.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [CSRF Protection](./CSRF_PROTECTION.md)
|
||||
- [Authentication System](../lib/auth.ts)
|
||||
|
||||
@ -28,6 +28,7 @@ X-CSRF-Token: <csrf-token>
|
||||
```
|
||||
|
||||
Get CSRF token:
|
||||
|
||||
```http
|
||||
GET /api/csrf-token
|
||||
```
|
||||
@ -35,55 +36,64 @@ GET /api/csrf-token
|
||||
## API Endpoints Overview
|
||||
|
||||
### Public Endpoints
|
||||
- `POST /api/csp-report` - CSP violation reporting (no auth required)
|
||||
- `OPTIONS /api/csp-report` - CORS preflight
|
||||
|
||||
- `POST /api/csp-report` - CSP violation reporting (no auth required)
|
||||
- `OPTIONS /api/csp-report` - CORS preflight
|
||||
|
||||
### Authentication Endpoints
|
||||
- `POST /api/auth/[...nextauth]` - NextAuth.js authentication
|
||||
- `GET /api/csrf-token` - Get CSRF token
|
||||
- `POST /api/register` - User registration
|
||||
- `POST /api/forgot-password` - Password reset request
|
||||
- `POST /api/reset-password` - Password reset completion
|
||||
|
||||
- `POST /api/auth/[...nextauth]` - NextAuth.js authentication
|
||||
- `GET /api/csrf-token` - Get CSRF token
|
||||
- `POST /api/register` - User registration
|
||||
- `POST /api/forgot-password` - Password reset request
|
||||
- `POST /api/reset-password` - Password reset completion
|
||||
|
||||
### Admin Endpoints (ADMIN role required)
|
||||
- `GET /api/admin/audit-logs` - Retrieve audit logs
|
||||
- `POST /api/admin/audit-logs/retention` - Manage audit log retention
|
||||
- `GET /api/admin/batch-monitoring` - Batch processing monitoring
|
||||
- `POST /api/admin/batch-monitoring/{id}/retry` - Retry failed batch job
|
||||
|
||||
- `GET /api/admin/audit-logs` - Retrieve audit logs
|
||||
- `POST /api/admin/audit-logs/retention` - Manage audit log retention
|
||||
- `GET /api/admin/batch-monitoring` - Batch processing monitoring
|
||||
- `POST /api/admin/batch-monitoring/{id}/retry` - Retry failed batch job
|
||||
|
||||
### Platform Admin Endpoints (Platform admin only)
|
||||
- `GET /api/admin/security-monitoring` - Security monitoring metrics
|
||||
- `POST /api/admin/security-monitoring` - Update security configuration
|
||||
- `GET /api/admin/security-monitoring/alerts` - Alert management
|
||||
- `POST /api/admin/security-monitoring/alerts` - Acknowledge alerts
|
||||
- `GET /api/admin/security-monitoring/export` - Export security data
|
||||
- `POST /api/admin/security-monitoring/threat-analysis` - Threat analysis
|
||||
|
||||
- `GET /api/admin/security-monitoring` - Security monitoring metrics
|
||||
- `POST /api/admin/security-monitoring` - Update security configuration
|
||||
- `GET /api/admin/security-monitoring/alerts` - Alert management
|
||||
- `POST /api/admin/security-monitoring/alerts` - Acknowledge alerts
|
||||
- `GET /api/admin/security-monitoring/export` - Export security data
|
||||
- `POST /api/admin/security-monitoring/threat-analysis` - Threat analysis
|
||||
|
||||
### Security Monitoring Endpoints
|
||||
- `GET /api/csp-metrics` - CSP violation metrics
|
||||
- `POST /api/csp-report` - CSP violation reporting
|
||||
|
||||
- `GET /api/csp-metrics` - CSP violation metrics
|
||||
- `POST /api/csp-report` - CSP violation reporting
|
||||
|
||||
### Dashboard Endpoints
|
||||
- `GET /api/dashboard/sessions` - Session data
|
||||
- `GET /api/dashboard/session/{id}` - Individual session details
|
||||
- `GET /api/dashboard/metrics` - Dashboard metrics
|
||||
- `GET /api/dashboard/config` - Dashboard configuration
|
||||
|
||||
- `GET /api/dashboard/sessions` - Session data
|
||||
- `GET /api/dashboard/session/{id}` - Individual session details
|
||||
- `GET /api/dashboard/metrics` - Dashboard metrics
|
||||
- `GET /api/dashboard/config` - Dashboard configuration
|
||||
|
||||
### Platform Management
|
||||
- `GET /api/platform/companies` - Company management
|
||||
- `POST /api/platform/companies` - Create company
|
||||
- `GET /api/platform/companies/{id}` - Company details
|
||||
- `GET /api/platform/companies/{id}/users` - Company users
|
||||
- `POST /api/platform/companies/{id}/users` - Add company user
|
||||
|
||||
- `GET /api/platform/companies` - Company management
|
||||
- `POST /api/platform/companies` - Create company
|
||||
- `GET /api/platform/companies/{id}` - Company details
|
||||
- `GET /api/platform/companies/{id}/users` - Company users
|
||||
- `POST /api/platform/companies/{id}/users` - Add company user
|
||||
|
||||
### tRPC Endpoints
|
||||
- `POST /api/trpc/[trpc]` - tRPC procedure calls
|
||||
|
||||
- `POST /api/trpc/[trpc]` - tRPC procedure calls
|
||||
|
||||
## Detailed Endpoint Documentation
|
||||
|
||||
### Admin Audit Logs
|
||||
|
||||
#### Get Audit Logs
|
||||
|
||||
```http
|
||||
GET /api/admin/audit-logs
|
||||
```
|
||||
@ -91,16 +101,18 @@ GET /api/admin/audit-logs
|
||||
**Authorization**: ADMIN role required
|
||||
|
||||
**Query Parameters**:
|
||||
- `page` (number, optional): Page number (default: 1)
|
||||
- `limit` (number, optional): Records per page, max 100 (default: 50)
|
||||
- `eventType` (string, optional): Filter by event type
|
||||
- `outcome` (string, optional): Filter by outcome (SUCCESS, FAILURE, BLOCKED, etc.)
|
||||
- `severity` (string, optional): Filter by severity (LOW, MEDIUM, HIGH, CRITICAL)
|
||||
- `userId` (string, optional): Filter by user ID
|
||||
- `startDate` (string, optional): Start date (ISO 8601)
|
||||
- `endDate` (string, optional): End date (ISO 8601)
|
||||
|
||||
- `page` (number, optional): Page number (default: 1)
|
||||
- `limit` (number, optional): Records per page, max 100 (default: 50)
|
||||
- `eventType` (string, optional): Filter by event type
|
||||
- `outcome` (string, optional): Filter by outcome (SUCCESS, FAILURE, BLOCKED, etc.)
|
||||
- `severity` (string, optional): Filter by severity (LOW, MEDIUM, HIGH, CRITICAL)
|
||||
- `userId` (string, optional): Filter by user ID
|
||||
- `startDate` (string, optional): Start date (ISO 8601)
|
||||
- `endDate` (string, optional): End date (ISO 8601)
|
||||
|
||||
**Response**:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@ -121,6 +133,7 @@ GET /api/admin/audit-logs
|
||||
**Rate Limit**: Inherits from auth rate limiting
|
||||
|
||||
#### Manage Audit Log Retention
|
||||
|
||||
```http
|
||||
POST /api/admin/audit-logs/retention
|
||||
```
|
||||
@ -128,6 +141,7 @@ POST /api/admin/audit-logs/retention
|
||||
**Authorization**: ADMIN role required
|
||||
|
||||
**Request Body**:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "cleanup" | "configure" | "status",
|
||||
@ -137,6 +151,7 @@ POST /api/admin/audit-logs/retention
|
||||
```
|
||||
|
||||
**Response**:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@ -152,6 +167,7 @@ POST /api/admin/audit-logs/retention
|
||||
### Security Monitoring
|
||||
|
||||
#### Get Security Metrics
|
||||
|
||||
```http
|
||||
GET /api/admin/security-monitoring
|
||||
```
|
||||
@ -159,12 +175,14 @@ GET /api/admin/security-monitoring
|
||||
**Authorization**: Platform admin required
|
||||
|
||||
**Query Parameters**:
|
||||
- `startDate` (string, optional): Start date (ISO 8601)
|
||||
- `endDate` (string, optional): End date (ISO 8601)
|
||||
- `companyId` (string, optional): Filter by company
|
||||
- `severity` (string, optional): Filter by severity
|
||||
|
||||
- `startDate` (string, optional): Start date (ISO 8601)
|
||||
- `endDate` (string, optional): End date (ISO 8601)
|
||||
- `companyId` (string, optional): Filter by company
|
||||
- `severity` (string, optional): Filter by severity
|
||||
|
||||
**Response**:
|
||||
|
||||
```json
|
||||
{
|
||||
"metrics": {
|
||||
@ -180,6 +198,7 @@ GET /api/admin/security-monitoring
|
||||
```
|
||||
|
||||
#### Update Security Configuration
|
||||
|
||||
```http
|
||||
POST /api/admin/security-monitoring
|
||||
```
|
||||
@ -187,6 +206,7 @@ POST /api/admin/security-monitoring
|
||||
**Authorization**: Platform admin required
|
||||
|
||||
**Request Body**:
|
||||
|
||||
```json
|
||||
{
|
||||
"thresholds": {
|
||||
@ -203,6 +223,7 @@ POST /api/admin/security-monitoring
|
||||
### CSP Monitoring
|
||||
|
||||
#### CSP Violation Reporting
|
||||
|
||||
```http
|
||||
POST /api/csp-report
|
||||
```
|
||||
@ -210,9 +231,11 @@ POST /api/csp-report
|
||||
**Authorization**: None (public endpoint)
|
||||
|
||||
**Headers**:
|
||||
- `Content-Type`: `application/csp-report` or `application/json`
|
||||
|
||||
- `Content-Type`: `application/csp-report` or `application/json`
|
||||
|
||||
**Request Body** (automatic from browser):
|
||||
|
||||
```json
|
||||
{
|
||||
"csp-report": {
|
||||
@ -230,6 +253,7 @@ POST /api/csp-report
|
||||
**Response**: `204 No Content`
|
||||
|
||||
#### Get CSP Metrics
|
||||
|
||||
```http
|
||||
GET /api/csp-metrics
|
||||
```
|
||||
@ -237,12 +261,14 @@ GET /api/csp-metrics
|
||||
**Authorization**: Admin role required
|
||||
|
||||
**Query Parameters**:
|
||||
- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d)
|
||||
- `format` (string, optional): Response format (json, csv)
|
||||
- `groupBy` (string, optional): Group by field (hour, directive, etc.)
|
||||
- `includeDetails` (boolean, optional): Include violation details
|
||||
|
||||
- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d)
|
||||
- `format` (string, optional): Response format (json, csv)
|
||||
- `groupBy` (string, optional): Group by field (hour, directive, etc.)
|
||||
- `includeDetails` (boolean, optional): Include violation details
|
||||
|
||||
**Response**:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@ -264,6 +290,7 @@ GET /api/csp-metrics
|
||||
### Batch Monitoring
|
||||
|
||||
#### Get Batch Monitoring Data
|
||||
|
||||
```http
|
||||
GET /api/admin/batch-monitoring
|
||||
```
|
||||
@ -271,14 +298,16 @@ GET /api/admin/batch-monitoring
|
||||
**Authorization**: ADMIN role required
|
||||
|
||||
**Query Parameters**:
|
||||
- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d)
|
||||
- `status` (string, optional): Filter by status (pending, completed, failed)
|
||||
- `jobType` (string, optional): Filter by job type
|
||||
- `includeDetails` (boolean, optional): Include detailed job information
|
||||
- `page` (number, optional): Page number
|
||||
- `limit` (number, optional): Records per page
|
||||
|
||||
- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d)
|
||||
- `status` (string, optional): Filter by status (pending, completed, failed)
|
||||
- `jobType` (string, optional): Filter by job type
|
||||
- `includeDetails` (boolean, optional): Include detailed job information
|
||||
- `page` (number, optional): Page number
|
||||
- `limit` (number, optional): Records per page
|
||||
|
||||
**Response**:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@ -297,6 +326,7 @@ GET /api/admin/batch-monitoring
|
||||
```
|
||||
|
||||
#### Retry Batch Job
|
||||
|
||||
```http
|
||||
POST /api/admin/batch-monitoring/{jobId}/retry
|
||||
```
|
||||
@ -304,6 +334,7 @@ POST /api/admin/batch-monitoring/{jobId}/retry
|
||||
**Authorization**: ADMIN role required
|
||||
|
||||
**Response**:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@ -318,6 +349,7 @@ POST /api/admin/batch-monitoring/{jobId}/retry
|
||||
### CSRF Token
|
||||
|
||||
#### Get CSRF Token
|
||||
|
||||
```http
|
||||
GET /api/csrf-token
|
||||
```
|
||||
@ -325,6 +357,7 @@ GET /api/csrf-token
|
||||
**Authorization**: None
|
||||
|
||||
**Response**:
|
||||
|
||||
```json
|
||||
{
|
||||
"csrfToken": "abc123..."
|
||||
@ -332,11 +365,13 @@ GET /api/csrf-token
|
||||
```
|
||||
|
||||
**Headers Set**:
|
||||
- `Set-Cookie`: HTTP-only CSRF token cookie
|
||||
|
||||
- `Set-Cookie`: HTTP-only CSRF token cookie
|
||||
|
||||
### Authentication
|
||||
|
||||
#### User Registration
|
||||
|
||||
```http
|
||||
POST /api/register
|
||||
```
|
||||
@ -344,9 +379,11 @@ POST /api/register
|
||||
**Authorization**: None
|
||||
|
||||
**Headers Required**:
|
||||
- `X-CSRF-Token`: CSRF token
|
||||
|
||||
- `X-CSRF-Token`: CSRF token
|
||||
|
||||
**Request Body**:
|
||||
|
||||
```json
|
||||
{
|
||||
"email": "user@example.com",
|
||||
@ -359,6 +396,7 @@ POST /api/register
|
||||
**Rate Limit**: 3 attempts per hour per IP
|
||||
|
||||
**Response**:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@ -368,6 +406,7 @@ POST /api/register
|
||||
```
|
||||
|
||||
#### Password Reset Request
|
||||
|
||||
```http
|
||||
POST /api/forgot-password
|
||||
```
|
||||
@ -375,9 +414,11 @@ POST /api/forgot-password
|
||||
**Authorization**: None
|
||||
|
||||
**Headers Required**:
|
||||
- `X-CSRF-Token`: CSRF token
|
||||
|
||||
- `X-CSRF-Token`: CSRF token
|
||||
|
||||
**Request Body**:
|
||||
|
||||
```json
|
||||
{
|
||||
"email": "user@example.com"
|
||||
@ -387,6 +428,7 @@ POST /api/forgot-password
|
||||
**Rate Limit**: 5 attempts per 15 minutes per IP
|
||||
|
||||
**Response**:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@ -395,6 +437,7 @@ POST /api/forgot-password
|
||||
```
|
||||
|
||||
#### Password Reset Completion
|
||||
|
||||
```http
|
||||
POST /api/reset-password
|
||||
```
|
||||
@ -402,9 +445,11 @@ POST /api/reset-password
|
||||
**Authorization**: None
|
||||
|
||||
**Headers Required**:
|
||||
- `X-CSRF-Token`: CSRF token
|
||||
|
||||
- `X-CSRF-Token`: CSRF token
|
||||
|
||||
**Request Body**:
|
||||
|
||||
```json
|
||||
{
|
||||
"token": "reset-token-123",
|
||||
@ -413,6 +458,7 @@ POST /api/reset-password
|
||||
```
|
||||
|
||||
**Response**:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@ -464,18 +510,21 @@ POST /api/reset-password
|
||||
## Rate Limiting
|
||||
|
||||
### Authentication Endpoints
|
||||
- **Login**: 5 attempts per 15 minutes per IP
|
||||
- **Registration**: 3 attempts per hour per IP
|
||||
- **Password Reset**: 5 attempts per 15 minutes per IP
|
||||
|
||||
- **Login**: 5 attempts per 15 minutes per IP
|
||||
- **Registration**: 3 attempts per hour per IP
|
||||
- **Password Reset**: 5 attempts per 15 minutes per IP
|
||||
|
||||
### Security Endpoints
|
||||
- **CSP Reports**: 10 reports per minute per IP
|
||||
- **Admin Endpoints**: 60 requests per minute per user
|
||||
- **Security Monitoring**: 30 requests per minute per user
|
||||
|
||||
- **CSP Reports**: 10 reports per minute per IP
|
||||
- **Admin Endpoints**: 60 requests per minute per user
|
||||
- **Security Monitoring**: 30 requests per minute per user
|
||||
|
||||
### General API
|
||||
- **Dashboard Endpoints**: 120 requests per minute per user
|
||||
- **Platform Management**: 60 requests per minute per user
|
||||
|
||||
- **Dashboard Endpoints**: 120 requests per minute per user
|
||||
- **Platform Management**: 60 requests per minute per user
|
||||
|
||||
## Security Headers
|
||||
|
||||
@ -492,14 +541,17 @@ Content-Security-Policy: [CSP directives]
|
||||
## CORS Configuration
|
||||
|
||||
### Allowed Origins
|
||||
- Development: `http://localhost:3000`
|
||||
- Production: `https://your-domain.com`
|
||||
|
||||
- Development: `http://localhost:3000`
|
||||
- Production: `https://your-domain.com`
|
||||
|
||||
### Allowed Methods
|
||||
- `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS`
|
||||
|
||||
- `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS`
|
||||
|
||||
### Allowed Headers
|
||||
- `Content-Type`, `Authorization`, `X-CSRF-Token`, `X-Requested-With`
|
||||
|
||||
- `Content-Type`, `Authorization`, `X-CSRF-Token`, `X-Requested-With`
|
||||
|
||||
## Pagination
|
||||
|
||||
@ -520,25 +572,29 @@ Content-Security-Policy: [CSP directives]
|
||||
```
|
||||
|
||||
### Pagination Parameters
|
||||
- `page`: Page number (1-based, default: 1)
|
||||
- `limit`: Records per page (default: 50, max: 100)
|
||||
|
||||
- `page`: Page number (1-based, default: 1)
|
||||
- `limit`: Records per page (default: 50, max: 100)
|
||||
|
||||
## Filtering and Sorting
|
||||
|
||||
### Common Filter Parameters
|
||||
- `startDate` / `endDate`: Date range filtering (ISO 8601)
|
||||
- `status`: Status filtering
|
||||
- `userId` / `companyId`: Entity filtering
|
||||
- `eventType`: Event type filtering
|
||||
- `severity`: Severity level filtering
|
||||
|
||||
- `startDate` / `endDate`: Date range filtering (ISO 8601)
|
||||
- `status`: Status filtering
|
||||
- `userId` / `companyId`: Entity filtering
|
||||
- `eventType`: Event type filtering
|
||||
- `severity`: Severity level filtering
|
||||
|
||||
### Sorting Parameters
|
||||
- `sortBy`: Field to sort by
|
||||
- `sortOrder`: `asc` or `desc` (default: `desc`)
|
||||
|
||||
- `sortBy`: Field to sort by
|
||||
- `sortOrder`: `asc` or `desc` (default: `desc`)
|
||||
|
||||
## Response Caching
|
||||
|
||||
### Cache Headers
|
||||
|
||||
```http
|
||||
Cache-Control: no-cache, no-store, must-revalidate
|
||||
Pragma: no-cache
|
||||
@ -546,20 +602,23 @@ Expires: 0
|
||||
```
|
||||
|
||||
### Cache Strategy
|
||||
- **Security data**: Never cached
|
||||
- **Static data**: Browser cache for 5 minutes
|
||||
- **User data**: No cache for security
|
||||
|
||||
- **Security data**: Never cached
|
||||
- **Static data**: Browser cache for 5 minutes
|
||||
- **User data**: No cache for security
|
||||
|
||||
## API Versioning
|
||||
|
||||
### Current Version
|
||||
- Version: `v1` (implied, no version prefix required)
|
||||
- Introduced: January 2025
|
||||
|
||||
- Version: `v1` (implied, no version prefix required)
|
||||
- Introduced: January 2025
|
||||
|
||||
### Future Versioning
|
||||
- Breaking changes will introduce new versions
|
||||
- Format: `/api/v2/endpoint`
|
||||
- Backward compatibility maintained for 12 months
|
||||
|
||||
- Breaking changes will introduce new versions
|
||||
- Format: `/api/v2/endpoint`
|
||||
- Backward compatibility maintained for 12 months
|
||||
|
||||
## SDK and Client Libraries
|
||||
|
||||
@ -639,10 +698,10 @@ describe('Admin Audit Logs API', () => {
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Admin Audit Logs API](./admin-audit-logs-api.md)
|
||||
- [CSP Metrics API](./csp-metrics-api.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [CSRF Protection](./CSRF_PROTECTION.md)
|
||||
- [Batch Monitoring Dashboard](./batch-monitoring-dashboard.md)
|
||||
- [Admin Audit Logs API](./admin-audit-logs-api.md)
|
||||
- [CSP Metrics API](./csp-metrics-api.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [CSRF Protection](./CSRF_PROTECTION.md)
|
||||
- [Batch Monitoring Dashboard](./batch-monitoring-dashboard.md)
|
||||
|
||||
This API reference provides comprehensive documentation for all endpoints in the LiveDash-Node application. For specific implementation details, refer to the individual documentation files for each feature area.
|
||||
This API reference provides comprehensive documentation for all endpoints in the LiveDash-Node application. For specific implementation details, refer to the individual documentation files for each feature area.
|
||||
|
||||
@ -9,22 +9,25 @@ The Batch Monitoring Dashboard provides real-time visibility into the OpenAI Bat
|
||||
## Features
|
||||
|
||||
### Real-time Monitoring
|
||||
- **Job Status Tracking**: Monitor batch jobs from creation to completion
|
||||
- **Queue Management**: View pending, running, and completed batch queues
|
||||
- **Processing Metrics**: Track throughput, success rates, and error patterns
|
||||
- **Cost Analysis**: Monitor API costs and savings compared to individual requests
|
||||
|
||||
- **Job Status Tracking**: Monitor batch jobs from creation to completion
|
||||
- **Queue Management**: View pending, running, and completed batch queues
|
||||
- **Processing Metrics**: Track throughput, success rates, and error patterns
|
||||
- **Cost Analysis**: Monitor API costs and savings compared to individual requests
|
||||
|
||||
### Performance Analytics
|
||||
- **Batch Efficiency**: Analyze batch size optimization and processing times
|
||||
- **Success Rates**: Track completion and failure rates across different job types
|
||||
- **Resource Utilization**: Monitor API quota usage and rate limiting
|
||||
- **Historical Trends**: View processing patterns over time
|
||||
|
||||
- **Batch Efficiency**: Analyze batch size optimization and processing times
|
||||
- **Success Rates**: Track completion and failure rates across different job types
|
||||
- **Resource Utilization**: Monitor API quota usage and rate limiting
|
||||
- **Historical Trends**: View processing patterns over time
|
||||
|
||||
### Administrative Controls
|
||||
- **Manual Intervention**: Pause, resume, or cancel batch operations
|
||||
- **Priority Management**: Adjust processing priorities for urgent requests
|
||||
- **Error Handling**: Review and retry failed batch operations
|
||||
- **Configuration Management**: Adjust batch parameters and thresholds
|
||||
|
||||
- **Manual Intervention**: Pause, resume, or cancel batch operations
|
||||
- **Priority Management**: Adjust processing priorities for urgent requests
|
||||
- **Error Handling**: Review and retry failed batch operations
|
||||
- **Configuration Management**: Adjust batch parameters and thresholds
|
||||
|
||||
## API Endpoints
|
||||
|
||||
@ -132,6 +135,7 @@ const data = await response.json();
|
||||
The main dashboard component (`components/admin/BatchMonitoringDashboard.tsx`) provides:
|
||||
|
||||
#### Key Metrics Cards
|
||||
|
||||
```tsx
|
||||
// Real-time overview cards
|
||||
<MetricCard
|
||||
@ -157,6 +161,7 @@ The main dashboard component (`components/admin/BatchMonitoringDashboard.tsx`) p
|
||||
```
|
||||
|
||||
#### Queue Status Visualization
|
||||
|
||||
```tsx
|
||||
// Visual representation of batch job queues
|
||||
<QueueStatusChart
|
||||
@ -168,6 +173,7 @@ The main dashboard component (`components/admin/BatchMonitoringDashboard.tsx`) p
|
||||
```
|
||||
|
||||
#### Performance Charts
|
||||
|
||||
```tsx
|
||||
// Processing throughput over time
|
||||
<ThroughputChart
|
||||
@ -183,6 +189,7 @@ The main dashboard component (`components/admin/BatchMonitoringDashboard.tsx`) p
|
||||
```
|
||||
|
||||
#### Job Management Table
|
||||
|
||||
```tsx
|
||||
// Detailed job listing with actions
|
||||
<BatchJobTable
|
||||
@ -400,6 +407,7 @@ async function configureAlerts(alertConfig) {
|
||||
### Common Issues
|
||||
|
||||
#### High Error Rates
|
||||
|
||||
```javascript
|
||||
// Investigate high error rates
|
||||
async function investigateErrors() {
|
||||
@ -424,6 +432,7 @@ async function investigateErrors() {
|
||||
```
|
||||
|
||||
#### Slow Processing
|
||||
|
||||
```javascript
|
||||
// Analyze processing bottlenecks
|
||||
async function analyzePerformance() {
|
||||
@ -457,6 +466,7 @@ async function analyzePerformance() {
|
||||
### Performance Optimization
|
||||
|
||||
#### Batch Size Optimization
|
||||
|
||||
```javascript
|
||||
// Analyze optimal batch sizes
|
||||
async function optimizeBatchSizes() {
|
||||
@ -497,6 +507,7 @@ async function optimizeBatchSizes() {
|
||||
## Integration with Existing Systems
|
||||
|
||||
### Security Audit Integration
|
||||
|
||||
All batch monitoring activities are logged through the security audit system:
|
||||
|
||||
```javascript
|
||||
@ -510,6 +521,7 @@ await securityAuditLogger.logPlatformAdmin(
|
||||
```
|
||||
|
||||
### Rate Limiting Integration
|
||||
|
||||
Monitoring API endpoints use the existing rate limiting system:
|
||||
|
||||
```javascript
|
||||
@ -523,9 +535,9 @@ const rateLimitResult = await rateLimiter.check(
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Batch Processing Optimizations](./batch-processing-optimizations.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [Admin Audit Logs API](./admin-audit-logs-api.md)
|
||||
- [OpenAI Batch API Integration](../lib/batchProcessor.ts)
|
||||
- [Batch Processing Optimizations](./batch-processing-optimizations.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [Admin Audit Logs API](./admin-audit-logs-api.md)
|
||||
- [OpenAI Batch API Integration](../lib/batchProcessor.ts)
|
||||
|
||||
The batch monitoring dashboard provides comprehensive visibility into the AI processing pipeline, enabling administrators to optimize performance, monitor costs, and ensure reliable operation of the batch processing system.
|
||||
The batch monitoring dashboard provides comprehensive visibility into the AI processing pipeline, enabling administrators to optimize performance, monitor costs, and ensure reliable operation of the batch processing system.
|
||||
|
||||
@ -6,11 +6,11 @@ This document outlines the database query optimizations implemented to improve t
|
||||
|
||||
The batch processing system was optimized to reduce database load and improve response times through several key strategies:
|
||||
|
||||
1. **Database Index Optimization**
|
||||
2. **Query Pattern Improvements**
|
||||
3. **Company Caching**
|
||||
4. **Batch Operations**
|
||||
5. **Integration Layer with Fallback**
|
||||
1. **Database Index Optimization**
|
||||
2. **Query Pattern Improvements**
|
||||
3. **Company Caching**
|
||||
4. **Batch Operations**
|
||||
5. **Integration Layer with Fallback**
|
||||
|
||||
## Database Index Improvements
|
||||
|
||||
@ -32,15 +32,17 @@ The following composite indexes were added to the `AIProcessingRequest` table in
|
||||
### Query Performance Impact
|
||||
|
||||
These indexes specifically optimize:
|
||||
- Finding pending requests by status and creation time
|
||||
- Batch-related lookups by batch ID
|
||||
- Combined status and batch filtering operations
|
||||
|
||||
- Finding pending requests by status and creation time
|
||||
- Batch-related lookups by batch ID
|
||||
- Combined status and batch filtering operations
|
||||
|
||||
## Query Optimization Strategies
|
||||
|
||||
### 1. Selective Data Fetching
|
||||
|
||||
**Before:**
|
||||
|
||||
```typescript
|
||||
// Loaded full session with all messages
|
||||
include: {
|
||||
@ -55,6 +57,7 @@ include: {
|
||||
```
|
||||
|
||||
**After:**
|
||||
|
||||
```typescript
|
||||
// Only essential data with message count
|
||||
include: {
|
||||
@ -86,6 +89,7 @@ class CompanyCache {
|
||||
### 3. Batch Operations
|
||||
|
||||
**Before:** N+1 queries for each company
|
||||
|
||||
```typescript
|
||||
// Sequential processing per company
|
||||
for (const company of companies) {
|
||||
@ -95,6 +99,7 @@ for (const company of companies) {
|
||||
```
|
||||
|
||||
**After:** Single query for all companies
|
||||
|
||||
```typescript
|
||||
// Batch query for all companies at once
|
||||
const allRequests = await prisma.aIProcessingRequest.findMany({
|
||||
@ -114,10 +119,10 @@ const requestsByCompany = groupByCompany(allRequests);
|
||||
|
||||
### Query Count Reduction
|
||||
|
||||
- **Company lookups:** Reduced from 4 separate queries per scheduler run to 1 cached lookup
|
||||
- **Pending requests:** Reduced from N queries (one per company) to 1 batch query
|
||||
- **Status checks:** Reduced from N queries to 1 batch query
|
||||
- **Failed requests:** Reduced from N queries to 1 batch query
|
||||
- **Company lookups:** Reduced from 4 separate queries per scheduler run to 1 cached lookup
|
||||
- **Pending requests:** Reduced from N queries (one per company) to 1 batch query
|
||||
- **Status checks:** Reduced from N queries to 1 batch query
|
||||
- **Failed requests:** Reduced from N queries to 1 batch query
|
||||
|
||||
### Parallel Processing
|
||||
|
||||
@ -133,9 +138,9 @@ const SCHEDULER_CONFIG = {
|
||||
|
||||
### Memory Optimization
|
||||
|
||||
- Eliminated loading unnecessary message content
|
||||
- Used `select` instead of `include` where possible
|
||||
- Implemented automatic cache cleanup
|
||||
- Eliminated loading unnecessary message content
|
||||
- Used `select` instead of `include` where possible
|
||||
- Implemented automatic cache cleanup
|
||||
|
||||
## Integration Layer
|
||||
|
||||
@ -169,45 +174,47 @@ class PerformanceTracker {
|
||||
## Files Modified
|
||||
|
||||
### New Files
|
||||
- `lib/batchProcessorOptimized.ts` - Optimized query implementations
|
||||
- `lib/batchSchedulerOptimized.ts` - Optimized scheduler
|
||||
- `lib/batchProcessorIntegration.ts` - Integration layer with fallback
|
||||
|
||||
- `lib/batchProcessorOptimized.ts` - Optimized query implementations
|
||||
- `lib/batchSchedulerOptimized.ts` - Optimized scheduler
|
||||
- `lib/batchProcessorIntegration.ts` - Integration layer with fallback
|
||||
|
||||
### Modified Files
|
||||
- `prisma/schema.prisma` - Added composite indexes
|
||||
- `server.ts` - Updated to use integration layer
|
||||
- `app/api/admin/batch-monitoring/route.ts` - Updated import
|
||||
|
||||
- `prisma/schema.prisma` - Added composite indexes
|
||||
- `server.ts` - Updated to use integration layer
|
||||
- `app/api/admin/batch-monitoring/route.ts` - Updated import
|
||||
|
||||
## Monitoring
|
||||
|
||||
The optimizations include comprehensive logging and monitoring:
|
||||
|
||||
- Performance metrics for each operation type
|
||||
- Cache hit/miss statistics
|
||||
- Fallback events tracking
|
||||
- Query execution time monitoring
|
||||
- Performance metrics for each operation type
|
||||
- Cache hit/miss statistics
|
||||
- Fallback events tracking
|
||||
- Query execution time monitoring
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
The integration layer allows for easy rollback:
|
||||
|
||||
1. Set `ENABLE_BATCH_OPTIMIZATION=false`
|
||||
2. System automatically uses original implementation
|
||||
3. No database schema changes needed for rollback
|
||||
4. Indexes remain beneficial for manual queries
|
||||
1. Set `ENABLE_BATCH_OPTIMIZATION=false`
|
||||
2. System automatically uses original implementation
|
||||
3. No database schema changes needed for rollback
|
||||
4. Indexes remain beneficial for manual queries
|
||||
|
||||
## Expected Performance Gains
|
||||
|
||||
- **Database Query Count:** 60-80% reduction in scheduler operations
|
||||
- **Memory Usage:** 40-60% reduction from selective data loading
|
||||
- **Response Time:** 30-50% improvement for batch operations
|
||||
- **Cache Hit Rate:** 95%+ for company lookups after warmup
|
||||
- **Database Query Count:** 60-80% reduction in scheduler operations
|
||||
- **Memory Usage:** 40-60% reduction from selective data loading
|
||||
- **Response Time:** 30-50% improvement for batch operations
|
||||
- **Cache Hit Rate:** 95%+ for company lookups after warmup
|
||||
|
||||
## Testing
|
||||
|
||||
Performance improvements can be validated by:
|
||||
|
||||
1. Monitoring the batch monitoring dashboard
|
||||
2. Checking performance metrics in logs
|
||||
3. Comparing execution times before/after optimization
|
||||
4. Load testing with multiple companies and large batches
|
||||
1. Monitoring the batch monitoring dashboard
|
||||
2. Checking performance metrics in logs
|
||||
3. Comparing execution times before/after optimization
|
||||
4. Load testing with multiple companies and large batches
|
||||
|
||||
@ -6,11 +6,11 @@ This document describes the Content Security Policy (CSP) metrics and violation
|
||||
|
||||
The CSP Metrics API provides comprehensive monitoring of Content Security Policy violations, including:
|
||||
|
||||
- Real-time violation tracking and metrics
|
||||
- Bypass attempt detection and risk assessment
|
||||
- Policy optimization recommendations
|
||||
- Historical trend analysis
|
||||
- Export capabilities for security analysis
|
||||
- Real-time violation tracking and metrics
|
||||
- Bypass attempt detection and risk assessment
|
||||
- Policy optimization recommendations
|
||||
- Historical trend analysis
|
||||
- Export capabilities for security analysis
|
||||
|
||||
## API Endpoints
|
||||
|
||||
@ -24,7 +24,7 @@ POST /api/csp-report
|
||||
|
||||
#### Request Headers
|
||||
|
||||
- `Content-Type`: `application/csp-report` or `application/json`
|
||||
- `Content-Type`: `application/csp-report` or `application/json`
|
||||
|
||||
#### Request Body (Automatic from Browser)
|
||||
|
||||
@ -43,10 +43,10 @@ POST /api/csp-report
|
||||
|
||||
#### Features
|
||||
|
||||
- **Rate Limiting**: 10 reports per minute per IP
|
||||
- **Risk Assessment**: Automatic classification of violation severity
|
||||
- **Bypass Detection**: Identifies potential CSP bypass attempts
|
||||
- **Real-time Processing**: Immediate analysis and alerting
|
||||
- **Rate Limiting**: 10 reports per minute per IP
|
||||
- **Risk Assessment**: Automatic classification of violation severity
|
||||
- **Bypass Detection**: Identifies potential CSP bypass attempts
|
||||
- **Real-time Processing**: Immediate analysis and alerting
|
||||
|
||||
### CSP Metrics API
|
||||
|
||||
@ -67,11 +67,11 @@ GET /api/csp-metrics
|
||||
|
||||
#### Time Range Options
|
||||
|
||||
- `1h` - Last 1 hour
|
||||
- `6h` - Last 6 hours
|
||||
- `24h` - Last 24 hours (default)
|
||||
- `7d` - Last 7 days
|
||||
- `30d` - Last 30 days
|
||||
- `1h` - Last 1 hour
|
||||
- `6h` - Last 6 hours
|
||||
- `24h` - Last 24 hours (default)
|
||||
- `7d` - Last 7 days
|
||||
- `30d` - Last 30 days
|
||||
|
||||
#### Example Request
|
||||
|
||||
@ -165,11 +165,11 @@ console.log(result.recommendations); // array of suggestions
|
||||
|
||||
The service automatically assesses violation risk based on:
|
||||
|
||||
- **Directive Type**: Script violations are higher risk than style violations
|
||||
- **Source Pattern**: External domains vs inline vs data URIs
|
||||
- **Bypass Indicators**: Known CSP bypass techniques
|
||||
- **Frequency**: Repeated violations from same source
|
||||
- **Geographic Factors**: Unusual source locations
|
||||
- **Directive Type**: Script violations are higher risk than style violations
|
||||
- **Source Pattern**: External domains vs inline vs data URIs
|
||||
- **Bypass Indicators**: Known CSP bypass techniques
|
||||
- **Frequency**: Repeated violations from same source
|
||||
- **Geographic Factors**: Unusual source locations
|
||||
|
||||
#### 3. Bypass Detection
|
||||
|
||||
@ -191,10 +191,10 @@ const bypassPatterns = [
|
||||
|
||||
Based on violation patterns, the service provides actionable recommendations:
|
||||
|
||||
- **Tighten Policies**: Suggest removing broad allowlists
|
||||
- **Add Domains**: Recommend allowing legitimate external resources
|
||||
- **Implement Nonces**: Suggest nonce-based policies for inline content
|
||||
- **Upgrade Directives**: Recommend modern CSP features
|
||||
- **Tighten Policies**: Suggest removing broad allowlists
|
||||
- **Add Domains**: Recommend allowing legitimate external resources
|
||||
- **Implement Nonces**: Suggest nonce-based policies for inline content
|
||||
- **Upgrade Directives**: Recommend modern CSP features
|
||||
|
||||
## Violation Analysis
|
||||
|
||||
@ -405,48 +405,50 @@ CSP_ALERT_THRESHOLD=5 # violations per 10 minutes
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
- **10 reports per minute per IP** prevents spam attacks
|
||||
- **Exponential backoff** for repeated violations from same source
|
||||
- **Memory cleanup** removes old violations automatically
|
||||
- **10 reports per minute per IP** prevents spam attacks
|
||||
- **Exponential backoff** for repeated violations from same source
|
||||
- **Memory cleanup** removes old violations automatically
|
||||
|
||||
### Memory Management
|
||||
|
||||
- **Violation buffer** limited to 1 hour of data in memory
|
||||
- **Automatic cleanup** runs every 100 requests (1% probability)
|
||||
- **Efficient storage** using Map data structures
|
||||
- **Violation buffer** limited to 1 hour of data in memory
|
||||
- **Automatic cleanup** runs every 100 requests (1% probability)
|
||||
- **Efficient storage** using Map data structures
|
||||
|
||||
### Database Impact
|
||||
|
||||
- **No persistent storage** for real-time metrics (memory only)
|
||||
- **Optional logging** to database for long-term analysis
|
||||
- **Indexed queries** for historical data retrieval
|
||||
- **No persistent storage** for real-time metrics (memory only)
|
||||
- **Optional logging** to database for long-term analysis
|
||||
- **Indexed queries** for historical data retrieval
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Privacy Protection
|
||||
|
||||
**⚠️ Data Collection Notice:**
|
||||
- **IP addresses** are collected and stored in memory for security monitoring
|
||||
- **User agent strings** are stored for browser compatibility analysis
|
||||
- **Legal basis**: Legitimate interest for security incident detection and prevention
|
||||
- **Retention**: In-memory storage only, automatically purged after 7 days or application restart
|
||||
- **Data minimization**: Only violation-related metadata is retained, not page content
|
||||
|
||||
- **IP addresses** are collected and stored in memory for security monitoring
|
||||
- **User agent strings** are stored for browser compatibility analysis
|
||||
- **Legal basis**: Legitimate interest for security incident detection and prevention
|
||||
- **Retention**: In-memory storage only, automatically purged after 7 days or application restart
|
||||
- **Data minimization**: Only violation-related metadata is retained, not page content
|
||||
|
||||
**Planned Privacy Enhancements:**
|
||||
- IP anonymization options for GDPR compliance (roadmap)
|
||||
- User agent sanitization to remove sensitive information (roadmap)
|
||||
|
||||
### Rate Limiting Protection
|
||||
- IP anonymization options for GDPR compliance (roadmap)
|
||||
- User agent sanitization to remove sensitive information (roadmap)
|
||||
|
||||
- **Per-IP limits** prevent DoS attacks on reporting endpoint
|
||||
- **Content-type validation** ensures proper report format
|
||||
- **Request size limits** prevent memory exhaustion
|
||||
### Rate-Limiting Protection
|
||||
|
||||
- **Per-IP limits** prevent DoS attacks on reporting endpoint
|
||||
- **Content-type validation** ensures proper report format
|
||||
- **Request size limits** prevent memory exhaustion
|
||||
|
||||
### False Positive Handling
|
||||
|
||||
- **Learning mode** for new deployments
|
||||
- **Whitelist support** for known legitimate violations
|
||||
- **Risk score adjustment** based on historical patterns
|
||||
- **Learning mode** for new deployments
|
||||
- **Whitelist support** for known legitimate violations
|
||||
- **Risk score adjustment** based on historical patterns
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@ -495,10 +497,10 @@ if (duration > 2000) {
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Enhanced CSP Implementation](./security/enhanced-csp.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [Security Headers](./security-headers.md)
|
||||
- [Rate Limiting](../lib/rateLimiter.ts)
|
||||
- [Enhanced CSP Implementation](./security/enhanced-csp.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [Security Headers](./security-headers.md)
|
||||
- [Rate Limiting](../lib/rateLimiter.ts)
|
||||
|
||||
## API Reference Summary
|
||||
|
||||
|
||||
@ -12,10 +12,10 @@ The WordCloud component visualizes categories or topics based on their frequency
|
||||
|
||||
**Features:**
|
||||
|
||||
- Dynamic sizing based on frequency
|
||||
- Colorful display with a pleasing color palette
|
||||
- Responsive design
|
||||
- Interactive hover effects
|
||||
- Dynamic sizing based on frequency
|
||||
- Colorful display with a pleasing color palette
|
||||
- Responsive design
|
||||
- Interactive hover effects
|
||||
|
||||
### 2. GeographicMap
|
||||
|
||||
@ -25,10 +25,10 @@ This component displays a world map with circles representing the number of sess
|
||||
|
||||
**Features:**
|
||||
|
||||
- Interactive map using React Leaflet
|
||||
- Circle sizes scaled by session count
|
||||
- Tooltips showing country names and session counts
|
||||
- Responsive design
|
||||
- Interactive map using React Leaflet
|
||||
- Circle sizes scaled by session count
|
||||
- Tooltips showing country names and session counts
|
||||
- Responsive design
|
||||
|
||||
### 3. MetricCard
|
||||
|
||||
@ -38,10 +38,10 @@ A modern, visually appealing card for displaying key metrics.
|
||||
|
||||
**Features:**
|
||||
|
||||
- Multiple design variants (default, primary, success, warning, danger)
|
||||
- Support for trend indicators
|
||||
- Icons and descriptions
|
||||
- Clean, modern styling
|
||||
- Multiple design variants (default, primary, success, warning, danger)
|
||||
- Support for trend indicators
|
||||
- Icons and descriptions
|
||||
- Clean, modern styling
|
||||
|
||||
### 4. DonutChart
|
||||
|
||||
@ -51,10 +51,10 @@ An enhanced donut chart with better styling and a central text display capabilit
|
||||
|
||||
**Features:**
|
||||
|
||||
- Customizable colors
|
||||
- Center text area for displaying summaries
|
||||
- Interactive tooltips with percentages
|
||||
- Well-balanced legend display
|
||||
- Customizable colors
|
||||
- Center text area for displaying summaries
|
||||
- Interactive tooltips with percentages
|
||||
- Well-balanced legend display
|
||||
|
||||
### 5. ResponseTimeDistribution
|
||||
|
||||
@ -64,10 +64,10 @@ Visualizes the distribution of response times as a histogram.
|
||||
|
||||
**Features:**
|
||||
|
||||
- Color-coded bars (green for fast, yellow for medium, red for slow)
|
||||
- Target time indicator
|
||||
- Automatic binning of response times
|
||||
- Clear labeling and scales
|
||||
- Color-coded bars (green for fast, yellow for medium, red for slow)
|
||||
- Target time indicator
|
||||
- Automatic binning of response times
|
||||
- Clear labeling and scales
|
||||
|
||||
## Dashboard Enhancements
|
||||
|
||||
@ -85,7 +85,7 @@ The dashboard has been enhanced with:
|
||||
|
||||
## Usage Notes
|
||||
|
||||
- The geographic map and response time distribution use simulated data where actual data is not available
|
||||
- All components are responsive and will adjust to different screen sizes
|
||||
- The dashboard automatically refreshes data when using the refresh button
|
||||
- Admin users have access to additional controls at the bottom of the dashboard
|
||||
- The geographic map and response time distribution use simulated data where actual data is not available
|
||||
- All components are responsive and will adjust to different screen sizes
|
||||
- The dashboard automatically refreshes data when using the refresh button
|
||||
- Admin users have access to additional controls at the bottom of the dashboard
|
||||
|
||||
@ -6,8 +6,8 @@ This document explains how to optimize database connection pooling for better pe
|
||||
|
||||
The application now supports two connection pooling modes:
|
||||
|
||||
1. **Standard Pooling**: Default Prisma client connection pooling
|
||||
2. **Enhanced Pooling**: Advanced PostgreSQL connection pooling with custom configuration
|
||||
1. **Standard Pooling**: Default Prisma client connection pooling
|
||||
2. **Enhanced Pooling**: Advanced PostgreSQL connection pooling with custom configuration
|
||||
|
||||
## Configuration
|
||||
|
||||
@ -29,42 +29,42 @@ DATABASE_URL="postgresql://user:pass@host:5432/db?connection_limit=20&pool_timeo
|
||||
|
||||
#### Standard Pooling (Default)
|
||||
|
||||
- Uses Prisma's built-in connection pooling
|
||||
- Simpler configuration
|
||||
- Good for development and small-scale deployments
|
||||
- Uses Prisma's built-in connection pooling
|
||||
- Simpler configuration
|
||||
- Good for development and small-scale deployments
|
||||
|
||||
#### Enhanced Pooling (Recommended for Production)
|
||||
|
||||
- Uses PostgreSQL native connection pooling with `@prisma/adapter-pg`
|
||||
- Advanced monitoring and health checks
|
||||
- Better resource management
|
||||
- Detailed connection metrics
|
||||
- Uses PostgreSQL native connection pooling with `@prisma/adapter-pg`
|
||||
- Advanced monitoring and health checks
|
||||
- Better resource management
|
||||
- Detailed connection metrics
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Fixed Issues
|
||||
|
||||
1. **Multiple PrismaClient Instances**:
|
||||
- ❌ Before: Each scheduler created its own PrismaClient
|
||||
- ✅ After: All modules use singleton pattern from `lib/prisma.ts`
|
||||
1. **Multiple PrismaClient Instances**:
|
||||
- ❌ Before: Each scheduler created its own PrismaClient
|
||||
- ✅ After: All modules use singleton pattern from `lib/prisma.ts`
|
||||
|
||||
2. **No Connection Management**:
|
||||
- ❌ Before: No graceful shutdown or connection cleanup
|
||||
- ✅ After: Proper cleanup on process termination
|
||||
2. **No Connection Management**:
|
||||
- ❌ Before: No graceful shutdown or connection cleanup
|
||||
- ✅ After: Proper cleanup on process termination
|
||||
|
||||
3. **No Monitoring**:
|
||||
- ❌ Before: No visibility into connection usage
|
||||
- ✅ After: Health check endpoint and connection metrics
|
||||
3. **No Monitoring**:
|
||||
- ❌ Before: No visibility into connection usage
|
||||
- ✅ After: Health check endpoint and connection metrics
|
||||
|
||||
### Key Files Modified
|
||||
|
||||
- `lib/prisma.ts` - Enhanced singleton with pooling options
|
||||
- `lib/database-pool.ts` - Advanced pooling configuration
|
||||
- `lib/processingScheduler.ts` - Fixed to use singleton
|
||||
- `lib/importProcessor.ts` - Fixed to use singleton
|
||||
- `lib/processingStatusManager.ts` - Fixed to use singleton
|
||||
- `lib/schedulers.ts` - Added graceful shutdown
|
||||
- `app/api/admin/database-health/route.ts` - Monitoring endpoint
|
||||
- `lib/prisma.ts` - Enhanced singleton with pooling options
|
||||
- `lib/database-pool.ts` - Advanced pooling configuration
|
||||
- `lib/processingScheduler.ts` - Fixed to use singleton
|
||||
- `lib/importProcessor.ts` - Fixed to use singleton
|
||||
- `lib/processingStatusManager.ts` - Fixed to use singleton
|
||||
- `lib/schedulers.ts` - Added graceful shutdown
|
||||
- `app/api/admin/database-health/route.ts` - Monitoring endpoint
|
||||
|
||||
## Monitoring
|
||||
|
||||
@ -79,36 +79,36 @@ curl -H "Authorization: Bearer your-token" \
|
||||
|
||||
Response includes:
|
||||
|
||||
- Connection status
|
||||
- Pool statistics (if enhanced pooling enabled)
|
||||
- Basic metrics (session counts, etc.)
|
||||
- Configuration details
|
||||
- Connection status
|
||||
- Pool statistics (if enhanced pooling enabled)
|
||||
- Basic metrics (session counts, etc.)
|
||||
- Configuration details
|
||||
|
||||
### Connection Metrics
|
||||
|
||||
With enhanced pooling enabled, you'll see console logs for:
|
||||
|
||||
- Connection acquisitions/releases
|
||||
- Pool size changes
|
||||
- Error events
|
||||
- Health check results
|
||||
- Connection acquisitions/releases
|
||||
- Pool size changes
|
||||
- Error events
|
||||
- Health check results
|
||||
|
||||
## Performance Benefits
|
||||
|
||||
### Before Optimization
|
||||
|
||||
- Multiple connection pools (one per scheduler)
|
||||
- Potential connection exhaustion under load
|
||||
- No connection monitoring
|
||||
- Resource waste from idle connections
|
||||
- Multiple connection pools (one per scheduler)
|
||||
- Potential connection exhaustion under load
|
||||
- No connection monitoring
|
||||
- Resource waste from idle connections
|
||||
|
||||
### After Optimization
|
||||
|
||||
- Single shared connection pool
|
||||
- Configurable pool size and timeouts
|
||||
- Connection health monitoring
|
||||
- Graceful shutdown and cleanup
|
||||
- Better resource utilization
|
||||
- Single shared connection pool
|
||||
- Configurable pool size and timeouts
|
||||
- Connection health monitoring
|
||||
- Graceful shutdown and cleanup
|
||||
- Better resource utilization
|
||||
|
||||
## Recommended Settings
|
||||
|
||||
@ -142,40 +142,40 @@ USE_ENHANCED_POOLING=true
|
||||
|
||||
If you see "too many connections" errors:
|
||||
|
||||
1. Increase `DATABASE_CONNECTION_LIMIT`
|
||||
2. Check for connection leaks in application code
|
||||
3. Monitor the health endpoint for pool statistics
|
||||
1. Increase `DATABASE_CONNECTION_LIMIT`
|
||||
2. Check for connection leaks in application code
|
||||
3. Monitor the health endpoint for pool statistics
|
||||
|
||||
### Slow Database Queries
|
||||
|
||||
If queries are timing out:
|
||||
|
||||
1. Decrease `DATABASE_POOL_TIMEOUT`
|
||||
2. Check database query performance
|
||||
3. Consider connection pooling at the infrastructure level (PgBouncer)
|
||||
1. Decrease `DATABASE_POOL_TIMEOUT`
|
||||
2. Check database query performance
|
||||
3. Consider connection pooling at the infrastructure level (PgBouncer)
|
||||
|
||||
### Memory Usage
|
||||
|
||||
If memory usage is high:
|
||||
|
||||
1. Decrease `DATABASE_CONNECTION_LIMIT`
|
||||
2. Enable enhanced pooling for better resource management
|
||||
3. Monitor idle connection cleanup
|
||||
1. Decrease `DATABASE_CONNECTION_LIMIT`
|
||||
2. Enable enhanced pooling for better resource management
|
||||
3. Monitor idle connection cleanup
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always use the singleton**: Import `prisma` from `lib/prisma.ts`
|
||||
2. **Monitor connection usage**: Use the health endpoint regularly
|
||||
3. **Set appropriate limits**: Don't over-provision connections
|
||||
4. **Enable enhanced pooling in production**: Better resource management
|
||||
5. **Implement graceful shutdown**: Ensure connections are properly closed
|
||||
6. **Log connection events**: Monitor for issues and optimize accordingly
|
||||
1. **Always use the singleton**: Import `prisma` from `lib/prisma.ts`
|
||||
2. **Monitor connection usage**: Use the health endpoint regularly
|
||||
3. **Set appropriate limits**: Don't over-provision connections
|
||||
4. **Enable enhanced pooling in production**: Better resource management
|
||||
5. **Implement graceful shutdown**: Ensure connections are properly closed
|
||||
6. **Log connection events**: Monitor for issues and optimize accordingly
|
||||
|
||||
## Next Steps
|
||||
|
||||
Consider implementing:
|
||||
|
||||
1. **Connection pooling middleware**: PgBouncer or similar
|
||||
2. **Read replicas**: For read-heavy workloads
|
||||
3. **Connection retry logic**: For handling temporary failures
|
||||
4. **Metrics collection**: Prometheus/Grafana for detailed monitoring
|
||||
1. **Connection pooling middleware**: PgBouncer or similar
|
||||
2. **Read replicas**: For read-heavy workloads
|
||||
3. **Connection retry logic**: For handling temporary failures
|
||||
4. **Metrics collection**: Prometheus/Grafana for detailed monitoring
|
||||
|
||||
@ -6,10 +6,10 @@ This document outlines the comprehensive database performance optimizations impl
|
||||
|
||||
The optimization focuses on the most frequently queried patterns in the application, particularly around:
|
||||
|
||||
- AI processing request tracking and batching
|
||||
- Session analytics and filtering
|
||||
- Security audit log analysis
|
||||
- Multi-tenant data isolation performance
|
||||
- AI processing request tracking and batching
|
||||
- Session analytics and filtering
|
||||
- Security audit log analysis
|
||||
- Multi-tenant data isolation performance
|
||||
|
||||
## Applied Optimizations
|
||||
|
||||
@ -31,9 +31,9 @@ INCLUDE ("processingStatus", "batchId", "requestedAt", "sessionId");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~70% faster batch job queries
|
||||
- Reduced I/O for cost analysis reports
|
||||
- Improved scheduler performance
|
||||
- ~70% faster batch job queries
|
||||
- Reduced I/O for cost analysis reports
|
||||
- Improved scheduler performance
|
||||
|
||||
### 2. Session Analytics Optimizations
|
||||
|
||||
@ -54,9 +54,9 @@ INCLUDE ("startTime", "messagesSent");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~85% faster dashboard load times
|
||||
- Efficient date range filtering
|
||||
- Optimized sentiment analysis queries
|
||||
- ~85% faster dashboard load times
|
||||
- Efficient date range filtering
|
||||
- Optimized sentiment analysis queries
|
||||
|
||||
### 3. Security Audit Log Optimizations
|
||||
|
||||
@ -77,9 +77,9 @@ INCLUDE ("eventType", "severity", "userId", "companyId");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~90% faster security monitoring
|
||||
- Efficient threat detection
|
||||
- Improved compliance reporting
|
||||
- ~90% faster security monitoring
|
||||
- Efficient threat detection
|
||||
- Improved compliance reporting
|
||||
|
||||
### 4. Message Processing Optimizations
|
||||
|
||||
@ -95,8 +95,8 @@ INCLUDE ("content");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~60% faster conversation loading
|
||||
- Reduced memory usage for message queries
|
||||
- ~60% faster conversation loading
|
||||
- Reduced memory usage for message queries
|
||||
|
||||
### 5. Processing Pipeline Optimizations
|
||||
|
||||
@ -118,29 +118,29 @@ INCLUDE ("sessionId", "errorMessage", "retryCount", "startedAt");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~75% faster processing monitoring
|
||||
- Efficient error tracking
|
||||
- Improved retry logic performance
|
||||
- ~75% faster processing monitoring
|
||||
- Efficient error tracking
|
||||
- Improved retry logic performance
|
||||
|
||||
## Index Strategy Principles
|
||||
|
||||
### 1. Composite Index Design
|
||||
|
||||
- **Leading column**: Most selective filter (usually companyId for multi-tenancy)
|
||||
- **Secondary columns**: Common WHERE clause filters
|
||||
- **Covering columns**: SELECT list columns via INCLUDE
|
||||
- **Leading column**: Most selective filter (usually companyId for multi-tenancy)
|
||||
- **Secondary columns**: Common WHERE clause filters
|
||||
- **Covering columns**: SELECT list columns via INCLUDE
|
||||
|
||||
### 2. Partial Indexes
|
||||
|
||||
- Used for error analysis and specific status filtering
|
||||
- Reduces index size and maintenance overhead
|
||||
- Improves write performance
|
||||
- Used for error analysis and specific status filtering
|
||||
- Reduces index size and maintenance overhead
|
||||
- Improves write performance
|
||||
|
||||
### 3. Covering Indexes
|
||||
|
||||
- Include frequently accessed columns to avoid table lookups
|
||||
- Reduces I/O for read-heavy operations
|
||||
- Particularly effective for dashboard queries
|
||||
- Include frequently accessed columns to avoid table lookups
|
||||
- Reduces I/O for read-heavy operations
|
||||
- Particularly effective for dashboard queries
|
||||
|
||||
## Query Pattern Analysis
|
||||
|
||||
@ -166,29 +166,29 @@ INCLUDE ("sessionId", "errorMessage", "retryCount", "startedAt");
|
||||
|
||||
### Index Monitoring
|
||||
|
||||
- Monitor index usage with `pg_stat_user_indexes`
|
||||
- Track bloat with `pg_stat_user_tables`
|
||||
- Regular ANALYZE after bulk operations
|
||||
- Monitor index usage with `pg_stat_user_indexes`
|
||||
- Track bloat with `pg_stat_user_tables`
|
||||
- Regular ANALYZE after bulk operations
|
||||
|
||||
### Write Performance Impact
|
||||
|
||||
- Composite indexes add ~15% write overhead
|
||||
- Offset by dramatic read performance gains
|
||||
- Monitored via slow query logs
|
||||
- Composite indexes add ~15% write overhead
|
||||
- Offset by dramatic read performance gains
|
||||
- Monitored via slow query logs
|
||||
|
||||
### Storage Impact
|
||||
|
||||
- Indexes add ~25% to total storage
|
||||
- Covering indexes reduce need for table scans
|
||||
- Partial indexes minimize storage overhead
|
||||
- Indexes add ~25% to total storage
|
||||
- Covering indexes reduce need for table scans
|
||||
- Partial indexes minimize storage overhead
|
||||
|
||||
## Migration Safety
|
||||
|
||||
### CONCURRENTLY Operations
|
||||
|
||||
- All indexes created with `CREATE INDEX CONCURRENTLY`
|
||||
- No table locks during creation
|
||||
- Production-safe deployment
|
||||
- All indexes created with `CREATE INDEX CONCURRENTLY`
|
||||
- No table locks during creation
|
||||
- Production-safe deployment
|
||||
|
||||
### Rollback Strategy
|
||||
|
||||
@ -238,18 +238,18 @@ LIMIT 10;
|
||||
|
||||
### Monitoring Strategy
|
||||
|
||||
- Set up automated index usage monitoring
|
||||
- Track slow query evolution
|
||||
- Monitor storage growth patterns
|
||||
- Implement performance alerting
|
||||
- Set up automated index usage monitoring
|
||||
- Track slow query evolution
|
||||
- Monitor storage growth patterns
|
||||
- Implement performance alerting
|
||||
|
||||
## Conclusion
|
||||
|
||||
These database optimizations provide:
|
||||
|
||||
- **70-90% improvement** in query performance
|
||||
- **Reduced server load** through efficient indexing
|
||||
- **Better user experience** with faster dashboards
|
||||
- **Scalable foundation** for future growth
|
||||
- **70-90% improvement** in query performance
|
||||
- **Reduced server load** through efficient indexing
|
||||
- **Better user experience** with faster dashboards
|
||||
- **Scalable foundation** for future growth
|
||||
|
||||
The optimizations are designed to be production-safe and monitoring-friendly, ensuring both immediate performance gains and long-term maintainability.
|
||||
|
||||
@ -15,21 +15,21 @@ Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.ne
|
||||
|
||||
### 1. Neon Connection Limits
|
||||
|
||||
- **Free Tier**: 20 concurrent connections
|
||||
- **Pro Tier**: 100 concurrent connections
|
||||
- **Multiple schedulers** can quickly exhaust connections
|
||||
- **Free Tier**: 20 concurrent connections
|
||||
- **Pro Tier**: 100 concurrent connections
|
||||
- **Multiple schedulers** can quickly exhaust connections
|
||||
|
||||
### 2. Connection Pooling Issues
|
||||
|
||||
- Each scheduler was creating separate PrismaClient instances
|
||||
- No connection reuse between operations
|
||||
- No retry logic for temporary failures
|
||||
- Each scheduler was creating separate PrismaClient instances
|
||||
- No connection reuse between operations
|
||||
- No retry logic for temporary failures
|
||||
|
||||
### 3. Neon-Specific Challenges
|
||||
|
||||
- **Auto-pause**: Databases pause after inactivity
|
||||
- **Cold starts**: First connection after pause takes longer
|
||||
- **Regional latency**: eu-central-1 may have variable latency
|
||||
- **Auto-pause**: Databases pause after inactivity
|
||||
- **Cold starts**: First connection after pause takes longer
|
||||
- **Regional latency**: eu-central-1 may have variable latency
|
||||
|
||||
## Solutions Implemented
|
||||
|
||||
@ -106,9 +106,9 @@ curl -H "Authorization: Bearer YOUR_API_TOKEN" \
|
||||
|
||||
### 2. Neon Dashboard Monitoring
|
||||
|
||||
- Monitor "Active connections" in Neon dashboard
|
||||
- Check for connection spikes during scheduler runs
|
||||
- Review query performance and slow queries
|
||||
- Monitor "Active connections" in Neon dashboard
|
||||
- Check for connection spikes during scheduler runs
|
||||
- Review query performance and slow queries
|
||||
|
||||
### 3. Application Logs
|
||||
|
||||
@ -158,9 +158,9 @@ const prisma = new PrismaClient({
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Neon database auto-paused
|
||||
- Connection limit exceeded
|
||||
- Network issues
|
||||
- Neon database auto-paused
|
||||
- Connection limit exceeded
|
||||
- Network issues
|
||||
|
||||
**Solutions:**
|
||||
|
||||
@ -173,9 +173,9 @@ const prisma = new PrismaClient({
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Idle connection timeout
|
||||
- Neon maintenance
|
||||
- Long-running transactions
|
||||
- Idle connection timeout
|
||||
- Neon maintenance
|
||||
- Long-running transactions
|
||||
|
||||
**Solutions:**
|
||||
|
||||
@ -187,9 +187,9 @@ const prisma = new PrismaClient({
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Blocking database operations
|
||||
- Scheduler overlap
|
||||
- High CPU usage
|
||||
- Blocking database operations
|
||||
- Scheduler overlap
|
||||
- High CPU usage
|
||||
|
||||
**Solutions:**
|
||||
|
||||
@ -230,10 +230,10 @@ SESSION_PROCESSING_INTERVAL="0 */2 * * *"
|
||||
|
||||
## Monitoring Checklist
|
||||
|
||||
- [ ] Check Neon dashboard for connection spikes
|
||||
- [ ] Monitor scheduler execution times
|
||||
- [ ] Review error logs for connection patterns
|
||||
- [ ] Test health endpoint regularly
|
||||
- [ ] Set up alerts for connection failures
|
||||
- [ ] Check Neon dashboard for connection spikes
|
||||
- [ ] Monitor scheduler execution times
|
||||
- [ ] Review error logs for connection patterns
|
||||
- [ ] Test health endpoint regularly
|
||||
- [ ] Set up alerts for connection failures
|
||||
|
||||
With these optimizations, your Neon database connections should be much more stable and efficient!
|
||||
|
||||
@ -17,48 +17,48 @@ Successfully migrated the livedash-node application from SQLite to PostgreSQL us
|
||||
|
||||
#### Production/Development
|
||||
|
||||
- **Provider**: PostgreSQL (Neon)
|
||||
- **Environment Variable**: `DATABASE_URL`
|
||||
- **Connection**: Neon PostgreSQL cluster
|
||||
- **Provider**: PostgreSQL (Neon)
|
||||
- **Environment Variable**: `DATABASE_URL`
|
||||
- **Connection**: Neon PostgreSQL cluster
|
||||
|
||||
#### Testing
|
||||
|
||||
- **Provider**: PostgreSQL (Neon - separate database)
|
||||
- **Environment Variable**: `DATABASE_URL_TEST`
|
||||
- **Test Setup**: Automatically switches to test database during test runs
|
||||
- **Provider**: PostgreSQL (Neon - separate database)
|
||||
- **Environment Variable**: `DATABASE_URL_TEST`
|
||||
- **Test Setup**: Automatically switches to test database during test runs
|
||||
|
||||
### Files Modified
|
||||
|
||||
1. **`prisma/schema.prisma`**
|
||||
|
||||
- Changed provider from `sqlite` to `postgresql`
|
||||
- Updated URL to use `env("DATABASE_URL")`
|
||||
- Changed provider from `sqlite` to `postgresql`
|
||||
- Updated URL to use `env("DATABASE_URL")`
|
||||
|
||||
2. **`tests/setup.ts`**
|
||||
|
||||
- Added logic to use `DATABASE_URL_TEST` when available
|
||||
- Ensures test isolation with separate database
|
||||
- Added logic to use `DATABASE_URL_TEST` when available
|
||||
- Ensures test isolation with separate database
|
||||
|
||||
3. **`.env`** (created)
|
||||
|
||||
- Contains `DATABASE_URL` for Prisma CLI operations
|
||||
- Contains `DATABASE_URL` for Prisma CLI operations
|
||||
|
||||
4. **`.env.local`** (existing)
|
||||
|
||||
- Contains both `DATABASE_URL` and `DATABASE_URL_TEST`
|
||||
- Contains both `DATABASE_URL` and `DATABASE_URL_TEST`
|
||||
|
||||
### Database Schema
|
||||
|
||||
All existing models and relationships were preserved:
|
||||
|
||||
- **Company**: Multi-tenant root entity
|
||||
- **User**: Authentication and authorization
|
||||
- **Session**: Processed session data
|
||||
- **SessionImport**: Raw CSV import data
|
||||
- **Message**: Individual conversation messages
|
||||
- **Question**: Normalized question storage
|
||||
- **SessionQuestion**: Session-question relationships
|
||||
- **AIProcessingRequest**: AI cost tracking
|
||||
- **Company**: Multi-tenant root entity
|
||||
- **User**: Authentication and authorization
|
||||
- **Session**: Processed session data
|
||||
- **SessionImport**: Raw CSV import data
|
||||
- **Message**: Individual conversation messages
|
||||
- **Question**: Normalized question storage
|
||||
- **SessionQuestion**: Session-question relationships
|
||||
- **AIProcessingRequest**: AI cost tracking
|
||||
|
||||
### Migration Process
|
||||
|
||||
@ -103,11 +103,11 @@ if (process.env.DATABASE_URL_TEST) {
|
||||
|
||||
All tests pass successfully:
|
||||
|
||||
- ✅ Environment configuration tests
|
||||
- ✅ Transcript fetcher tests
|
||||
- ✅ Database connection tests
|
||||
- ✅ Schema validation tests
|
||||
- ✅ CRUD operation tests
|
||||
- ✅ Environment configuration tests
|
||||
- ✅ Transcript fetcher tests
|
||||
- ✅ Database connection tests
|
||||
- ✅ Schema validation tests
|
||||
- ✅ CRUD operation tests
|
||||
|
||||
### Next Steps
|
||||
|
||||
|
||||
@ -7,21 +7,24 @@ Successfully refactored the session processing pipeline from a simple status-bas
|
||||
## Problems Solved
|
||||
|
||||
### Original Issues
|
||||
1. **Inconsistent Status Tracking**: The old system used a simple enum on SessionImport that didn't properly track the multi-stage processing pipeline
|
||||
2. **Poor Error Visibility**: Error messages were buried in the SessionImport table and not easily accessible
|
||||
3. **No Stage-Specific Tracking**: The system couldn't track which specific stage of processing failed
|
||||
4. **Difficult Recovery**: Failed sessions were hard to identify and retry
|
||||
5. **Linting Errors**: Multiple TypeScript files referencing removed database fields
|
||||
|
||||
1. **Inconsistent Status Tracking**: The old system used a simple enum on SessionImport that didn't properly track the multi-stage processing pipeline
|
||||
2. **Poor Error Visibility**: Error messages were buried in the SessionImport table and not easily accessible
|
||||
3. **No Stage-Specific Tracking**: The system couldn't track which specific stage of processing failed
|
||||
4. **Difficult Recovery**: Failed sessions were hard to identify and retry
|
||||
5. **Linting Errors**: Multiple TypeScript files referencing removed database fields
|
||||
|
||||
### Schema Changes Made
|
||||
- **Removed** old `status`, `errorMsg`, and `processedAt` columns from SessionImport
|
||||
- **Removed** `processed` field from Session
|
||||
- **Added** new `SessionProcessingStatus` table with granular stage tracking
|
||||
- **Added** `ProcessingStage` and `ProcessingStatus` enums
|
||||
|
||||
- **Removed** old `status`, `errorMsg`, and `processedAt` columns from SessionImport
|
||||
- **Removed** `processed` field from Session
|
||||
- **Added** new `SessionProcessingStatus` table with granular stage tracking
|
||||
- **Added** `ProcessingStage` and `ProcessingStatus` enums
|
||||
|
||||
## New Processing Pipeline
|
||||
|
||||
### Processing Stages
|
||||
|
||||
```typescript
|
||||
enum ProcessingStage {
|
||||
CSV_IMPORT // SessionImport created
|
||||
@ -39,95 +42,107 @@ enum ProcessingStatus {
|
||||
### Key Components
|
||||
|
||||
#### 1. ProcessingStatusManager
|
||||
|
||||
Centralized class for managing processing status with methods:
|
||||
- `initializeSession()` - Set up processing status for new sessions
|
||||
- `startStage()`, `completeStage()`, `failStage()`, `skipStage()` - Stage management
|
||||
- `getSessionsNeedingProcessing()` - Query sessions by stage and status
|
||||
- `getPipelineStatus()` - Get overview of entire pipeline
|
||||
- `getFailedSessions()` - Find sessions needing retry
|
||||
- `resetStageForRetry()` - Reset failed stages
|
||||
|
||||
- `initializeSession()` - Set up processing status for new sessions
|
||||
- `startStage()`, `completeStage()`, `failStage()`, `skipStage()` - Stage management
|
||||
- `getSessionsNeedingProcessing()` - Query sessions by stage and status
|
||||
- `getPipelineStatus()` - Get overview of entire pipeline
|
||||
- `getFailedSessions()` - Find sessions needing retry
|
||||
- `resetStageForRetry()` - Reset failed stages
|
||||
|
||||
#### 2. Updated Processing Scheduler
|
||||
- Integrated with new `ProcessingStatusManager`
|
||||
- Tracks AI analysis and question extraction stages
|
||||
- Records detailed processing metadata
|
||||
- Proper error handling and retry capabilities
|
||||
|
||||
- Integrated with new `ProcessingStatusManager`
|
||||
- Tracks AI analysis and question extraction stages
|
||||
- Records detailed processing metadata
|
||||
- Proper error handling and retry capabilities
|
||||
|
||||
#### 3. Migration System
|
||||
- Successfully migrated all 109 existing sessions
|
||||
- Determined current state based on existing data
|
||||
- Preserved all existing functionality
|
||||
|
||||
- Successfully migrated all 109 existing sessions
|
||||
- Determined current state based on existing data
|
||||
- Preserved all existing functionality
|
||||
|
||||
## Current Pipeline Status
|
||||
|
||||
After migration and refactoring:
|
||||
- **CSV_IMPORT**: 109 completed
|
||||
- **TRANSCRIPT_FETCH**: 109 completed
|
||||
- **SESSION_CREATION**: 109 completed
|
||||
- **AI_ANALYSIS**: 16 completed, 93 pending
|
||||
- **QUESTION_EXTRACTION**: 11 completed, 98 pending
|
||||
|
||||
- **CSV_IMPORT**: 109 completed
|
||||
- **TRANSCRIPT_FETCH**: 109 completed
|
||||
- **SESSION_CREATION**: 109 completed
|
||||
- **AI_ANALYSIS**: 16 completed, 93 pending
|
||||
- **QUESTION_EXTRACTION**: 11 completed, 98 pending
|
||||
|
||||
## Files Updated/Created
|
||||
|
||||
### New Files
|
||||
- `lib/processingStatusManager.ts` - Core processing status management
|
||||
- `check-refactored-pipeline-status.ts` - New pipeline status checker
|
||||
- `migrate-to-refactored-system.ts` - Migration script
|
||||
- `docs/processing-system-refactor.md` - This documentation
|
||||
|
||||
- `lib/processingStatusManager.ts` - Core processing status management
|
||||
- `check-refactored-pipeline-status.ts` - New pipeline status checker
|
||||
- `migrate-to-refactored-system.ts` - Migration script
|
||||
- `docs/processing-system-refactor.md` - This documentation
|
||||
|
||||
### Updated Files
|
||||
- `prisma/schema.prisma` - Added new processing status tables
|
||||
- `lib/processingScheduler.ts` - Integrated with new status system
|
||||
- `debug-import-status.ts` - Updated to use new system
|
||||
- `fix-import-status.ts` - Updated to use new system
|
||||
|
||||
- `prisma/schema.prisma` - Added new processing status tables
|
||||
- `lib/processingScheduler.ts` - Integrated with new status system
|
||||
- `debug-import-status.ts` - Updated to use new system
|
||||
- `fix-import-status.ts` - Updated to use new system
|
||||
|
||||
### Removed Files
|
||||
- `check-pipeline-status.ts` - Replaced by refactored version
|
||||
|
||||
- `check-pipeline-status.ts` - Replaced by refactored version
|
||||
|
||||
## Benefits Achieved
|
||||
|
||||
1. **Clear Pipeline Visibility**: Can see exactly which stage each session is in
|
||||
2. **Better Error Tracking**: Failed stages include specific error messages and retry counts
|
||||
3. **Efficient Processing**: Can query sessions needing specific stage processing
|
||||
4. **Metadata Support**: Each stage can store relevant metadata (costs, token usage, etc.)
|
||||
5. **Easy Recovery**: Failed sessions can be easily identified and retried
|
||||
6. **Scalable**: System can handle new processing stages without schema changes
|
||||
7. **No Linting Errors**: All TypeScript compilation issues resolved
|
||||
1. **Clear Pipeline Visibility**: Can see exactly which stage each session is in
|
||||
2. **Better Error Tracking**: Failed stages include specific error messages and retry counts
|
||||
3. **Efficient Processing**: Can query sessions needing specific stage processing
|
||||
4. **Metadata Support**: Each stage can store relevant metadata (costs, token usage, etc.)
|
||||
5. **Easy Recovery**: Failed sessions can be easily identified and retried
|
||||
6. **Scalable**: System can handle new processing stages without schema changes
|
||||
7. **No Linting Errors**: All TypeScript compilation issues resolved
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Check Pipeline Status
|
||||
|
||||
```bash
|
||||
npx tsx check-refactored-pipeline-status.ts
|
||||
```
|
||||
|
||||
### Debug Processing Issues
|
||||
|
||||
```bash
|
||||
npx tsx debug-import-status.ts
|
||||
```
|
||||
|
||||
### Fix/Retry Failed Sessions
|
||||
|
||||
```bash
|
||||
npx tsx fix-import-status.ts
|
||||
```
|
||||
|
||||
### Process Sessions
|
||||
|
||||
```bash
|
||||
npx tsx test-ai-processing.ts
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Test AI Processing**: Run AI processing on pending sessions
|
||||
2. **Monitor Performance**: Watch for any issues with the new system
|
||||
3. **Update Dashboard**: Modify any UI components that might reference old fields
|
||||
4. **Documentation**: Update any API documentation that references the old system
|
||||
1. **Test AI Processing**: Run AI processing on pending sessions
|
||||
2. **Monitor Performance**: Watch for any issues with the new system
|
||||
3. **Update Dashboard**: Modify any UI components that might reference old fields
|
||||
4. **Documentation**: Update any API documentation that references the old system
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- All existing data preserved
|
||||
- No data loss during migration
|
||||
- Backward compatibility maintained where possible
|
||||
- System ready for production use
|
||||
- All existing data preserved
|
||||
- No data loss during migration
|
||||
- Backward compatibility maintained where possible
|
||||
- System ready for production use
|
||||
|
||||
The refactored system provides much better visibility into the processing pipeline and makes it easy to identify and resolve any issues that arise during session processing.
|
||||
|
||||
@ -6,11 +6,11 @@ This document describes the extracted scheduler architecture that enables horizo
|
||||
|
||||
The scheduler system has been refactored from a monolithic approach to a service-oriented architecture that supports:
|
||||
|
||||
- **Individual Scheduler Services** - Each scheduler runs as a separate service
|
||||
- **Horizontal Scaling** - Multiple instances of the same scheduler can run across different machines
|
||||
- **Health Monitoring** - Built-in health checks for load balancers and orchestrators
|
||||
- **Graceful Shutdown** - Proper handling of shutdown signals for zero-downtime deployments
|
||||
- **Centralized Management** - Optional scheduler manager for coordinated operations
|
||||
- **Individual Scheduler Services** - Each scheduler runs as a separate service
|
||||
- **Horizontal Scaling** - Multiple instances of the same scheduler can run across different machines
|
||||
- **Health Monitoring** - Built-in health checks for load balancers and orchestrators
|
||||
- **Graceful Shutdown** - Proper handling of shutdown signals for zero-downtime deployments
|
||||
- **Centralized Management** - Optional scheduler manager for coordinated operations
|
||||
|
||||
## Components
|
||||
|
||||
@ -34,11 +34,11 @@ export abstract class BaseSchedulerService extends EventEmitter {
|
||||
|
||||
**Features:**
|
||||
|
||||
- Status management (STOPPED, STARTING, RUNNING, PAUSED, ERROR)
|
||||
- Metrics collection (run counts, timing, success/failure rates)
|
||||
- Event emission for monitoring
|
||||
- Configurable intervals and timeouts
|
||||
- Automatic retry handling
|
||||
- Status management (STOPPED, STARTING, RUNNING, PAUSED, ERROR)
|
||||
- Metrics collection (run counts, timing, success/failure rates)
|
||||
- Event emission for monitoring
|
||||
- Configurable intervals and timeouts
|
||||
- Automatic retry handling
|
||||
|
||||
### 2. Individual Scheduler Services
|
||||
|
||||
@ -57,16 +57,16 @@ const csvScheduler = new CsvImportSchedulerService({
|
||||
|
||||
**Features:**
|
||||
|
||||
- Batch processing with configurable concurrency
|
||||
- Duplicate detection
|
||||
- Company-specific error handling
|
||||
- Progress monitoring
|
||||
- Batch processing with configurable concurrency
|
||||
- Duplicate detection
|
||||
- Company-specific error handling
|
||||
- Progress monitoring
|
||||
|
||||
#### Additional Schedulers (To Be Implemented)
|
||||
|
||||
- `ImportProcessingSchedulerService` - Process imported CSV data into sessions
|
||||
- `SessionProcessingSchedulerService` - AI analysis and categorization
|
||||
- `BatchProcessingSchedulerService` - OpenAI Batch API integration
|
||||
- `ImportProcessingSchedulerService` - Process imported CSV data into sessions
|
||||
- `SessionProcessingSchedulerService` - AI analysis and categorization
|
||||
- `BatchProcessingSchedulerService` - OpenAI Batch API integration
|
||||
|
||||
### 3. SchedulerManager
|
||||
|
||||
@ -88,10 +88,10 @@ await manager.startAll();
|
||||
|
||||
**Features:**
|
||||
|
||||
- Automatic restart of failed critical schedulers
|
||||
- Health monitoring across all schedulers
|
||||
- Coordinated start/stop operations
|
||||
- Event aggregation and logging
|
||||
- Automatic restart of failed critical schedulers
|
||||
- Health monitoring across all schedulers
|
||||
- Coordinated start/stop operations
|
||||
- Event aggregation and logging
|
||||
|
||||
### 4. Standalone Scheduler Runner
|
||||
|
||||
@ -107,10 +107,10 @@ npx tsx lib/services/schedulers/StandaloneSchedulerRunner.ts --list
|
||||
|
||||
**Features:**
|
||||
|
||||
- Independent process execution
|
||||
- Environment variable configuration
|
||||
- Graceful shutdown handling
|
||||
- Health reporting for monitoring
|
||||
- Independent process execution
|
||||
- Environment variable configuration
|
||||
- Graceful shutdown handling
|
||||
- Health reporting for monitoring
|
||||
|
||||
## Deployment Patterns
|
||||
|
||||
@ -127,15 +127,15 @@ await initializeSchedulers();
|
||||
|
||||
**Pros:**
|
||||
|
||||
- Simple deployment
|
||||
- Lower resource usage
|
||||
- Easy local development
|
||||
- Simple deployment
|
||||
- Lower resource usage
|
||||
- Easy local development
|
||||
|
||||
**Cons:**
|
||||
|
||||
- Limited scalability
|
||||
- Single point of failure
|
||||
- Resource contention
|
||||
- Limited scalability
|
||||
- Single point of failure
|
||||
- Resource contention
|
||||
|
||||
### 2. Separate Processes
|
||||
|
||||
@ -154,15 +154,15 @@ npm run scheduler:session-processing
|
||||
|
||||
**Pros:**
|
||||
|
||||
- Independent scaling
|
||||
- Fault isolation
|
||||
- Resource optimization per scheduler
|
||||
- Independent scaling
|
||||
- Fault isolation
|
||||
- Resource optimization per scheduler
|
||||
|
||||
**Cons:**
|
||||
|
||||
- More complex deployment
|
||||
- Higher resource overhead
|
||||
- Inter-process coordination needed
|
||||
- More complex deployment
|
||||
- Higher resource overhead
|
||||
- Inter-process coordination needed
|
||||
|
||||
### 3. Container Orchestration (Recommended for Production)
|
||||
|
||||
@ -193,16 +193,16 @@ services:
|
||||
|
||||
**Pros:**
|
||||
|
||||
- Full horizontal scaling
|
||||
- Independent resource allocation
|
||||
- Health monitoring integration
|
||||
- Zero-downtime deployments
|
||||
- Full horizontal scaling
|
||||
- Independent resource allocation
|
||||
- Health monitoring integration
|
||||
- Zero-downtime deployments
|
||||
|
||||
**Cons:**
|
||||
|
||||
- Complex orchestration setup
|
||||
- Network latency considerations
|
||||
- Distributed system challenges
|
||||
- Complex orchestration setup
|
||||
- Network latency considerations
|
||||
- Distributed system challenges
|
||||
|
||||
## Configuration
|
||||
|
||||
@ -379,46 +379,46 @@ csv-import-scheduler-eu:
|
||||
|
||||
### From Current Architecture
|
||||
|
||||
1. **Phase 1: Extract Schedulers**
|
||||
- ✅ Create BaseSchedulerService
|
||||
- ✅ Implement CsvImportSchedulerService
|
||||
- ✅ Create SchedulerManager
|
||||
- ⏳ Implement remaining scheduler services
|
||||
1. **Phase 1: Extract Schedulers**
|
||||
- ✅ Create BaseSchedulerService
|
||||
- ✅ Implement CsvImportSchedulerService
|
||||
- ✅ Create SchedulerManager
|
||||
- ⏳ Implement remaining scheduler services
|
||||
|
||||
2. **Phase 2: Deployment Options**
|
||||
- ✅ Add ServerSchedulerIntegration for backwards compatibility
|
||||
- ✅ Create StandaloneSchedulerRunner
|
||||
- ✅ Add health check endpoints
|
||||
2. **Phase 2: Deployment Options**
|
||||
- ✅ Add ServerSchedulerIntegration for backwards compatibility
|
||||
- ✅ Create StandaloneSchedulerRunner
|
||||
- ✅ Add health check endpoints
|
||||
|
||||
3. **Phase 3: Container Support**
|
||||
- ⏳ Create Dockerfile for scheduler containers
|
||||
- ⏳ Add Kubernetes manifests
|
||||
- ⏳ Implement distributed coordination
|
||||
3. **Phase 3: Container Support**
|
||||
- ⏳ Create Dockerfile for scheduler containers
|
||||
- ⏳ Add Kubernetes manifests
|
||||
- ⏳ Implement distributed coordination
|
||||
|
||||
4. **Phase 4: Production Migration**
|
||||
- ⏳ Deploy separate scheduler containers
|
||||
- ⏳ Monitor performance and stability
|
||||
- ⏳ Gradually increase horizontal scaling
|
||||
4. **Phase 4: Production Migration**
|
||||
- ⏳ Deploy separate scheduler containers
|
||||
- ⏳ Monitor performance and stability
|
||||
- ⏳ Gradually increase horizontal scaling
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
- Scheduler initialization moved from `server.ts` to `ServerSchedulerIntegration`
|
||||
- Individual scheduler functions replaced with service classes
|
||||
- Configuration moved to environment variables
|
||||
- Scheduler initialization moved from `server.ts` to `ServerSchedulerIntegration`
|
||||
- Individual scheduler functions replaced with service classes
|
||||
- Configuration moved to environment variables
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Scalability**: Independent scaling of different scheduler types
|
||||
2. **Reliability**: Fault isolation prevents cascading failures
|
||||
3. **Performance**: Optimized resource allocation per scheduler
|
||||
4. **Monitoring**: Granular health checks and metrics
|
||||
5. **Deployment**: Zero-downtime updates and rollbacks
|
||||
6. **Development**: Easier testing and debugging of individual schedulers
|
||||
1. **Scalability**: Independent scaling of different scheduler types
|
||||
2. **Reliability**: Fault isolation prevents cascading failures
|
||||
3. **Performance**: Optimized resource allocation per scheduler
|
||||
4. **Monitoring**: Granular health checks and metrics
|
||||
5. **Deployment**: Zero-downtime updates and rollbacks
|
||||
6. **Development**: Easier testing and debugging of individual schedulers
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Implement remaining scheduler services (ImportProcessing, SessionProcessing, BatchProcessing)
|
||||
2. Add distributed coordination for multi-instance schedulers
|
||||
3. Create Kubernetes operators for automatic scaling
|
||||
4. Implement scheduler-specific metrics and dashboards
|
||||
5. Add scheduler performance optimization tools
|
||||
1. Implement remaining scheduler services (ImportProcessing, SessionProcessing, BatchProcessing)
|
||||
2. Add distributed coordination for multi-instance schedulers
|
||||
3. Create Kubernetes operators for automatic scaling
|
||||
4. Implement scheduler-specific metrics and dashboards
|
||||
5. Add scheduler performance optimization tools
|
||||
|
||||
@ -8,8 +8,8 @@
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs
|
||||
- Removed the invalid company record from the database using `fix_companies.js`
|
||||
- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs
|
||||
- Removed the invalid company record from the database using `fix_companies.js`
|
||||
|
||||
### 2. Transcript Fetching Errors
|
||||
|
||||
@ -17,10 +17,10 @@
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Improved error handling in `fetchTranscriptContent()` function
|
||||
- Added probabilistic logging (only ~10% of errors logged) to prevent log spam
|
||||
- Added timeout (10 seconds) for transcript fetching
|
||||
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
|
||||
- Improved error handling in `fetchTranscriptContent()` function
|
||||
- Added probabilistic logging (only ~10% of errors logged) to prevent log spam
|
||||
- Added timeout (10 seconds) for transcript fetching
|
||||
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
|
||||
|
||||
### 3. CSV Fetching Errors
|
||||
|
||||
@ -28,8 +28,8 @@
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Added URL validation to skip companies with `example.com` URLs
|
||||
- Improved error logging to be more descriptive
|
||||
- Added URL validation to skip companies with `example.com` URLs
|
||||
- Improved error logging to be more descriptive
|
||||
|
||||
## Current Status
|
||||
|
||||
@ -42,23 +42,23 @@
|
||||
|
||||
After cleanup, only valid companies remain:
|
||||
|
||||
- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`)
|
||||
- CSV URL: `https://proto.notso.ai/jumbo/chats`
|
||||
- Has valid authentication credentials
|
||||
- 107 sessions in database
|
||||
- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`)
|
||||
- CSV URL: `https://proto.notso.ai/jumbo/chats`
|
||||
- Has valid authentication credentials
|
||||
- 107 sessions in database
|
||||
|
||||
## Files Modified
|
||||
|
||||
1. **lib/csvFetcher.js**
|
||||
|
||||
- Added company URL validation
|
||||
- Improved transcript fetching error handling
|
||||
- Reduced error log verbosity
|
||||
- Added company URL validation
|
||||
- Improved transcript fetching error handling
|
||||
- Reduced error log verbosity
|
||||
|
||||
2. **fix_companies.js** (cleanup script)
|
||||
|
||||
- Removes invalid company records
|
||||
- Can be run again if needed
|
||||
- Removes invalid company records
|
||||
- Can be run again if needed
|
||||
|
||||
## Monitoring
|
||||
|
||||
|
||||
@ -99,12 +99,14 @@ node scripts/manual-triggers.js both
|
||||
```
|
||||
|
||||
2. **If "Sessions with transcript" is 0:**
|
||||
- Sessions exist but transcripts haven't been fetched yet
|
||||
- Run session refresh: `node scripts/manual-triggers.js refresh`
|
||||
|
||||
- Sessions exist but transcripts haven't been fetched yet
|
||||
- Run session refresh: `node scripts/manual-triggers.js refresh`
|
||||
|
||||
3. **If "Ready for processing" is 0 but "Sessions with transcript" > 0:**
|
||||
- All sessions with transcripts have already been processed
|
||||
- Check if `OPENAI_API_KEY` is set in environment
|
||||
|
||||
- All sessions with transcripts have already been processed
|
||||
- Check if `OPENAI_API_KEY` is set in environment
|
||||
|
||||
### Common Issues
|
||||
|
||||
|
||||
@ -12,59 +12,59 @@ The security audit logging system provides comprehensive tracking of security-cr
|
||||
|
||||
The system logs the following event types:
|
||||
|
||||
- **Authentication Events**: Login attempts, password changes, session management
|
||||
- **Authorization Events**: Permission checks, access denied events
|
||||
- **User Management**: User creation, modification, deletion, invitations
|
||||
- **Company Management**: Company suspension, settings changes
|
||||
- **Rate Limiting**: Abuse prevention and rate limit violations
|
||||
- **CSRF Protection**: Cross-site request forgery protection events
|
||||
- **Security Headers**: Security header violations
|
||||
- **Password Reset**: Password reset flows and token validation
|
||||
- **Platform Admin**: Administrative activities by platform users
|
||||
- **Data Privacy**: Data export and privacy-related events
|
||||
- **System Configuration**: System setting changes
|
||||
- **API Security**: API-related security events
|
||||
- **Authentication Events**: Login attempts, password changes, session management
|
||||
- **Authorization Events**: Permission checks, access denied events
|
||||
- **User Management**: User creation, modification, deletion, invitations
|
||||
- **Company Management**: Company suspension, settings changes
|
||||
- **Rate Limiting**: Abuse prevention and rate limit violations
|
||||
- **CSRF Protection**: Cross-site request forgery protection events
|
||||
- **Security Headers**: Security header violations
|
||||
- **Password Reset**: Password reset flows and token validation
|
||||
- **Platform Admin**: Administrative activities by platform users
|
||||
- **Data Privacy**: Data export and privacy-related events
|
||||
- **System Configuration**: System setting changes
|
||||
- **API Security**: API-related security events
|
||||
|
||||
### 2. Structured Logging
|
||||
|
||||
Each audit log entry includes:
|
||||
|
||||
- **Event Type**: Categorizes the security event
|
||||
- **Action**: Specific action performed
|
||||
- **Outcome**: Success, failure, blocked, rate limited, or suspicious
|
||||
- **Severity**: Info, low, medium, high, or critical
|
||||
- **Context**: User ID, company ID, platform user ID, IP address, user agent
|
||||
- **Metadata**: Structured additional information
|
||||
- **Timestamp**: Immutable timestamp for chronological ordering
|
||||
- **Event Type**: Categorizes the security event
|
||||
- **Action**: Specific action performed
|
||||
- **Outcome**: Success, failure, blocked, rate limited, or suspicious
|
||||
- **Severity**: Info, low, medium, high, or critical
|
||||
- **Context**: User ID, company ID, platform user ID, IP address, user agent
|
||||
- **Metadata**: Structured additional information
|
||||
- **Timestamp**: Immutable timestamp for chronological ordering
|
||||
|
||||
### 3. Multi-Tenant Security
|
||||
|
||||
- Company-scoped audit logs ensure data isolation
|
||||
- Platform admin actions tracked separately
|
||||
- Role-based access controls for audit log viewing
|
||||
- Company-scoped audit logs ensure data isolation
|
||||
- Platform admin actions tracked separately
|
||||
- Role-based access controls for audit log viewing
|
||||
|
||||
### 4. Log Retention and Management
|
||||
|
||||
- **Configurable Retention Policies**: Different retention periods based on event type and severity
|
||||
- **Automatic Archival**: Critical and high-severity events archived before deletion
|
||||
- **Scheduled Cleanup**: Weekly automated retention policy execution
|
||||
- **Manual Controls**: Admin interface for manual retention execution
|
||||
- **Configurable Retention Policies**: Different retention periods based on event type and severity
|
||||
- **Automatic Archival**: Critical and high-severity events archived before deletion
|
||||
- **Scheduled Cleanup**: Weekly automated retention policy execution
|
||||
- **Manual Controls**: Admin interface for manual retention execution
|
||||
|
||||
### 5. Administrative Interface
|
||||
|
||||
- **Audit Log Viewer**: Comprehensive filtering and search capabilities
|
||||
- **Retention Management**: View statistics and execute retention policies
|
||||
- **Real-time Monitoring**: Track security events as they occur
|
||||
- **Audit Log Viewer**: Comprehensive filtering and search capabilities
|
||||
- **Retention Management**: View statistics and execute retention policies
|
||||
- **Real-time Monitoring**: Track security events as they occur
|
||||
|
||||
## Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
1. **SecurityAuditLogger** (`lib/securityAuditLogger.ts`): Centralized logging service
|
||||
2. **AuditLogRetentionManager** (`lib/auditLogRetention.ts`): Retention policy management
|
||||
3. **AuditLogScheduler** (`lib/auditLogScheduler.ts`): Scheduled retention execution
|
||||
4. **Admin API** (`app/api/admin/audit-logs/`): REST API for audit log access
|
||||
5. **Admin UI** (`app/dashboard/audit-logs/`): Administrative interface
|
||||
1. **SecurityAuditLogger** (`lib/securityAuditLogger.ts`): Centralized logging service
|
||||
2. **AuditLogRetentionManager** (`lib/auditLogRetention.ts`): Retention policy management
|
||||
3. **AuditLogScheduler** (`lib/auditLogScheduler.ts`): Scheduled retention execution
|
||||
4. **Admin API** (`app/api/admin/audit-logs/`): REST API for audit log access
|
||||
5. **Admin UI** (`app/dashboard/audit-logs/`): Administrative interface
|
||||
|
||||
### Database Schema
|
||||
|
||||
@ -126,18 +126,18 @@ await securityAuditLogger.logAuthorization(
|
||||
|
||||
Administrators can access audit logs through:
|
||||
|
||||
1. **Dashboard UI**: Navigate to "Audit Logs" in the sidebar
|
||||
2. **API Access**: GET `/api/admin/audit-logs` with filtering parameters
|
||||
3. **Retention Management**: GET/POST `/api/admin/audit-logs/retention`
|
||||
1. **Dashboard UI**: Navigate to "Audit Logs" in the sidebar
|
||||
2. **API Access**: GET `/api/admin/audit-logs` with filtering parameters
|
||||
3. **Retention Management**: GET/POST `/api/admin/audit-logs/retention`
|
||||
|
||||
### Filtering Options
|
||||
|
||||
- Event type (authentication, authorization, etc.)
|
||||
- Outcome (success, failure, blocked, etc.)
|
||||
- Severity level (info, low, medium, high, critical)
|
||||
- Date range
|
||||
- User ID
|
||||
- Pagination support
|
||||
- Event type (authentication, authorization, etc.)
|
||||
- Outcome (success, failure, blocked, etc.)
|
||||
- Severity level (info, low, medium, high, critical)
|
||||
- Date range
|
||||
- User ID
|
||||
- Pagination support
|
||||
|
||||
## Configuration
|
||||
|
||||
@ -159,105 +159,105 @@ AUDIT_LOG_RETENTION_DRY_RUN=false
|
||||
|
||||
### Default Retention Policies
|
||||
|
||||
1. **Critical Events**: 7 years retention with archival
|
||||
2. **High Severity Events**: 3 years retention with archival
|
||||
3. **Authentication Events**: 2 years retention with archival
|
||||
4. **Platform Admin Events**: 3 years retention with archival
|
||||
5. **User Management Events**: 2 years retention with archival
|
||||
6. **General Events**: 1 year retention without archival
|
||||
1. **Critical Events**: 7 years retention with archival
|
||||
2. **High Severity Events**: 3 years retention with archival
|
||||
3. **Authentication Events**: 2 years retention with archival
|
||||
4. **Platform Admin Events**: 3 years retention with archival
|
||||
5. **User Management Events**: 2 years retention with archival
|
||||
6. **General Events**: 1 year retention without archival
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Data Protection
|
||||
|
||||
- **IP Address Storage**: Client IP addresses stored for geographic analysis
|
||||
- **Sensitive Data Redaction**: Passwords, tokens, and emails marked as `[REDACTED]`
|
||||
- **Metadata Sanitization**: Complex objects sanitized to prevent data leakage
|
||||
- **IP Address Storage**: Client IP addresses stored for geographic analysis
|
||||
- **Sensitive Data Redaction**: Passwords, tokens, and emails marked as `[REDACTED]`
|
||||
- **Metadata Sanitization**: Complex objects sanitized to prevent data leakage
|
||||
|
||||
### Access Controls
|
||||
|
||||
- **Admin-Only Access**: Only users with `ADMIN` role can view audit logs
|
||||
- **Company Isolation**: Users can only view logs for their own company
|
||||
- **Platform Separation**: Platform admin logs tracked separately
|
||||
- **Admin-Only Access**: Only users with `ADMIN` role can view audit logs
|
||||
- **Company Isolation**: Users can only view logs for their own company
|
||||
- **Platform Separation**: Platform admin logs tracked separately
|
||||
|
||||
### Performance
|
||||
|
||||
- **Async Logging**: All logging operations are asynchronous to avoid blocking
|
||||
- **Error Handling**: Logging failures don't affect application functionality
|
||||
- **Indexed Queries**: Database indexes optimize common query patterns
|
||||
- **Batch Operations**: Retention policies use batch operations for efficiency
|
||||
- **Async Logging**: All logging operations are asynchronous to avoid blocking
|
||||
- **Error Handling**: Logging failures don't affect application functionality
|
||||
- **Indexed Queries**: Database indexes optimize common query patterns
|
||||
- **Batch Operations**: Retention policies use batch operations for efficiency
|
||||
|
||||
## Compliance Features
|
||||
|
||||
### Audit Standards
|
||||
|
||||
- **Immutable Records**: Audit logs cannot be modified after creation
|
||||
- **Chronological Ordering**: Precise timestamps for event sequencing
|
||||
- **Non-Repudiation**: User actions clearly attributed and timestamped
|
||||
- **Comprehensive Coverage**: All security-relevant events logged
|
||||
- **Immutable Records**: Audit logs cannot be modified after creation
|
||||
- **Chronological Ordering**: Precise timestamps for event sequencing
|
||||
- **Non-Repudiation**: User actions clearly attributed and timestamped
|
||||
- **Comprehensive Coverage**: All security-relevant events logged
|
||||
|
||||
### Reporting
|
||||
|
||||
- **Event Statistics**: Summary statistics by event type, severity, and time period
|
||||
- **Export Capabilities**: Structured data export for compliance reporting
|
||||
- **Retention Tracking**: Detailed logging of retention policy execution
|
||||
- **Event Statistics**: Summary statistics by event type, severity, and time period
|
||||
- **Export Capabilities**: Structured data export for compliance reporting
|
||||
- **Retention Tracking**: Detailed logging of retention policy execution
|
||||
|
||||
## Monitoring and Alerting
|
||||
|
||||
### System Health
|
||||
|
||||
- **Scheduler Status**: Monitor retention scheduler health
|
||||
- **Error Tracking**: Log retention and audit logging errors
|
||||
- **Performance Metrics**: Track logging performance and database impact
|
||||
- **Scheduler Status**: Monitor retention scheduler health
|
||||
- **Error Tracking**: Log retention and audit logging errors
|
||||
- **Performance Metrics**: Track logging performance and database impact
|
||||
|
||||
### Security Monitoring
|
||||
|
||||
- **Failed Authentication Patterns**: Track repeated login failures
|
||||
- **Privilege Escalation**: Monitor administrative action patterns
|
||||
- **Suspicious Activity**: Identify unusual access patterns
|
||||
- **Failed Authentication Patterns**: Track repeated login failures
|
||||
- **Privilege Escalation**: Monitor administrative action patterns
|
||||
- **Suspicious Activity**: Identify unusual access patterns
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Audit Logging Disabled**: Check `AUDIT_LOGGING_ENABLED` environment variable
|
||||
2. **Retention Not Running**: Verify `AUDIT_LOG_RETENTION_ENABLED` and scheduler status
|
||||
3. **Access Denied**: Ensure user has `ADMIN` role for audit log access
|
||||
4. **Performance Issues**: Review retention policies and database indexes
|
||||
1. **Audit Logging Disabled**: Check `AUDIT_LOGGING_ENABLED` environment variable
|
||||
2. **Retention Not Running**: Verify `AUDIT_LOG_RETENTION_ENABLED` and scheduler status
|
||||
3. **Access Denied**: Ensure user has `ADMIN` role for audit log access
|
||||
4. **Performance Issues**: Review retention policies and database indexes
|
||||
|
||||
### Debug Information
|
||||
|
||||
- Check application logs for scheduler startup messages
|
||||
- Monitor database query performance for audit log operations
|
||||
- Review retention policy validation warnings
|
||||
- Check application logs for scheduler startup messages
|
||||
- Monitor database query performance for audit log operations
|
||||
- Review retention policy validation warnings
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Implementation
|
||||
|
||||
1. **Always use the centralized logger**: Don't bypass the `securityAuditLogger`
|
||||
2. **Include relevant context**: Provide user, company, and IP information
|
||||
3. **Use appropriate severity levels**: Follow the severity assignment guidelines
|
||||
4. **Sanitize sensitive data**: Use `createAuditMetadata()` for safe metadata
|
||||
1. **Always use the centralized logger**: Don't bypass the `securityAuditLogger`
|
||||
2. **Include relevant context**: Provide user, company, and IP information
|
||||
3. **Use appropriate severity levels**: Follow the severity assignment guidelines
|
||||
4. **Sanitize sensitive data**: Use `createAuditMetadata()` for safe metadata
|
||||
|
||||
### Operations
|
||||
|
||||
1. **Regular retention review**: Monitor retention statistics and adjust policies
|
||||
2. **Archive critical data**: Ensure important logs are archived before deletion
|
||||
3. **Monitor storage usage**: Track audit log database growth
|
||||
4. **Test restoration**: Verify archived data can be restored when needed
|
||||
1. **Regular retention review**: Monitor retention statistics and adjust policies
|
||||
2. **Archive critical data**: Ensure important logs are archived before deletion
|
||||
3. **Monitor storage usage**: Track audit log database growth
|
||||
4. **Test restoration**: Verify archived data can be restored when needed
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
|
||||
- **Real-time Alerting**: Immediate notifications for critical security events
|
||||
- **Advanced Analytics**: ML-based anomaly detection and pattern recognition
|
||||
- **Export Formats**: Additional export formats for compliance reporting
|
||||
- **External Integration**: SIEM and security tool integrations
|
||||
- **Real-time Alerting**: Immediate notifications for critical security events
|
||||
- **Advanced Analytics**: ML-based anomaly detection and pattern recognition
|
||||
- **Export Formats**: Additional export formats for compliance reporting
|
||||
- **External Integration**: SIEM and security tool integrations
|
||||
|
||||
### Performance Optimizations
|
||||
|
||||
- **Log Partitioning**: Database partitioning for improved query performance
|
||||
- **Compression**: Log compression for storage efficiency
|
||||
- **Streaming**: Real-time log streaming for external systems
|
||||
- **Log Partitioning**: Database partitioning for improved query performance
|
||||
- **Compression**: Log compression for storage efficiency
|
||||
- **Streaming**: Real-time log streaming for external systems
|
||||
|
||||
@ -12,33 +12,33 @@ The application implements multiple layers of HTTP security headers to provide d
|
||||
|
||||
#### X-Content-Type-Options: nosniff
|
||||
|
||||
- **Purpose**: Prevents MIME type sniffing attacks
|
||||
- **Protection**: Stops browsers from interpreting files as different MIME types than declared
|
||||
- **Value**: `nosniff`
|
||||
- **Purpose**: Prevents MIME type sniffing attacks
|
||||
- **Protection**: Stops browsers from interpreting files as different MIME types than declared
|
||||
- **Value**: `nosniff`
|
||||
|
||||
#### X-Frame-Options: DENY
|
||||
|
||||
- **Purpose**: Prevents clickjacking attacks
|
||||
- **Protection**: Blocks embedding the site in frames/iframes
|
||||
- **Value**: `DENY`
|
||||
- **Purpose**: Prevents clickjacking attacks
|
||||
- **Protection**: Blocks embedding the site in frames/iframes
|
||||
- **Value**: `DENY`
|
||||
|
||||
#### X-XSS-Protection: 1; mode=block
|
||||
|
||||
- **Purpose**: Enables XSS protection in legacy browsers
|
||||
- **Protection**: Activates built-in XSS filtering (primarily for older browsers)
|
||||
- **Value**: `1; mode=block`
|
||||
- **Purpose**: Enables XSS protection in legacy browsers
|
||||
- **Protection**: Activates built-in XSS filtering (primarily for older browsers)
|
||||
- **Value**: `1; mode=block`
|
||||
|
||||
#### Referrer-Policy: strict-origin-when-cross-origin
|
||||
|
||||
- **Purpose**: Controls referrer information leakage
|
||||
- **Protection**: Limits referrer data sent to external sites
|
||||
- **Value**: `strict-origin-when-cross-origin`
|
||||
- **Purpose**: Controls referrer information leakage
|
||||
- **Protection**: Limits referrer data sent to external sites
|
||||
- **Value**: `strict-origin-when-cross-origin`
|
||||
|
||||
#### X-DNS-Prefetch-Control: off
|
||||
|
||||
- **Purpose**: Prevents DNS rebinding attacks
|
||||
- **Protection**: Disables DNS prefetching to reduce attack surface
|
||||
- **Value**: `off`
|
||||
- **Purpose**: Prevents DNS rebinding attacks
|
||||
- **Protection**: Disables DNS prefetching to reduce attack surface
|
||||
- **Value**: `off`
|
||||
|
||||
### Content Security Policy (CSP)
|
||||
|
||||
@ -48,15 +48,15 @@ Comprehensive CSP implementation with the following directives:
|
||||
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self' https:; frame-ancestors 'none'; base-uri 'self'; form-action 'self'; object-src 'none'; upgrade-insecure-requests
|
||||
```
|
||||
|
||||
#### Key CSP Directives:
|
||||
#### Key CSP Directives
|
||||
|
||||
- **default-src 'self'**: Restrictive default for all resource types
|
||||
- **script-src 'self' 'unsafe-eval' 'unsafe-inline'**: Allows Next.js dev tools and React functionality
|
||||
- **style-src 'self' 'unsafe-inline'**: Enables TailwindCSS and component styles
|
||||
- **img-src 'self' data: https:**: Allows secure image sources
|
||||
- **frame-ancestors 'none'**: Prevents embedding (reinforces X-Frame-Options)
|
||||
- **object-src 'none'**: Blocks dangerous plugins and embeds
|
||||
- **upgrade-insecure-requests**: Automatically upgrades HTTP to HTTPS
|
||||
- **default-src 'self'**: Restrictive default for all resource types
|
||||
- **script-src 'self' 'unsafe-eval' 'unsafe-inline'**: Allows Next.js dev tools and React functionality
|
||||
- **style-src 'self' 'unsafe-inline'**: Enables TailwindCSS and component styles
|
||||
- **img-src 'self' data: https:**: Allows secure image sources
|
||||
- **frame-ancestors 'none'**: Prevents embedding (reinforces X-Frame-Options)
|
||||
- **object-src 'none'**: Blocks dangerous plugins and embeds
|
||||
- **upgrade-insecure-requests**: Automatically upgrades HTTP to HTTPS
|
||||
|
||||
### Permissions Policy
|
||||
|
||||
@ -66,19 +66,19 @@ Controls browser feature access:
|
||||
Permissions-Policy: camera=(), microphone=(), geolocation=(), interest-cohort=(), browsing-topics=()
|
||||
```
|
||||
|
||||
- **camera=()**: Disables camera access
|
||||
- **microphone=()**: Disables microphone access
|
||||
- **geolocation=()**: Disables location tracking
|
||||
- **interest-cohort=()**: Blocks FLoC (privacy protection)
|
||||
- **browsing-topics=()**: Blocks Topics API (privacy protection)
|
||||
- **camera=()**: Disables camera access
|
||||
- **microphone=()**: Disables microphone access
|
||||
- **geolocation=()**: Disables location tracking
|
||||
- **interest-cohort=()**: Blocks FLoC (privacy protection)
|
||||
- **browsing-topics=()**: Blocks Topics API (privacy protection)
|
||||
|
||||
### Strict Transport Security (HSTS)
|
||||
|
||||
**Production Only**: `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload`
|
||||
|
||||
- **max-age=31536000**: 1 year HSTS policy
|
||||
- **includeSubDomains**: Applies to all subdomains
|
||||
- **preload**: Ready for HSTS preload list inclusion
|
||||
- **max-age=31536000**: 1 year HSTS policy
|
||||
- **includeSubDomains**: Applies to all subdomains
|
||||
- **preload**: Ready for HSTS preload list inclusion
|
||||
|
||||
## Configuration
|
||||
|
||||
@ -110,8 +110,8 @@ headers: async () => {
|
||||
|
||||
### Environment-Specific Behavior
|
||||
|
||||
- **Development**: All headers except HSTS
|
||||
- **Production**: All headers including HSTS
|
||||
- **Development**: All headers except HSTS
|
||||
- **Production**: All headers including HSTS
|
||||
|
||||
## Testing
|
||||
|
||||
@ -121,11 +121,11 @@ Location: `tests/unit/http-security-headers.test.ts`
|
||||
|
||||
Tests cover:
|
||||
|
||||
- Individual header validation
|
||||
- CSP directive verification
|
||||
- Permissions Policy validation
|
||||
- Environment-specific configuration
|
||||
- Next.js compatibility checks
|
||||
- Individual header validation
|
||||
- CSP directive verification
|
||||
- Permissions Policy validation
|
||||
- Environment-specific configuration
|
||||
- Next.js compatibility checks
|
||||
|
||||
### Integration Tests
|
||||
|
||||
@ -133,9 +133,9 @@ Location: `tests/integration/security-headers-basic.test.ts`
|
||||
|
||||
Tests cover:
|
||||
|
||||
- Next.js configuration validation
|
||||
- Header generation verification
|
||||
- Environment-based header differences
|
||||
- Next.js configuration validation
|
||||
- Header generation verification
|
||||
- Environment-based header differences
|
||||
|
||||
### Manual Testing
|
||||
|
||||
@ -153,41 +153,41 @@ pnpm test:security-headers https://your-domain.com
|
||||
|
||||
### Protection Against OWASP Top 10
|
||||
|
||||
1. **A03:2021 - Injection**: CSP prevents script injection
|
||||
2. **A05:2021 - Security Misconfiguration**: Comprehensive headers reduce attack surface
|
||||
3. **A06:2021 - Vulnerable Components**: CSP limits execution context
|
||||
4. **A07:2021 - Identification and Authentication Failures**: HSTS prevents downgrade attacks
|
||||
1. **A03:2021 - Injection**: CSP prevents script injection
|
||||
2. **A05:2021 - Security Misconfiguration**: Comprehensive headers reduce attack surface
|
||||
3. **A06:2021 - Vulnerable Components**: CSP limits execution context
|
||||
4. **A07:2021 - Identification and Authentication Failures**: HSTS prevents downgrade attacks
|
||||
|
||||
### Additional Security Benefits
|
||||
|
||||
- **Clickjacking Protection**: X-Frame-Options + CSP frame-ancestors
|
||||
- **MIME Sniffing Prevention**: X-Content-Type-Options
|
||||
- **Information Leakage Reduction**: Referrer-Policy
|
||||
- **Privacy Protection**: Permissions Policy restrictions
|
||||
- **Transport Security**: HSTS enforcement
|
||||
- **Clickjacking Protection**: X-Frame-Options + CSP frame-ancestors
|
||||
- **MIME Sniffing Prevention**: X-Content-Type-Options
|
||||
- **Information Leakage Reduction**: Referrer-Policy
|
||||
- **Privacy Protection**: Permissions Policy restrictions
|
||||
- **Transport Security**: HSTS enforcement
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Regular Reviews
|
||||
|
||||
1. **Quarterly CSP Review**: Analyze CSP violations and tighten policies
|
||||
2. **Annual Header Audit**: Review new security headers and standards
|
||||
3. **Dependency Updates**: Ensure compatibility with framework updates
|
||||
1. **Quarterly CSP Review**: Analyze CSP violations and tighten policies
|
||||
2. **Annual Header Audit**: Review new security headers and standards
|
||||
3. **Dependency Updates**: Ensure compatibility with framework updates
|
||||
|
||||
### Monitoring
|
||||
|
||||
- Monitor CSP violation reports (when implemented)
|
||||
- Use online tools like securityheaders.com for validation
|
||||
- Include security header tests in CI/CD pipeline
|
||||
- Monitor CSP violation reports (when implemented)
|
||||
- Use online tools like securityheaders.com for validation
|
||||
- Include security header tests in CI/CD pipeline
|
||||
|
||||
### Future Enhancements
|
||||
|
||||
Planned improvements:
|
||||
|
||||
1. CSP violation reporting endpoint
|
||||
2. Nonce-based CSP for inline scripts
|
||||
3. Additional Permissions Policy restrictions
|
||||
4. Content-Type validation middleware
|
||||
1. CSP violation reporting endpoint
|
||||
2. Nonce-based CSP for inline scripts
|
||||
3. Additional Permissions Policy restrictions
|
||||
4. Content-Type validation middleware
|
||||
|
||||
## Compatibility
|
||||
|
||||
@ -195,37 +195,37 @@ Planned improvements:
|
||||
|
||||
Headers are configured to be compatible with:
|
||||
|
||||
- Next.js 15+ App Router
|
||||
- React 19 development tools
|
||||
- TailwindCSS 4 styling system
|
||||
- Development hot reload functionality
|
||||
- Next.js 15+ App Router
|
||||
- React 19 development tools
|
||||
- TailwindCSS 4 styling system
|
||||
- Development hot reload functionality
|
||||
|
||||
### Browser Support
|
||||
|
||||
Security headers are supported by:
|
||||
|
||||
- All modern browsers (Chrome 60+, Firefox 60+, Safari 12+)
|
||||
- Graceful degradation for older browsers
|
||||
- Progressive enhancement approach
|
||||
- All modern browsers (Chrome 60+, Firefox 60+, Safari 12+)
|
||||
- Graceful degradation for older browsers
|
||||
- Progressive enhancement approach
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **CSP Violations**: Check browser console for CSP errors
|
||||
2. **Styling Issues**: Verify style-src allows 'unsafe-inline'
|
||||
3. **Script Errors**: Ensure script-src permits necessary scripts
|
||||
4. **Development Issues**: Use `pnpm dev:next-only` to isolate Next.js
|
||||
1. **CSP Violations**: Check browser console for CSP errors
|
||||
2. **Styling Issues**: Verify style-src allows 'unsafe-inline'
|
||||
3. **Script Errors**: Ensure script-src permits necessary scripts
|
||||
4. **Development Issues**: Use `pnpm dev:next-only` to isolate Next.js
|
||||
|
||||
### Debug Tools
|
||||
|
||||
- Browser DevTools Security tab
|
||||
- CSP Evaluator: https://csp-evaluator.withgoogle.com/
|
||||
- Security Headers Scanner: https://securityheaders.com/
|
||||
- Browser DevTools Security tab
|
||||
- CSP Evaluator: <https://csp-evaluator.withgoogle.com/>
|
||||
- Security Headers Scanner: <https://securityheaders.com/>
|
||||
|
||||
## References
|
||||
|
||||
- [OWASP Secure Headers Project](https://owasp.org/www-project-secure-headers/)
|
||||
- [MDN Security Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#security)
|
||||
- [Next.js Security Headers](https://nextjs.org/docs/app/api-reference/config/headers)
|
||||
- [Content Security Policy Reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)
|
||||
- [OWASP Secure Headers Project](https://owasp.org/www-project-secure-headers/)
|
||||
- [MDN Security Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#security)
|
||||
- [Next.js Security Headers](https://nextjs.org/docs/app/api-reference/config/headers)
|
||||
- [Content Security Policy Reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)
|
||||
|
||||
@ -8,39 +8,39 @@ The Security Monitoring and Alerting System provides comprehensive real-time sec
|
||||
|
||||
### Core Components
|
||||
|
||||
1. **Security Monitoring Service** (`lib/securityMonitoring.ts`)
|
||||
- Real-time event processing
|
||||
- Anomaly detection algorithms
|
||||
- Alert generation and management
|
||||
- Security score calculation
|
||||
- Threat level assessment
|
||||
1. **Security Monitoring Service** (`lib/securityMonitoring.ts`)
|
||||
- Real-time event processing
|
||||
- Anomaly detection algorithms
|
||||
- Alert generation and management
|
||||
- Security score calculation
|
||||
- Threat level assessment
|
||||
|
||||
2. **Enhanced Security Logging** (`enhancedSecurityLog`)
|
||||
- Integrates with existing audit logger
|
||||
- Processes events through monitoring system
|
||||
- Triggers immediate threat detection
|
||||
2. **Enhanced Security Logging** (`enhancedSecurityLog`)
|
||||
- Integrates with existing audit logger
|
||||
- Processes events through monitoring system
|
||||
- Triggers immediate threat detection
|
||||
|
||||
3. **API Endpoints** (`app/api/admin/security-monitoring/`)
|
||||
- `/api/admin/security-monitoring` - Main metrics and configuration
|
||||
- `/api/admin/security-monitoring/alerts` - Alert management
|
||||
- `/api/admin/security-monitoring/export` - Data export
|
||||
- `/api/admin/security-monitoring/threat-analysis` - Threat analysis
|
||||
3. **API Endpoints** (`app/api/admin/security-monitoring/`)
|
||||
- `/api/admin/security-monitoring` - Main metrics and configuration
|
||||
- `/api/admin/security-monitoring/alerts` - Alert management
|
||||
- `/api/admin/security-monitoring/export` - Data export
|
||||
- `/api/admin/security-monitoring/threat-analysis` - Threat analysis
|
||||
|
||||
4. **Dashboard UI** (`app/platform/security/page.tsx`)
|
||||
- Real-time security metrics
|
||||
- Active alerts management
|
||||
- Threat analysis visualization
|
||||
- Configuration management
|
||||
4. **Dashboard UI** (`app/platform/security/page.tsx`)
|
||||
- Real-time security metrics
|
||||
- Active alerts management
|
||||
- Threat analysis visualization
|
||||
- Configuration management
|
||||
|
||||
## Features
|
||||
|
||||
### Real-time Monitoring
|
||||
|
||||
- **Authentication Events**: Login attempts, failures, brute force attacks
|
||||
- **Rate Limiting**: Excessive request patterns, API abuse
|
||||
- **Admin Activity**: Unusual administrative actions
|
||||
- **Geographic Anomalies**: Logins from unusual locations
|
||||
- **Temporal Anomalies**: Activity spikes outside normal patterns
|
||||
- **Authentication Events**: Login attempts, failures, brute force attacks
|
||||
- **Rate Limiting**: Excessive request patterns, API abuse
|
||||
- **Admin Activity**: Unusual administrative actions
|
||||
- **Geographic Anomalies**: Logins from unusual locations
|
||||
- **Temporal Anomalies**: Activity spikes outside normal patterns
|
||||
|
||||
### Alert Types
|
||||
|
||||
@ -68,30 +68,30 @@ enum AlertType {
|
||||
|
||||
The system implements several anomaly detection algorithms:
|
||||
|
||||
1. **Geographic Anomaly Detection**
|
||||
- Detects logins from unusual countries
|
||||
- Compares against historical user patterns
|
||||
- Confidence scoring based on deviation
|
||||
1. **Geographic Anomaly Detection**
|
||||
- Detects logins from unusual countries
|
||||
- Compares against historical user patterns
|
||||
- Confidence scoring based on deviation
|
||||
|
||||
2. **Temporal Anomaly Detection**
|
||||
- Identifies activity spikes during unusual hours
|
||||
- Compares current activity to historical averages
|
||||
- Configurable thresholds for different event types
|
||||
2. **Temporal Anomaly Detection**
|
||||
- Identifies activity spikes during unusual hours
|
||||
- Compares current activity to historical averages
|
||||
- Configurable thresholds for different event types
|
||||
|
||||
3. **Behavioral Anomaly Detection**
|
||||
- Multiple failed login attempts
|
||||
- Rapid succession of actions
|
||||
- Pattern deviation analysis
|
||||
3. **Behavioral Anomaly Detection**
|
||||
- Multiple failed login attempts
|
||||
- Rapid succession of actions
|
||||
- Pattern deviation analysis
|
||||
|
||||
### Security Scoring
|
||||
|
||||
The system calculates a real-time security score (0-100) based on:
|
||||
|
||||
- Critical security events (weight: 25)
|
||||
- Active unresolved alerts (weight: 30)
|
||||
- High-severity threats (weight: 20)
|
||||
- Overall event volume (weight: 15)
|
||||
- System stability factors (weight: 10)
|
||||
- Critical security events (weight: 25)
|
||||
- Active unresolved alerts (weight: 30)
|
||||
- High-severity threats (weight: 20)
|
||||
- Overall event volume (weight: 15)
|
||||
- System stability factors (weight: 10)
|
||||
|
||||
### Threat Levels
|
||||
|
||||
@ -255,117 +255,117 @@ await enhancedSecurityLog(
|
||||
|
||||
### Security Overview
|
||||
|
||||
- Real-time security score (0-100)
|
||||
- Current threat level indicator
|
||||
- Active alerts count
|
||||
- Security events summary
|
||||
- Real-time security score (0-100)
|
||||
- Current threat level indicator
|
||||
- Active alerts count
|
||||
- Security events summary
|
||||
|
||||
### Alert Management
|
||||
|
||||
- View active and historical alerts
|
||||
- Filter by severity and type
|
||||
- Acknowledge alerts with tracking
|
||||
- Detailed alert context and metadata
|
||||
- View active and historical alerts
|
||||
- Filter by severity and type
|
||||
- Acknowledge alerts with tracking
|
||||
- Detailed alert context and metadata
|
||||
|
||||
### Threat Analysis
|
||||
|
||||
- Geographic distribution of events
|
||||
- Top threat types and patterns
|
||||
- User risk scoring
|
||||
- IP threat level analysis
|
||||
- Geographic distribution of events
|
||||
- Top threat types and patterns
|
||||
- User risk scoring
|
||||
- IP threat level analysis
|
||||
|
||||
### Configuration Management
|
||||
|
||||
- Adjust detection thresholds
|
||||
- Configure alerting channels
|
||||
- Set data retention policies
|
||||
- Export capabilities
|
||||
- Adjust detection thresholds
|
||||
- Configure alerting channels
|
||||
- Set data retention policies
|
||||
- Export capabilities
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Memory Management
|
||||
|
||||
- Event buffer limited to 1 hour of data
|
||||
- Automatic cleanup of old alerts (configurable)
|
||||
- Efficient in-memory storage for real-time analysis
|
||||
- Event buffer limited to 1 hour of data
|
||||
- Automatic cleanup of old alerts (configurable)
|
||||
- Efficient in-memory storage for real-time analysis
|
||||
|
||||
### Database Impact
|
||||
|
||||
- Leverages existing audit log indexes
|
||||
- Optimized queries for time-range filtering
|
||||
- Background processing to avoid blocking operations
|
||||
- Leverages existing audit log indexes
|
||||
- Optimized queries for time-range filtering
|
||||
- Background processing to avoid blocking operations
|
||||
|
||||
### Scalability
|
||||
|
||||
- Stateless architecture (except for buffering)
|
||||
- Horizontal scaling support
|
||||
- Configurable processing intervals
|
||||
- Stateless architecture (except for buffering)
|
||||
- Horizontal scaling support
|
||||
- Configurable processing intervals
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Access Control
|
||||
|
||||
- Platform admin authentication required
|
||||
- Role-based access to security endpoints
|
||||
- Audit logging of all monitoring activities
|
||||
- Platform admin authentication required
|
||||
- Role-based access to security endpoints
|
||||
- Audit logging of all monitoring activities
|
||||
|
||||
### Data Privacy
|
||||
|
||||
- Sensitive data redaction in logs
|
||||
- IP address anonymization options
|
||||
- Configurable data retention periods
|
||||
- Sensitive data redaction in logs
|
||||
- IP address anonymization options
|
||||
- Configurable data retention periods
|
||||
|
||||
### Alert Suppression
|
||||
|
||||
- Duplicate alert suppression (configurable window)
|
||||
- Rate limiting on alert generation
|
||||
- Escalation policies for critical threats
|
||||
- Duplicate alert suppression (configurable window)
|
||||
- Rate limiting on alert generation
|
||||
- Escalation policies for critical threats
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### Health Checks
|
||||
|
||||
- Monitor service availability
|
||||
- Check alert generation pipeline
|
||||
- Verify data export functionality
|
||||
- Monitor service availability
|
||||
- Check alert generation pipeline
|
||||
- Verify data export functionality
|
||||
|
||||
### Regular Tasks
|
||||
|
||||
- Review and adjust thresholds quarterly
|
||||
- Analyze false positive rates
|
||||
- Update threat detection patterns
|
||||
- Clean up old alert data
|
||||
- Review and adjust thresholds quarterly
|
||||
- Analyze false positive rates
|
||||
- Update threat detection patterns
|
||||
- Clean up old alert data
|
||||
|
||||
### Performance Metrics
|
||||
|
||||
- Alert response time
|
||||
- False positive/negative rates
|
||||
- System resource usage
|
||||
- User engagement with alerts
|
||||
- Alert response time
|
||||
- False positive/negative rates
|
||||
- System resource usage
|
||||
- User engagement with alerts
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
|
||||
1. **Machine Learning Integration**
|
||||
- Behavioral pattern recognition
|
||||
- Adaptive threshold adjustment
|
||||
- Predictive threat modeling
|
||||
1. **Machine Learning Integration**
|
||||
- Behavioral pattern recognition
|
||||
- Adaptive threshold adjustment
|
||||
- Predictive threat modeling
|
||||
|
||||
2. **Advanced Analytics**
|
||||
- Threat intelligence integration
|
||||
- Cross-correlation analysis
|
||||
- Risk trend analysis
|
||||
2. **Advanced Analytics**
|
||||
- Threat intelligence integration
|
||||
- Cross-correlation analysis
|
||||
- Risk trend analysis
|
||||
|
||||
3. **Integration Enhancements**
|
||||
- SIEM system connectors
|
||||
- Webhook customization
|
||||
- Mobile app notifications
|
||||
3. **Integration Enhancements**
|
||||
- SIEM system connectors
|
||||
- Webhook customization
|
||||
- Mobile app notifications
|
||||
|
||||
4. **Automated Response**
|
||||
- IP blocking automation
|
||||
- Account suspension workflows
|
||||
- Incident response orchestration
|
||||
4. **Automated Response**
|
||||
- IP blocking automation
|
||||
- Account suspension workflows
|
||||
- Incident response orchestration
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@ -373,27 +373,27 @@ await enhancedSecurityLog(
|
||||
|
||||
**High False Positive Rate**
|
||||
|
||||
- Review and adjust detection thresholds
|
||||
- Analyze user behavior patterns
|
||||
- Consider geographical variations
|
||||
- Review and adjust detection thresholds
|
||||
- Analyze user behavior patterns
|
||||
- Consider geographical variations
|
||||
|
||||
**Missing Alerts**
|
||||
|
||||
- Check service configuration
|
||||
- Verify audit log integration
|
||||
- Review threshold settings
|
||||
- Check service configuration
|
||||
- Verify audit log integration
|
||||
- Review threshold settings
|
||||
|
||||
**Performance Issues**
|
||||
|
||||
- Monitor memory usage
|
||||
- Adjust cleanup intervals
|
||||
- Optimize database queries
|
||||
- Monitor memory usage
|
||||
- Adjust cleanup intervals
|
||||
- Optimize database queries
|
||||
|
||||
**Export Failures**
|
||||
|
||||
- Check file permissions
|
||||
- Verify date range validity
|
||||
- Monitor server resources
|
||||
- Check file permissions
|
||||
- Verify date range validity
|
||||
- Monitor server resources
|
||||
|
||||
### Debugging
|
||||
|
||||
@ -419,24 +419,24 @@ console.log("Active alerts:", alerts.length);
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- Alert generation logic
|
||||
- Anomaly detection algorithms
|
||||
- Configuration management
|
||||
- Data export functionality
|
||||
- Alert generation logic
|
||||
- Anomaly detection algorithms
|
||||
- Configuration management
|
||||
- Data export functionality
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- API endpoint security
|
||||
- Database integration
|
||||
- Real-time event processing
|
||||
- Alert acknowledgment flow
|
||||
- API endpoint security
|
||||
- Database integration
|
||||
- Real-time event processing
|
||||
- Alert acknowledgment flow
|
||||
|
||||
### Load Testing
|
||||
|
||||
- High-volume event processing
|
||||
- Concurrent alert generation
|
||||
- Database performance under load
|
||||
- Memory usage patterns
|
||||
- High-volume event processing
|
||||
- Concurrent alert generation
|
||||
- Database performance under load
|
||||
- Memory usage patterns
|
||||
|
||||
Run tests:
|
||||
|
||||
|
||||
@ -8,54 +8,54 @@ This document outlines the comprehensive Content Security Policy implementation
|
||||
|
||||
The enhanced CSP implementation provides:
|
||||
|
||||
- **Nonce-based script execution** for maximum security in production
|
||||
- **Strict mode policies** with configurable external domain allowlists
|
||||
- **Environment-specific configurations** for development vs production
|
||||
- **CSP violation reporting and monitoring** system with real-time analysis
|
||||
- **Advanced bypass detection and alerting** capabilities with risk assessment
|
||||
- **Comprehensive testing framework** with automated validation
|
||||
- **Performance metrics and policy recommendations**
|
||||
- **Framework compatibility** with Next.js, TailwindCSS, and Leaflet maps
|
||||
- **Nonce-based script execution** for maximum security in production
|
||||
- **Strict mode policies** with configurable external domain allowlists
|
||||
- **Environment-specific configurations** for development vs production
|
||||
- **CSP violation reporting and monitoring** system with real-time analysis
|
||||
- **Advanced bypass detection and alerting** capabilities with risk assessment
|
||||
- **Comprehensive testing framework** with automated validation
|
||||
- **Performance metrics and policy recommendations**
|
||||
- **Framework compatibility** with Next.js, TailwindCSS, and Leaflet maps
|
||||
|
||||
## Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
1. **CSP Utility Library** (`lib/csp.ts`)
|
||||
- Nonce generation with cryptographic security
|
||||
- Dynamic CSP building based on environment
|
||||
- Violation parsing and bypass detection
|
||||
- Policy validation and testing
|
||||
1. **CSP Utility Library** (`lib/csp.ts`)
|
||||
- Nonce generation with cryptographic security
|
||||
- Dynamic CSP building based on environment
|
||||
- Violation parsing and bypass detection
|
||||
- Policy validation and testing
|
||||
|
||||
2. **Middleware Implementation** (`middleware.ts`)
|
||||
- Automatic nonce generation per request
|
||||
- Environment-aware policy application
|
||||
- Enhanced security headers
|
||||
- Route-based CSP filtering
|
||||
2. **Middleware Implementation** (`middleware.ts`)
|
||||
- Automatic nonce generation per request
|
||||
- Environment-aware policy application
|
||||
- Enhanced security headers
|
||||
- Route-based CSP filtering
|
||||
|
||||
3. **Violation Reporting** (`app/api/csp-report/route.ts`)
|
||||
- Real-time violation monitoring with intelligent analysis
|
||||
- Rate-limited endpoint protection (10 reports/minute per IP)
|
||||
- Advanced bypass attempt detection with risk assessment
|
||||
- Automated alerting for critical violations with recommendations
|
||||
3. **Violation Reporting** (`app/api/csp-report/route.ts`)
|
||||
- Real-time violation monitoring with intelligent analysis
|
||||
- Rate-limited endpoint protection (10 reports/minute per IP)
|
||||
- Advanced bypass attempt detection with risk assessment
|
||||
- Automated alerting for critical violations with recommendations
|
||||
|
||||
4. **Monitoring Service** (`lib/csp-monitoring.ts`)
|
||||
- Violation tracking and metrics collection
|
||||
- Policy recommendation engine based on violation patterns
|
||||
- Export capabilities for external analysis (JSON/CSV)
|
||||
- Automatic cleanup of old violation data
|
||||
4. **Monitoring Service** (`lib/csp-monitoring.ts`)
|
||||
- Violation tracking and metrics collection
|
||||
- Policy recommendation engine based on violation patterns
|
||||
- Export capabilities for external analysis (JSON/CSV)
|
||||
- Automatic cleanup of old violation data
|
||||
|
||||
5. **Metrics API** (`app/api/csp-metrics/route.ts`)
|
||||
- Real-time CSP violation metrics (1h, 6h, 24h, 7d, 30d ranges)
|
||||
- Top violated directives and blocked URIs analysis
|
||||
- Violation trend tracking and visualization data
|
||||
- Policy optimization recommendations
|
||||
5. **Metrics API** (`app/api/csp-metrics/route.ts`)
|
||||
- Real-time CSP violation metrics (1h, 6h, 24h, 7d, 30d ranges)
|
||||
- Top violated directives and blocked URIs analysis
|
||||
- Violation trend tracking and visualization data
|
||||
- Policy optimization recommendations
|
||||
|
||||
6. **Testing Framework**
|
||||
- Comprehensive unit and integration tests
|
||||
- Enhanced CSP validation tools with security scoring
|
||||
- Automated compliance verification
|
||||
- Real-world scenario testing for application compatibility
|
||||
6. **Testing Framework**
|
||||
- Comprehensive unit and integration tests
|
||||
- Enhanced CSP validation tools with security scoring
|
||||
- Automated compliance verification
|
||||
- Real-world scenario testing for application compatibility
|
||||
|
||||
## CSP Policies
|
||||
|
||||
@ -118,9 +118,9 @@ const developmentCSP = {
|
||||
|
||||
### 1. Nonce-Based Script Execution
|
||||
|
||||
- **128-bit cryptographically secure nonces** generated per request
|
||||
- **Strict-dynamic policy** prevents inline script execution
|
||||
- **Automatic nonce injection** into layout components
|
||||
- **128-bit cryptographically secure nonces** generated per request
|
||||
- **Strict-dynamic policy** prevents inline script execution
|
||||
- **Automatic nonce injection** into layout components
|
||||
|
||||
```tsx
|
||||
// Layout with nonce support
|
||||
@ -149,27 +149,32 @@ export default async function RootLayout({ children }: { children: ReactNode })
|
||||
### 2. Content Source Restrictions
|
||||
|
||||
#### Script Sources
|
||||
- **Production**: Only `'self'` and nonce-approved scripts
|
||||
- **Development**: Additional `'unsafe-eval'` for dev tools
|
||||
- **Blocked**: All external CDNs, inline scripts without nonce
|
||||
|
||||
- **Production**: Only `'self'` and nonce-approved scripts
|
||||
- **Development**: Additional `'unsafe-eval'` for dev tools
|
||||
- **Blocked**: All external CDNs, inline scripts without nonce
|
||||
|
||||
#### Style Sources
|
||||
- **Production**: Nonce-based inline styles preferred
|
||||
- **Fallback**: `'unsafe-inline'` for TailwindCSS compatibility
|
||||
- **External**: Only self-hosted stylesheets
|
||||
|
||||
- **Production**: Nonce-based inline styles preferred
|
||||
- **Fallback**: `'unsafe-inline'` for TailwindCSS compatibility
|
||||
- **External**: Only self-hosted stylesheets
|
||||
|
||||
#### Image Sources
|
||||
- **Allowed**: Self, data URIs, schema.org, application domain
|
||||
- **Blocked**: All other external domains
|
||||
|
||||
- **Allowed**: Self, data URIs, schema.org, application domain
|
||||
- **Blocked**: All other external domains
|
||||
|
||||
#### Connection Sources
|
||||
- **Production**: Self, OpenAI API, application domain
|
||||
- **Development**: Additional WebSocket for HMR
|
||||
- **Blocked**: All other external connections
|
||||
|
||||
- **Production**: Self, OpenAI API, application domain
|
||||
- **Development**: Additional WebSocket for HMR
|
||||
- **Blocked**: All other external connections
|
||||
|
||||
### 3. XSS Protection Mechanisms
|
||||
|
||||
#### Inline Script Prevention
|
||||
|
||||
```javascript
|
||||
// Blocked by CSP
|
||||
<script>alert('xss')</script>
|
||||
@ -179,18 +184,21 @@ export default async function RootLayout({ children }: { children: ReactNode })
|
||||
```
|
||||
|
||||
#### Object Injection Prevention
|
||||
|
||||
```javascript
|
||||
// Completely blocked
|
||||
object-src 'none'
|
||||
```
|
||||
|
||||
#### Base Tag Injection Prevention
|
||||
|
||||
```javascript
|
||||
// Restricted to same origin
|
||||
base-uri 'self'
|
||||
```
|
||||
|
||||
#### Clickjacking Protection
|
||||
|
||||
```javascript
|
||||
// No framing allowed
|
||||
frame-ancestors 'none'
|
||||
@ -230,20 +238,21 @@ CSP violations are automatically reported to `/api/csp-report`:
|
||||
|
||||
### Violation Processing
|
||||
|
||||
1. **Rate Limiting**: 10 reports per minute per IP
|
||||
2. **Parsing**: Extract violation details and context
|
||||
3. **Risk Assessment**: Classify as low/medium/high risk
|
||||
4. **Bypass Detection**: Check for known attack patterns
|
||||
5. **Alerting**: Immediate notifications for critical violations
|
||||
1. **Rate Limiting**: 10 reports per minute per IP
|
||||
2. **Parsing**: Extract violation details and context
|
||||
3. **Risk Assessment**: Classify as low/medium/high risk
|
||||
4. **Bypass Detection**: Check for known attack patterns
|
||||
5. **Alerting**: Immediate notifications for critical violations
|
||||
|
||||
### Monitoring Dashboard
|
||||
|
||||
Violations are logged with:
|
||||
- Timestamp and source IP
|
||||
- User agent and referer
|
||||
- Violation type and blocked content
|
||||
- Risk level and bypass indicators
|
||||
- Response actions taken
|
||||
|
||||
- Timestamp and source IP
|
||||
- User agent and referer
|
||||
- Violation type and blocked content
|
||||
- Risk level and bypass indicators
|
||||
- Response actions taken
|
||||
|
||||
## Testing and Validation
|
||||
|
||||
@ -262,19 +271,20 @@ pnpm test:csp:full
|
||||
|
||||
### Manual Testing
|
||||
|
||||
1. **Nonce Validation**: Verify unique nonces per request
|
||||
2. **Policy Compliance**: Check all required directives
|
||||
3. **Bypass Resistance**: Test common XSS techniques
|
||||
4. **Framework Compatibility**: Ensure Next.js/TailwindCSS work
|
||||
5. **Performance Impact**: Measure overhead
|
||||
1. **Nonce Validation**: Verify unique nonces per request
|
||||
2. **Policy Compliance**: Check all required directives
|
||||
3. **Bypass Resistance**: Test common XSS techniques
|
||||
4. **Framework Compatibility**: Ensure Next.js/TailwindCSS work
|
||||
5. **Performance Impact**: Measure overhead
|
||||
|
||||
### Security Scoring
|
||||
|
||||
The validation framework provides a security score:
|
||||
- **90-100%**: Excellent implementation
|
||||
- **80-89%**: Good with minor improvements needed
|
||||
- **70-79%**: Needs attention
|
||||
- **<70%**: Serious security issues
|
||||
|
||||
- **90-100%**: Excellent implementation
|
||||
- **80-89%**: Good with minor improvements needed
|
||||
- **70-79%**: Needs attention
|
||||
- **<70%**: Serious security issues
|
||||
|
||||
## Deployment Considerations
|
||||
|
||||
@ -288,74 +298,74 @@ NODE_ENV=development # Enables permissive CSP
|
||||
|
||||
### Performance Impact
|
||||
|
||||
- **Nonce generation**: ~0.1ms per request
|
||||
- **Header processing**: ~0.05ms per request
|
||||
- **Total overhead**: <1ms per request
|
||||
- **Nonce generation**: ~0.1ms per request
|
||||
- **Header processing**: ~0.05ms per request
|
||||
- **Total overhead**: <1ms per request
|
||||
|
||||
### Browser Compatibility
|
||||
|
||||
- **Modern browsers**: Full CSP Level 3 support
|
||||
- **Legacy browsers**: Graceful degradation with X-XSS-Protection
|
||||
- **Reporting**: Supported in all major browsers
|
||||
- **Modern browsers**: Full CSP Level 3 support
|
||||
- **Legacy browsers**: Graceful degradation with X-XSS-Protection
|
||||
- **Reporting**: Supported in all major browsers
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Regular Reviews
|
||||
|
||||
1. **Monthly**: Review violation reports and patterns
|
||||
2. **Quarterly**: Update content source restrictions
|
||||
3. **Per release**: Validate CSP with new features
|
||||
4. **Annually**: Security audit and penetration testing
|
||||
1. **Monthly**: Review violation reports and patterns
|
||||
2. **Quarterly**: Update content source restrictions
|
||||
3. **Per release**: Validate CSP with new features
|
||||
4. **Annually**: Security audit and penetration testing
|
||||
|
||||
### Updates and Modifications
|
||||
|
||||
When adding new content sources:
|
||||
|
||||
1. Update `buildCSP()` function in `lib/csp.ts`
|
||||
2. Add tests for new directives
|
||||
3. Validate security impact
|
||||
4. Update documentation
|
||||
1. Update `buildCSP()` function in `lib/csp.ts`
|
||||
2. Add tests for new directives
|
||||
3. Validate security impact
|
||||
4. Update documentation
|
||||
|
||||
### Incident Response
|
||||
|
||||
For CSP violations:
|
||||
|
||||
1. **High-risk violations**: Immediate investigation
|
||||
2. **Bypass attempts**: Security team notification
|
||||
3. **Mass violations**: Check for policy issues
|
||||
4. **False positives**: Adjust policies as needed
|
||||
1. **High-risk violations**: Immediate investigation
|
||||
2. **Bypass attempts**: Security team notification
|
||||
3. **Mass violations**: Check for policy issues
|
||||
4. **False positives**: Adjust policies as needed
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Development
|
||||
|
||||
- Always test CSP changes in development first
|
||||
- Use nonce provider for new inline scripts
|
||||
- Validate external resources before adding
|
||||
- Monitor console for CSP violations
|
||||
- Always test CSP changes in development first
|
||||
- Use nonce provider for new inline scripts
|
||||
- Validate external resources before adding
|
||||
- Monitor console for CSP violations
|
||||
|
||||
### Production
|
||||
|
||||
- Never disable CSP in production
|
||||
- Monitor violation rates and patterns
|
||||
- Keep nonce generation entropy high
|
||||
- Regular security audits
|
||||
- Never disable CSP in production
|
||||
- Monitor violation rates and patterns
|
||||
- Keep nonce generation entropy high
|
||||
- Regular security audits
|
||||
|
||||
### Code Review
|
||||
|
||||
- Check all inline scripts have nonce
|
||||
- Verify external resources are approved
|
||||
- Ensure CSP tests pass
|
||||
- Document any policy changes
|
||||
- Check all inline scripts have nonce
|
||||
- Verify external resources are approved
|
||||
- Ensure CSP tests pass
|
||||
- Document any policy changes
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Inline styles blocked**: Use nonce or move to external CSS
|
||||
2. **Third-party scripts blocked**: Add to approved sources
|
||||
3. **Dev tools not working**: Ensure development CSP allows unsafe-eval
|
||||
4. **Images not loading**: Check image source restrictions
|
||||
1. **Inline styles blocked**: Use nonce or move to external CSS
|
||||
2. **Third-party scripts blocked**: Add to approved sources
|
||||
3. **Dev tools not working**: Ensure development CSP allows unsafe-eval
|
||||
4. **Images not loading**: Check image source restrictions
|
||||
|
||||
### Debug Tools
|
||||
|
||||
@ -374,19 +384,19 @@ curl -X POST /api/csp-report -d '{"csp-report": {...}}'
|
||||
|
||||
If CSP breaks production:
|
||||
|
||||
1. Check violation reports for patterns
|
||||
2. Identify blocking directive
|
||||
3. Test fix in staging environment
|
||||
4. Deploy emergency policy update
|
||||
5. Monitor for resolved issues
|
||||
1. Check violation reports for patterns
|
||||
2. Identify blocking directive
|
||||
3. Test fix in staging environment
|
||||
4. Deploy emergency policy update
|
||||
5. Monitor for resolved issues
|
||||
|
||||
## Compliance
|
||||
|
||||
This CSP implementation addresses:
|
||||
|
||||
- **OWASP Top 10**: XSS prevention
|
||||
- **CSP Level 3**: Modern security standards
|
||||
- **GDPR**: Privacy-preserving monitoring
|
||||
- **SOC 2**: Security controls documentation
|
||||
- **OWASP Top 10**: XSS prevention
|
||||
- **CSP Level 3**: Modern security standards
|
||||
- **GDPR**: Privacy-preserving monitoring
|
||||
- **SOC 2**: Security controls documentation
|
||||
|
||||
The enhanced CSP provides defense-in-depth against XSS attacks while maintaining application functionality and performance.
|
||||
The enhanced CSP provides defense-in-depth against XSS attacks while maintaining application functionality and performance.
|
||||
|
||||
@ -15,22 +15,22 @@ The system now includes an automated process for analyzing chat session transcri
|
||||
|
||||
### Session Fetching
|
||||
|
||||
- The system fetches session data from configured CSV URLs for each company
|
||||
- Unlike the previous implementation, it now only adds sessions that don't already exist in the database
|
||||
- This prevents duplicate sessions and allows for incremental updates
|
||||
- The system fetches session data from configured CSV URLs for each company
|
||||
- Unlike the previous implementation, it now only adds sessions that don't already exist in the database
|
||||
- This prevents duplicate sessions and allows for incremental updates
|
||||
|
||||
### Transcript Processing
|
||||
|
||||
- For sessions with transcript content that haven't been processed yet, the system calls OpenAI's API
|
||||
- The API analyzes the transcript and extracts the following information:
|
||||
- Primary language used (ISO 639-1 code)
|
||||
- Number of messages sent by the user
|
||||
- Overall sentiment (positive, neutral, negative)
|
||||
- Whether the conversation was escalated
|
||||
- Whether HR contact was mentioned or provided
|
||||
- Best-fitting category for the conversation
|
||||
- Up to 5 paraphrased questions asked by the user
|
||||
- A brief summary of the conversation
|
||||
- For sessions with transcript content that haven't been processed yet, the system calls OpenAI's API
|
||||
- The API analyzes the transcript and extracts the following information:
|
||||
- Primary language used (ISO 639-1 code)
|
||||
- Number of messages sent by the user
|
||||
- Overall sentiment (positive, neutral, negative)
|
||||
- Whether the conversation was escalated
|
||||
- Whether HR contact was mentioned or provided
|
||||
- Best-fitting category for the conversation
|
||||
- Up to 5 paraphrased questions asked by the user
|
||||
- A brief summary of the conversation
|
||||
|
||||
### Scheduling
|
||||
|
||||
@ -43,10 +43,10 @@ The system includes two schedulers:
|
||||
|
||||
The Session model has been updated with new fields to store the processed data:
|
||||
|
||||
- `processed`: Boolean flag indicating whether the session has been processed
|
||||
- `sentimentCategory`: String value ("positive", "neutral", "negative") from OpenAI
|
||||
- `questions`: JSON array of questions asked by the user
|
||||
- `summary`: Brief summary of the conversation
|
||||
- `processed`: Boolean flag indicating whether the session has been processed
|
||||
- `sentimentCategory`: String value ("positive", "neutral", "negative") from OpenAI
|
||||
- `questions`: JSON array of questions asked by the user
|
||||
- `summary`: Brief summary of the conversation
|
||||
|
||||
## Configuration
|
||||
|
||||
@ -62,9 +62,9 @@ OPENAI_API_KEY=your_api_key_here
|
||||
|
||||
To run the application with schedulers enabled:
|
||||
|
||||
- Development: `npm run dev`
|
||||
- Development (with schedulers disabled): `npm run dev:no-schedulers`
|
||||
- Production: `npm run start`
|
||||
- Development: `npm run dev`
|
||||
- Development (with schedulers disabled): `npm run dev:no-schedulers`
|
||||
- Production: `npm run start`
|
||||
|
||||
Note: These commands will start a custom Next.js server with the schedulers enabled. You'll need to have an OpenAI API key set in your `.env.local` file for the session processing to work.
|
||||
|
||||
@ -82,5 +82,5 @@ This will process all unprocessed sessions that have transcript content.
|
||||
|
||||
The processing logic can be customized by modifying:
|
||||
|
||||
- `lib/processingScheduler.ts`: Contains the OpenAI processing logic
|
||||
- `scripts/process_sessions.ts`: Standalone script for manual processing
|
||||
- `lib/processingScheduler.ts`: Contains the OpenAI processing logic
|
||||
- `scripts/process_sessions.ts`: Standalone script for manual processing
|
||||
|
||||
@ -5,9 +5,11 @@ This document outlines the fixes applied to resolve TypeScript compilation error
|
||||
## Issues Resolved
|
||||
|
||||
### 1. Missing Type Imports
|
||||
|
||||
**Problem:** `lib/api/index.ts` was missing required type imports
|
||||
**Error:** `Cannot find name 'APIHandler'`, `Cannot find name 'Permission'`
|
||||
**Fix:** Added proper imports at the top of the file
|
||||
|
||||
```typescript
|
||||
import type { APIContext, APIHandler, APIHandlerOptions } from "./handler";
|
||||
import { createAPIHandler } from "./handler";
|
||||
@ -15,9 +17,11 @@ import { Permission, createPermissionChecker } from "./authorization";
|
||||
```
|
||||
|
||||
### 2. Zod API Breaking Change
|
||||
|
||||
**Problem:** Zod error property name changed from `errors` to `issues`
|
||||
**Error:** `Property 'errors' does not exist on type 'ZodError'`
|
||||
**Fix:** Updated all references to use `error.issues` instead of `error.errors`
|
||||
|
||||
```typescript
|
||||
// Before
|
||||
error.errors.map((e) => `${e.path.join(".")}: ${e.message}`)
|
||||
@ -26,17 +30,21 @@ error.issues.map((e) => `${e.path.join(".")}: ${e.message}`)
|
||||
```
|
||||
|
||||
### 3. Missing LRU Cache Dependency
|
||||
|
||||
**Problem:** `lru-cache` package was missing from dependencies
|
||||
**Error:** `Cannot find module 'lru-cache'`
|
||||
**Fix:** Installed the missing dependency
|
||||
|
||||
```bash
|
||||
pnpm add lru-cache
|
||||
```
|
||||
|
||||
### 4. LRU Cache Generic Type Constraints
|
||||
|
||||
**Problem:** TypeScript generic constraints not satisfied
|
||||
**Error:** `Type 'K' does not satisfy the constraint '{}'`
|
||||
**Fix:** Added proper generic type constraints
|
||||
|
||||
```typescript
|
||||
// Before
|
||||
<K = string, V = any>
|
||||
@ -45,9 +53,11 @@ pnpm add lru-cache
|
||||
```
|
||||
|
||||
### 5. Map Iteration ES5 Compatibility
|
||||
|
||||
**Problem:** Map iteration requires downlevel iteration flag
|
||||
**Error:** `can only be iterated through when using the '--downlevelIteration' flag`
|
||||
**Fix:** Used `Array.from()` pattern for compatibility
|
||||
|
||||
```typescript
|
||||
// Before
|
||||
for (const [key, value] of map) { ... }
|
||||
@ -56,9 +66,11 @@ for (const [key, value] of Array.from(map.entries())) { ... }
|
||||
```
|
||||
|
||||
### 6. Redis Configuration Issues
|
||||
|
||||
**Problem:** Invalid Redis socket options
|
||||
**Error:** Redis connection failed with unsupported options
|
||||
**Fix:** Simplified Redis configuration to only include supported options
|
||||
|
||||
```typescript
|
||||
this.client = createClient({
|
||||
url: env.REDIS_URL,
|
||||
@ -69,9 +81,11 @@ this.client = createClient({
|
||||
```
|
||||
|
||||
### 7. Prisma Relationship Naming Mismatches
|
||||
|
||||
**Problem:** Code referenced non-existent Prisma relationships
|
||||
**Error:** `securityAuditLogs` and `sessionImport` don't exist
|
||||
**Fix:** Used correct relationship names
|
||||
|
||||
```typescript
|
||||
// Before
|
||||
user.securityAuditLogs
|
||||
@ -82,17 +96,21 @@ session.import
|
||||
```
|
||||
|
||||
### 8. Missing Schema Fields
|
||||
|
||||
**Problem:** Code referenced fields that don't exist in the database schema
|
||||
**Error:** `Property 'userId' does not exist on type`
|
||||
**Fix:** Applied type casting where schema fields were missing
|
||||
|
||||
```typescript
|
||||
userId: (session as any).userId || null
|
||||
```
|
||||
|
||||
### 9. Deprecated Package Dependencies
|
||||
|
||||
**Problem:** `critters` package is deprecated and caused build failures
|
||||
**Error:** `Cannot find module 'critters'`
|
||||
**Fix:** Disabled CSS optimization feature that required critters
|
||||
|
||||
```javascript
|
||||
experimental: {
|
||||
optimizeCss: false, // Disabled due to critters dependency
|
||||
@ -100,9 +118,11 @@ experimental: {
|
||||
```
|
||||
|
||||
### 10. ESLint vs Biome Conflict
|
||||
|
||||
**Problem:** ESLint warnings treated as build errors
|
||||
**Error:** Build failed due to linting warnings
|
||||
**Fix:** Disabled ESLint during build since Biome is used for linting
|
||||
|
||||
```javascript
|
||||
eslint: {
|
||||
ignoreDuringBuilds: true,
|
||||
@ -112,6 +132,7 @@ eslint: {
|
||||
## Schema Enhancements
|
||||
|
||||
### Enhanced User Management
|
||||
|
||||
Added comprehensive user management fields to the User model:
|
||||
|
||||
```prisma
|
||||
@ -137,38 +158,45 @@ model User {
|
||||
```
|
||||
|
||||
### Updated Repository Methods
|
||||
|
||||
Enhanced UserRepository with new methods:
|
||||
- `updateLastLogin()` - Tracks user login times
|
||||
- `incrementFailedLoginAttempts()` - Security feature for account locking
|
||||
- `verifyEmail()` - Email verification management
|
||||
- `deactivateUser()` - Account management
|
||||
- `unlockUser()` - Security administration
|
||||
- `updatePreferences()` - User settings management
|
||||
- `findInactiveUsers()` - Now uses `lastLoginAt` instead of `createdAt`
|
||||
|
||||
- `updateLastLogin()` - Tracks user login times
|
||||
- `incrementFailedLoginAttempts()` - Security feature for account locking
|
||||
- `verifyEmail()` - Email verification management
|
||||
- `deactivateUser()` - Account management
|
||||
- `unlockUser()` - Security administration
|
||||
- `updatePreferences()` - User settings management
|
||||
- `findInactiveUsers()` - Now uses `lastLoginAt` instead of `createdAt`
|
||||
|
||||
## Prevention Measures
|
||||
|
||||
### 1. Regular Dependency Updates
|
||||
- Monitor for breaking changes in dependencies like Zod
|
||||
- Use `pnpm outdated` to check for deprecated packages
|
||||
- Test builds after dependency updates
|
||||
|
||||
- Monitor for breaking changes in dependencies like Zod
|
||||
- Use `pnpm outdated` to check for deprecated packages
|
||||
- Test builds after dependency updates
|
||||
|
||||
### 2. TypeScript Strict Checking
|
||||
- Enable strict TypeScript checking to catch type errors early
|
||||
- Use proper type imports and exports
|
||||
- Avoid `any` types where possible
|
||||
|
||||
- Enable strict TypeScript checking to catch type errors early
|
||||
- Use proper type imports and exports
|
||||
- Avoid `any` types where possible
|
||||
|
||||
### 3. Build Pipeline Validation
|
||||
- Run `pnpm build` before committing
|
||||
- Include type checking in CI/CD pipeline
|
||||
- Separate linting from build process
|
||||
|
||||
- Run `pnpm build` before committing
|
||||
- Include type checking in CI/CD pipeline
|
||||
- Separate linting from build process
|
||||
|
||||
### 4. Schema Management
|
||||
- Regenerate Prisma client after schema changes: `pnpm prisma:generate`
|
||||
- Validate schema changes with database migrations
|
||||
- Use proper TypeScript types for database operations
|
||||
|
||||
- Regenerate Prisma client after schema changes: `pnpm prisma:generate`
|
||||
- Validate schema changes with database migrations
|
||||
- Use proper TypeScript types for database operations
|
||||
|
||||
### 5. Development Workflow
|
||||
|
||||
```bash
|
||||
# Recommended development workflow
|
||||
pnpm prisma:generate # After schema changes
|
||||
@ -206,4 +234,4 @@ pnpm install
|
||||
---
|
||||
|
||||
*Last updated: 2025-07-12*
|
||||
*Build Status: ✅ Success (47/47 pages generated)*
|
||||
*Build Status: ✅ Success (47/47 pages generated)*
|
||||
|
||||
Reference in New Issue
Block a user