mirror of
https://github.com/kjanat/livedash-node.git
synced 2026-01-16 13:52:16 +01:00
refactor: fix biome linting issues and update project documentation
- Fix 36+ biome linting issues reducing errors/warnings from 227 to 191 - Replace explicit 'any' types with proper TypeScript interfaces - Fix React hooks dependencies and useCallback patterns - Resolve unused variables and parameter assignment issues - Improve accessibility with proper label associations - Add comprehensive API documentation for admin and security features - Update README.md with accurate PostgreSQL setup and current tech stack - Create complete documentation for audit logging, CSP monitoring, and batch processing - Fix outdated project information and missing developer workflows
This commit is contained in:
393
docs/admin-audit-logs-api.md
Normal file
393
docs/admin-audit-logs-api.md
Normal file
@ -0,0 +1,393 @@
|
||||
# Admin Audit Logs API
|
||||
|
||||
This document describes the Admin Audit Logs API endpoints for retrieving and managing security audit logs in the LiveDash application.
|
||||
|
||||
## Overview
|
||||
|
||||
The Admin Audit Logs API provides secure access to security audit trails for administrative users. It includes comprehensive filtering, pagination, and retention management capabilities.
|
||||
|
||||
## Authentication & Authorization
|
||||
|
||||
- **Authentication**: NextAuth.js session required
|
||||
- **Authorization**: ADMIN role required for all endpoints
|
||||
- **Rate Limiting**: Integrated with existing auth rate limiting system
|
||||
- **Audit Trail**: All API access is logged for security monitoring
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Get Audit Logs
|
||||
|
||||
Retrieve paginated audit logs with optional filtering.
|
||||
|
||||
```http
|
||||
GET /api/admin/audit-logs
|
||||
```
|
||||
|
||||
#### Query Parameters
|
||||
|
||||
| Parameter | Type | Description | Default | Example |
|
||||
|-----------|------|-------------|---------|---------|
|
||||
| `page` | number | Page number (1-based) | 1 | `?page=2` |
|
||||
| `limit` | number | Records per page (max 100) | 50 | `?limit=25` |
|
||||
| `eventType` | string | Filter by event type | - | `?eventType=login_attempt` |
|
||||
| `outcome` | string | Filter by outcome | - | `?outcome=FAILURE` |
|
||||
| `severity` | string | Filter by severity level | - | `?severity=HIGH` |
|
||||
| `userId` | string | Filter by specific user ID | - | `?userId=user-123` |
|
||||
| `startDate` | string | Filter from date (ISO 8601) | - | `?startDate=2024-01-01T00:00:00Z` |
|
||||
| `endDate` | string | Filter to date (ISO 8601) | - | `?endDate=2024-01-02T00:00:00Z` |
|
||||
|
||||
#### Example Request
|
||||
|
||||
```javascript
|
||||
const response = await fetch('/api/admin/audit-logs?' + new URLSearchParams({
|
||||
page: '1',
|
||||
limit: '25',
|
||||
eventType: 'login_attempt',
|
||||
outcome: 'FAILURE',
|
||||
startDate: '2024-01-01T00:00:00Z',
|
||||
endDate: '2024-01-02T00:00:00Z'
|
||||
}));
|
||||
|
||||
const data = await response.json();
|
||||
```
|
||||
|
||||
#### Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"auditLogs": [
|
||||
{
|
||||
"id": "log-123",
|
||||
"eventType": "login_attempt",
|
||||
"outcome": "FAILURE",
|
||||
"severity": "HIGH",
|
||||
"userId": "user-456",
|
||||
"companyId": "company-789",
|
||||
"ipAddress": "192.168.1.100",
|
||||
"userAgent": "Mozilla/5.0...",
|
||||
"timestamp": "2024-01-01T12:00:00Z",
|
||||
"description": "Failed login attempt",
|
||||
"metadata": {
|
||||
"error": "invalid_password",
|
||||
"endpoint": "/api/auth/signin"
|
||||
},
|
||||
"user": {
|
||||
"id": "user-456",
|
||||
"email": "user@example.com",
|
||||
"name": "John Doe",
|
||||
"role": "USER"
|
||||
},
|
||||
"platformUser": null
|
||||
}
|
||||
],
|
||||
"pagination": {
|
||||
"page": 1,
|
||||
"limit": 25,
|
||||
"totalCount": 150,
|
||||
"totalPages": 6,
|
||||
"hasNext": true,
|
||||
"hasPrev": false
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Error Responses
|
||||
|
||||
```json
|
||||
// Unauthorized (401)
|
||||
{
|
||||
"success": false,
|
||||
"error": "Unauthorized"
|
||||
}
|
||||
|
||||
// Insufficient permissions (403)
|
||||
{
|
||||
"success": false,
|
||||
"error": "Insufficient permissions"
|
||||
}
|
||||
|
||||
// Server error (500)
|
||||
{
|
||||
"success": false,
|
||||
"error": "Internal server error"
|
||||
}
|
||||
```
|
||||
|
||||
### Audit Log Retention Management
|
||||
|
||||
Manage audit log retention policies and cleanup.
|
||||
|
||||
```http
|
||||
POST /api/admin/audit-logs/retention
|
||||
```
|
||||
|
||||
#### Request Body
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "cleanup" | "configure" | "status",
|
||||
"retentionDays": 90,
|
||||
"dryRun": true
|
||||
}
|
||||
```
|
||||
|
||||
#### Parameters
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `action` | string | Yes | Action to perform: `cleanup`, `configure`, or `status` |
|
||||
| `retentionDays` | number | No | Retention period in days (for configure action) |
|
||||
| `dryRun` | boolean | No | Preview changes without executing (for cleanup) |
|
||||
|
||||
#### Example Requests
|
||||
|
||||
**Check retention status:**
|
||||
```javascript
|
||||
const response = await fetch('/api/admin/audit-logs/retention', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ action: 'status' })
|
||||
});
|
||||
```
|
||||
|
||||
**Configure retention policy:**
|
||||
```javascript
|
||||
const response = await fetch('/api/admin/audit-logs/retention', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
action: 'configure',
|
||||
retentionDays: 365
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
**Cleanup old logs (dry run):**
|
||||
```javascript
|
||||
const response = await fetch('/api/admin/audit-logs/retention', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
action: 'cleanup',
|
||||
dryRun: true
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
## Security Features
|
||||
|
||||
### Access Control
|
||||
- **Role-based Access**: Only ADMIN users can access audit logs
|
||||
- **Company Isolation**: Users only see logs for their company
|
||||
- **Session Validation**: Active NextAuth session required
|
||||
|
||||
### Audit Trail
|
||||
- **Access Logging**: All audit log access is recorded
|
||||
- **Metadata Tracking**: Request parameters and results are logged
|
||||
- **IP Tracking**: Client IP addresses are recorded for all requests
|
||||
|
||||
### Rate Limiting
|
||||
- **Integrated Protection**: Uses existing authentication rate limiting
|
||||
- **Abuse Prevention**: Protects against excessive API usage
|
||||
- **Error Tracking**: Failed attempts are monitored
|
||||
|
||||
## Event Types
|
||||
|
||||
Common event types available for filtering:
|
||||
|
||||
| Event Type | Description |
|
||||
|------------|-------------|
|
||||
| `login_attempt` | User login attempts |
|
||||
| `login_success` | Successful logins |
|
||||
| `logout` | User logouts |
|
||||
| `password_reset_request` | Password reset requests |
|
||||
| `password_reset_complete` | Password reset completions |
|
||||
| `user_creation` | New user registrations |
|
||||
| `user_modification` | User profile changes |
|
||||
| `admin_action` | Administrative actions |
|
||||
| `data_export` | Data export activities |
|
||||
| `security_violation` | Security policy violations |
|
||||
|
||||
## Outcome Types
|
||||
|
||||
| Outcome | Description |
|
||||
|---------|-------------|
|
||||
| `SUCCESS` | Operation completed successfully |
|
||||
| `FAILURE` | Operation failed |
|
||||
| `BLOCKED` | Operation was blocked by security policy |
|
||||
| `WARNING` | Operation completed with warnings |
|
||||
| `RATE_LIMITED` | Operation was rate limited |
|
||||
|
||||
## Severity Levels
|
||||
|
||||
| Severity | Description | Use Case |
|
||||
|----------|-------------|----------|
|
||||
| `LOW` | Informational events | Normal operations |
|
||||
| `MEDIUM` | Notable events | Configuration changes |
|
||||
| `HIGH` | Security events | Failed logins, violations |
|
||||
| `CRITICAL` | Critical security events | Breaches, attacks |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Daily Security Report
|
||||
|
||||
```javascript
|
||||
async function getDailySecurityReport() {
|
||||
const yesterday = new Date();
|
||||
yesterday.setDate(yesterday.getDate() - 1);
|
||||
yesterday.setHours(0, 0, 0, 0);
|
||||
|
||||
const today = new Date();
|
||||
today.setHours(0, 0, 0, 0);
|
||||
|
||||
const response = await fetch('/api/admin/audit-logs?' + new URLSearchParams({
|
||||
startDate: yesterday.toISOString(),
|
||||
endDate: today.toISOString(),
|
||||
limit: '100'
|
||||
}));
|
||||
|
||||
const data = await response.json();
|
||||
return data.data.auditLogs;
|
||||
}
|
||||
```
|
||||
|
||||
### Failed Login Analysis
|
||||
|
||||
```javascript
|
||||
async function getFailedLogins(hours = 24) {
|
||||
const since = new Date();
|
||||
since.setHours(since.getHours() - hours);
|
||||
|
||||
const response = await fetch('/api/admin/audit-logs?' + new URLSearchParams({
|
||||
eventType: 'login_attempt',
|
||||
outcome: 'FAILURE',
|
||||
startDate: since.toISOString(),
|
||||
limit: '100'
|
||||
}));
|
||||
|
||||
const data = await response.json();
|
||||
return data.data.auditLogs;
|
||||
}
|
||||
```
|
||||
|
||||
### User Activity Tracking
|
||||
|
||||
```javascript
|
||||
async function getUserActivity(userId, days = 7) {
|
||||
const since = new Date();
|
||||
since.setDate(since.getDate() - days);
|
||||
|
||||
const response = await fetch('/api/admin/audit-logs?' + new URLSearchParams({
|
||||
userId: userId,
|
||||
startDate: since.toISOString(),
|
||||
limit: '50'
|
||||
}));
|
||||
|
||||
const data = await response.json();
|
||||
return data.data.auditLogs;
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Database Optimization
|
||||
- **Indexed Queries**: All filter columns are properly indexed
|
||||
- **Pagination**: Efficient offset-based pagination with limits
|
||||
- **Time Range Filtering**: Optimized for date range queries
|
||||
|
||||
### Memory Usage
|
||||
- **Limited Results**: Maximum 100 records per request
|
||||
- **Streaming**: Large exports use streaming for memory efficiency
|
||||
- **Connection Pooling**: Database connections are pooled
|
||||
|
||||
### Caching Considerations
|
||||
- **No Caching**: Audit logs are never cached for security reasons
|
||||
- **Fresh Data**: All queries hit the database for real-time results
|
||||
- **Read Replicas**: Consider using read replicas for heavy reporting
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
```javascript
|
||||
try {
|
||||
const response = await fetch('/api/admin/audit-logs');
|
||||
const data = await response.json();
|
||||
|
||||
if (!data.success) {
|
||||
switch (response.status) {
|
||||
case 401:
|
||||
console.error('User not authenticated');
|
||||
break;
|
||||
case 403:
|
||||
console.error('User lacks admin permissions');
|
||||
break;
|
||||
case 500:
|
||||
console.error('Server error:', data.error);
|
||||
break;
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Network error:', error);
|
||||
}
|
||||
```
|
||||
|
||||
### Rate Limiting Handling
|
||||
|
||||
```javascript
|
||||
async function fetchWithRetry(url, options = {}) {
|
||||
const response = await fetch(url, options);
|
||||
|
||||
if (response.status === 429) {
|
||||
// Rate limited, wait and retry
|
||||
await new Promise(resolve => setTimeout(resolve, 5000));
|
||||
return fetchWithRetry(url, options);
|
||||
}
|
||||
|
||||
return response;
|
||||
}
|
||||
```
|
||||
|
||||
## Monitoring and Alerting
|
||||
|
||||
### Key Metrics to Monitor
|
||||
- **Request Volume**: Track API usage patterns
|
||||
- **Error Rates**: Monitor authentication and authorization failures
|
||||
- **Query Performance**: Track slow queries and optimize
|
||||
- **Data Growth**: Monitor audit log size and plan retention
|
||||
|
||||
### Alert Conditions
|
||||
- **High Error Rates**: >5% of requests failing
|
||||
- **Unusual Access Patterns**: Off-hours access, high volume
|
||||
- **Performance Degradation**: Query times >2 seconds
|
||||
- **Security Events**: Multiple failed admin access attempts
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Security
|
||||
- Always validate user permissions before displaying UI
|
||||
- Log all administrative access to audit logs
|
||||
- Use HTTPS in production environments
|
||||
- Implement proper error handling to avoid information leakage
|
||||
|
||||
### Performance
|
||||
- Use appropriate page sizes (25-50 records typical)
|
||||
- Implement client-side pagination for better UX
|
||||
- Cache results only in memory, never persist
|
||||
- Use date range filters to limit query scope
|
||||
|
||||
### User Experience
|
||||
- Provide clear filtering options in the UI
|
||||
- Show loading states for long-running queries
|
||||
- Implement export functionality for reports
|
||||
- Provide search and sort capabilities
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Security Audit Logging](./security-audit-logging.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [CSRF Protection](./CSRF_PROTECTION.md)
|
||||
- [Authentication System](../lib/auth.ts)
|
||||
648
docs/api-reference.md
Normal file
648
docs/api-reference.md
Normal file
@ -0,0 +1,648 @@
|
||||
# LiveDash-Node API Reference
|
||||
|
||||
This document provides a comprehensive reference for all API endpoints in the LiveDash-Node application, including authentication, security monitoring, audit logging, and administrative functions.
|
||||
|
||||
## Base URL
|
||||
|
||||
```
|
||||
Local Development: http://localhost:3000
|
||||
Production: https://your-domain.com
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
All API endpoints (except public endpoints) require authentication via NextAuth.js session cookies.
|
||||
|
||||
### Authentication Headers
|
||||
|
||||
```http
|
||||
Cookie: next-auth.session-token=<session-token>
|
||||
```
|
||||
|
||||
### CSRF Protection
|
||||
|
||||
State-changing endpoints require CSRF tokens:
|
||||
|
||||
```http
|
||||
X-CSRF-Token: <csrf-token>
|
||||
```
|
||||
|
||||
Get CSRF token:
|
||||
```http
|
||||
GET /api/csrf-token
|
||||
```
|
||||
|
||||
## API Endpoints Overview
|
||||
|
||||
### Public Endpoints
|
||||
- `POST /api/csp-report` - CSP violation reporting (no auth required)
|
||||
- `OPTIONS /api/csp-report` - CORS preflight
|
||||
|
||||
### Authentication Endpoints
|
||||
- `POST /api/auth/[...nextauth]` - NextAuth.js authentication
|
||||
- `GET /api/csrf-token` - Get CSRF token
|
||||
- `POST /api/register` - User registration
|
||||
- `POST /api/forgot-password` - Password reset request
|
||||
- `POST /api/reset-password` - Password reset completion
|
||||
|
||||
### Admin Endpoints (ADMIN role required)
|
||||
- `GET /api/admin/audit-logs` - Retrieve audit logs
|
||||
- `POST /api/admin/audit-logs/retention` - Manage audit log retention
|
||||
- `GET /api/admin/batch-monitoring` - Batch processing monitoring
|
||||
- `POST /api/admin/batch-monitoring/{id}/retry` - Retry failed batch job
|
||||
|
||||
### Platform Admin Endpoints (Platform admin only)
|
||||
- `GET /api/admin/security-monitoring` - Security monitoring metrics
|
||||
- `POST /api/admin/security-monitoring` - Update security configuration
|
||||
- `GET /api/admin/security-monitoring/alerts` - Alert management
|
||||
- `POST /api/admin/security-monitoring/alerts` - Acknowledge alerts
|
||||
- `GET /api/admin/security-monitoring/export` - Export security data
|
||||
- `POST /api/admin/security-monitoring/threat-analysis` - Threat analysis
|
||||
|
||||
### Security Monitoring Endpoints
|
||||
- `GET /api/csp-metrics` - CSP violation metrics
|
||||
- `POST /api/csp-report` - CSP violation reporting
|
||||
|
||||
### Dashboard Endpoints
|
||||
- `GET /api/dashboard/sessions` - Session data
|
||||
- `GET /api/dashboard/session/{id}` - Individual session details
|
||||
- `GET /api/dashboard/metrics` - Dashboard metrics
|
||||
- `GET /api/dashboard/config` - Dashboard configuration
|
||||
|
||||
### Platform Management
|
||||
- `GET /api/platform/companies` - Company management
|
||||
- `POST /api/platform/companies` - Create company
|
||||
- `GET /api/platform/companies/{id}` - Company details
|
||||
- `GET /api/platform/companies/{id}/users` - Company users
|
||||
- `POST /api/platform/companies/{id}/users` - Add company user
|
||||
|
||||
### tRPC Endpoints
|
||||
- `POST /api/trpc/[trpc]` - tRPC procedure calls
|
||||
|
||||
## Detailed Endpoint Documentation
|
||||
|
||||
### Admin Audit Logs
|
||||
|
||||
#### Get Audit Logs
|
||||
```http
|
||||
GET /api/admin/audit-logs
|
||||
```
|
||||
|
||||
**Authorization**: ADMIN role required
|
||||
|
||||
**Query Parameters**:
|
||||
- `page` (number, optional): Page number (default: 1)
|
||||
- `limit` (number, optional): Records per page, max 100 (default: 50)
|
||||
- `eventType` (string, optional): Filter by event type
|
||||
- `outcome` (string, optional): Filter by outcome (SUCCESS, FAILURE, BLOCKED, etc.)
|
||||
- `severity` (string, optional): Filter by severity (LOW, MEDIUM, HIGH, CRITICAL)
|
||||
- `userId` (string, optional): Filter by user ID
|
||||
- `startDate` (string, optional): Start date (ISO 8601)
|
||||
- `endDate` (string, optional): End date (ISO 8601)
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"auditLogs": [...],
|
||||
"pagination": {
|
||||
"page": 1,
|
||||
"limit": 50,
|
||||
"totalCount": 150,
|
||||
"totalPages": 3,
|
||||
"hasNext": true,
|
||||
"hasPrev": false
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Rate Limit**: Inherits from auth rate limiting
|
||||
|
||||
#### Manage Audit Log Retention
|
||||
```http
|
||||
POST /api/admin/audit-logs/retention
|
||||
```
|
||||
|
||||
**Authorization**: ADMIN role required
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"action": "cleanup" | "configure" | "status",
|
||||
"retentionDays": 90,
|
||||
"dryRun": true
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"action": "cleanup",
|
||||
"recordsAffected": 1250,
|
||||
"retentionDays": 90,
|
||||
"dryRun": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Security Monitoring
|
||||
|
||||
#### Get Security Metrics
|
||||
```http
|
||||
GET /api/admin/security-monitoring
|
||||
```
|
||||
|
||||
**Authorization**: Platform admin required
|
||||
|
||||
**Query Parameters**:
|
||||
- `startDate` (string, optional): Start date (ISO 8601)
|
||||
- `endDate` (string, optional): End date (ISO 8601)
|
||||
- `companyId` (string, optional): Filter by company
|
||||
- `severity` (string, optional): Filter by severity
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"metrics": {
|
||||
"securityScore": 85,
|
||||
"threatLevel": "LOW",
|
||||
"eventCounts": {...},
|
||||
"anomalies": [...]
|
||||
},
|
||||
"alerts": [...],
|
||||
"config": {...},
|
||||
"timeRange": {...}
|
||||
}
|
||||
```
|
||||
|
||||
#### Update Security Configuration
|
||||
```http
|
||||
POST /api/admin/security-monitoring
|
||||
```
|
||||
|
||||
**Authorization**: Platform admin required
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"thresholds": {
|
||||
"failedLoginsPerMinute": 5,
|
||||
"rateLimitViolationsPerMinute": 10
|
||||
},
|
||||
"alerting": {
|
||||
"enabled": true,
|
||||
"channels": ["EMAIL", "WEBHOOK"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### CSP Monitoring
|
||||
|
||||
#### CSP Violation Reporting
|
||||
```http
|
||||
POST /api/csp-report
|
||||
```
|
||||
|
||||
**Authorization**: None (public endpoint)
|
||||
|
||||
**Headers**:
|
||||
- `Content-Type`: `application/csp-report` or `application/json`
|
||||
|
||||
**Request Body** (automatic from browser):
|
||||
```json
|
||||
{
|
||||
"csp-report": {
|
||||
"document-uri": "https://example.com/page",
|
||||
"violated-directive": "script-src 'self'",
|
||||
"blocked-uri": "https://malicious.com/script.js",
|
||||
"source-file": "https://example.com/page",
|
||||
"line-number": 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Rate Limit**: 10 reports per minute per IP
|
||||
|
||||
**Response**: `204 No Content`
|
||||
|
||||
#### Get CSP Metrics
|
||||
```http
|
||||
GET /api/csp-metrics
|
||||
```
|
||||
|
||||
**Authorization**: Admin role required
|
||||
|
||||
**Query Parameters**:
|
||||
- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d)
|
||||
- `format` (string, optional): Response format (json, csv)
|
||||
- `groupBy` (string, optional): Group by field (hour, directive, etc.)
|
||||
- `includeDetails` (boolean, optional): Include violation details
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"summary": {
|
||||
"totalViolations": 45,
|
||||
"uniqueViolations": 12,
|
||||
"highRiskViolations": 3,
|
||||
"bypassAttempts": 1
|
||||
},
|
||||
"trends": {...},
|
||||
"topViolations": [...],
|
||||
"riskAnalysis": {...},
|
||||
"violations": [...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Batch Monitoring
|
||||
|
||||
#### Get Batch Monitoring Data
|
||||
```http
|
||||
GET /api/admin/batch-monitoring
|
||||
```
|
||||
|
||||
**Authorization**: ADMIN role required
|
||||
|
||||
**Query Parameters**:
|
||||
- `timeRange` (string, optional): Time range (1h, 6h, 24h, 7d, 30d)
|
||||
- `status` (string, optional): Filter by status (pending, completed, failed)
|
||||
- `jobType` (string, optional): Filter by job type
|
||||
- `includeDetails` (boolean, optional): Include detailed job information
|
||||
- `page` (number, optional): Page number
|
||||
- `limit` (number, optional): Records per page
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"summary": {
|
||||
"totalJobs": 156,
|
||||
"completedJobs": 142,
|
||||
"failedJobs": 8,
|
||||
"costSavings": {...}
|
||||
},
|
||||
"queues": {...},
|
||||
"performance": {...},
|
||||
"jobs": [...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Retry Batch Job
|
||||
```http
|
||||
POST /api/admin/batch-monitoring/{jobId}/retry
|
||||
```
|
||||
|
||||
**Authorization**: ADMIN role required
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"jobId": "batch-job-123",
|
||||
"status": "retrying",
|
||||
"message": "Job queued for retry"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### CSRF Token
|
||||
|
||||
#### Get CSRF Token
|
||||
```http
|
||||
GET /api/csrf-token
|
||||
```
|
||||
|
||||
**Authorization**: None
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"csrfToken": "abc123..."
|
||||
}
|
||||
```
|
||||
|
||||
**Headers Set**:
|
||||
- `Set-Cookie`: HTTP-only CSRF token cookie
|
||||
|
||||
### Authentication
|
||||
|
||||
#### User Registration
|
||||
```http
|
||||
POST /api/register
|
||||
```
|
||||
|
||||
**Authorization**: None
|
||||
|
||||
**Headers Required**:
|
||||
- `X-CSRF-Token`: CSRF token
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"email": "user@example.com",
|
||||
"password": "SecurePassword123!",
|
||||
"name": "John Doe",
|
||||
"companyName": "Acme Corp"
|
||||
}
|
||||
```
|
||||
|
||||
**Rate Limit**: 3 attempts per hour per IP
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "User registered successfully",
|
||||
"userId": "user-123"
|
||||
}
|
||||
```
|
||||
|
||||
#### Password Reset Request
|
||||
```http
|
||||
POST /api/forgot-password
|
||||
```
|
||||
|
||||
**Authorization**: None
|
||||
|
||||
**Headers Required**:
|
||||
- `X-CSRF-Token`: CSRF token
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"email": "user@example.com"
|
||||
}
|
||||
```
|
||||
|
||||
**Rate Limit**: 5 attempts per 15 minutes per IP
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Password reset email sent"
|
||||
}
|
||||
```
|
||||
|
||||
#### Password Reset Completion
|
||||
```http
|
||||
POST /api/reset-password
|
||||
```
|
||||
|
||||
**Authorization**: None
|
||||
|
||||
**Headers Required**:
|
||||
- `X-CSRF-Token`: CSRF token
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"token": "reset-token-123",
|
||||
"password": "NewSecurePassword123!"
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Password reset successfully"
|
||||
}
|
||||
```
|
||||
|
||||
## Error Responses
|
||||
|
||||
### Standard Error Format
|
||||
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"error": "Error message",
|
||||
"code": "ERROR_CODE",
|
||||
"details": {...}
|
||||
}
|
||||
```
|
||||
|
||||
### Common HTTP Status Codes
|
||||
|
||||
| Status | Description | Common Causes |
|
||||
|--------|-------------|---------------|
|
||||
| 200 | OK | Successful request |
|
||||
| 201 | Created | Resource created successfully |
|
||||
| 204 | No Content | Successful request with no response body |
|
||||
| 400 | Bad Request | Invalid request parameters or body |
|
||||
| 401 | Unauthorized | Authentication required or invalid |
|
||||
| 403 | Forbidden | Insufficient permissions |
|
||||
| 404 | Not Found | Resource not found |
|
||||
| 409 | Conflict | Resource already exists or conflict |
|
||||
| 422 | Unprocessable Entity | Validation errors |
|
||||
| 429 | Too Many Requests | Rate limit exceeded |
|
||||
| 500 | Internal Server Error | Server error |
|
||||
|
||||
### Error Codes
|
||||
|
||||
| Code | Description | Resolution |
|
||||
|------|-------------|------------|
|
||||
| `UNAUTHORIZED` | No valid session | Login required |
|
||||
| `FORBIDDEN` | Insufficient permissions | Check user role |
|
||||
| `VALIDATION_ERROR` | Invalid input data | Check request format |
|
||||
| `RATE_LIMITED` | Too many requests | Wait and retry |
|
||||
| `CSRF_INVALID` | Invalid CSRF token | Get new token |
|
||||
| `NOT_FOUND` | Resource not found | Check resource ID |
|
||||
| `CONFLICT` | Resource conflict | Check existing data |
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
### Authentication Endpoints
|
||||
- **Login**: 5 attempts per 15 minutes per IP
|
||||
- **Registration**: 3 attempts per hour per IP
|
||||
- **Password Reset**: 5 attempts per 15 minutes per IP
|
||||
|
||||
### Security Endpoints
|
||||
- **CSP Reports**: 10 reports per minute per IP
|
||||
- **Admin Endpoints**: 60 requests per minute per user
|
||||
- **Security Monitoring**: 30 requests per minute per user
|
||||
|
||||
### General API
|
||||
- **Dashboard Endpoints**: 120 requests per minute per user
|
||||
- **Platform Management**: 60 requests per minute per user
|
||||
|
||||
## Security Headers
|
||||
|
||||
All API responses include security headers:
|
||||
|
||||
```http
|
||||
X-Content-Type-Options: nosniff
|
||||
X-Frame-Options: DENY
|
||||
X-XSS-Protection: 1; mode=block
|
||||
Referrer-Policy: strict-origin-when-cross-origin
|
||||
Content-Security-Policy: [CSP directives]
|
||||
```
|
||||
|
||||
## CORS Configuration
|
||||
|
||||
### Allowed Origins
|
||||
- Development: `http://localhost:3000`
|
||||
- Production: `https://your-domain.com`
|
||||
|
||||
### Allowed Methods
|
||||
- `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS`
|
||||
|
||||
### Allowed Headers
|
||||
- `Content-Type`, `Authorization`, `X-CSRF-Token`, `X-Requested-With`
|
||||
|
||||
## Pagination
|
||||
|
||||
### Standard Pagination Format
|
||||
|
||||
```json
|
||||
{
|
||||
"data": [...],
|
||||
"pagination": {
|
||||
"page": 1,
|
||||
"limit": 50,
|
||||
"totalCount": 150,
|
||||
"totalPages": 3,
|
||||
"hasNext": true,
|
||||
"hasPrev": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pagination Parameters
|
||||
- `page`: Page number (1-based, default: 1)
|
||||
- `limit`: Records per page (default: 50, max: 100)
|
||||
|
||||
## Filtering and Sorting
|
||||
|
||||
### Common Filter Parameters
|
||||
- `startDate` / `endDate`: Date range filtering (ISO 8601)
|
||||
- `status`: Status filtering
|
||||
- `userId` / `companyId`: Entity filtering
|
||||
- `eventType`: Event type filtering
|
||||
- `severity`: Severity level filtering
|
||||
|
||||
### Sorting Parameters
|
||||
- `sortBy`: Field to sort by
|
||||
- `sortOrder`: `asc` or `desc` (default: `desc`)
|
||||
|
||||
## Response Caching
|
||||
|
||||
### Cache Headers
|
||||
```http
|
||||
Cache-Control: no-cache, no-store, must-revalidate
|
||||
Pragma: no-cache
|
||||
Expires: 0
|
||||
```
|
||||
|
||||
### Cache Strategy
|
||||
- **Security data**: Never cached
|
||||
- **Static data**: Browser cache for 5 minutes
|
||||
- **User data**: No cache for security
|
||||
|
||||
## API Versioning
|
||||
|
||||
### Current Version
|
||||
- Version: `v1` (implied, no version prefix required)
|
||||
- Introduced: January 2025
|
||||
|
||||
### Future Versioning
|
||||
- Breaking changes will introduce new versions
|
||||
- Format: `/api/v2/endpoint`
|
||||
- Backward compatibility maintained for 12 months
|
||||
|
||||
## SDK and Client Libraries
|
||||
|
||||
### JavaScript/TypeScript Client
|
||||
|
||||
```javascript
|
||||
// Initialize client
|
||||
const client = new LiveDashClient({
|
||||
baseURL: 'https://your-domain.com',
|
||||
apiKey: 'your-api-key' // For future API key auth
|
||||
});
|
||||
|
||||
// Get audit logs
|
||||
const auditLogs = await client.admin.getAuditLogs({
|
||||
page: 1,
|
||||
limit: 50,
|
||||
eventType: 'login_attempt'
|
||||
});
|
||||
|
||||
// Get security metrics
|
||||
const metrics = await client.security.getMetrics({
|
||||
timeRange: '24h'
|
||||
});
|
||||
```
|
||||
|
||||
### tRPC Client
|
||||
|
||||
```javascript
|
||||
import { createTRPCNext } from '@trpc/next';
|
||||
|
||||
const trpc = createTRPCNext({
|
||||
config() {
|
||||
return {
|
||||
url: '/api/trpc',
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
// Use tRPC procedures
|
||||
const { data: user } = trpc.auth.getUser.useQuery();
|
||||
const updateProfile = trpc.user.updateProfile.useMutation();
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### API Testing Tools
|
||||
|
||||
```bash
|
||||
# Test with curl
|
||||
curl -X GET "http://localhost:3000/api/admin/audit-logs" \
|
||||
-H "Cookie: next-auth.session-token=..." \
|
||||
-H "X-CSRF-Token: ..."
|
||||
|
||||
# Test with HTTPie
|
||||
http GET localhost:3000/api/csp-metrics \
|
||||
timeRange==24h \
|
||||
Cookie:next-auth.session-token=...
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
```javascript
|
||||
// Example test
|
||||
describe('Admin Audit Logs API', () => {
|
||||
test('should return paginated audit logs', async () => {
|
||||
const response = await request(app)
|
||||
.get('/api/admin/audit-logs?page=1&limit=10')
|
||||
.set('Cookie', 'next-auth.session-token=...')
|
||||
.expect(200);
|
||||
|
||||
expect(response.body.success).toBe(true);
|
||||
expect(response.body.data.auditLogs).toHaveLength(10);
|
||||
expect(response.body.data.pagination.page).toBe(1);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Admin Audit Logs API](./admin-audit-logs-api.md)
|
||||
- [CSP Metrics API](./csp-metrics-api.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [CSRF Protection](./CSRF_PROTECTION.md)
|
||||
- [Batch Monitoring Dashboard](./batch-monitoring-dashboard.md)
|
||||
|
||||
This API reference provides comprehensive documentation for all endpoints in the LiveDash-Node application. For specific implementation details, refer to the individual documentation files for each feature area.
|
||||
531
docs/batch-monitoring-dashboard.md
Normal file
531
docs/batch-monitoring-dashboard.md
Normal file
@ -0,0 +1,531 @@
|
||||
# Batch Processing Monitoring Dashboard
|
||||
|
||||
This document describes the batch processing monitoring dashboard and API endpoints for tracking OpenAI Batch API operations in the LiveDash application.
|
||||
|
||||
## Overview
|
||||
|
||||
The Batch Monitoring Dashboard provides real-time visibility into the OpenAI Batch API processing pipeline, including job status tracking, cost analysis, and performance monitoring. This system enables 50% cost reduction on AI processing while maintaining comprehensive oversight.
|
||||
|
||||
## Features
|
||||
|
||||
### Real-time Monitoring
|
||||
- **Job Status Tracking**: Monitor batch jobs from creation to completion
|
||||
- **Queue Management**: View pending, running, and completed batch queues
|
||||
- **Processing Metrics**: Track throughput, success rates, and error patterns
|
||||
- **Cost Analysis**: Monitor API costs and savings compared to individual requests
|
||||
|
||||
### Performance Analytics
|
||||
- **Batch Efficiency**: Analyze batch size optimization and processing times
|
||||
- **Success Rates**: Track completion and failure rates across different job types
|
||||
- **Resource Utilization**: Monitor API quota usage and rate limiting
|
||||
- **Historical Trends**: View processing patterns over time
|
||||
|
||||
### Administrative Controls
|
||||
- **Manual Intervention**: Pause, resume, or cancel batch operations
|
||||
- **Priority Management**: Adjust processing priorities for urgent requests
|
||||
- **Error Handling**: Review and retry failed batch operations
|
||||
- **Configuration Management**: Adjust batch parameters and thresholds
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Batch Monitoring API
|
||||
|
||||
Retrieve comprehensive batch processing metrics and status information.
|
||||
|
||||
```http
|
||||
GET /api/admin/batch-monitoring
|
||||
```
|
||||
|
||||
#### Query Parameters
|
||||
|
||||
| Parameter | Type | Description | Default | Example |
|
||||
|-----------|------|-------------|---------|---------|
|
||||
| `timeRange` | string | Time range for metrics | `24h` | `?timeRange=7d` |
|
||||
| `status` | string | Filter by batch status | - | `?status=completed` |
|
||||
| `jobType` | string | Filter by job type | - | `?jobType=ai_analysis` |
|
||||
| `includeDetails` | boolean | Include detailed job information | `false` | `?includeDetails=true` |
|
||||
| `page` | number | Page number for pagination | 1 | `?page=2` |
|
||||
| `limit` | number | Records per page (max 100) | 50 | `?limit=25` |
|
||||
|
||||
#### Example Request
|
||||
|
||||
```javascript
|
||||
const response = await fetch('/api/admin/batch-monitoring?' + new URLSearchParams({
|
||||
timeRange: '24h',
|
||||
status: 'completed',
|
||||
includeDetails: 'true'
|
||||
}));
|
||||
|
||||
const data = await response.json();
|
||||
```
|
||||
|
||||
#### Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"summary": {
|
||||
"totalJobs": 156,
|
||||
"completedJobs": 142,
|
||||
"failedJobs": 8,
|
||||
"pendingJobs": 6,
|
||||
"totalRequests": 15600,
|
||||
"processedRequests": 14200,
|
||||
"costSavings": {
|
||||
"currentPeriod": 234.56,
|
||||
"projectedMonthly": 7038.45,
|
||||
"savingsPercentage": 48.2
|
||||
},
|
||||
"averageProcessingTime": 1800000,
|
||||
"successRate": 95.2
|
||||
},
|
||||
"queues": {
|
||||
"pending": 12,
|
||||
"processing": 3,
|
||||
"completed": 142,
|
||||
"failed": 8
|
||||
},
|
||||
"performance": {
|
||||
"throughput": {
|
||||
"requestsPerHour": 650,
|
||||
"jobsPerHour": 6.5,
|
||||
"averageBatchSize": 100
|
||||
},
|
||||
"efficiency": {
|
||||
"batchUtilization": 87.3,
|
||||
"processingEfficiency": 92.1,
|
||||
"errorRate": 4.8
|
||||
}
|
||||
},
|
||||
"jobs": [
|
||||
{
|
||||
"id": "batch-job-123",
|
||||
"batchId": "batch_abc123",
|
||||
"status": "completed",
|
||||
"jobType": "ai_analysis",
|
||||
"requestCount": 100,
|
||||
"completedCount": 98,
|
||||
"failedCount": 2,
|
||||
"createdAt": "2024-01-01T10:00:00Z",
|
||||
"startedAt": "2024-01-01T10:05:00Z",
|
||||
"completedAt": "2024-01-01T10:35:00Z",
|
||||
"processingTimeMs": 1800000,
|
||||
"costEstimate": 12.50,
|
||||
"errorSummary": [
|
||||
{
|
||||
"error": "token_limit_exceeded",
|
||||
"count": 2,
|
||||
"percentage": 2.0
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Dashboard Components
|
||||
|
||||
### BatchMonitoringDashboard Component
|
||||
|
||||
The main dashboard component (`components/admin/BatchMonitoringDashboard.tsx`) provides:
|
||||
|
||||
#### Key Metrics Cards
|
||||
```tsx
|
||||
// Real-time overview cards
|
||||
<MetricCard
|
||||
title="Total Jobs"
|
||||
value={data.summary.totalJobs}
|
||||
change={"+12 from yesterday"}
|
||||
trend="up"
|
||||
/>
|
||||
|
||||
<MetricCard
|
||||
title="Success Rate"
|
||||
value={`${data.summary.successRate}%`}
|
||||
change={"+2.1% from last week"}
|
||||
trend="up"
|
||||
/>
|
||||
|
||||
<MetricCard
|
||||
title="Cost Savings"
|
||||
value={`$${data.summary.costSavings.currentPeriod}`}
|
||||
change={`${data.summary.costSavings.savingsPercentage}% vs individual API`}
|
||||
trend="up"
|
||||
/>
|
||||
```
|
||||
|
||||
#### Queue Status Visualization
|
||||
```tsx
|
||||
// Visual representation of batch job queues
|
||||
<QueueStatusChart
|
||||
pending={data.queues.pending}
|
||||
processing={data.queues.processing}
|
||||
completed={data.queues.completed}
|
||||
failed={data.queues.failed}
|
||||
/>
|
||||
```
|
||||
|
||||
#### Performance Charts
|
||||
```tsx
|
||||
// Processing throughput over time
|
||||
<ThroughputChart
|
||||
data={data.performance.throughput}
|
||||
timeRange={timeRange}
|
||||
/>
|
||||
|
||||
// Cost savings trend
|
||||
<CostSavingsChart
|
||||
savings={data.summary.costSavings}
|
||||
historical={data.historical}
|
||||
/>
|
||||
```
|
||||
|
||||
#### Job Management Table
|
||||
```tsx
|
||||
// Detailed job listing with actions
|
||||
<BatchJobTable
|
||||
jobs={data.jobs}
|
||||
onRetry={handleRetryJob}
|
||||
onCancel={handleCancelJob}
|
||||
onViewDetails={handleViewDetails}
|
||||
/>
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Monitor Batch Performance
|
||||
|
||||
```javascript
|
||||
async function monitorBatchPerformance() {
|
||||
const response = await fetch('/api/admin/batch-monitoring?timeRange=24h');
|
||||
const data = await response.json();
|
||||
|
||||
const performance = data.data.performance;
|
||||
|
||||
// Check if performance is within acceptable ranges
|
||||
if (performance.efficiency.errorRate > 10) {
|
||||
console.warn('High error rate detected:', performance.efficiency.errorRate + '%');
|
||||
|
||||
// Get failed jobs for analysis
|
||||
const failedJobs = await fetch('/api/admin/batch-monitoring?status=failed');
|
||||
const failures = await failedJobs.json();
|
||||
|
||||
// Analyze common failure patterns
|
||||
const errorSummary = failures.data.jobs.reduce((acc, job) => {
|
||||
job.errorSummary?.forEach(error => {
|
||||
acc[error.error] = (acc[error.error] || 0) + error.count;
|
||||
});
|
||||
return acc;
|
||||
}, {});
|
||||
|
||||
console.log('Error patterns:', errorSummary);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cost Savings Analysis
|
||||
|
||||
```javascript
|
||||
async function analyzeCostSavings() {
|
||||
const response = await fetch('/api/admin/batch-monitoring?timeRange=30d&includeDetails=true');
|
||||
const data = await response.json();
|
||||
|
||||
const savings = data.data.summary.costSavings;
|
||||
|
||||
return {
|
||||
currentSavings: savings.currentPeriod,
|
||||
projectedAnnual: savings.projectedMonthly * 12,
|
||||
savingsRate: savings.savingsPercentage,
|
||||
totalProcessed: data.data.summary.processedRequests,
|
||||
averageCostPerRequest: savings.currentPeriod / data.data.summary.processedRequests
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### Retry Failed Jobs
|
||||
|
||||
```javascript
|
||||
async function retryFailedJobs() {
|
||||
// Get failed jobs
|
||||
const response = await fetch('/api/admin/batch-monitoring?status=failed');
|
||||
const data = await response.json();
|
||||
|
||||
const retryableJobs = data.data.jobs.filter(job => {
|
||||
// Only retry jobs that failed due to temporary issues
|
||||
const hasRetryableErrors = job.errorSummary?.some(error =>
|
||||
['rate_limit_exceeded', 'temporary_error', 'timeout'].includes(error.error)
|
||||
);
|
||||
return hasRetryableErrors;
|
||||
});
|
||||
|
||||
// Retry jobs individually
|
||||
for (const job of retryableJobs) {
|
||||
try {
|
||||
await fetch(`/api/admin/batch-monitoring/${job.id}/retry`, {
|
||||
method: 'POST'
|
||||
});
|
||||
console.log(`Retried job ${job.id}`);
|
||||
} catch (error) {
|
||||
console.error(`Failed to retry job ${job.id}:`, error);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Real-time Dashboard Updates
|
||||
|
||||
```javascript
|
||||
function useRealtimeBatchMonitoring() {
|
||||
const [data, setData] = useState(null);
|
||||
const [isLoading, setIsLoading] = useState(true);
|
||||
|
||||
useEffect(() => {
|
||||
const fetchData = async () => {
|
||||
try {
|
||||
const response = await fetch('/api/admin/batch-monitoring?timeRange=1h');
|
||||
const result = await response.json();
|
||||
setData(result.data);
|
||||
} catch (error) {
|
||||
console.error('Failed to fetch batch monitoring data:', error);
|
||||
} finally {
|
||||
setIsLoading(false);
|
||||
}
|
||||
};
|
||||
|
||||
// Initial fetch
|
||||
fetchData();
|
||||
|
||||
// Update every 30 seconds
|
||||
const interval = setInterval(fetchData, 30000);
|
||||
|
||||
return () => clearInterval(interval);
|
||||
}, []);
|
||||
|
||||
return { data, isLoading };
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Batch Processing Settings
|
||||
|
||||
Configure batch processing parameters in environment variables:
|
||||
|
||||
```bash
|
||||
# Batch Processing Configuration
|
||||
BATCH_PROCESSING_ENABLED="true"
|
||||
BATCH_CREATE_INTERVAL="*/5 * * * *" # Create batches every 5 minutes
|
||||
BATCH_STATUS_CHECK_INTERVAL="*/2 * * * *" # Check status every 2 minutes
|
||||
BATCH_RESULT_PROCESSING_INTERVAL="*/1 * * * *" # Process results every minute
|
||||
|
||||
# Batch Size and Limits
|
||||
BATCH_MAX_REQUESTS="1000" # Maximum requests per batch
|
||||
BATCH_TIMEOUT_HOURS="24" # Batch timeout in hours
|
||||
BATCH_MIN_SIZE="10" # Minimum batch size
|
||||
|
||||
# Monitoring Configuration
|
||||
BATCH_MONITORING_RETENTION_DAYS="30" # How long to keep monitoring data
|
||||
BATCH_ALERT_THRESHOLD_ERROR_RATE="10" # Alert if error rate exceeds 10%
|
||||
BATCH_ALERT_THRESHOLD_PROCESSING_TIME="3600" # Alert if processing takes >1 hour
|
||||
```
|
||||
|
||||
### Dashboard Refresh Settings
|
||||
|
||||
```javascript
|
||||
// Configure dashboard update intervals
|
||||
const DASHBOARD_CONFIG = {
|
||||
refreshInterval: 30000, // 30 seconds
|
||||
alertRefreshInterval: 10000, // 10 seconds for alerts
|
||||
detailRefreshInterval: 60000, // 1 minute for detailed views
|
||||
maxRetries: 3, // Maximum retry attempts
|
||||
retryDelay: 5000 // Delay between retries
|
||||
};
|
||||
```
|
||||
|
||||
## Alerts and Notifications
|
||||
|
||||
### Automated Alerts
|
||||
|
||||
The system automatically generates alerts for:
|
||||
|
||||
```javascript
|
||||
const alertConditions = {
|
||||
highErrorRate: {
|
||||
threshold: 10, // Error rate > 10%
|
||||
severity: 'high',
|
||||
notification: 'immediate'
|
||||
},
|
||||
longProcessingTime: {
|
||||
threshold: 3600000, // > 1 hour
|
||||
severity: 'medium',
|
||||
notification: 'hourly'
|
||||
},
|
||||
lowThroughput: {
|
||||
threshold: 0.5, // < 0.5 jobs per hour
|
||||
severity: 'medium',
|
||||
notification: 'daily'
|
||||
},
|
||||
batchFailure: {
|
||||
threshold: 1, // Any complete batch failure
|
||||
severity: 'critical',
|
||||
notification: 'immediate'
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### Custom Alert Configuration
|
||||
|
||||
```javascript
|
||||
// Configure custom alerts through the admin interface
|
||||
async function configureAlerts(alertConfig) {
|
||||
const response = await fetch('/api/admin/batch-monitoring/alerts', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
errorRateThreshold: alertConfig.errorRate,
|
||||
processingTimeThreshold: alertConfig.processingTime,
|
||||
notificationChannels: alertConfig.channels,
|
||||
alertSuppression: alertConfig.suppression
|
||||
})
|
||||
});
|
||||
|
||||
return response.json();
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### High Error Rates
|
||||
```javascript
|
||||
// Investigate high error rates
|
||||
async function investigateErrors() {
|
||||
const response = await fetch('/api/admin/batch-monitoring?status=failed&includeDetails=true');
|
||||
const data = await response.json();
|
||||
|
||||
// Group errors by type
|
||||
const errorAnalysis = data.data.jobs.reduce((acc, job) => {
|
||||
job.errorSummary?.forEach(error => {
|
||||
if (!acc[error.error]) {
|
||||
acc[error.error] = { count: 0, jobs: [] };
|
||||
}
|
||||
acc[error.error].count += error.count;
|
||||
acc[error.error].jobs.push(job.id);
|
||||
});
|
||||
return acc;
|
||||
}, {});
|
||||
|
||||
console.log('Error analysis:', errorAnalysis);
|
||||
return errorAnalysis;
|
||||
}
|
||||
```
|
||||
|
||||
#### Slow Processing
|
||||
```javascript
|
||||
// Analyze processing bottlenecks
|
||||
async function analyzePerformance() {
|
||||
const response = await fetch('/api/admin/batch-monitoring?timeRange=24h&includeDetails=true');
|
||||
const data = await response.json();
|
||||
|
||||
const slowJobs = data.data.jobs
|
||||
.filter(job => job.processingTimeMs > 3600000) // > 1 hour
|
||||
.sort((a, b) => b.processingTimeMs - a.processingTimeMs);
|
||||
|
||||
console.log('Slowest jobs:', slowJobs.slice(0, 5));
|
||||
|
||||
// Analyze patterns
|
||||
const avgByType = slowJobs.reduce((acc, job) => {
|
||||
if (!acc[job.jobType]) {
|
||||
acc[job.jobType] = { total: 0, count: 0 };
|
||||
}
|
||||
acc[job.jobType].total += job.processingTimeMs;
|
||||
acc[job.jobType].count++;
|
||||
return acc;
|
||||
}, {});
|
||||
|
||||
Object.keys(avgByType).forEach(type => {
|
||||
avgByType[type].average = avgByType[type].total / avgByType[type].count;
|
||||
});
|
||||
|
||||
return avgByType;
|
||||
}
|
||||
```
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
#### Batch Size Optimization
|
||||
```javascript
|
||||
// Analyze optimal batch sizes
|
||||
async function optimizeBatchSizes() {
|
||||
const response = await fetch('/api/admin/batch-monitoring?timeRange=7d&includeDetails=true');
|
||||
const data = await response.json();
|
||||
|
||||
// Group by batch size ranges
|
||||
const sizePerformance = data.data.jobs.reduce((acc, job) => {
|
||||
const sizeRange = Math.floor(job.requestCount / 50) * 50; // Group by 50s
|
||||
if (!acc[sizeRange]) {
|
||||
acc[sizeRange] = {
|
||||
jobs: 0,
|
||||
totalTime: 0,
|
||||
totalRequests: 0,
|
||||
successRate: 0
|
||||
};
|
||||
}
|
||||
|
||||
acc[sizeRange].jobs++;
|
||||
acc[sizeRange].totalTime += job.processingTimeMs;
|
||||
acc[sizeRange].totalRequests += job.requestCount;
|
||||
acc[sizeRange].successRate += job.completedCount / job.requestCount;
|
||||
|
||||
return acc;
|
||||
}, {});
|
||||
|
||||
// Calculate averages
|
||||
Object.keys(sizePerformance).forEach(range => {
|
||||
const perf = sizePerformance[range];
|
||||
perf.avgTimePerRequest = perf.totalTime / perf.totalRequests;
|
||||
perf.avgSuccessRate = perf.successRate / perf.jobs;
|
||||
});
|
||||
|
||||
return sizePerformance;
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with Existing Systems
|
||||
|
||||
### Security Audit Integration
|
||||
All batch monitoring activities are logged through the security audit system:
|
||||
|
||||
```javascript
|
||||
// Automatic audit logging for monitoring activities
|
||||
await securityAuditLogger.logPlatformAdmin(
|
||||
'batch_monitoring_access',
|
||||
AuditOutcome.SUCCESS,
|
||||
context,
|
||||
'Admin accessed batch monitoring dashboard'
|
||||
);
|
||||
```
|
||||
|
||||
### Rate Limiting Integration
|
||||
Monitoring API endpoints use the existing rate limiting system:
|
||||
|
||||
```javascript
|
||||
// Protected by admin rate limiting
|
||||
const rateLimitResult = await rateLimiter.check(
|
||||
`admin-batch-monitoring:${userId}`,
|
||||
60, // 60 requests
|
||||
60 * 1000 // per minute
|
||||
);
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Batch Processing Optimizations](./batch-processing-optimizations.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [Admin Audit Logs API](./admin-audit-logs-api.md)
|
||||
- [OpenAI Batch API Integration](../lib/batchProcessor.ts)
|
||||
|
||||
The batch monitoring dashboard provides comprehensive visibility into the AI processing pipeline, enabling administrators to optimize performance, monitor costs, and ensure reliable operation of the batch processing system.
|
||||
213
docs/batch-processing-optimizations.md
Normal file
213
docs/batch-processing-optimizations.md
Normal file
@ -0,0 +1,213 @@
|
||||
# Batch Processing Database Query Optimizations
|
||||
|
||||
This document outlines the database query optimizations implemented to improve the performance of the OpenAI Batch API processing pipeline.
|
||||
|
||||
## Overview
|
||||
|
||||
The batch processing system was optimized to reduce database load and improve response times through several key strategies:
|
||||
|
||||
1. **Database Index Optimization**
|
||||
2. **Query Pattern Improvements**
|
||||
3. **Company Caching**
|
||||
4. **Batch Operations**
|
||||
5. **Integration Layer with Fallback**
|
||||
|
||||
## Database Index Improvements
|
||||
|
||||
### New Indexes Added
|
||||
|
||||
The following composite indexes were added to the `AIProcessingRequest` table in the Prisma schema:
|
||||
|
||||
```sql
|
||||
-- Optimize time-based status queries
|
||||
@@index([processingStatus, requestedAt])
|
||||
|
||||
-- Optimize batch-related queries
|
||||
@@index([batchId])
|
||||
|
||||
-- Composite index for batch status filtering
|
||||
@@index([processingStatus, batchId])
|
||||
```
|
||||
|
||||
### Query Performance Impact
|
||||
|
||||
These indexes specifically optimize:
|
||||
- Finding pending requests by status and creation time
|
||||
- Batch-related lookups by batch ID
|
||||
- Combined status and batch filtering operations
|
||||
|
||||
## Query Optimization Strategies
|
||||
|
||||
### 1. Selective Data Fetching
|
||||
|
||||
**Before:**
|
||||
```typescript
|
||||
// Loaded full session with all messages
|
||||
include: {
|
||||
session: {
|
||||
include: {
|
||||
messages: {
|
||||
orderBy: { order: "asc" },
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**After:**
|
||||
```typescript
|
||||
// Only essential data with message count
|
||||
include: {
|
||||
session: {
|
||||
select: {
|
||||
id: true,
|
||||
companyId: true,
|
||||
_count: { select: { messages: true } }
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Company Caching
|
||||
|
||||
Implemented a 5-minute TTL cache for active companies to eliminate redundant database lookups:
|
||||
|
||||
```typescript
|
||||
class CompanyCache {
|
||||
private readonly CACHE_TTL = 5 * 60 * 1000; // 5 minutes
|
||||
|
||||
async getActiveCompanies(): Promise<CachedCompany[]> {
|
||||
// Returns cached data if available and fresh
|
||||
// Otherwise refreshes from database
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Batch Operations
|
||||
|
||||
**Before:** N+1 queries for each company
|
||||
```typescript
|
||||
// Sequential processing per company
|
||||
for (const company of companies) {
|
||||
const requests = await getPendingRequests(company.id);
|
||||
// Process each company separately
|
||||
}
|
||||
```
|
||||
|
||||
**After:** Single query for all companies
|
||||
```typescript
|
||||
// Batch query for all companies at once
|
||||
const allRequests = await prisma.aIProcessingRequest.findMany({
|
||||
where: {
|
||||
session: {
|
||||
companyId: { in: companies.map(c => c.id) },
|
||||
},
|
||||
processingStatus: AIRequestStatus.PENDING_BATCHING,
|
||||
},
|
||||
});
|
||||
|
||||
// Group results by company in memory
|
||||
const requestsByCompany = groupByCompany(allRequests);
|
||||
```
|
||||
|
||||
## Performance Improvements
|
||||
|
||||
### Query Count Reduction
|
||||
|
||||
- **Company lookups:** Reduced from 4 separate queries per scheduler run to 1 cached lookup
|
||||
- **Pending requests:** Reduced from N queries (one per company) to 1 batch query
|
||||
- **Status checks:** Reduced from N queries to 1 batch query
|
||||
- **Failed requests:** Reduced from N queries to 1 batch query
|
||||
|
||||
### Parallel Processing
|
||||
|
||||
Added configurable parallel processing with batching:
|
||||
|
||||
```typescript
|
||||
const SCHEDULER_CONFIG = {
|
||||
MAX_CONCURRENT_COMPANIES: 5,
|
||||
USE_BATCH_OPERATIONS: true,
|
||||
PARALLEL_COMPANY_PROCESSING: true,
|
||||
};
|
||||
```
|
||||
|
||||
### Memory Optimization
|
||||
|
||||
- Eliminated loading unnecessary message content
|
||||
- Used `select` instead of `include` where possible
|
||||
- Implemented automatic cache cleanup
|
||||
|
||||
## Integration Layer
|
||||
|
||||
Created a unified interface that can switch between original and optimized implementations:
|
||||
|
||||
### Environment Configuration
|
||||
|
||||
```bash
|
||||
# Enable optimizations (default: true)
|
||||
ENABLE_BATCH_OPTIMIZATION=true
|
||||
ENABLE_BATCH_OPERATIONS=true
|
||||
ENABLE_PARALLEL_PROCESSING=true
|
||||
|
||||
# Fallback behavior
|
||||
FALLBACK_ON_ERRORS=true
|
||||
```
|
||||
|
||||
### Performance Tracking
|
||||
|
||||
The integration layer automatically tracks performance metrics and can fall back to the original implementation if optimizations fail:
|
||||
|
||||
```typescript
|
||||
class PerformanceTracker {
|
||||
shouldUseOptimized(): boolean {
|
||||
// Uses optimized if faster and success rate > 90%
|
||||
return optimizedAvg < originalAvg && optimizedSuccess > 0.9;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Files Modified
|
||||
|
||||
### New Files
|
||||
- `lib/batchProcessorOptimized.ts` - Optimized query implementations
|
||||
- `lib/batchSchedulerOptimized.ts` - Optimized scheduler
|
||||
- `lib/batchProcessorIntegration.ts` - Integration layer with fallback
|
||||
|
||||
### Modified Files
|
||||
- `prisma/schema.prisma` - Added composite indexes
|
||||
- `server.ts` - Updated to use integration layer
|
||||
- `app/api/admin/batch-monitoring/route.ts` - Updated import
|
||||
|
||||
## Monitoring
|
||||
|
||||
The optimizations include comprehensive logging and monitoring:
|
||||
|
||||
- Performance metrics for each operation type
|
||||
- Cache hit/miss statistics
|
||||
- Fallback events tracking
|
||||
- Query execution time monitoring
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
The integration layer allows for easy rollback:
|
||||
|
||||
1. Set `ENABLE_BATCH_OPTIMIZATION=false`
|
||||
2. System automatically uses original implementation
|
||||
3. No database schema changes needed for rollback
|
||||
4. Indexes remain beneficial for manual queries
|
||||
|
||||
## Expected Performance Gains
|
||||
|
||||
- **Database Query Count:** 60-80% reduction in scheduler operations
|
||||
- **Memory Usage:** 40-60% reduction from selective data loading
|
||||
- **Response Time:** 30-50% improvement for batch operations
|
||||
- **Cache Hit Rate:** 95%+ for company lookups after warmup
|
||||
|
||||
## Testing
|
||||
|
||||
Performance improvements can be validated by:
|
||||
|
||||
1. Monitoring the batch monitoring dashboard
|
||||
2. Checking performance metrics in logs
|
||||
3. Comparing execution times before/after optimization
|
||||
4. Load testing with multiple companies and large batches
|
||||
494
docs/csp-metrics-api.md
Normal file
494
docs/csp-metrics-api.md
Normal file
@ -0,0 +1,494 @@
|
||||
# CSP Metrics and Monitoring API
|
||||
|
||||
This document describes the Content Security Policy (CSP) metrics and violation reporting APIs that provide real-time monitoring and analysis of CSP violations.
|
||||
|
||||
## Overview
|
||||
|
||||
The CSP Metrics API provides comprehensive monitoring of Content Security Policy violations, including:
|
||||
- Real-time violation tracking and metrics
|
||||
- Bypass attempt detection and risk assessment
|
||||
- Policy optimization recommendations
|
||||
- Historical trend analysis
|
||||
- Export capabilities for security analysis
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### CSP Violation Reporting
|
||||
|
||||
Endpoint for browsers to report CSP violations (automatic).
|
||||
|
||||
```http
|
||||
POST /api/csp-report
|
||||
```
|
||||
|
||||
#### Request Headers
|
||||
- `Content-Type`: `application/csp-report` or `application/json`
|
||||
|
||||
#### Request Body (Automatic from Browser)
|
||||
|
||||
```json
|
||||
{
|
||||
"csp-report": {
|
||||
"document-uri": "https://example.com/page",
|
||||
"violated-directive": "script-src 'self'",
|
||||
"blocked-uri": "https://malicious.com/script.js",
|
||||
"source-file": "https://example.com/page",
|
||||
"line-number": 42,
|
||||
"script-sample": "eval(maliciousCode)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Features
|
||||
- **Rate Limiting**: 10 reports per minute per IP
|
||||
- **Risk Assessment**: Automatic classification of violation severity
|
||||
- **Bypass Detection**: Identifies potential CSP bypass attempts
|
||||
- **Real-time Processing**: Immediate analysis and alerting
|
||||
|
||||
### CSP Metrics API
|
||||
|
||||
Retrieve CSP violation metrics and analytics.
|
||||
|
||||
```http
|
||||
GET /api/csp-metrics
|
||||
```
|
||||
|
||||
#### Query Parameters
|
||||
|
||||
| Parameter | Type | Description | Default | Example |
|
||||
|-----------|------|-------------|---------|---------|
|
||||
| `timeRange` | string | Time range for metrics | `24h` | `?timeRange=7d` |
|
||||
| `format` | string | Response format | `json` | `?format=csv` |
|
||||
| `groupBy` | string | Group results by field | `hour` | `?groupBy=directive` |
|
||||
| `includeDetails` | boolean | Include violation details | `false` | `?includeDetails=true` |
|
||||
|
||||
#### Time Range Options
|
||||
- `1h` - Last 1 hour
|
||||
- `6h` - Last 6 hours
|
||||
- `24h` - Last 24 hours (default)
|
||||
- `7d` - Last 7 days
|
||||
- `30d` - Last 30 days
|
||||
|
||||
#### Example Request
|
||||
|
||||
```javascript
|
||||
const response = await fetch('/api/csp-metrics?' + new URLSearchParams({
|
||||
timeRange: '24h',
|
||||
groupBy: 'directive',
|
||||
includeDetails: 'true'
|
||||
}));
|
||||
|
||||
const metrics = await response.json();
|
||||
```
|
||||
|
||||
#### Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"summary": {
|
||||
"totalViolations": 45,
|
||||
"uniqueViolations": 12,
|
||||
"highRiskViolations": 3,
|
||||
"bypassAttempts": 1,
|
||||
"timeRange": "24h",
|
||||
"generatedAt": "2024-01-01T12:00:00Z"
|
||||
},
|
||||
"trends": {
|
||||
"hourlyCount": [
|
||||
{ "hour": "2024-01-01T11:00:00Z", "count": 5 },
|
||||
{ "hour": "2024-01-01T12:00:00Z", "count": 8 }
|
||||
],
|
||||
"trendDirection": "increasing",
|
||||
"changePercent": 25.5
|
||||
},
|
||||
"topViolations": [
|
||||
{
|
||||
"directive": "script-src",
|
||||
"count": 15,
|
||||
"percentage": 33.3,
|
||||
"riskLevel": "medium",
|
||||
"topBlockedUris": [
|
||||
"https://malicious.com/script.js",
|
||||
"inline"
|
||||
]
|
||||
}
|
||||
],
|
||||
"riskAnalysis": {
|
||||
"overallRiskScore": 65,
|
||||
"riskLevel": "medium",
|
||||
"criticalIssues": 1,
|
||||
"recommendations": [
|
||||
"Review script-src policy for external domains",
|
||||
"Consider implementing nonce-based CSP"
|
||||
]
|
||||
},
|
||||
"violations": [
|
||||
{
|
||||
"timestamp": "2024-01-01T12:00:00Z",
|
||||
"directive": "script-src",
|
||||
"blockedUri": "https://malicious.com/script.js",
|
||||
"sourceFile": "https://example.com/page",
|
||||
"riskLevel": "high",
|
||||
"bypassAttempt": true,
|
||||
"ipAddress": "192.168.1.100",
|
||||
"userAgent": "Mozilla/5.0..."
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## CSP Monitoring Service
|
||||
|
||||
The monitoring service (`lib/csp-monitoring.ts`) provides advanced violation analysis.
|
||||
|
||||
### Key Features
|
||||
|
||||
#### 1. Real-time Violation Processing
|
||||
|
||||
```javascript
|
||||
// Automatic processing when violations are reported
|
||||
const result = await cspMonitoring.processViolation(
|
||||
violationReport,
|
||||
clientIP,
|
||||
userAgent
|
||||
);
|
||||
|
||||
console.log(result.alertLevel); // low, medium, high, critical
|
||||
console.log(result.shouldAlert); // boolean
|
||||
console.log(result.recommendations); // array of suggestions
|
||||
```
|
||||
|
||||
#### 2. Risk Assessment
|
||||
|
||||
The service automatically assesses violation risk based on:
|
||||
|
||||
- **Directive Type**: Script violations are higher risk than style violations
|
||||
- **Source Pattern**: External domains vs inline vs data URIs
|
||||
- **Bypass Indicators**: Known CSP bypass techniques
|
||||
- **Frequency**: Repeated violations from same source
|
||||
- **Geographic Factors**: Unusual source locations
|
||||
|
||||
#### 3. Bypass Detection
|
||||
|
||||
Automatic detection of common CSP bypass attempts:
|
||||
|
||||
```javascript
|
||||
const bypassPatterns = [
|
||||
/javascript:/i, // javascript: protocol injection
|
||||
/data:text\/html/i, // HTML data URI injection
|
||||
/eval\(/i, // Direct eval() calls
|
||||
/Function\(/i, // Function constructor
|
||||
/setTimeout.*string/i, // Timer string execution
|
||||
/location\s*=/i, // Location manipulation
|
||||
/document\.write/i, // Document.write injection
|
||||
];
|
||||
```
|
||||
|
||||
#### 4. Policy Recommendations
|
||||
|
||||
Based on violation patterns, the service provides actionable recommendations:
|
||||
|
||||
- **Tighten Policies**: Suggest removing broad allowlists
|
||||
- **Add Domains**: Recommend allowing legitimate external resources
|
||||
- **Implement Nonces**: Suggest nonce-based policies for inline content
|
||||
- **Upgrade Directives**: Recommend modern CSP features
|
||||
|
||||
## Violation Analysis
|
||||
|
||||
### Risk Levels
|
||||
|
||||
| Risk Level | Score | Description | Action |
|
||||
|------------|-------|-------------|---------|
|
||||
| **Critical** | 90-100 | Active bypass attempts, known attack patterns | Immediate investigation |
|
||||
| **High** | 70-89 | Suspicious patterns, potential security risks | Urgent review |
|
||||
| **Medium** | 40-69 | Policy violations, may need attention | Regular monitoring |
|
||||
| **Low** | 0-39 | Minor violations, likely legitimate | Log for trends |
|
||||
|
||||
### Alert Conditions
|
||||
|
||||
```javascript
|
||||
// High-risk violations trigger immediate alerts
|
||||
const alertConditions = {
|
||||
critical: {
|
||||
bypassAttempt: true,
|
||||
unknownExternalDomain: true,
|
||||
suspiciousUserAgent: true
|
||||
},
|
||||
high: {
|
||||
repeatedViolations: '>5 in 10 minutes',
|
||||
scriptInjectionAttempt: true,
|
||||
dataUriWithScript: true
|
||||
},
|
||||
medium: {
|
||||
newExternalDomain: true,
|
||||
inlineScriptViolation: true,
|
||||
unknownSource: true
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Real-time Violation Monitoring
|
||||
|
||||
```javascript
|
||||
// Monitor violations in real-time
|
||||
async function monitorViolations() {
|
||||
const metrics = await fetch('/api/csp-metrics?timeRange=1h');
|
||||
const data = await metrics.json();
|
||||
|
||||
if (data.data.summary.highRiskViolations > 0) {
|
||||
console.warn('High-risk CSP violations detected:',
|
||||
data.data.summary.highRiskViolations);
|
||||
|
||||
// Get violation details
|
||||
const details = await fetch('/api/csp-metrics?includeDetails=true');
|
||||
const violations = await details.json();
|
||||
|
||||
violations.data.violations
|
||||
.filter(v => v.riskLevel === 'high')
|
||||
.forEach(violation => {
|
||||
console.error('High-risk violation:', {
|
||||
directive: violation.directive,
|
||||
blockedUri: violation.blockedUri,
|
||||
timestamp: violation.timestamp
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Run every 5 minutes
|
||||
setInterval(monitorViolations, 5 * 60 * 1000);
|
||||
```
|
||||
|
||||
### Security Dashboard Integration
|
||||
|
||||
```javascript
|
||||
// Get CSP metrics for security dashboard
|
||||
async function getCSPDashboardData() {
|
||||
const [current, previous] = await Promise.all([
|
||||
fetch('/api/csp-metrics?timeRange=24h').then(r => r.json()),
|
||||
fetch('/api/csp-metrics?timeRange=24h&offset=24h').then(r => r.json())
|
||||
]);
|
||||
|
||||
return {
|
||||
currentViolations: current.data.summary.totalViolations,
|
||||
previousViolations: previous.data.summary.totalViolations,
|
||||
trend: current.data.trends.trendDirection,
|
||||
riskScore: current.data.riskAnalysis.overallRiskScore,
|
||||
recommendations: current.data.riskAnalysis.recommendations.slice(0, 3)
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### Export Violation Data
|
||||
|
||||
```javascript
|
||||
// Export violations for external analysis
|
||||
async function exportViolations(format = 'csv', timeRange = '7d') {
|
||||
const response = await fetch(`/api/csp-metrics?format=${format}&timeRange=${timeRange}`);
|
||||
|
||||
if (format === 'csv') {
|
||||
const csvData = await response.text();
|
||||
downloadFile(csvData, `csp-violations-${timeRange}.csv`, 'text/csv');
|
||||
} else {
|
||||
const jsonData = await response.json();
|
||||
downloadFile(JSON.stringify(jsonData, null, 2),
|
||||
`csp-violations-${timeRange}.json`,
|
||||
'application/json');
|
||||
}
|
||||
}
|
||||
|
||||
function downloadFile(content, filename, contentType) {
|
||||
const blob = new Blob([content], { type: contentType });
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = filename;
|
||||
a.click();
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
```
|
||||
|
||||
### Policy Optimization
|
||||
|
||||
```javascript
|
||||
// Analyze violations to optimize CSP policy
|
||||
async function optimizeCSPPolicy() {
|
||||
const metrics = await fetch('/api/csp-metrics?timeRange=30d&includeDetails=true');
|
||||
const data = await metrics.json();
|
||||
|
||||
// Group violations by directive
|
||||
const violationsByDirective = data.data.violations.reduce((acc, violation) => {
|
||||
if (!acc[violation.directive]) {
|
||||
acc[violation.directive] = [];
|
||||
}
|
||||
acc[violation.directive].push(violation);
|
||||
return acc;
|
||||
}, {});
|
||||
|
||||
// Generate recommendations
|
||||
const recommendations = Object.entries(violationsByDirective).map(([directive, violations]) => {
|
||||
const uniqueDomains = [...new Set(violations.map(v => v.blockedUri))];
|
||||
const legitimateCount = violations.filter(v => v.riskLevel === 'low').length;
|
||||
|
||||
if (legitimateCount > violations.length * 0.8) {
|
||||
return {
|
||||
directive,
|
||||
action: 'allow',
|
||||
domains: uniqueDomains.slice(0, 5),
|
||||
confidence: 'high'
|
||||
};
|
||||
} else {
|
||||
return {
|
||||
directive,
|
||||
action: 'investigate',
|
||||
riskDomains: uniqueDomains.filter((_, i) =>
|
||||
violations.find(v => v.blockedUri === uniqueDomains[i])?.riskLevel === 'high'
|
||||
),
|
||||
confidence: 'medium'
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
return recommendations;
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration and Setup
|
||||
|
||||
### CSP Header Configuration
|
||||
|
||||
The CSP metrics system requires proper CSP headers with reporting:
|
||||
|
||||
```javascript
|
||||
// In next.config.js or middleware
|
||||
const cspDirectives = {
|
||||
'default-src': "'self'",
|
||||
'script-src': "'self' 'nonce-{NONCE}'",
|
||||
'report-uri': '/api/csp-report',
|
||||
'report-to': 'csp-endpoint'
|
||||
};
|
||||
```
|
||||
|
||||
### Report-To Header
|
||||
|
||||
For modern browsers, configure the Report-To header:
|
||||
|
||||
```javascript
|
||||
const reportToHeader = JSON.stringify({
|
||||
group: 'csp-endpoint',
|
||||
max_age: 86400,
|
||||
endpoints: [{ url: '/api/csp-report' }]
|
||||
});
|
||||
|
||||
// Add to response headers
|
||||
headers['Report-To'] = reportToHeader;
|
||||
```
|
||||
|
||||
### Environment Configuration
|
||||
|
||||
```bash
|
||||
# Enable CSP monitoring in production
|
||||
NODE_ENV=production
|
||||
|
||||
# Optional: Configure monitoring sensitivity
|
||||
CSP_MONITORING_SENSITIVITY=medium # low, medium, high
|
||||
CSP_ALERT_THRESHOLD=5 # violations per 10 minutes
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Rate Limiting
|
||||
- **10 reports per minute per IP** prevents spam attacks
|
||||
- **Exponential backoff** for repeated violations from same source
|
||||
- **Memory cleanup** removes old violations automatically
|
||||
|
||||
### Memory Management
|
||||
- **Violation buffer** limited to 1 hour of data in memory
|
||||
- **Automatic cleanup** runs every 100 requests (1% probability)
|
||||
- **Efficient storage** using Map data structures
|
||||
|
||||
### Database Impact
|
||||
- **No persistent storage** for real-time metrics (memory only)
|
||||
- **Optional logging** to database for long-term analysis
|
||||
- **Indexed queries** for historical data retrieval
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Privacy Protection
|
||||
- **IP anonymization** option for GDPR compliance
|
||||
- **User agent sanitization** removes sensitive information
|
||||
- **No personal data** stored in violation reports
|
||||
|
||||
### Rate Limiting Protection
|
||||
- **Per-IP limits** prevent DoS attacks on reporting endpoint
|
||||
- **Content-type validation** ensures proper report format
|
||||
- **Request size limits** prevent memory exhaustion
|
||||
|
||||
### False Positive Handling
|
||||
- **Learning mode** for new deployments
|
||||
- **Whitelist support** for known legitimate violations
|
||||
- **Risk score adjustment** based on historical patterns
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### High False Positive Rate
|
||||
```javascript
|
||||
// Check for legitimate violations being flagged
|
||||
const metrics = await fetch('/api/csp-metrics?includeDetails=true');
|
||||
const data = await metrics.json();
|
||||
|
||||
const falsePositives = data.data.violations.filter(v =>
|
||||
v.riskLevel === 'high' &&
|
||||
v.blockedUri.includes('legitimate-domain.com')
|
||||
);
|
||||
|
||||
if (falsePositives.length > 0) {
|
||||
console.log('Consider whitelisting:', falsePositives[0].blockedUri);
|
||||
}
|
||||
```
|
||||
|
||||
#### Missing Violation Reports
|
||||
```javascript
|
||||
// Check if CSP headers are properly configured
|
||||
fetch('/').then(response => {
|
||||
const csp = response.headers.get('Content-Security-Policy');
|
||||
if (!csp.includes('report-uri')) {
|
||||
console.error('CSP report-uri directive missing');
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
#### Performance Issues
|
||||
```javascript
|
||||
// Monitor API response times
|
||||
const start = performance.now();
|
||||
const response = await fetch('/api/csp-metrics');
|
||||
const duration = performance.now() - start;
|
||||
|
||||
if (duration > 2000) {
|
||||
console.warn('CSP metrics API slow response:', duration + 'ms');
|
||||
}
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Enhanced CSP Implementation](./security/enhanced-csp.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [Security Headers](./security-headers.md)
|
||||
- [Rate Limiting](../lib/rateLimiter.ts)
|
||||
|
||||
## API Reference Summary
|
||||
|
||||
| Endpoint | Method | Purpose | Auth Required |
|
||||
|----------|--------|---------|---------------|
|
||||
| `/api/csp-report` | POST | Receive CSP violation reports | No (public) |
|
||||
| `/api/csp-metrics` | GET | Get violation metrics and analytics | Admin |
|
||||
|
||||
Both APIs are production-ready and provide comprehensive CSP monitoring capabilities for enterprise security requirements.
|
||||
263
docs/security-audit-logging.md
Normal file
263
docs/security-audit-logging.md
Normal file
@ -0,0 +1,263 @@
|
||||
# Security Audit Logging System
|
||||
|
||||
This document provides an overview of the comprehensive security audit logging system implemented in LiveDash.
|
||||
|
||||
## Overview
|
||||
|
||||
The security audit logging system provides comprehensive tracking of security-critical events, authentication activities, and administrative actions across the platform. It is designed for compliance, incident investigation, and security monitoring.
|
||||
|
||||
## Features
|
||||
|
||||
### 1. Comprehensive Event Tracking
|
||||
|
||||
The system logs the following event types:
|
||||
|
||||
- **Authentication Events**: Login attempts, password changes, session management
|
||||
- **Authorization Events**: Permission checks, access denied events
|
||||
- **User Management**: User creation, modification, deletion, invitations
|
||||
- **Company Management**: Company suspension, settings changes
|
||||
- **Rate Limiting**: Abuse prevention and rate limit violations
|
||||
- **CSRF Protection**: Cross-site request forgery protection events
|
||||
- **Security Headers**: Security header violations
|
||||
- **Password Reset**: Password reset flows and token validation
|
||||
- **Platform Admin**: Administrative activities by platform users
|
||||
- **Data Privacy**: Data export and privacy-related events
|
||||
- **System Configuration**: System setting changes
|
||||
- **API Security**: API-related security events
|
||||
|
||||
### 2. Structured Logging
|
||||
|
||||
Each audit log entry includes:
|
||||
|
||||
- **Event Type**: Categorizes the security event
|
||||
- **Action**: Specific action performed
|
||||
- **Outcome**: Success, failure, blocked, rate limited, or suspicious
|
||||
- **Severity**: Info, low, medium, high, or critical
|
||||
- **Context**: User ID, company ID, platform user ID, IP address, user agent
|
||||
- **Metadata**: Structured additional information
|
||||
- **Timestamp**: Immutable timestamp for chronological ordering
|
||||
|
||||
### 3. Multi-Tenant Security
|
||||
|
||||
- Company-scoped audit logs ensure data isolation
|
||||
- Platform admin actions tracked separately
|
||||
- Role-based access controls for audit log viewing
|
||||
|
||||
### 4. Log Retention and Management
|
||||
|
||||
- **Configurable Retention Policies**: Different retention periods based on event type and severity
|
||||
- **Automatic Archival**: Critical and high-severity events archived before deletion
|
||||
- **Scheduled Cleanup**: Weekly automated retention policy execution
|
||||
- **Manual Controls**: Admin interface for manual retention execution
|
||||
|
||||
### 5. Administrative Interface
|
||||
|
||||
- **Audit Log Viewer**: Comprehensive filtering and search capabilities
|
||||
- **Retention Management**: View statistics and execute retention policies
|
||||
- **Real-time Monitoring**: Track security events as they occur
|
||||
|
||||
## Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
1. **SecurityAuditLogger** (`lib/securityAuditLogger.ts`): Centralized logging service
|
||||
2. **AuditLogRetentionManager** (`lib/auditLogRetention.ts`): Retention policy management
|
||||
3. **AuditLogScheduler** (`lib/auditLogScheduler.ts`): Scheduled retention execution
|
||||
4. **Admin API** (`app/api/admin/audit-logs/`): REST API for audit log access
|
||||
5. **Admin UI** (`app/dashboard/audit-logs/`): Administrative interface
|
||||
|
||||
### Database Schema
|
||||
|
||||
The `SecurityAuditLog` model includes:
|
||||
|
||||
```prisma
|
||||
model SecurityAuditLog {
|
||||
id String @id @default(uuid())
|
||||
eventType SecurityEventType
|
||||
action String @db.VarChar(255)
|
||||
outcome AuditOutcome
|
||||
severity AuditSeverity @default(INFO)
|
||||
userId String?
|
||||
companyId String?
|
||||
platformUserId String?
|
||||
ipAddress String? @db.Inet
|
||||
userAgent String?
|
||||
country String? @db.VarChar(3)
|
||||
metadata Json?
|
||||
errorMessage String?
|
||||
sessionId String? @db.VarChar(255)
|
||||
requestId String? @db.VarChar(255)
|
||||
timestamp DateTime @default(now()) @db.Timestamptz(6)
|
||||
|
||||
// Relations and indexes...
|
||||
}
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Logging Security Events
|
||||
|
||||
```typescript
|
||||
import { securityAuditLogger, AuditOutcome } from "./lib/securityAuditLogger";
|
||||
|
||||
// Log authentication event
|
||||
await securityAuditLogger.logAuthentication("user_login_success", AuditOutcome.SUCCESS, {
|
||||
userId: "user-123",
|
||||
companyId: "company-456",
|
||||
ipAddress: "192.168.1.1",
|
||||
userAgent: "Mozilla/5.0...",
|
||||
metadata: { loginMethod: "password" },
|
||||
});
|
||||
|
||||
// Log authorization failure
|
||||
await securityAuditLogger.logAuthorization(
|
||||
"admin_access_denied",
|
||||
AuditOutcome.BLOCKED,
|
||||
{
|
||||
userId: "user-123",
|
||||
companyId: "company-456",
|
||||
metadata: { requiredRole: "ADMIN", currentRole: "USER" },
|
||||
},
|
||||
"Insufficient permissions for admin access"
|
||||
);
|
||||
```
|
||||
|
||||
### Viewing Audit Logs
|
||||
|
||||
Administrators can access audit logs through:
|
||||
|
||||
1. **Dashboard UI**: Navigate to "Audit Logs" in the sidebar
|
||||
2. **API Access**: GET `/api/admin/audit-logs` with filtering parameters
|
||||
3. **Retention Management**: GET/POST `/api/admin/audit-logs/retention`
|
||||
|
||||
### Filtering Options
|
||||
|
||||
- Event type (authentication, authorization, etc.)
|
||||
- Outcome (success, failure, blocked, etc.)
|
||||
- Severity level (info, low, medium, high, critical)
|
||||
- Date range
|
||||
- User ID
|
||||
- Pagination support
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Enable/disable audit logging (default: true)
|
||||
AUDIT_LOGGING_ENABLED=true
|
||||
|
||||
# Enable/disable retention scheduler (default: true)
|
||||
AUDIT_LOG_RETENTION_ENABLED=true
|
||||
|
||||
# Retention schedule (cron format, default: 2 AM every Sunday)
|
||||
AUDIT_LOG_RETENTION_SCHEDULE="0 2 * * 0"
|
||||
|
||||
# Dry run mode for retention (default: false)
|
||||
AUDIT_LOG_RETENTION_DRY_RUN=false
|
||||
```
|
||||
|
||||
### Default Retention Policies
|
||||
|
||||
1. **Critical Events**: 7 years retention with archival
|
||||
2. **High Severity Events**: 3 years retention with archival
|
||||
3. **Authentication Events**: 2 years retention with archival
|
||||
4. **Platform Admin Events**: 3 years retention with archival
|
||||
5. **User Management Events**: 2 years retention with archival
|
||||
6. **General Events**: 1 year retention without archival
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Data Protection
|
||||
|
||||
- **IP Address Storage**: Client IP addresses stored for geographic analysis
|
||||
- **Sensitive Data Redaction**: Passwords, tokens, and emails marked as `[REDACTED]`
|
||||
- **Metadata Sanitization**: Complex objects sanitized to prevent data leakage
|
||||
|
||||
### Access Controls
|
||||
|
||||
- **Admin-Only Access**: Only users with `ADMIN` role can view audit logs
|
||||
- **Company Isolation**: Users can only view logs for their own company
|
||||
- **Platform Separation**: Platform admin logs tracked separately
|
||||
|
||||
### Performance
|
||||
|
||||
- **Async Logging**: All logging operations are asynchronous to avoid blocking
|
||||
- **Error Handling**: Logging failures don't affect application functionality
|
||||
- **Indexed Queries**: Database indexes optimize common query patterns
|
||||
- **Batch Operations**: Retention policies use batch operations for efficiency
|
||||
|
||||
## Compliance Features
|
||||
|
||||
### Audit Standards
|
||||
|
||||
- **Immutable Records**: Audit logs cannot be modified after creation
|
||||
- **Chronological Ordering**: Precise timestamps for event sequencing
|
||||
- **Non-Repudiation**: User actions clearly attributed and timestamped
|
||||
- **Comprehensive Coverage**: All security-relevant events logged
|
||||
|
||||
### Reporting
|
||||
|
||||
- **Event Statistics**: Summary statistics by event type, severity, and time period
|
||||
- **Export Capabilities**: Structured data export for compliance reporting
|
||||
- **Retention Tracking**: Detailed logging of retention policy execution
|
||||
|
||||
## Monitoring and Alerting
|
||||
|
||||
### System Health
|
||||
|
||||
- **Scheduler Status**: Monitor retention scheduler health
|
||||
- **Error Tracking**: Log retention and audit logging errors
|
||||
- **Performance Metrics**: Track logging performance and database impact
|
||||
|
||||
### Security Monitoring
|
||||
|
||||
- **Failed Authentication Patterns**: Track repeated login failures
|
||||
- **Privilege Escalation**: Monitor administrative action patterns
|
||||
- **Suspicious Activity**: Identify unusual access patterns
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Audit Logging Disabled**: Check `AUDIT_LOGGING_ENABLED` environment variable
|
||||
2. **Retention Not Running**: Verify `AUDIT_LOG_RETENTION_ENABLED` and scheduler status
|
||||
3. **Access Denied**: Ensure user has `ADMIN` role for audit log access
|
||||
4. **Performance Issues**: Review retention policies and database indexes
|
||||
|
||||
### Debug Information
|
||||
|
||||
- Check application logs for scheduler startup messages
|
||||
- Monitor database query performance for audit log operations
|
||||
- Review retention policy validation warnings
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Implementation
|
||||
|
||||
1. **Always use the centralized logger**: Don't bypass the `securityAuditLogger`
|
||||
2. **Include relevant context**: Provide user, company, and IP information
|
||||
3. **Use appropriate severity levels**: Follow the severity assignment guidelines
|
||||
4. **Sanitize sensitive data**: Use `createAuditMetadata()` for safe metadata
|
||||
|
||||
### Operations
|
||||
|
||||
1. **Regular retention review**: Monitor retention statistics and adjust policies
|
||||
2. **Archive critical data**: Ensure important logs are archived before deletion
|
||||
3. **Monitor storage usage**: Track audit log database growth
|
||||
4. **Test restoration**: Verify archived data can be restored when needed
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
|
||||
- **Real-time Alerting**: Immediate notifications for critical security events
|
||||
- **Advanced Analytics**: ML-based anomaly detection and pattern recognition
|
||||
- **Export Formats**: Additional export formats for compliance reporting
|
||||
- **External Integration**: SIEM and security tool integrations
|
||||
|
||||
### Performance Optimizations
|
||||
|
||||
- **Log Partitioning**: Database partitioning for improved query performance
|
||||
- **Compression**: Log compression for storage efficiency
|
||||
- **Streaming**: Real-time log streaming for external systems
|
||||
@ -11,26 +11,31 @@ The application implements multiple layers of HTTP security headers to provide d
|
||||
### Core Security Headers
|
||||
|
||||
#### X-Content-Type-Options: nosniff
|
||||
|
||||
- **Purpose**: Prevents MIME type sniffing attacks
|
||||
- **Protection**: Stops browsers from interpreting files as different MIME types than declared
|
||||
- **Value**: `nosniff`
|
||||
|
||||
#### X-Frame-Options: DENY
|
||||
|
||||
- **Purpose**: Prevents clickjacking attacks
|
||||
- **Protection**: Blocks embedding the site in frames/iframes
|
||||
- **Value**: `DENY`
|
||||
|
||||
#### X-XSS-Protection: 1; mode=block
|
||||
|
||||
- **Purpose**: Enables XSS protection in legacy browsers
|
||||
- **Protection**: Activates built-in XSS filtering (primarily for older browsers)
|
||||
- **Value**: `1; mode=block`
|
||||
|
||||
#### Referrer-Policy: strict-origin-when-cross-origin
|
||||
|
||||
- **Purpose**: Controls referrer information leakage
|
||||
- **Protection**: Limits referrer data sent to external sites
|
||||
- **Value**: `strict-origin-when-cross-origin`
|
||||
|
||||
#### X-DNS-Prefetch-Control: off
|
||||
|
||||
- **Purpose**: Prevents DNS rebinding attacks
|
||||
- **Protection**: Disables DNS prefetching to reduce attack surface
|
||||
- **Value**: `off`
|
||||
@ -44,6 +49,7 @@ Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'un
|
||||
```
|
||||
|
||||
#### Key CSP Directives:
|
||||
|
||||
- **default-src 'self'**: Restrictive default for all resource types
|
||||
- **script-src 'self' 'unsafe-eval' 'unsafe-inline'**: Allows Next.js dev tools and React functionality
|
||||
- **style-src 'self' 'unsafe-inline'**: Enables TailwindCSS and component styles
|
||||
@ -91,12 +97,15 @@ headers: async () => {
|
||||
},
|
||||
{
|
||||
source: "/(.*)",
|
||||
headers: process.env.NODE_ENV === "production" ? [
|
||||
// HSTS header for production only
|
||||
] : [],
|
||||
headers:
|
||||
process.env.NODE_ENV === "production"
|
||||
? [
|
||||
// HSTS header for production only
|
||||
]
|
||||
: [],
|
||||
},
|
||||
];
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### Environment-Specific Behavior
|
||||
@ -111,6 +120,7 @@ headers: async () => {
|
||||
Location: `tests/unit/http-security-headers.test.ts`
|
||||
|
||||
Tests cover:
|
||||
|
||||
- Individual header validation
|
||||
- CSP directive verification
|
||||
- Permissions Policy validation
|
||||
@ -122,6 +132,7 @@ Tests cover:
|
||||
Location: `tests/integration/security-headers-basic.test.ts`
|
||||
|
||||
Tests cover:
|
||||
|
||||
- Next.js configuration validation
|
||||
- Header generation verification
|
||||
- Environment-based header differences
|
||||
@ -172,6 +183,7 @@ pnpm test:security-headers https://your-domain.com
|
||||
### Future Enhancements
|
||||
|
||||
Planned improvements:
|
||||
|
||||
1. CSP violation reporting endpoint
|
||||
2. Nonce-based CSP for inline scripts
|
||||
3. Additional Permissions Policy restrictions
|
||||
@ -182,6 +194,7 @@ Planned improvements:
|
||||
### Next.js Compatibility
|
||||
|
||||
Headers are configured to be compatible with:
|
||||
|
||||
- Next.js 15+ App Router
|
||||
- React 19 development tools
|
||||
- TailwindCSS 4 styling system
|
||||
@ -190,6 +203,7 @@ Headers are configured to be compatible with:
|
||||
### Browser Support
|
||||
|
||||
Security headers are supported by:
|
||||
|
||||
- All modern browsers (Chrome 60+, Firefox 60+, Safari 12+)
|
||||
- Graceful degradation for older browsers
|
||||
- Progressive enhancement approach
|
||||
@ -214,4 +228,4 @@ Security headers are supported by:
|
||||
- [OWASP Secure Headers Project](https://owasp.org/www-project-secure-headers/)
|
||||
- [MDN Security Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#security)
|
||||
- [Next.js Security Headers](https://nextjs.org/docs/app/api-reference/config/headers)
|
||||
- [Content Security Policy Reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)
|
||||
- [Content Security Policy Reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)
|
||||
|
||||
446
docs/security-monitoring.md
Normal file
446
docs/security-monitoring.md
Normal file
@ -0,0 +1,446 @@
|
||||
# Security Monitoring and Alerting System
|
||||
|
||||
## Overview
|
||||
|
||||
The Security Monitoring and Alerting System provides comprehensive real-time security monitoring, anomaly detection, and threat alerting for the LiveDash-Node application. It integrates with the existing audit logging system to provide proactive security monitoring and incident response capabilities.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
1. **Security Monitoring Service** (`lib/securityMonitoring.ts`)
|
||||
- Real-time event processing
|
||||
- Anomaly detection algorithms
|
||||
- Alert generation and management
|
||||
- Security score calculation
|
||||
- Threat level assessment
|
||||
|
||||
2. **Enhanced Security Logging** (`enhancedSecurityLog`)
|
||||
- Integrates with existing audit logger
|
||||
- Processes events through monitoring system
|
||||
- Triggers immediate threat detection
|
||||
|
||||
3. **API Endpoints** (`app/api/admin/security-monitoring/`)
|
||||
- `/api/admin/security-monitoring` - Main metrics and configuration
|
||||
- `/api/admin/security-monitoring/alerts` - Alert management
|
||||
- `/api/admin/security-monitoring/export` - Data export
|
||||
- `/api/admin/security-monitoring/threat-analysis` - Threat analysis
|
||||
|
||||
4. **Dashboard UI** (`app/platform/security/page.tsx`)
|
||||
- Real-time security metrics
|
||||
- Active alerts management
|
||||
- Threat analysis visualization
|
||||
- Configuration management
|
||||
|
||||
## Features
|
||||
|
||||
### Real-time Monitoring
|
||||
|
||||
- **Authentication Events**: Login attempts, failures, brute force attacks
|
||||
- **Rate Limiting**: Excessive request patterns, API abuse
|
||||
- **Admin Activity**: Unusual administrative actions
|
||||
- **Geographic Anomalies**: Logins from unusual locations
|
||||
- **Temporal Anomalies**: Activity spikes outside normal patterns
|
||||
|
||||
### Alert Types
|
||||
|
||||
```typescript
|
||||
enum AlertType {
|
||||
AUTHENTICATION_ANOMALY = "AUTHENTICATION_ANOMALY",
|
||||
RATE_LIMIT_BREACH = "RATE_LIMIT_BREACH",
|
||||
MULTIPLE_FAILED_LOGINS = "MULTIPLE_FAILED_LOGINS",
|
||||
SUSPICIOUS_IP_ACTIVITY = "SUSPICIOUS_IP_ACTIVITY",
|
||||
PRIVILEGE_ESCALATION = "PRIVILEGE_ESCALATION",
|
||||
DATA_BREACH_ATTEMPT = "DATA_BREACH_ATTEMPT",
|
||||
CSRF_ATTACK = "CSRF_ATTACK",
|
||||
CSP_VIOLATION_SPIKE = "CSP_VIOLATION_SPIKE",
|
||||
ACCOUNT_ENUMERATION = "ACCOUNT_ENUMERATION",
|
||||
BRUTE_FORCE_ATTACK = "BRUTE_FORCE_ATTACK",
|
||||
UNUSUAL_ADMIN_ACTIVITY = "UNUSUAL_ADMIN_ACTIVITY",
|
||||
GEOLOCATION_ANOMALY = "GEOLOCATION_ANOMALY",
|
||||
MASS_DATA_ACCESS = "MASS_DATA_ACCESS",
|
||||
SUSPICIOUS_USER_AGENT = "SUSPICIOUS_USER_AGENT",
|
||||
SESSION_HIJACKING = "SESSION_HIJACKING",
|
||||
}
|
||||
```
|
||||
|
||||
### Anomaly Detection
|
||||
|
||||
The system implements several anomaly detection algorithms:
|
||||
|
||||
1. **Geographic Anomaly Detection**
|
||||
- Detects logins from unusual countries
|
||||
- Compares against historical user patterns
|
||||
- Confidence scoring based on deviation
|
||||
|
||||
2. **Temporal Anomaly Detection**
|
||||
- Identifies activity spikes during unusual hours
|
||||
- Compares current activity to historical averages
|
||||
- Configurable thresholds for different event types
|
||||
|
||||
3. **Behavioral Anomaly Detection**
|
||||
- Multiple failed login attempts
|
||||
- Rapid succession of actions
|
||||
- Pattern deviation analysis
|
||||
|
||||
### Security Scoring
|
||||
|
||||
The system calculates a real-time security score (0-100) based on:
|
||||
|
||||
- Critical security events (weight: 25)
|
||||
- Active unresolved alerts (weight: 30)
|
||||
- High-severity threats (weight: 20)
|
||||
- Overall event volume (weight: 15)
|
||||
- System stability factors (weight: 10)
|
||||
|
||||
### Threat Levels
|
||||
|
||||
```typescript
|
||||
enum ThreatLevel {
|
||||
LOW = "LOW", // Score: 85-100
|
||||
MODERATE = "MODERATE", // Score: 70-84
|
||||
HIGH = "HIGH", // Score: 50-69
|
||||
CRITICAL = "CRITICAL", // Score: 0-49
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Default Thresholds
|
||||
|
||||
```typescript
|
||||
const defaultThresholds = {
|
||||
failedLoginsPerMinute: 5,
|
||||
failedLoginsPerHour: 20,
|
||||
rateLimitViolationsPerMinute: 10,
|
||||
cspViolationsPerMinute: 15,
|
||||
adminActionsPerHour: 25,
|
||||
massDataAccessThreshold: 100,
|
||||
suspiciousIPThreshold: 10,
|
||||
};
|
||||
```
|
||||
|
||||
### Alerting Configuration
|
||||
|
||||
```typescript
|
||||
const alertingConfig = {
|
||||
enabled: true,
|
||||
channels: ["EMAIL", "WEBHOOK", "SLACK", "DISCORD", "PAGERDUTY"],
|
||||
suppressDuplicateMinutes: 10,
|
||||
escalationTimeoutMinutes: 60,
|
||||
};
|
||||
```
|
||||
|
||||
### Data Retention
|
||||
|
||||
```typescript
|
||||
const retentionConfig = {
|
||||
alertRetentionDays: 90,
|
||||
metricsRetentionDays: 365,
|
||||
};
|
||||
```
|
||||
|
||||
## API Usage
|
||||
|
||||
### Get Security Metrics
|
||||
|
||||
```javascript
|
||||
const response = await fetch(
|
||||
"/api/admin/security-monitoring?startDate=2024-01-01T00:00:00Z&endDate=2024-01-02T00:00:00Z"
|
||||
);
|
||||
const data = await response.json();
|
||||
|
||||
console.log(data.metrics.securityScore); // 0-100
|
||||
console.log(data.metrics.threatLevel); // LOW, MODERATE, HIGH, CRITICAL
|
||||
console.log(data.alerts); // Active alerts array
|
||||
```
|
||||
|
||||
### Acknowledge Alert
|
||||
|
||||
```javascript
|
||||
await fetch("/api/admin/security-monitoring/alerts", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
alertId: "alert-123",
|
||||
action: "acknowledge",
|
||||
}),
|
||||
});
|
||||
```
|
||||
|
||||
### Export Security Data
|
||||
|
||||
```javascript
|
||||
// Export alerts as CSV
|
||||
const response = await fetch(
|
||||
"/api/admin/security-monitoring/export?format=csv&type=alerts&startDate=2024-01-01T00:00:00Z&endDate=2024-01-02T00:00:00Z"
|
||||
);
|
||||
const csvData = await response.text();
|
||||
|
||||
// Export metrics as JSON
|
||||
const response = await fetch(
|
||||
"/api/admin/security-monitoring/export?format=json&type=metrics&startDate=2024-01-01T00:00:00Z&endDate=2024-01-02T00:00:00Z"
|
||||
);
|
||||
const jsonData = await response.json();
|
||||
```
|
||||
|
||||
### Perform Threat Analysis
|
||||
|
||||
```javascript
|
||||
const analysis = await fetch("/api/admin/security-monitoring/threat-analysis", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
ipAddress: "192.168.1.100",
|
||||
timeRange: {
|
||||
start: "2024-01-01T00:00:00Z",
|
||||
end: "2024-01-02T00:00:00Z",
|
||||
},
|
||||
}),
|
||||
});
|
||||
|
||||
const data = await analysis.json();
|
||||
console.log(data.ipThreatAnalysis.threatLevel);
|
||||
console.log(data.ipThreatAnalysis.riskFactors);
|
||||
console.log(data.ipThreatAnalysis.recommendations);
|
||||
```
|
||||
|
||||
## Integration with Existing Systems
|
||||
|
||||
### Enhanced Security Logging
|
||||
|
||||
Replace existing `securityAuditLogger.log()` calls with `enhancedSecurityLog()`:
|
||||
|
||||
```typescript
|
||||
// Before
|
||||
await securityAuditLogger.logAuthentication(
|
||||
"login_attempt",
|
||||
AuditOutcome.FAILURE,
|
||||
context,
|
||||
"Invalid password"
|
||||
);
|
||||
|
||||
// After
|
||||
await enhancedSecurityLog(
|
||||
SecurityEventType.AUTHENTICATION,
|
||||
"login_attempt",
|
||||
AuditOutcome.FAILURE,
|
||||
context,
|
||||
AuditSeverity.HIGH,
|
||||
"Invalid password",
|
||||
{
|
||||
attemptType: "invalid_password",
|
||||
endpoint: "/api/auth/signin",
|
||||
}
|
||||
);
|
||||
```
|
||||
|
||||
### Rate Limiting Integration
|
||||
|
||||
The system automatically integrates with existing rate limiting middleware:
|
||||
|
||||
```typescript
|
||||
// middleware/authRateLimit.ts
|
||||
await enhancedSecurityLog(
|
||||
SecurityEventType.RATE_LIMITING,
|
||||
"auth_rate_limit_exceeded",
|
||||
AuditOutcome.RATE_LIMITED,
|
||||
context,
|
||||
AuditSeverity.HIGH,
|
||||
"Authentication rate limit exceeded"
|
||||
);
|
||||
```
|
||||
|
||||
## Dashboard Features
|
||||
|
||||
### Security Overview
|
||||
|
||||
- Real-time security score (0-100)
|
||||
- Current threat level indicator
|
||||
- Active alerts count
|
||||
- Security events summary
|
||||
|
||||
### Alert Management
|
||||
|
||||
- View active and historical alerts
|
||||
- Filter by severity and type
|
||||
- Acknowledge alerts with tracking
|
||||
- Detailed alert context and metadata
|
||||
|
||||
### Threat Analysis
|
||||
|
||||
- Geographic distribution of events
|
||||
- Top threat types and patterns
|
||||
- User risk scoring
|
||||
- IP threat level analysis
|
||||
|
||||
### Configuration Management
|
||||
|
||||
- Adjust detection thresholds
|
||||
- Configure alerting channels
|
||||
- Set data retention policies
|
||||
- Export capabilities
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Memory Management
|
||||
|
||||
- Event buffer limited to 1 hour of data
|
||||
- Automatic cleanup of old alerts (configurable)
|
||||
- Efficient in-memory storage for real-time analysis
|
||||
|
||||
### Database Impact
|
||||
|
||||
- Leverages existing audit log indexes
|
||||
- Optimized queries for time-range filtering
|
||||
- Background processing to avoid blocking operations
|
||||
|
||||
### Scalability
|
||||
|
||||
- Stateless architecture (except for buffering)
|
||||
- Horizontal scaling support
|
||||
- Configurable processing intervals
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Access Control
|
||||
|
||||
- Platform admin authentication required
|
||||
- Role-based access to security endpoints
|
||||
- Audit logging of all monitoring activities
|
||||
|
||||
### Data Privacy
|
||||
|
||||
- Sensitive data redaction in logs
|
||||
- IP address anonymization options
|
||||
- Configurable data retention periods
|
||||
|
||||
### Alert Suppression
|
||||
|
||||
- Duplicate alert suppression (configurable window)
|
||||
- Rate limiting on alert generation
|
||||
- Escalation policies for critical threats
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### Health Checks
|
||||
|
||||
- Monitor service availability
|
||||
- Check alert generation pipeline
|
||||
- Verify data export functionality
|
||||
|
||||
### Regular Tasks
|
||||
|
||||
- Review and adjust thresholds quarterly
|
||||
- Analyze false positive rates
|
||||
- Update threat detection patterns
|
||||
- Clean up old alert data
|
||||
|
||||
### Performance Metrics
|
||||
|
||||
- Alert response time
|
||||
- False positive/negative rates
|
||||
- System resource usage
|
||||
- User engagement with alerts
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
|
||||
1. **Machine Learning Integration**
|
||||
- Behavioral pattern recognition
|
||||
- Adaptive threshold adjustment
|
||||
- Predictive threat modeling
|
||||
|
||||
2. **Advanced Analytics**
|
||||
- Threat intelligence integration
|
||||
- Cross-correlation analysis
|
||||
- Risk trend analysis
|
||||
|
||||
3. **Integration Enhancements**
|
||||
- SIEM system connectors
|
||||
- Webhook customization
|
||||
- Mobile app notifications
|
||||
|
||||
4. **Automated Response**
|
||||
- IP blocking automation
|
||||
- Account suspension workflows
|
||||
- Incident response orchestration
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**High False Positive Rate**
|
||||
|
||||
- Review and adjust detection thresholds
|
||||
- Analyze user behavior patterns
|
||||
- Consider geographical variations
|
||||
|
||||
**Missing Alerts**
|
||||
|
||||
- Check service configuration
|
||||
- Verify audit log integration
|
||||
- Review threshold settings
|
||||
|
||||
**Performance Issues**
|
||||
|
||||
- Monitor memory usage
|
||||
- Adjust cleanup intervals
|
||||
- Optimize database queries
|
||||
|
||||
**Export Failures**
|
||||
|
||||
- Check file permissions
|
||||
- Verify date range validity
|
||||
- Monitor server resources
|
||||
|
||||
### Debugging
|
||||
|
||||
Enable debug logging:
|
||||
|
||||
```typescript
|
||||
securityMonitoring.updateConfig({
|
||||
alerting: {
|
||||
enabled: true,
|
||||
debugMode: true,
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
Check alert generation:
|
||||
|
||||
```typescript
|
||||
const alerts = securityMonitoring.getActiveAlerts();
|
||||
console.log("Active alerts:", alerts.length);
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- Alert generation logic
|
||||
- Anomaly detection algorithms
|
||||
- Configuration management
|
||||
- Data export functionality
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- API endpoint security
|
||||
- Database integration
|
||||
- Real-time event processing
|
||||
- Alert acknowledgment flow
|
||||
|
||||
### Load Testing
|
||||
|
||||
- High-volume event processing
|
||||
- Concurrent alert generation
|
||||
- Database performance under load
|
||||
- Memory usage patterns
|
||||
|
||||
Run tests:
|
||||
|
||||
```bash
|
||||
pnpm test tests/unit/security-monitoring.test.ts
|
||||
pnpm test tests/integration/security-monitoring-api.test.ts
|
||||
```
|
||||
392
docs/security/enhanced-csp.md
Normal file
392
docs/security/enhanced-csp.md
Normal file
@ -0,0 +1,392 @@
|
||||
# Enhanced Content Security Policy (CSP) Implementation
|
||||
|
||||
> **Task 5 Completed**: Refined and strengthened Content Security Policy for maximum XSS protection while maintaining functionality
|
||||
|
||||
This document outlines the comprehensive Content Security Policy implementation for maximum XSS protection while maintaining application functionality.
|
||||
|
||||
## Overview
|
||||
|
||||
The enhanced CSP implementation provides:
|
||||
|
||||
- **Nonce-based script execution** for maximum security in production
|
||||
- **Strict mode policies** with configurable external domain allowlists
|
||||
- **Environment-specific configurations** for development vs production
|
||||
- **CSP violation reporting and monitoring** system with real-time analysis
|
||||
- **Advanced bypass detection and alerting** capabilities with risk assessment
|
||||
- **Comprehensive testing framework** with automated validation
|
||||
- **Performance metrics and policy recommendations**
|
||||
- **Framework compatibility** with Next.js, TailwindCSS, and Leaflet maps
|
||||
|
||||
## Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
1. **CSP Utility Library** (`lib/csp.ts`)
|
||||
- Nonce generation with cryptographic security
|
||||
- Dynamic CSP building based on environment
|
||||
- Violation parsing and bypass detection
|
||||
- Policy validation and testing
|
||||
|
||||
2. **Middleware Implementation** (`middleware.ts`)
|
||||
- Automatic nonce generation per request
|
||||
- Environment-aware policy application
|
||||
- Enhanced security headers
|
||||
- Route-based CSP filtering
|
||||
|
||||
3. **Violation Reporting** (`app/api/csp-report/route.ts`)
|
||||
- Real-time violation monitoring with intelligent analysis
|
||||
- Rate-limited endpoint protection (10 reports/minute per IP)
|
||||
- Advanced bypass attempt detection with risk assessment
|
||||
- Automated alerting for critical violations with recommendations
|
||||
|
||||
4. **Monitoring Service** (`lib/csp-monitoring.ts`)
|
||||
- Violation tracking and metrics collection
|
||||
- Policy recommendation engine based on violation patterns
|
||||
- Export capabilities for external analysis (JSON/CSV)
|
||||
- Automatic cleanup of old violation data
|
||||
|
||||
5. **Metrics API** (`app/api/csp-metrics/route.ts`)
|
||||
- Real-time CSP violation metrics (1h, 6h, 24h, 7d, 30d ranges)
|
||||
- Top violated directives and blocked URIs analysis
|
||||
- Violation trend tracking and visualization data
|
||||
- Policy optimization recommendations
|
||||
|
||||
6. **Testing Framework**
|
||||
- Comprehensive unit and integration tests
|
||||
- Enhanced CSP validation tools with security scoring
|
||||
- Automated compliance verification
|
||||
- Real-world scenario testing for application compatibility
|
||||
|
||||
## CSP Policies
|
||||
|
||||
### Production Environment (Standard Mode)
|
||||
|
||||
```javascript
|
||||
// Nonce-based CSP with broad HTTPS allowlist
|
||||
const productionCSP = {
|
||||
"default-src": ["'self'"],
|
||||
"script-src": ["'self'", "'nonce-{generated}'", "'strict-dynamic'"],
|
||||
"style-src": ["'self'", "'nonce-{generated}'"],
|
||||
"img-src": ["'self'", "data:", "https://schema.org", "https://livedash.notso.ai",
|
||||
"https://*.basemaps.cartocdn.com", "https://*.openstreetmap.org"],
|
||||
"font-src": ["'self'", "data:"],
|
||||
"connect-src": ["'self'", "https://api.openai.com", "https://livedash.notso.ai", "https:"],
|
||||
"object-src": ["'none'"],
|
||||
"base-uri": ["'self'"],
|
||||
"form-action": ["'self'"],
|
||||
"frame-ancestors": ["'none'"],
|
||||
"upgrade-insecure-requests": true,
|
||||
"report-uri": ["/api/csp-report"],
|
||||
"report-to": ["csp-endpoint"]
|
||||
};
|
||||
```
|
||||
|
||||
### Production Environment (Strict Mode)
|
||||
|
||||
```javascript
|
||||
// Strict CSP with minimal external domain allowlist
|
||||
const strictCSP = buildCSP({
|
||||
isDevelopment: false,
|
||||
nonce: generateNonce(),
|
||||
strictMode: true,
|
||||
allowedExternalDomains: [
|
||||
"https://api.openai.com",
|
||||
"https://schema.org"
|
||||
],
|
||||
reportUri: "/api/csp-report"
|
||||
});
|
||||
|
||||
// Results in:
|
||||
// connect-src 'self' https://api.openai.com https://livedash.notso.ai https://schema.org
|
||||
// (No broad "https:" allowlist)
|
||||
```
|
||||
|
||||
### Development Environment
|
||||
|
||||
```javascript
|
||||
// Permissive CSP for development tools
|
||||
const developmentCSP = {
|
||||
"default-src": ["'self'"],
|
||||
"script-src": ["'self'", "'unsafe-eval'", "'unsafe-inline'"], // HMR & dev tools
|
||||
"style-src": ["'self'", "'unsafe-inline'"], // Hot reload
|
||||
"connect-src": ["'self'", "https:", "wss:", "ws:"], // Dev server
|
||||
// ... other directives remain strict
|
||||
};
|
||||
```
|
||||
|
||||
## Security Features
|
||||
|
||||
### 1. Nonce-Based Script Execution
|
||||
|
||||
- **128-bit cryptographically secure nonces** generated per request
|
||||
- **Strict-dynamic policy** prevents inline script execution
|
||||
- **Automatic nonce injection** into layout components
|
||||
|
||||
```tsx
|
||||
// Layout with nonce support
|
||||
export default async function RootLayout({ children }: { children: ReactNode }) {
|
||||
const nonce = await getNonce();
|
||||
|
||||
return (
|
||||
<html>
|
||||
<head>
|
||||
<script
|
||||
type="application/ld+json"
|
||||
nonce={nonce}
|
||||
dangerouslySetInnerHTML={{ __html: JSON.stringify(jsonLd) }}
|
||||
/>
|
||||
</head>
|
||||
<body>
|
||||
<NonceProvider nonce={nonce}>
|
||||
{children}
|
||||
</NonceProvider>
|
||||
</body>
|
||||
</html>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Content Source Restrictions
|
||||
|
||||
#### Script Sources
|
||||
- **Production**: Only `'self'` and nonce-approved scripts
|
||||
- **Development**: Additional `'unsafe-eval'` for dev tools
|
||||
- **Blocked**: All external CDNs, inline scripts without nonce
|
||||
|
||||
#### Style Sources
|
||||
- **Production**: Nonce-based inline styles preferred
|
||||
- **Fallback**: `'unsafe-inline'` for TailwindCSS compatibility
|
||||
- **External**: Only self-hosted stylesheets
|
||||
|
||||
#### Image Sources
|
||||
- **Allowed**: Self, data URIs, schema.org, application domain
|
||||
- **Blocked**: All other external domains
|
||||
|
||||
#### Connection Sources
|
||||
- **Production**: Self, OpenAI API, application domain
|
||||
- **Development**: Additional WebSocket for HMR
|
||||
- **Blocked**: All other external connections
|
||||
|
||||
### 3. XSS Protection Mechanisms
|
||||
|
||||
#### Inline Script Prevention
|
||||
```javascript
|
||||
// Blocked by CSP
|
||||
<script>alert('xss')</script>
|
||||
|
||||
// Allowed with nonce
|
||||
<script nonce="abc123">legitCode()</script>
|
||||
```
|
||||
|
||||
#### Object Injection Prevention
|
||||
```javascript
|
||||
// Completely blocked
|
||||
object-src 'none'
|
||||
```
|
||||
|
||||
#### Base Tag Injection Prevention
|
||||
```javascript
|
||||
// Restricted to same origin
|
||||
base-uri 'self'
|
||||
```
|
||||
|
||||
#### Clickjacking Protection
|
||||
```javascript
|
||||
// No framing allowed
|
||||
frame-ancestors 'none'
|
||||
```
|
||||
|
||||
### 4. Bypass Detection
|
||||
|
||||
The system actively monitors for common CSP bypass attempts:
|
||||
|
||||
```javascript
|
||||
const bypassPatterns = [
|
||||
/javascript:/i, // Protocol injection
|
||||
/data:text\/html/i, // Data URI injection
|
||||
/eval\(/i, // Code evaluation
|
||||
/Function\(/i, // Constructor injection
|
||||
/setTimeout.*string/i, // Timer string execution
|
||||
];
|
||||
```
|
||||
|
||||
## Violation Reporting
|
||||
|
||||
### Report Format
|
||||
|
||||
CSP violations are automatically reported to `/api/csp-report`:
|
||||
|
||||
```json
|
||||
{
|
||||
"csp-report": {
|
||||
"document-uri": "https://example.com/page",
|
||||
"violated-directive": "script-src 'self'",
|
||||
"blocked-uri": "https://evil.com/script.js",
|
||||
"source-file": "https://example.com/page",
|
||||
"line-number": 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Violation Processing
|
||||
|
||||
1. **Rate Limiting**: 10 reports per minute per IP
|
||||
2. **Parsing**: Extract violation details and context
|
||||
3. **Risk Assessment**: Classify as low/medium/high risk
|
||||
4. **Bypass Detection**: Check for known attack patterns
|
||||
5. **Alerting**: Immediate notifications for critical violations
|
||||
|
||||
### Monitoring Dashboard
|
||||
|
||||
Violations are logged with:
|
||||
- Timestamp and source IP
|
||||
- User agent and referer
|
||||
- Violation type and blocked content
|
||||
- Risk level and bypass indicators
|
||||
- Response actions taken
|
||||
|
||||
## Testing and Validation
|
||||
|
||||
### Automated Testing
|
||||
|
||||
```bash
|
||||
# Run CSP-specific tests
|
||||
pnpm test:csp
|
||||
|
||||
# Validate CSP implementation
|
||||
pnpm test:csp:validate
|
||||
|
||||
# Full CSP test suite
|
||||
pnpm test:csp:full
|
||||
```
|
||||
|
||||
### Manual Testing
|
||||
|
||||
1. **Nonce Validation**: Verify unique nonces per request
|
||||
2. **Policy Compliance**: Check all required directives
|
||||
3. **Bypass Resistance**: Test common XSS techniques
|
||||
4. **Framework Compatibility**: Ensure Next.js/TailwindCSS work
|
||||
5. **Performance Impact**: Measure overhead
|
||||
|
||||
### Security Scoring
|
||||
|
||||
The validation framework provides a security score:
|
||||
- **90-100%**: Excellent implementation
|
||||
- **80-89%**: Good with minor improvements needed
|
||||
- **70-79%**: Needs attention
|
||||
- **<70%**: Serious security issues
|
||||
|
||||
## Deployment Considerations
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# CSP is automatically environment-aware
|
||||
NODE_ENV=production # Enables strict CSP
|
||||
NODE_ENV=development # Enables permissive CSP
|
||||
```
|
||||
|
||||
### Performance Impact
|
||||
|
||||
- **Nonce generation**: ~0.1ms per request
|
||||
- **Header processing**: ~0.05ms per request
|
||||
- **Total overhead**: <1ms per request
|
||||
|
||||
### Browser Compatibility
|
||||
|
||||
- **Modern browsers**: Full CSP Level 3 support
|
||||
- **Legacy browsers**: Graceful degradation with X-XSS-Protection
|
||||
- **Reporting**: Supported in all major browsers
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Regular Reviews
|
||||
|
||||
1. **Monthly**: Review violation reports and patterns
|
||||
2. **Quarterly**: Update content source restrictions
|
||||
3. **Per release**: Validate CSP with new features
|
||||
4. **Annually**: Security audit and penetration testing
|
||||
|
||||
### Updates and Modifications
|
||||
|
||||
When adding new content sources:
|
||||
|
||||
1. Update `buildCSP()` function in `lib/csp.ts`
|
||||
2. Add tests for new directives
|
||||
3. Validate security impact
|
||||
4. Update documentation
|
||||
|
||||
### Incident Response
|
||||
|
||||
For CSP violations:
|
||||
|
||||
1. **High-risk violations**: Immediate investigation
|
||||
2. **Bypass attempts**: Security team notification
|
||||
3. **Mass violations**: Check for policy issues
|
||||
4. **False positives**: Adjust policies as needed
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Development
|
||||
|
||||
- Always test CSP changes in development first
|
||||
- Use nonce provider for new inline scripts
|
||||
- Validate external resources before adding
|
||||
- Monitor console for CSP violations
|
||||
|
||||
### Production
|
||||
|
||||
- Never disable CSP in production
|
||||
- Monitor violation rates and patterns
|
||||
- Keep nonce generation entropy high
|
||||
- Regular security audits
|
||||
|
||||
### Code Review
|
||||
|
||||
- Check all inline scripts have nonce
|
||||
- Verify external resources are approved
|
||||
- Ensure CSP tests pass
|
||||
- Document any policy changes
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Inline styles blocked**: Use nonce or move to external CSS
|
||||
2. **Third-party scripts blocked**: Add to approved sources
|
||||
3. **Dev tools not working**: Ensure development CSP allows unsafe-eval
|
||||
4. **Images not loading**: Check image source restrictions
|
||||
|
||||
### Debug Tools
|
||||
|
||||
```bash
|
||||
# Test CSP generation
|
||||
pnpm test:csp
|
||||
|
||||
# Validate current implementation
|
||||
pnpm test:csp:validate
|
||||
|
||||
# Check specific violations
|
||||
curl -X POST /api/csp-report -d '{"csp-report": {...}}'
|
||||
```
|
||||
|
||||
### Emergency Procedures
|
||||
|
||||
If CSP breaks production:
|
||||
|
||||
1. Check violation reports for patterns
|
||||
2. Identify blocking directive
|
||||
3. Test fix in staging environment
|
||||
4. Deploy emergency policy update
|
||||
5. Monitor for resolved issues
|
||||
|
||||
## Compliance
|
||||
|
||||
This CSP implementation addresses:
|
||||
|
||||
- **OWASP Top 10**: XSS prevention
|
||||
- **CSP Level 3**: Modern security standards
|
||||
- **GDPR**: Privacy-preserving monitoring
|
||||
- **SOC 2**: Security controls documentation
|
||||
|
||||
The enhanced CSP provides defense-in-depth against XSS attacks while maintaining application functionality and performance.
|
||||
Reference in New Issue
Block a user