mirror of
https://github.com/kjanat/livedash-node.git
synced 2026-01-16 12:32:10 +01:00
style: formatted the docs with prettier
This commit is contained in:
@ -24,7 +24,7 @@ POST /api/csp-report
|
||||
|
||||
#### Request Headers
|
||||
|
||||
- `Content-Type`: `application/csp-report` or `application/json`
|
||||
- `Content-Type`: `application/csp-report` or `application/json`
|
||||
|
||||
#### Request Body (Automatic from Browser)
|
||||
|
||||
@ -59,7 +59,7 @@ GET /api/csp-metrics
|
||||
#### Query Parameters
|
||||
|
||||
| Parameter | Type | Description | Default | Example |
|
||||
| ---------------- | ------- | ------------------------- | ------- | ---------------------- |
|
||||
| ---------------- | ------- | ------------------------------------------------------------------------------- | ------- | ---------------------- |
|
||||
| `timeRange` | string | Time range for metrics | `24h` | `?timeRange=7d` |
|
||||
| `format` | string | Response format | `json` | `?format=csv` |
|
||||
| `groupBy` | string | Group results by field | `hour` | `?groupBy=directive` |
|
||||
@ -68,11 +68,11 @@ GET /api/csp-metrics
|
||||
|
||||
#### Time Range Options
|
||||
|
||||
- `1h` - Last 1 hour
|
||||
- `6h` - Last 6 hours
|
||||
- `24h` - Last 24 hours (default)
|
||||
- `7d` - Last 7 days
|
||||
- `30d` - Last 30 days
|
||||
- `1h` - Last 1 hour
|
||||
- `6h` - Last 6 hours
|
||||
- `24h` - Last 24 hours (default)
|
||||
- `7d` - Last 7 days
|
||||
- `30d` - Last 30 days
|
||||
|
||||
#### Example Request
|
||||
|
||||
@ -166,11 +166,11 @@ console.log(result.recommendations); // array of suggestions
|
||||
|
||||
The service automatically assesses violation risk based on:
|
||||
|
||||
- **Directive Type**: Script violations are higher risk than style violations
|
||||
- **Source Pattern**: External domains vs inline vs data URIs
|
||||
- **Bypass Indicators**: Known CSP bypass techniques
|
||||
- **Frequency**: Repeated violations from same source
|
||||
- **Geographic Factors**: Unusual source locations
|
||||
- **Directive Type**: Script violations are higher risk than style violations
|
||||
- **Source Pattern**: External domains vs inline vs data URIs
|
||||
- **Bypass Indicators**: Known CSP bypass techniques
|
||||
- **Frequency**: Repeated violations from same source
|
||||
- **Geographic Factors**: Unusual source locations
|
||||
|
||||
#### 3. Bypass Detection
|
||||
|
||||
@ -192,10 +192,10 @@ const bypassPatterns = [
|
||||
|
||||
Based on violation patterns, the service provides actionable recommendations:
|
||||
|
||||
- **Tighten Policies**: Suggest removing broad allowlists
|
||||
- **Add Domains**: Recommend allowing legitimate external resources
|
||||
- **Implement Nonces**: Suggest nonce-based policies for inline content
|
||||
- **Upgrade Directives**: Recommend modern CSP features
|
||||
- **Tighten Policies**: Suggest removing broad allowlists
|
||||
- **Add Domains**: Recommend allowing legitimate external resources
|
||||
- **Implement Nonces**: Suggest nonce-based policies for inline content
|
||||
- **Upgrade Directives**: Recommend modern CSP features
|
||||
|
||||
## Violation Analysis
|
||||
|
||||
@ -406,22 +406,22 @@ CSP_ALERT_THRESHOLD=5 # violations per 10 minutes
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
- **10 reports per minute per IP** prevents spam attacks
|
||||
- **Exponential backoff** for repeated violations from same source
|
||||
- **Memory cleanup** removes old violations automatically
|
||||
- **10 reports per minute per IP** prevents spam attacks
|
||||
- **Exponential backoff** for repeated violations from same source
|
||||
- **Memory cleanup** removes old violations automatically
|
||||
|
||||
### Memory Management
|
||||
|
||||
- **Violation buffer** limited to 7 days of data in memory
|
||||
- **Hard cap** of 10,000 violation entries to prevent memory exhaustion
|
||||
- **Automatic cleanup** runs every 100 requests (1% probability)
|
||||
- **Efficient storage** using Map data structures
|
||||
- **Violation buffer** limited to 7 days of data in memory
|
||||
- **Hard cap** of 10,000 violation entries to prevent memory exhaustion
|
||||
- **Automatic cleanup** runs every 100 requests (1% probability)
|
||||
- **Efficient storage** using Map data structures
|
||||
|
||||
### Database Impact
|
||||
|
||||
- **No persistent storage** for real-time metrics (memory only)
|
||||
- **Optional logging** to database for long-term analysis
|
||||
- **Indexed queries** for historical data retrieval
|
||||
- **No persistent storage** for real-time metrics (memory only)
|
||||
- **Optional logging** to database for long-term analysis
|
||||
- **Indexed queries** for historical data retrieval
|
||||
|
||||
## Security Considerations
|
||||
|
||||
@ -429,28 +429,28 @@ CSP_ALERT_THRESHOLD=5 # violations per 10 minutes
|
||||
|
||||
**⚠️ Data Collection Notice:**
|
||||
|
||||
- **IP addresses** are collected and stored in memory for security monitoring
|
||||
- **User agent strings** are stored for browser compatibility analysis
|
||||
- **Legal basis**: Legitimate interest for security incident detection and prevention
|
||||
- **Retention**: In-memory storage only, automatically purged after 7 days or application restart
|
||||
- **Data minimization**: Only violation-related metadata is retained, not page content
|
||||
- **IP addresses** are collected and stored in memory for security monitoring
|
||||
- **User agent strings** are stored for browser compatibility analysis
|
||||
- **Legal basis**: Legitimate interest for security incident detection and prevention
|
||||
- **Retention**: In-memory storage only, automatically purged after 7 days or application restart
|
||||
- **Data minimization**: Only violation-related metadata is retained, not page content
|
||||
|
||||
**Planned Privacy Enhancements:**
|
||||
|
||||
- IP anonymization options for GDPR compliance (roadmap)
|
||||
- User agent sanitization to remove sensitive information (roadmap)
|
||||
- IP anonymization options for GDPR compliance (roadmap)
|
||||
- User agent sanitization to remove sensitive information (roadmap)
|
||||
|
||||
### Rate-Limiting Protection
|
||||
|
||||
- **Per-IP limits** prevent DoS attacks on reporting endpoint
|
||||
- **Content-type validation** ensures proper report format
|
||||
- **Request size limits** prevent memory exhaustion
|
||||
- **Per-IP limits** prevent DoS attacks on reporting endpoint
|
||||
- **Content-type validation** ensures proper report format
|
||||
- **Request size limits** prevent memory exhaustion
|
||||
|
||||
### False Positive Handling
|
||||
|
||||
- **Learning mode** for new deployments
|
||||
- **Whitelist support** for known legitimate violations
|
||||
- **Risk score adjustment** based on historical patterns
|
||||
- **Learning mode** for new deployments
|
||||
- **Whitelist support** for known legitimate violations
|
||||
- **Risk score adjustment** based on historical patterns
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@ -499,10 +499,10 @@ if (duration > 2000) {
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Enhanced CSP Implementation](./security/enhanced-csp.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [Security Headers](./security-headers.md)
|
||||
- [Rate Limiting](../lib/rateLimiter.ts)
|
||||
- [Enhanced CSP Implementation](./security/enhanced-csp.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [Security Headers](./security-headers.md)
|
||||
- [Rate Limiting](../lib/rateLimiter.ts)
|
||||
|
||||
## API Reference Summary
|
||||
|
||||
|
||||
@ -45,16 +45,19 @@ DATABASE_URL="postgresql://user:pass@host:5432/db?connection_limit=20&pool_timeo
|
||||
### Fixed Issues
|
||||
|
||||
1. **Multiple PrismaClient Instances**:
|
||||
- ❌ Before: Each scheduler created its own PrismaClient
|
||||
- ✅ After: All modules use singleton pattern from `lib/prisma.ts`
|
||||
|
||||
- ❌ Before: Each scheduler created its own PrismaClient
|
||||
- ✅ After: All modules use singleton pattern from `lib/prisma.ts`
|
||||
|
||||
2. **No Connection Management**:
|
||||
- ❌ Before: No graceful shutdown or connection cleanup
|
||||
- ✅ After: Proper cleanup on process termination
|
||||
|
||||
- ❌ Before: No graceful shutdown or connection cleanup
|
||||
- ✅ After: Proper cleanup on process termination
|
||||
|
||||
3. **No Monitoring**:
|
||||
- ❌ Before: No visibility into connection usage
|
||||
- ✅ After: Health check endpoint and connection metrics
|
||||
|
||||
- ❌ Before: No visibility into connection usage
|
||||
- ✅ After: Health check endpoint and connection metrics
|
||||
|
||||
### Key Files Modified
|
||||
|
||||
|
||||
@ -6,10 +6,10 @@ This document outlines the comprehensive database performance optimizations impl
|
||||
|
||||
The optimization focuses on the most frequently queried patterns in the application, particularly around:
|
||||
|
||||
- AI processing request tracking and batching
|
||||
- Session analytics and filtering
|
||||
- Security audit log analysis
|
||||
- Multi-tenant data isolation performance
|
||||
- AI processing request tracking and batching
|
||||
- Session analytics and filtering
|
||||
- Security audit log analysis
|
||||
- Multi-tenant data isolation performance
|
||||
|
||||
## Applied Optimizations
|
||||
|
||||
@ -31,9 +31,9 @@ INCLUDE ("processingStatus", "batchId", "requestedAt", "sessionId");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~70% faster batch job queries
|
||||
- Reduced I/O for cost analysis reports
|
||||
- Improved scheduler performance
|
||||
- ~70% faster batch job queries
|
||||
- Reduced I/O for cost analysis reports
|
||||
- Improved scheduler performance
|
||||
|
||||
### 2. Session Analytics Optimizations
|
||||
|
||||
@ -54,9 +54,9 @@ INCLUDE ("startTime", "messagesSent");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~85% faster dashboard load times
|
||||
- Efficient date range filtering
|
||||
- Optimized sentiment analysis queries
|
||||
- ~85% faster dashboard load times
|
||||
- Efficient date range filtering
|
||||
- Optimized sentiment analysis queries
|
||||
|
||||
### 3. Security Audit Log Optimizations
|
||||
|
||||
@ -77,9 +77,9 @@ INCLUDE ("eventType", "severity", "userId", "companyId");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~90% faster security monitoring
|
||||
- Efficient threat detection
|
||||
- Improved compliance reporting
|
||||
- ~90% faster security monitoring
|
||||
- Efficient threat detection
|
||||
- Improved compliance reporting
|
||||
|
||||
### 4. Message Processing Optimizations
|
||||
|
||||
@ -95,8 +95,8 @@ INCLUDE ("content");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~60% faster conversation loading
|
||||
- Reduced memory usage for message queries
|
||||
- ~60% faster conversation loading
|
||||
- Reduced memory usage for message queries
|
||||
|
||||
### 5. Processing Pipeline Optimizations
|
||||
|
||||
@ -118,29 +118,29 @@ INCLUDE ("sessionId", "errorMessage", "retryCount", "startedAt");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~75% faster processing monitoring
|
||||
- Efficient error tracking
|
||||
- Improved retry logic performance
|
||||
- ~75% faster processing monitoring
|
||||
- Efficient error tracking
|
||||
- Improved retry logic performance
|
||||
|
||||
## Index Strategy Principles
|
||||
|
||||
### 1. Composite Index Design
|
||||
|
||||
- **Leading column**: Most selective filter (usually companyId for multi-tenancy)
|
||||
- **Secondary columns**: Common WHERE clause filters
|
||||
- **Covering columns**: SELECT list columns via INCLUDE
|
||||
- **Leading column**: Most selective filter (usually companyId for multi-tenancy)
|
||||
- **Secondary columns**: Common WHERE clause filters
|
||||
- **Covering columns**: SELECT list columns via INCLUDE
|
||||
|
||||
### 2. Partial Indexes
|
||||
|
||||
- Used for error analysis and specific status filtering
|
||||
- Reduces index size and maintenance overhead
|
||||
- Improves write performance
|
||||
- Used for error analysis and specific status filtering
|
||||
- Reduces index size and maintenance overhead
|
||||
- Improves write performance
|
||||
|
||||
### 3. Covering Indexes
|
||||
|
||||
- Include frequently accessed columns to avoid table lookups
|
||||
- Reduces I/O for read-heavy operations
|
||||
- Particularly effective for dashboard queries
|
||||
- Include frequently accessed columns to avoid table lookups
|
||||
- Reduces I/O for read-heavy operations
|
||||
- Particularly effective for dashboard queries
|
||||
|
||||
## Query Pattern Analysis
|
||||
|
||||
@ -166,29 +166,29 @@ INCLUDE ("sessionId", "errorMessage", "retryCount", "startedAt");
|
||||
|
||||
### Index Monitoring
|
||||
|
||||
- Monitor index usage with `pg_stat_user_indexes`
|
||||
- Track bloat with `pg_stat_user_tables`
|
||||
- Regular ANALYZE after bulk operations
|
||||
- Monitor index usage with `pg_stat_user_indexes`
|
||||
- Track bloat with `pg_stat_user_tables`
|
||||
- Regular ANALYZE after bulk operations
|
||||
|
||||
### Write Performance Impact
|
||||
|
||||
- Composite indexes add ~15% write overhead
|
||||
- Offset by dramatic read performance gains
|
||||
- Monitored via slow query logs
|
||||
- Composite indexes add ~15% write overhead
|
||||
- Offset by dramatic read performance gains
|
||||
- Monitored via slow query logs
|
||||
|
||||
### Storage Impact
|
||||
|
||||
- Indexes add ~25% to total storage
|
||||
- Covering indexes reduce need for table scans
|
||||
- Partial indexes minimize storage overhead
|
||||
- Indexes add ~25% to total storage
|
||||
- Covering indexes reduce need for table scans
|
||||
- Partial indexes minimize storage overhead
|
||||
|
||||
## Migration Safety
|
||||
|
||||
### CONCURRENTLY Operations
|
||||
|
||||
- All indexes created with `CREATE INDEX CONCURRENTLY`
|
||||
- No table locks during creation
|
||||
- Production-safe deployment
|
||||
- All indexes created with `CREATE INDEX CONCURRENTLY`
|
||||
- No table locks during creation
|
||||
- Production-safe deployment
|
||||
|
||||
### Rollback Strategy
|
||||
|
||||
@ -238,18 +238,18 @@ LIMIT 10;
|
||||
|
||||
### Monitoring Strategy
|
||||
|
||||
- Set up automated index usage monitoring
|
||||
- Track slow query evolution
|
||||
- Monitor storage growth patterns
|
||||
- Implement performance alerting
|
||||
- Set up automated index usage monitoring
|
||||
- Track slow query evolution
|
||||
- Monitor storage growth patterns
|
||||
- Implement performance alerting
|
||||
|
||||
## Conclusion
|
||||
|
||||
These database optimizations provide:
|
||||
|
||||
- **70-90% improvement** in query performance
|
||||
- **Reduced server load** through efficient indexing
|
||||
- **Better user experience** with faster dashboards
|
||||
- **Scalable foundation** for future growth
|
||||
- **70-90% improvement** in query performance
|
||||
- **Reduced server load** through efficient indexing
|
||||
- **Better user experience** with faster dashboards
|
||||
- **Scalable foundation** for future growth
|
||||
|
||||
The optimizations are designed to be production-safe and monitoring-friendly, ensuring both immediate performance gains and long-term maintainability.
|
||||
|
||||
@ -380,25 +380,29 @@ csv-import-scheduler-eu:
|
||||
### From Current Architecture
|
||||
|
||||
1. **Phase 1: Extract Schedulers**
|
||||
- ✅ Create BaseSchedulerService
|
||||
- ✅ Implement CsvImportSchedulerService
|
||||
- ✅ Create SchedulerManager
|
||||
- ⏳ Implement remaining scheduler services
|
||||
|
||||
- ✅ Create BaseSchedulerService
|
||||
- ✅ Implement CsvImportSchedulerService
|
||||
- ✅ Create SchedulerManager
|
||||
- ⏳ Implement remaining scheduler services
|
||||
|
||||
2. **Phase 2: Deployment Options**
|
||||
- ✅ Add ServerSchedulerIntegration for backwards compatibility
|
||||
- ✅ Create StandaloneSchedulerRunner
|
||||
- ✅ Add health check endpoints
|
||||
|
||||
- ✅ Add ServerSchedulerIntegration for backwards compatibility
|
||||
- ✅ Create StandaloneSchedulerRunner
|
||||
- ✅ Add health check endpoints
|
||||
|
||||
3. **Phase 3: Container Support**
|
||||
- ⏳ Create Dockerfile for scheduler containers
|
||||
- ⏳ Add Kubernetes manifests
|
||||
- ⏳ Implement distributed coordination
|
||||
|
||||
- ⏳ Create Dockerfile for scheduler containers
|
||||
- ⏳ Add Kubernetes manifests
|
||||
- ⏳ Implement distributed coordination
|
||||
|
||||
4. **Phase 4: Production Migration**
|
||||
- ⏳ Deploy separate scheduler containers
|
||||
- ⏳ Monitor performance and stability
|
||||
- ⏳ Gradually increase horizontal scaling
|
||||
|
||||
- ⏳ Deploy separate scheduler containers
|
||||
- ⏳ Monitor performance and stability
|
||||
- ⏳ Gradually increase horizontal scaling
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
|
||||
@ -9,28 +9,32 @@ The Security Monitoring and Alerting System provides comprehensive real-time sec
|
||||
### Core Components
|
||||
|
||||
1. **Security Monitoring Service** (`lib/securityMonitoring.ts`)
|
||||
- Real-time event processing
|
||||
- Anomaly detection algorithms
|
||||
- Alert generation and management
|
||||
- Security score calculation
|
||||
- Threat level assessment
|
||||
|
||||
- Real-time event processing
|
||||
- Anomaly detection algorithms
|
||||
- Alert generation and management
|
||||
- Security score calculation
|
||||
- Threat level assessment
|
||||
|
||||
2. **Enhanced Security Logging** (`enhancedSecurityLog`)
|
||||
- Integrates with existing audit logger
|
||||
- Processes events through monitoring system
|
||||
- Triggers immediate threat detection
|
||||
|
||||
- Integrates with existing audit logger
|
||||
- Processes events through monitoring system
|
||||
- Triggers immediate threat detection
|
||||
|
||||
3. **API Endpoints** (`app/api/admin/security-monitoring/`)
|
||||
- `/api/admin/security-monitoring` - Main metrics and configuration
|
||||
- `/api/admin/security-monitoring/alerts` - Alert management
|
||||
- `/api/admin/security-monitoring/export` - Data export
|
||||
- `/api/admin/security-monitoring/threat-analysis` - Threat analysis
|
||||
|
||||
- `/api/admin/security-monitoring` - Main metrics and configuration
|
||||
- `/api/admin/security-monitoring/alerts` - Alert management
|
||||
- `/api/admin/security-monitoring/export` - Data export
|
||||
- `/api/admin/security-monitoring/threat-analysis` - Threat analysis
|
||||
|
||||
4. **Dashboard UI** (`app/platform/security/page.tsx`)
|
||||
- Real-time security metrics
|
||||
- Active alerts management
|
||||
- Threat analysis visualization
|
||||
- Configuration management
|
||||
|
||||
- Real-time security metrics
|
||||
- Active alerts management
|
||||
- Threat analysis visualization
|
||||
- Configuration management
|
||||
|
||||
## Features
|
||||
|
||||
@ -69,19 +73,22 @@ enum AlertType {
|
||||
The system implements several anomaly detection algorithms:
|
||||
|
||||
1. **Geographic Anomaly Detection**
|
||||
- Detects logins from unusual countries
|
||||
- Compares against historical user patterns
|
||||
- Confidence scoring based on deviation
|
||||
|
||||
- Detects logins from unusual countries
|
||||
- Compares against historical user patterns
|
||||
- Confidence scoring based on deviation
|
||||
|
||||
2. **Temporal Anomaly Detection**
|
||||
- Identifies activity spikes during unusual hours
|
||||
- Compares current activity to historical averages
|
||||
- Configurable thresholds for different event types
|
||||
|
||||
- Identifies activity spikes during unusual hours
|
||||
- Compares current activity to historical averages
|
||||
- Configurable thresholds for different event types
|
||||
|
||||
3. **Behavioral Anomaly Detection**
|
||||
- Multiple failed login attempts
|
||||
- Rapid succession of actions
|
||||
- Pattern deviation analysis
|
||||
|
||||
- Multiple failed login attempts
|
||||
- Rapid succession of actions
|
||||
- Pattern deviation analysis
|
||||
|
||||
### Security Scoring
|
||||
|
||||
@ -348,24 +355,28 @@ await enhancedSecurityLog(
|
||||
### Planned Features
|
||||
|
||||
1. **Machine Learning Integration**
|
||||
- Behavioral pattern recognition
|
||||
- Adaptive threshold adjustment
|
||||
- Predictive threat modeling
|
||||
|
||||
- Behavioral pattern recognition
|
||||
- Adaptive threshold adjustment
|
||||
- Predictive threat modeling
|
||||
|
||||
2. **Advanced Analytics**
|
||||
- Threat intelligence integration
|
||||
- Cross-correlation analysis
|
||||
- Risk trend analysis
|
||||
|
||||
- Threat intelligence integration
|
||||
- Cross-correlation analysis
|
||||
- Risk trend analysis
|
||||
|
||||
3. **Integration Enhancements**
|
||||
- SIEM system connectors
|
||||
- Webhook customization
|
||||
- Mobile app notifications
|
||||
|
||||
- SIEM system connectors
|
||||
- Webhook customization
|
||||
- Mobile app notifications
|
||||
|
||||
4. **Automated Response**
|
||||
- IP blocking automation
|
||||
- Account suspension workflows
|
||||
- Incident response orchestration
|
||||
|
||||
- IP blocking automation
|
||||
- Account suspension workflows
|
||||
- Incident response orchestration
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
Reference in New Issue
Block a user