mirror of
https://github.com/kjanat/livedash-node.git
synced 2026-01-16 10:12:09 +01:00
style: formatted the docs with prettier
This commit is contained in:
@ -24,7 +24,7 @@ POST /api/csp-report
|
||||
|
||||
#### Request Headers
|
||||
|
||||
- `Content-Type`: `application/csp-report` or `application/json`
|
||||
- `Content-Type`: `application/csp-report` or `application/json`
|
||||
|
||||
#### Request Body (Automatic from Browser)
|
||||
|
||||
@ -58,21 +58,21 @@ GET /api/csp-metrics
|
||||
|
||||
#### Query Parameters
|
||||
|
||||
| Parameter | Type | Description | Default | Example |
|
||||
| ---------------- | ------- | ------------------------- | ------- | ---------------------- |
|
||||
| `timeRange` | string | Time range for metrics | `24h` | `?timeRange=7d` |
|
||||
| `format` | string | Response format | `json` | `?format=csv` |
|
||||
| `groupBy` | string | Group results by field | `hour` | `?groupBy=directive` |
|
||||
| `includeDetails` | boolean | Include violation details | `false` | `?includeDetails=true` |
|
||||
| `offset` | string | Shift the queried time-window backwards by the given duration (for comparisons) | `0` | `?offset=24h` |
|
||||
| Parameter | Type | Description | Default | Example |
|
||||
| ---------------- | ------- | ------------------------------------------------------------------------------- | ------- | ---------------------- |
|
||||
| `timeRange` | string | Time range for metrics | `24h` | `?timeRange=7d` |
|
||||
| `format` | string | Response format | `json` | `?format=csv` |
|
||||
| `groupBy` | string | Group results by field | `hour` | `?groupBy=directive` |
|
||||
| `includeDetails` | boolean | Include violation details | `false` | `?includeDetails=true` |
|
||||
| `offset` | string | Shift the queried time-window backwards by the given duration (for comparisons) | `0` | `?offset=24h` |
|
||||
|
||||
#### Time Range Options
|
||||
|
||||
- `1h` - Last 1 hour
|
||||
- `6h` - Last 6 hours
|
||||
- `24h` - Last 24 hours (default)
|
||||
- `7d` - Last 7 days
|
||||
- `30d` - Last 30 days
|
||||
- `1h` - Last 1 hour
|
||||
- `6h` - Last 6 hours
|
||||
- `24h` - Last 24 hours (default)
|
||||
- `7d` - Last 7 days
|
||||
- `30d` - Last 30 days
|
||||
|
||||
#### Example Request
|
||||
|
||||
@ -166,11 +166,11 @@ console.log(result.recommendations); // array of suggestions
|
||||
|
||||
The service automatically assesses violation risk based on:
|
||||
|
||||
- **Directive Type**: Script violations are higher risk than style violations
|
||||
- **Source Pattern**: External domains vs inline vs data URIs
|
||||
- **Bypass Indicators**: Known CSP bypass techniques
|
||||
- **Frequency**: Repeated violations from same source
|
||||
- **Geographic Factors**: Unusual source locations
|
||||
- **Directive Type**: Script violations are higher risk than style violations
|
||||
- **Source Pattern**: External domains vs inline vs data URIs
|
||||
- **Bypass Indicators**: Known CSP bypass techniques
|
||||
- **Frequency**: Repeated violations from same source
|
||||
- **Geographic Factors**: Unusual source locations
|
||||
|
||||
#### 3. Bypass Detection
|
||||
|
||||
@ -192,10 +192,10 @@ const bypassPatterns = [
|
||||
|
||||
Based on violation patterns, the service provides actionable recommendations:
|
||||
|
||||
- **Tighten Policies**: Suggest removing broad allowlists
|
||||
- **Add Domains**: Recommend allowing legitimate external resources
|
||||
- **Implement Nonces**: Suggest nonce-based policies for inline content
|
||||
- **Upgrade Directives**: Recommend modern CSP features
|
||||
- **Tighten Policies**: Suggest removing broad allowlists
|
||||
- **Add Domains**: Recommend allowing legitimate external resources
|
||||
- **Implement Nonces**: Suggest nonce-based policies for inline content
|
||||
- **Upgrade Directives**: Recommend modern CSP features
|
||||
|
||||
## Violation Analysis
|
||||
|
||||
@ -406,22 +406,22 @@ CSP_ALERT_THRESHOLD=5 # violations per 10 minutes
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
- **10 reports per minute per IP** prevents spam attacks
|
||||
- **Exponential backoff** for repeated violations from same source
|
||||
- **Memory cleanup** removes old violations automatically
|
||||
- **10 reports per minute per IP** prevents spam attacks
|
||||
- **Exponential backoff** for repeated violations from same source
|
||||
- **Memory cleanup** removes old violations automatically
|
||||
|
||||
### Memory Management
|
||||
|
||||
- **Violation buffer** limited to 7 days of data in memory
|
||||
- **Hard cap** of 10,000 violation entries to prevent memory exhaustion
|
||||
- **Automatic cleanup** runs every 100 requests (1% probability)
|
||||
- **Efficient storage** using Map data structures
|
||||
- **Violation buffer** limited to 7 days of data in memory
|
||||
- **Hard cap** of 10,000 violation entries to prevent memory exhaustion
|
||||
- **Automatic cleanup** runs every 100 requests (1% probability)
|
||||
- **Efficient storage** using Map data structures
|
||||
|
||||
### Database Impact
|
||||
|
||||
- **No persistent storage** for real-time metrics (memory only)
|
||||
- **Optional logging** to database for long-term analysis
|
||||
- **Indexed queries** for historical data retrieval
|
||||
- **No persistent storage** for real-time metrics (memory only)
|
||||
- **Optional logging** to database for long-term analysis
|
||||
- **Indexed queries** for historical data retrieval
|
||||
|
||||
## Security Considerations
|
||||
|
||||
@ -429,28 +429,28 @@ CSP_ALERT_THRESHOLD=5 # violations per 10 minutes
|
||||
|
||||
**⚠️ Data Collection Notice:**
|
||||
|
||||
- **IP addresses** are collected and stored in memory for security monitoring
|
||||
- **User agent strings** are stored for browser compatibility analysis
|
||||
- **Legal basis**: Legitimate interest for security incident detection and prevention
|
||||
- **Retention**: In-memory storage only, automatically purged after 7 days or application restart
|
||||
- **Data minimization**: Only violation-related metadata is retained, not page content
|
||||
- **IP addresses** are collected and stored in memory for security monitoring
|
||||
- **User agent strings** are stored for browser compatibility analysis
|
||||
- **Legal basis**: Legitimate interest for security incident detection and prevention
|
||||
- **Retention**: In-memory storage only, automatically purged after 7 days or application restart
|
||||
- **Data minimization**: Only violation-related metadata is retained, not page content
|
||||
|
||||
**Planned Privacy Enhancements:**
|
||||
|
||||
- IP anonymization options for GDPR compliance (roadmap)
|
||||
- User agent sanitization to remove sensitive information (roadmap)
|
||||
- IP anonymization options for GDPR compliance (roadmap)
|
||||
- User agent sanitization to remove sensitive information (roadmap)
|
||||
|
||||
### Rate-Limiting Protection
|
||||
|
||||
- **Per-IP limits** prevent DoS attacks on reporting endpoint
|
||||
- **Content-type validation** ensures proper report format
|
||||
- **Request size limits** prevent memory exhaustion
|
||||
- **Per-IP limits** prevent DoS attacks on reporting endpoint
|
||||
- **Content-type validation** ensures proper report format
|
||||
- **Request size limits** prevent memory exhaustion
|
||||
|
||||
### False Positive Handling
|
||||
|
||||
- **Learning mode** for new deployments
|
||||
- **Whitelist support** for known legitimate violations
|
||||
- **Risk score adjustment** based on historical patterns
|
||||
- **Learning mode** for new deployments
|
||||
- **Whitelist support** for known legitimate violations
|
||||
- **Risk score adjustment** based on historical patterns
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@ -499,10 +499,10 @@ if (duration > 2000) {
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Enhanced CSP Implementation](./security/enhanced-csp.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [Security Headers](./security-headers.md)
|
||||
- [Rate Limiting](../lib/rateLimiter.ts)
|
||||
- [Enhanced CSP Implementation](./security/enhanced-csp.md)
|
||||
- [Security Monitoring](./security-monitoring.md)
|
||||
- [Security Headers](./security-headers.md)
|
||||
- [Rate Limiting](../lib/rateLimiter.ts)
|
||||
|
||||
## API Reference Summary
|
||||
|
||||
|
||||
@ -12,10 +12,10 @@ The WordCloud component visualizes categories or topics based on their frequency
|
||||
|
||||
**Features:**
|
||||
|
||||
- Dynamic sizing based on frequency
|
||||
- Colorful display with a pleasing color palette
|
||||
- Responsive design
|
||||
- Interactive hover effects
|
||||
- Dynamic sizing based on frequency
|
||||
- Colorful display with a pleasing color palette
|
||||
- Responsive design
|
||||
- Interactive hover effects
|
||||
|
||||
### 2. GeographicMap
|
||||
|
||||
@ -25,10 +25,10 @@ This component displays a world map with circles representing the number of sess
|
||||
|
||||
**Features:**
|
||||
|
||||
- Interactive map using React Leaflet
|
||||
- Circle sizes scaled by session count
|
||||
- Tooltips showing country names and session counts
|
||||
- Responsive design
|
||||
- Interactive map using React Leaflet
|
||||
- Circle sizes scaled by session count
|
||||
- Tooltips showing country names and session counts
|
||||
- Responsive design
|
||||
|
||||
### 3. MetricCard
|
||||
|
||||
@ -38,10 +38,10 @@ A modern, visually appealing card for displaying key metrics.
|
||||
|
||||
**Features:**
|
||||
|
||||
- Multiple design variants (default, primary, success, warning, danger)
|
||||
- Support for trend indicators
|
||||
- Icons and descriptions
|
||||
- Clean, modern styling
|
||||
- Multiple design variants (default, primary, success, warning, danger)
|
||||
- Support for trend indicators
|
||||
- Icons and descriptions
|
||||
- Clean, modern styling
|
||||
|
||||
### 4. DonutChart
|
||||
|
||||
@ -51,10 +51,10 @@ An enhanced donut chart with better styling and a central text display capabilit
|
||||
|
||||
**Features:**
|
||||
|
||||
- Customizable colors
|
||||
- Center text area for displaying summaries
|
||||
- Interactive tooltips with percentages
|
||||
- Well-balanced legend display
|
||||
- Customizable colors
|
||||
- Center text area for displaying summaries
|
||||
- Interactive tooltips with percentages
|
||||
- Well-balanced legend display
|
||||
|
||||
### 5. ResponseTimeDistribution
|
||||
|
||||
@ -64,10 +64,10 @@ Visualizes the distribution of response times as a histogram.
|
||||
|
||||
**Features:**
|
||||
|
||||
- Color-coded bars (green for fast, yellow for medium, red for slow)
|
||||
- Target time indicator
|
||||
- Automatic binning of response times
|
||||
- Clear labeling and scales
|
||||
- Color-coded bars (green for fast, yellow for medium, red for slow)
|
||||
- Target time indicator
|
||||
- Automatic binning of response times
|
||||
- Clear labeling and scales
|
||||
|
||||
## Dashboard Enhancements
|
||||
|
||||
@ -85,7 +85,7 @@ The dashboard has been enhanced with:
|
||||
|
||||
## Usage Notes
|
||||
|
||||
- The geographic map and response time distribution use simulated data where actual data is not available
|
||||
- All components are responsive and will adjust to different screen sizes
|
||||
- The dashboard automatically refreshes data when using the refresh button
|
||||
- Admin users have access to additional controls at the bottom of the dashboard
|
||||
- The geographic map and response time distribution use simulated data where actual data is not available
|
||||
- All components are responsive and will adjust to different screen sizes
|
||||
- The dashboard automatically refreshes data when using the refresh button
|
||||
- Admin users have access to additional controls at the bottom of the dashboard
|
||||
|
||||
@ -29,42 +29,45 @@ DATABASE_URL="postgresql://user:pass@host:5432/db?connection_limit=20&pool_timeo
|
||||
|
||||
#### Standard Pooling (Default)
|
||||
|
||||
- Uses Prisma's built-in connection pooling
|
||||
- Simpler configuration
|
||||
- Good for development and small-scale deployments
|
||||
- Uses Prisma's built-in connection pooling
|
||||
- Simpler configuration
|
||||
- Good for development and small-scale deployments
|
||||
|
||||
#### Enhanced Pooling (Recommended for Production)
|
||||
|
||||
- Uses PostgreSQL native connection pooling with `@prisma/adapter-pg`
|
||||
- Advanced monitoring and health checks
|
||||
- Better resource management
|
||||
- Detailed connection metrics
|
||||
- Uses PostgreSQL native connection pooling with `@prisma/adapter-pg`
|
||||
- Advanced monitoring and health checks
|
||||
- Better resource management
|
||||
- Detailed connection metrics
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Fixed Issues
|
||||
|
||||
1. **Multiple PrismaClient Instances**:
|
||||
- ❌ Before: Each scheduler created its own PrismaClient
|
||||
- ✅ After: All modules use singleton pattern from `lib/prisma.ts`
|
||||
|
||||
- ❌ Before: Each scheduler created its own PrismaClient
|
||||
- ✅ After: All modules use singleton pattern from `lib/prisma.ts`
|
||||
|
||||
2. **No Connection Management**:
|
||||
- ❌ Before: No graceful shutdown or connection cleanup
|
||||
- ✅ After: Proper cleanup on process termination
|
||||
|
||||
- ❌ Before: No graceful shutdown or connection cleanup
|
||||
- ✅ After: Proper cleanup on process termination
|
||||
|
||||
3. **No Monitoring**:
|
||||
- ❌ Before: No visibility into connection usage
|
||||
- ✅ After: Health check endpoint and connection metrics
|
||||
|
||||
- ❌ Before: No visibility into connection usage
|
||||
- ✅ After: Health check endpoint and connection metrics
|
||||
|
||||
### Key Files Modified
|
||||
|
||||
- `lib/prisma.ts` - Enhanced singleton with pooling options
|
||||
- `lib/database-pool.ts` - Advanced pooling configuration
|
||||
- `lib/processingScheduler.ts` - Fixed to use singleton
|
||||
- `lib/importProcessor.ts` - Fixed to use singleton
|
||||
- `lib/processingStatusManager.ts` - Fixed to use singleton
|
||||
- `lib/schedulers.ts` - Added graceful shutdown
|
||||
- `app/api/admin/database-health/route.ts` - Monitoring endpoint
|
||||
- `lib/prisma.ts` - Enhanced singleton with pooling options
|
||||
- `lib/database-pool.ts` - Advanced pooling configuration
|
||||
- `lib/processingScheduler.ts` - Fixed to use singleton
|
||||
- `lib/importProcessor.ts` - Fixed to use singleton
|
||||
- `lib/processingStatusManager.ts` - Fixed to use singleton
|
||||
- `lib/schedulers.ts` - Added graceful shutdown
|
||||
- `app/api/admin/database-health/route.ts` - Monitoring endpoint
|
||||
|
||||
## Monitoring
|
||||
|
||||
@ -79,36 +82,36 @@ curl -H "Authorization: Bearer your-token" \
|
||||
|
||||
Response includes:
|
||||
|
||||
- Connection status
|
||||
- Pool statistics (if enhanced pooling enabled)
|
||||
- Basic metrics (session counts, etc.)
|
||||
- Configuration details
|
||||
- Connection status
|
||||
- Pool statistics (if enhanced pooling enabled)
|
||||
- Basic metrics (session counts, etc.)
|
||||
- Configuration details
|
||||
|
||||
### Connection Metrics
|
||||
|
||||
With enhanced pooling enabled, you'll see console logs for:
|
||||
|
||||
- Connection acquisitions/releases
|
||||
- Pool size changes
|
||||
- Error events
|
||||
- Health check results
|
||||
- Connection acquisitions/releases
|
||||
- Pool size changes
|
||||
- Error events
|
||||
- Health check results
|
||||
|
||||
## Performance Benefits
|
||||
|
||||
### Before Optimization
|
||||
|
||||
- Multiple connection pools (one per scheduler)
|
||||
- Potential connection exhaustion under load
|
||||
- No connection monitoring
|
||||
- Resource waste from idle connections
|
||||
- Multiple connection pools (one per scheduler)
|
||||
- Potential connection exhaustion under load
|
||||
- No connection monitoring
|
||||
- Resource waste from idle connections
|
||||
|
||||
### After Optimization
|
||||
|
||||
- Single shared connection pool
|
||||
- Configurable pool size and timeouts
|
||||
- Connection health monitoring
|
||||
- Graceful shutdown and cleanup
|
||||
- Better resource utilization
|
||||
- Single shared connection pool
|
||||
- Configurable pool size and timeouts
|
||||
- Connection health monitoring
|
||||
- Graceful shutdown and cleanup
|
||||
- Better resource utilization
|
||||
|
||||
## Recommended Settings
|
||||
|
||||
|
||||
@ -6,10 +6,10 @@ This document outlines the comprehensive database performance optimizations impl
|
||||
|
||||
The optimization focuses on the most frequently queried patterns in the application, particularly around:
|
||||
|
||||
- AI processing request tracking and batching
|
||||
- Session analytics and filtering
|
||||
- Security audit log analysis
|
||||
- Multi-tenant data isolation performance
|
||||
- AI processing request tracking and batching
|
||||
- Session analytics and filtering
|
||||
- Security audit log analysis
|
||||
- Multi-tenant data isolation performance
|
||||
|
||||
## Applied Optimizations
|
||||
|
||||
@ -31,9 +31,9 @@ INCLUDE ("processingStatus", "batchId", "requestedAt", "sessionId");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~70% faster batch job queries
|
||||
- Reduced I/O for cost analysis reports
|
||||
- Improved scheduler performance
|
||||
- ~70% faster batch job queries
|
||||
- Reduced I/O for cost analysis reports
|
||||
- Improved scheduler performance
|
||||
|
||||
### 2. Session Analytics Optimizations
|
||||
|
||||
@ -54,9 +54,9 @@ INCLUDE ("startTime", "messagesSent");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~85% faster dashboard load times
|
||||
- Efficient date range filtering
|
||||
- Optimized sentiment analysis queries
|
||||
- ~85% faster dashboard load times
|
||||
- Efficient date range filtering
|
||||
- Optimized sentiment analysis queries
|
||||
|
||||
### 3. Security Audit Log Optimizations
|
||||
|
||||
@ -77,9 +77,9 @@ INCLUDE ("eventType", "severity", "userId", "companyId");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~90% faster security monitoring
|
||||
- Efficient threat detection
|
||||
- Improved compliance reporting
|
||||
- ~90% faster security monitoring
|
||||
- Efficient threat detection
|
||||
- Improved compliance reporting
|
||||
|
||||
### 4. Message Processing Optimizations
|
||||
|
||||
@ -95,8 +95,8 @@ INCLUDE ("content");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~60% faster conversation loading
|
||||
- Reduced memory usage for message queries
|
||||
- ~60% faster conversation loading
|
||||
- Reduced memory usage for message queries
|
||||
|
||||
### 5. Processing Pipeline Optimizations
|
||||
|
||||
@ -118,29 +118,29 @@ INCLUDE ("sessionId", "errorMessage", "retryCount", "startedAt");
|
||||
|
||||
**Impact**:
|
||||
|
||||
- ~75% faster processing monitoring
|
||||
- Efficient error tracking
|
||||
- Improved retry logic performance
|
||||
- ~75% faster processing monitoring
|
||||
- Efficient error tracking
|
||||
- Improved retry logic performance
|
||||
|
||||
## Index Strategy Principles
|
||||
|
||||
### 1. Composite Index Design
|
||||
|
||||
- **Leading column**: Most selective filter (usually companyId for multi-tenancy)
|
||||
- **Secondary columns**: Common WHERE clause filters
|
||||
- **Covering columns**: SELECT list columns via INCLUDE
|
||||
- **Leading column**: Most selective filter (usually companyId for multi-tenancy)
|
||||
- **Secondary columns**: Common WHERE clause filters
|
||||
- **Covering columns**: SELECT list columns via INCLUDE
|
||||
|
||||
### 2. Partial Indexes
|
||||
|
||||
- Used for error analysis and specific status filtering
|
||||
- Reduces index size and maintenance overhead
|
||||
- Improves write performance
|
||||
- Used for error analysis and specific status filtering
|
||||
- Reduces index size and maintenance overhead
|
||||
- Improves write performance
|
||||
|
||||
### 3. Covering Indexes
|
||||
|
||||
- Include frequently accessed columns to avoid table lookups
|
||||
- Reduces I/O for read-heavy operations
|
||||
- Particularly effective for dashboard queries
|
||||
- Include frequently accessed columns to avoid table lookups
|
||||
- Reduces I/O for read-heavy operations
|
||||
- Particularly effective for dashboard queries
|
||||
|
||||
## Query Pattern Analysis
|
||||
|
||||
@ -166,29 +166,29 @@ INCLUDE ("sessionId", "errorMessage", "retryCount", "startedAt");
|
||||
|
||||
### Index Monitoring
|
||||
|
||||
- Monitor index usage with `pg_stat_user_indexes`
|
||||
- Track bloat with `pg_stat_user_tables`
|
||||
- Regular ANALYZE after bulk operations
|
||||
- Monitor index usage with `pg_stat_user_indexes`
|
||||
- Track bloat with `pg_stat_user_tables`
|
||||
- Regular ANALYZE after bulk operations
|
||||
|
||||
### Write Performance Impact
|
||||
|
||||
- Composite indexes add ~15% write overhead
|
||||
- Offset by dramatic read performance gains
|
||||
- Monitored via slow query logs
|
||||
- Composite indexes add ~15% write overhead
|
||||
- Offset by dramatic read performance gains
|
||||
- Monitored via slow query logs
|
||||
|
||||
### Storage Impact
|
||||
|
||||
- Indexes add ~25% to total storage
|
||||
- Covering indexes reduce need for table scans
|
||||
- Partial indexes minimize storage overhead
|
||||
- Indexes add ~25% to total storage
|
||||
- Covering indexes reduce need for table scans
|
||||
- Partial indexes minimize storage overhead
|
||||
|
||||
## Migration Safety
|
||||
|
||||
### CONCURRENTLY Operations
|
||||
|
||||
- All indexes created with `CREATE INDEX CONCURRENTLY`
|
||||
- No table locks during creation
|
||||
- Production-safe deployment
|
||||
- All indexes created with `CREATE INDEX CONCURRENTLY`
|
||||
- No table locks during creation
|
||||
- Production-safe deployment
|
||||
|
||||
### Rollback Strategy
|
||||
|
||||
@ -238,18 +238,18 @@ LIMIT 10;
|
||||
|
||||
### Monitoring Strategy
|
||||
|
||||
- Set up automated index usage monitoring
|
||||
- Track slow query evolution
|
||||
- Monitor storage growth patterns
|
||||
- Implement performance alerting
|
||||
- Set up automated index usage monitoring
|
||||
- Track slow query evolution
|
||||
- Monitor storage growth patterns
|
||||
- Implement performance alerting
|
||||
|
||||
## Conclusion
|
||||
|
||||
These database optimizations provide:
|
||||
|
||||
- **70-90% improvement** in query performance
|
||||
- **Reduced server load** through efficient indexing
|
||||
- **Better user experience** with faster dashboards
|
||||
- **Scalable foundation** for future growth
|
||||
- **70-90% improvement** in query performance
|
||||
- **Reduced server load** through efficient indexing
|
||||
- **Better user experience** with faster dashboards
|
||||
- **Scalable foundation** for future growth
|
||||
|
||||
The optimizations are designed to be production-safe and monitoring-friendly, ensuring both immediate performance gains and long-term maintainability.
|
||||
|
||||
@ -15,21 +15,21 @@ Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.ne
|
||||
|
||||
### 1. Neon Connection Limits
|
||||
|
||||
- **Free Tier**: 20 concurrent connections
|
||||
- **Pro Tier**: 100 concurrent connections
|
||||
- **Multiple schedulers** can quickly exhaust connections
|
||||
- **Free Tier**: 20 concurrent connections
|
||||
- **Pro Tier**: 100 concurrent connections
|
||||
- **Multiple schedulers** can quickly exhaust connections
|
||||
|
||||
### 2. Connection Pooling Issues
|
||||
|
||||
- Each scheduler was creating separate PrismaClient instances
|
||||
- No connection reuse between operations
|
||||
- No retry logic for temporary failures
|
||||
- Each scheduler was creating separate PrismaClient instances
|
||||
- No connection reuse between operations
|
||||
- No retry logic for temporary failures
|
||||
|
||||
### 3. Neon-Specific Challenges
|
||||
|
||||
- **Auto-pause**: Databases pause after inactivity
|
||||
- **Cold starts**: First connection after pause takes longer
|
||||
- **Regional latency**: eu-central-1 may have variable latency
|
||||
- **Auto-pause**: Databases pause after inactivity
|
||||
- **Cold starts**: First connection after pause takes longer
|
||||
- **Regional latency**: eu-central-1 may have variable latency
|
||||
|
||||
## Solutions Implemented
|
||||
|
||||
@ -106,9 +106,9 @@ curl -H "Authorization: Bearer YOUR_API_TOKEN" \
|
||||
|
||||
### 2. Neon Dashboard Monitoring
|
||||
|
||||
- Monitor "Active connections" in Neon dashboard
|
||||
- Check for connection spikes during scheduler runs
|
||||
- Review query performance and slow queries
|
||||
- Monitor "Active connections" in Neon dashboard
|
||||
- Check for connection spikes during scheduler runs
|
||||
- Review query performance and slow queries
|
||||
|
||||
### 3. Application Logs
|
||||
|
||||
@ -158,9 +158,9 @@ const prisma = new PrismaClient({
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Neon database auto-paused
|
||||
- Connection limit exceeded
|
||||
- Network issues
|
||||
- Neon database auto-paused
|
||||
- Connection limit exceeded
|
||||
- Network issues
|
||||
|
||||
**Solutions:**
|
||||
|
||||
@ -173,9 +173,9 @@ const prisma = new PrismaClient({
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Idle connection timeout
|
||||
- Neon maintenance
|
||||
- Long-running transactions
|
||||
- Idle connection timeout
|
||||
- Neon maintenance
|
||||
- Long-running transactions
|
||||
|
||||
**Solutions:**
|
||||
|
||||
@ -187,9 +187,9 @@ const prisma = new PrismaClient({
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Blocking database operations
|
||||
- Scheduler overlap
|
||||
- High CPU usage
|
||||
- Blocking database operations
|
||||
- Scheduler overlap
|
||||
- High CPU usage
|
||||
|
||||
**Solutions:**
|
||||
|
||||
@ -230,10 +230,10 @@ SESSION_PROCESSING_INTERVAL="0 */2 * * *"
|
||||
|
||||
## Monitoring Checklist
|
||||
|
||||
- [ ] Check Neon dashboard for connection spikes
|
||||
- [ ] Monitor scheduler execution times
|
||||
- [ ] Review error logs for connection patterns
|
||||
- [ ] Test health endpoint regularly
|
||||
- [ ] Set up alerts for connection failures
|
||||
- [ ] Check Neon dashboard for connection spikes
|
||||
- [ ] Monitor scheduler execution times
|
||||
- [ ] Review error logs for connection patterns
|
||||
- [ ] Test health endpoint regularly
|
||||
- [ ] Set up alerts for connection failures
|
||||
|
||||
With these optimizations, your Neon database connections should be much more stable and efficient!
|
||||
|
||||
@ -17,48 +17,48 @@ Successfully migrated the livedash-node application from SQLite to PostgreSQL us
|
||||
|
||||
#### Production/Development
|
||||
|
||||
- **Provider**: PostgreSQL (Neon)
|
||||
- **Environment Variable**: `DATABASE_URL`
|
||||
- **Connection**: Neon PostgreSQL cluster
|
||||
- **Provider**: PostgreSQL (Neon)
|
||||
- **Environment Variable**: `DATABASE_URL`
|
||||
- **Connection**: Neon PostgreSQL cluster
|
||||
|
||||
#### Testing
|
||||
|
||||
- **Provider**: PostgreSQL (Neon - separate database)
|
||||
- **Environment Variable**: `DATABASE_URL_TEST`
|
||||
- **Test Setup**: Automatically switches to test database during test runs
|
||||
- **Provider**: PostgreSQL (Neon - separate database)
|
||||
- **Environment Variable**: `DATABASE_URL_TEST`
|
||||
- **Test Setup**: Automatically switches to test database during test runs
|
||||
|
||||
### Files Modified
|
||||
|
||||
1. **`prisma/schema.prisma`**
|
||||
|
||||
- Changed provider from `sqlite` to `postgresql`
|
||||
- Updated URL to use `env("DATABASE_URL")`
|
||||
- Changed provider from `sqlite` to `postgresql`
|
||||
- Updated URL to use `env("DATABASE_URL")`
|
||||
|
||||
2. **`tests/setup.ts`**
|
||||
|
||||
- Added logic to use `DATABASE_URL_TEST` when available
|
||||
- Ensures test isolation with separate database
|
||||
- Added logic to use `DATABASE_URL_TEST` when available
|
||||
- Ensures test isolation with separate database
|
||||
|
||||
3. **`.env`** (created)
|
||||
|
||||
- Contains `DATABASE_URL` for Prisma CLI operations
|
||||
- Contains `DATABASE_URL` for Prisma CLI operations
|
||||
|
||||
4. **`.env.local`** (existing)
|
||||
|
||||
- Contains both `DATABASE_URL` and `DATABASE_URL_TEST`
|
||||
- Contains both `DATABASE_URL` and `DATABASE_URL_TEST`
|
||||
|
||||
### Database Schema
|
||||
|
||||
All existing models and relationships were preserved:
|
||||
|
||||
- **Company**: Multi-tenant root entity
|
||||
- **User**: Authentication and authorization
|
||||
- **Session**: Processed session data
|
||||
- **SessionImport**: Raw CSV import data
|
||||
- **Message**: Individual conversation messages
|
||||
- **Question**: Normalized question storage
|
||||
- **SessionQuestion**: Session-question relationships
|
||||
- **AIProcessingRequest**: AI cost tracking
|
||||
- **Company**: Multi-tenant root entity
|
||||
- **User**: Authentication and authorization
|
||||
- **Session**: Processed session data
|
||||
- **SessionImport**: Raw CSV import data
|
||||
- **Message**: Individual conversation messages
|
||||
- **Question**: Normalized question storage
|
||||
- **SessionQuestion**: Session-question relationships
|
||||
- **AIProcessingRequest**: AI cost tracking
|
||||
|
||||
### Migration Process
|
||||
|
||||
@ -103,11 +103,11 @@ if (process.env.DATABASE_URL_TEST) {
|
||||
|
||||
All tests pass successfully:
|
||||
|
||||
- ✅ Environment configuration tests
|
||||
- ✅ Transcript fetcher tests
|
||||
- ✅ Database connection tests
|
||||
- ✅ Schema validation tests
|
||||
- ✅ CRUD operation tests
|
||||
- ✅ Environment configuration tests
|
||||
- ✅ Transcript fetcher tests
|
||||
- ✅ Database connection tests
|
||||
- ✅ Schema validation tests
|
||||
- ✅ CRUD operation tests
|
||||
|
||||
### Next Steps
|
||||
|
||||
|
||||
@ -6,11 +6,11 @@ This document describes the extracted scheduler architecture that enables horizo
|
||||
|
||||
The scheduler system has been refactored from a monolithic approach to a service-oriented architecture that supports:
|
||||
|
||||
- **Individual Scheduler Services** - Each scheduler runs as a separate service
|
||||
- **Horizontal Scaling** - Multiple instances of the same scheduler can run across different machines
|
||||
- **Health Monitoring** - Built-in health checks for load balancers and orchestrators
|
||||
- **Graceful Shutdown** - Proper handling of shutdown signals for zero-downtime deployments
|
||||
- **Centralized Management** - Optional scheduler manager for coordinated operations
|
||||
- **Individual Scheduler Services** - Each scheduler runs as a separate service
|
||||
- **Horizontal Scaling** - Multiple instances of the same scheduler can run across different machines
|
||||
- **Health Monitoring** - Built-in health checks for load balancers and orchestrators
|
||||
- **Graceful Shutdown** - Proper handling of shutdown signals for zero-downtime deployments
|
||||
- **Centralized Management** - Optional scheduler manager for coordinated operations
|
||||
|
||||
## Components
|
||||
|
||||
@ -34,11 +34,11 @@ export abstract class BaseSchedulerService extends EventEmitter {
|
||||
|
||||
**Features:**
|
||||
|
||||
- Status management (STOPPED, STARTING, RUNNING, PAUSED, ERROR)
|
||||
- Metrics collection (run counts, timing, success/failure rates)
|
||||
- Event emission for monitoring
|
||||
- Configurable intervals and timeouts
|
||||
- Automatic retry handling
|
||||
- Status management (STOPPED, STARTING, RUNNING, PAUSED, ERROR)
|
||||
- Metrics collection (run counts, timing, success/failure rates)
|
||||
- Event emission for monitoring
|
||||
- Configurable intervals and timeouts
|
||||
- Automatic retry handling
|
||||
|
||||
### 2. Individual Scheduler Services
|
||||
|
||||
@ -57,16 +57,16 @@ const csvScheduler = new CsvImportSchedulerService({
|
||||
|
||||
**Features:**
|
||||
|
||||
- Batch processing with configurable concurrency
|
||||
- Duplicate detection
|
||||
- Company-specific error handling
|
||||
- Progress monitoring
|
||||
- Batch processing with configurable concurrency
|
||||
- Duplicate detection
|
||||
- Company-specific error handling
|
||||
- Progress monitoring
|
||||
|
||||
#### Additional Schedulers (To Be Implemented)
|
||||
|
||||
- `ImportProcessingSchedulerService` - Process imported CSV data into sessions
|
||||
- `SessionProcessingSchedulerService` - AI analysis and categorization
|
||||
- `BatchProcessingSchedulerService` - OpenAI Batch API integration
|
||||
- `ImportProcessingSchedulerService` - Process imported CSV data into sessions
|
||||
- `SessionProcessingSchedulerService` - AI analysis and categorization
|
||||
- `BatchProcessingSchedulerService` - OpenAI Batch API integration
|
||||
|
||||
### 3. SchedulerManager
|
||||
|
||||
@ -88,10 +88,10 @@ await manager.startAll();
|
||||
|
||||
**Features:**
|
||||
|
||||
- Automatic restart of failed critical schedulers
|
||||
- Health monitoring across all schedulers
|
||||
- Coordinated start/stop operations
|
||||
- Event aggregation and logging
|
||||
- Automatic restart of failed critical schedulers
|
||||
- Health monitoring across all schedulers
|
||||
- Coordinated start/stop operations
|
||||
- Event aggregation and logging
|
||||
|
||||
### 4. Standalone Scheduler Runner
|
||||
|
||||
@ -107,10 +107,10 @@ npx tsx lib/services/schedulers/StandaloneSchedulerRunner.ts --list
|
||||
|
||||
**Features:**
|
||||
|
||||
- Independent process execution
|
||||
- Environment variable configuration
|
||||
- Graceful shutdown handling
|
||||
- Health reporting for monitoring
|
||||
- Independent process execution
|
||||
- Environment variable configuration
|
||||
- Graceful shutdown handling
|
||||
- Health reporting for monitoring
|
||||
|
||||
## Deployment Patterns
|
||||
|
||||
@ -127,15 +127,15 @@ await initializeSchedulers();
|
||||
|
||||
**Pros:**
|
||||
|
||||
- Simple deployment
|
||||
- Lower resource usage
|
||||
- Easy local development
|
||||
- Simple deployment
|
||||
- Lower resource usage
|
||||
- Easy local development
|
||||
|
||||
**Cons:**
|
||||
|
||||
- Limited scalability
|
||||
- Single point of failure
|
||||
- Resource contention
|
||||
- Limited scalability
|
||||
- Single point of failure
|
||||
- Resource contention
|
||||
|
||||
### 2. Separate Processes
|
||||
|
||||
@ -154,15 +154,15 @@ npm run scheduler:session-processing
|
||||
|
||||
**Pros:**
|
||||
|
||||
- Independent scaling
|
||||
- Fault isolation
|
||||
- Resource optimization per scheduler
|
||||
- Independent scaling
|
||||
- Fault isolation
|
||||
- Resource optimization per scheduler
|
||||
|
||||
**Cons:**
|
||||
|
||||
- More complex deployment
|
||||
- Higher resource overhead
|
||||
- Inter-process coordination needed
|
||||
- More complex deployment
|
||||
- Higher resource overhead
|
||||
- Inter-process coordination needed
|
||||
|
||||
### 3. Container Orchestration (Recommended for Production)
|
||||
|
||||
@ -193,16 +193,16 @@ services:
|
||||
|
||||
**Pros:**
|
||||
|
||||
- Full horizontal scaling
|
||||
- Independent resource allocation
|
||||
- Health monitoring integration
|
||||
- Zero-downtime deployments
|
||||
- Full horizontal scaling
|
||||
- Independent resource allocation
|
||||
- Health monitoring integration
|
||||
- Zero-downtime deployments
|
||||
|
||||
**Cons:**
|
||||
|
||||
- Complex orchestration setup
|
||||
- Network latency considerations
|
||||
- Distributed system challenges
|
||||
- Complex orchestration setup
|
||||
- Network latency considerations
|
||||
- Distributed system challenges
|
||||
|
||||
## Configuration
|
||||
|
||||
@ -380,31 +380,35 @@ csv-import-scheduler-eu:
|
||||
### From Current Architecture
|
||||
|
||||
1. **Phase 1: Extract Schedulers**
|
||||
- ✅ Create BaseSchedulerService
|
||||
- ✅ Implement CsvImportSchedulerService
|
||||
- ✅ Create SchedulerManager
|
||||
- ⏳ Implement remaining scheduler services
|
||||
|
||||
- ✅ Create BaseSchedulerService
|
||||
- ✅ Implement CsvImportSchedulerService
|
||||
- ✅ Create SchedulerManager
|
||||
- ⏳ Implement remaining scheduler services
|
||||
|
||||
2. **Phase 2: Deployment Options**
|
||||
- ✅ Add ServerSchedulerIntegration for backwards compatibility
|
||||
- ✅ Create StandaloneSchedulerRunner
|
||||
- ✅ Add health check endpoints
|
||||
|
||||
- ✅ Add ServerSchedulerIntegration for backwards compatibility
|
||||
- ✅ Create StandaloneSchedulerRunner
|
||||
- ✅ Add health check endpoints
|
||||
|
||||
3. **Phase 3: Container Support**
|
||||
- ⏳ Create Dockerfile for scheduler containers
|
||||
- ⏳ Add Kubernetes manifests
|
||||
- ⏳ Implement distributed coordination
|
||||
|
||||
- ⏳ Create Dockerfile for scheduler containers
|
||||
- ⏳ Add Kubernetes manifests
|
||||
- ⏳ Implement distributed coordination
|
||||
|
||||
4. **Phase 4: Production Migration**
|
||||
- ⏳ Deploy separate scheduler containers
|
||||
- ⏳ Monitor performance and stability
|
||||
- ⏳ Gradually increase horizontal scaling
|
||||
|
||||
- ⏳ Deploy separate scheduler containers
|
||||
- ⏳ Monitor performance and stability
|
||||
- ⏳ Gradually increase horizontal scaling
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
- Scheduler initialization moved from `server.ts` to `ServerSchedulerIntegration`
|
||||
- Individual scheduler functions replaced with service classes
|
||||
- Configuration moved to environment variables
|
||||
- Scheduler initialization moved from `server.ts` to `ServerSchedulerIntegration`
|
||||
- Individual scheduler functions replaced with service classes
|
||||
- Configuration moved to environment variables
|
||||
|
||||
## Benefits
|
||||
|
||||
|
||||
@ -8,8 +8,8 @@
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs
|
||||
- Removed the invalid company record from the database using `fix_companies.js`
|
||||
- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs
|
||||
- Removed the invalid company record from the database using `fix_companies.js`
|
||||
|
||||
### 2. Transcript Fetching Errors
|
||||
|
||||
@ -17,10 +17,10 @@
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Improved error handling in `fetchTranscriptContent()` function
|
||||
- Added probabilistic logging (only ~10% of errors logged) to prevent log spam
|
||||
- Added timeout (10 seconds) for transcript fetching
|
||||
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
|
||||
- Improved error handling in `fetchTranscriptContent()` function
|
||||
- Added probabilistic logging (only ~10% of errors logged) to prevent log spam
|
||||
- Added timeout (10 seconds) for transcript fetching
|
||||
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
|
||||
|
||||
### 3. CSV Fetching Errors
|
||||
|
||||
@ -28,8 +28,8 @@
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Added URL validation to skip companies with `example.com` URLs
|
||||
- Improved error logging to be more descriptive
|
||||
- Added URL validation to skip companies with `example.com` URLs
|
||||
- Improved error logging to be more descriptive
|
||||
|
||||
## Current Status
|
||||
|
||||
@ -42,23 +42,23 @@
|
||||
|
||||
After cleanup, only valid companies remain:
|
||||
|
||||
- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`)
|
||||
- CSV URL: `https://proto.notso.ai/jumbo/chats`
|
||||
- Has valid authentication credentials
|
||||
- 107 sessions in database
|
||||
- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`)
|
||||
- CSV URL: `https://proto.notso.ai/jumbo/chats`
|
||||
- Has valid authentication credentials
|
||||
- 107 sessions in database
|
||||
|
||||
## Files Modified
|
||||
|
||||
1. **lib/csvFetcher.js**
|
||||
|
||||
- Added company URL validation
|
||||
- Improved transcript fetching error handling
|
||||
- Reduced error log verbosity
|
||||
- Added company URL validation
|
||||
- Improved transcript fetching error handling
|
||||
- Reduced error log verbosity
|
||||
|
||||
2. **fix_companies.js** (cleanup script)
|
||||
|
||||
- Removes invalid company records
|
||||
- Can be run again if needed
|
||||
- Removes invalid company records
|
||||
- Can be run again if needed
|
||||
|
||||
## Monitoring
|
||||
|
||||
|
||||
@ -12,49 +12,49 @@ The security audit logging system provides comprehensive tracking of security-cr
|
||||
|
||||
The system logs the following event types:
|
||||
|
||||
- **Authentication Events**: Login attempts, password changes, session management
|
||||
- **Authorization Events**: Permission checks, access denied events
|
||||
- **User Management**: User creation, modification, deletion, invitations
|
||||
- **Company Management**: Company suspension, settings changes
|
||||
- **Rate Limiting**: Abuse prevention and rate limit violations
|
||||
- **CSRF Protection**: Cross-site request forgery protection events
|
||||
- **Security Headers**: Security header violations
|
||||
- **Password Reset**: Password reset flows and token validation
|
||||
- **Platform Admin**: Administrative activities by platform users
|
||||
- **Data Privacy**: Data export and privacy-related events
|
||||
- **System Configuration**: System setting changes
|
||||
- **API Security**: API-related security events
|
||||
- **Authentication Events**: Login attempts, password changes, session management
|
||||
- **Authorization Events**: Permission checks, access denied events
|
||||
- **User Management**: User creation, modification, deletion, invitations
|
||||
- **Company Management**: Company suspension, settings changes
|
||||
- **Rate Limiting**: Abuse prevention and rate limit violations
|
||||
- **CSRF Protection**: Cross-site request forgery protection events
|
||||
- **Security Headers**: Security header violations
|
||||
- **Password Reset**: Password reset flows and token validation
|
||||
- **Platform Admin**: Administrative activities by platform users
|
||||
- **Data Privacy**: Data export and privacy-related events
|
||||
- **System Configuration**: System setting changes
|
||||
- **API Security**: API-related security events
|
||||
|
||||
### 2. Structured Logging
|
||||
|
||||
Each audit log entry includes:
|
||||
|
||||
- **Event Type**: Categorizes the security event
|
||||
- **Action**: Specific action performed
|
||||
- **Outcome**: Success, failure, blocked, rate limited, or suspicious
|
||||
- **Severity**: Info, low, medium, high, or critical
|
||||
- **Context**: User ID, company ID, platform user ID, IP address, user agent
|
||||
- **Metadata**: Structured additional information
|
||||
- **Timestamp**: Immutable timestamp for chronological ordering
|
||||
- **Event Type**: Categorizes the security event
|
||||
- **Action**: Specific action performed
|
||||
- **Outcome**: Success, failure, blocked, rate limited, or suspicious
|
||||
- **Severity**: Info, low, medium, high, or critical
|
||||
- **Context**: User ID, company ID, platform user ID, IP address, user agent
|
||||
- **Metadata**: Structured additional information
|
||||
- **Timestamp**: Immutable timestamp for chronological ordering
|
||||
|
||||
### 3. Multi-Tenant Security
|
||||
|
||||
- Company-scoped audit logs ensure data isolation
|
||||
- Platform admin actions tracked separately
|
||||
- Role-based access controls for audit log viewing
|
||||
- Company-scoped audit logs ensure data isolation
|
||||
- Platform admin actions tracked separately
|
||||
- Role-based access controls for audit log viewing
|
||||
|
||||
### 4. Log Retention and Management
|
||||
|
||||
- **Configurable Retention Policies**: Different retention periods based on event type and severity
|
||||
- **Automatic Archival**: Critical and high-severity events archived before deletion
|
||||
- **Scheduled Cleanup**: Weekly automated retention policy execution
|
||||
- **Manual Controls**: Admin interface for manual retention execution
|
||||
- **Configurable Retention Policies**: Different retention periods based on event type and severity
|
||||
- **Automatic Archival**: Critical and high-severity events archived before deletion
|
||||
- **Scheduled Cleanup**: Weekly automated retention policy execution
|
||||
- **Manual Controls**: Admin interface for manual retention execution
|
||||
|
||||
### 5. Administrative Interface
|
||||
|
||||
- **Audit Log Viewer**: Comprehensive filtering and search capabilities
|
||||
- **Retention Management**: View statistics and execute retention policies
|
||||
- **Real-time Monitoring**: Track security events as they occur
|
||||
- **Audit Log Viewer**: Comprehensive filtering and search capabilities
|
||||
- **Retention Management**: View statistics and execute retention policies
|
||||
- **Real-time Monitoring**: Track security events as they occur
|
||||
|
||||
## Architecture
|
||||
|
||||
@ -132,12 +132,12 @@ Administrators can access audit logs through:
|
||||
|
||||
### Filtering Options
|
||||
|
||||
- Event type (authentication, authorization, etc.)
|
||||
- Outcome (success, failure, blocked, etc.)
|
||||
- Severity level (info, low, medium, high, critical)
|
||||
- Date range
|
||||
- User ID
|
||||
- Pagination support
|
||||
- Event type (authentication, authorization, etc.)
|
||||
- Outcome (success, failure, blocked, etc.)
|
||||
- Severity level (info, low, medium, high, critical)
|
||||
- Date range
|
||||
- User ID
|
||||
- Pagination support
|
||||
|
||||
## Configuration
|
||||
|
||||
@ -170,51 +170,51 @@ AUDIT_LOG_RETENTION_DRY_RUN=false
|
||||
|
||||
### Data Protection
|
||||
|
||||
- **IP Address Storage**: Client IP addresses stored for geographic analysis
|
||||
- **Sensitive Data Redaction**: Passwords, tokens, and emails marked as `[REDACTED]`
|
||||
- **Metadata Sanitization**: Complex objects sanitized to prevent data leakage
|
||||
- **IP Address Storage**: Client IP addresses stored for geographic analysis
|
||||
- **Sensitive Data Redaction**: Passwords, tokens, and emails marked as `[REDACTED]`
|
||||
- **Metadata Sanitization**: Complex objects sanitized to prevent data leakage
|
||||
|
||||
### Access Controls
|
||||
|
||||
- **Admin-Only Access**: Only users with `ADMIN` role can view audit logs
|
||||
- **Company Isolation**: Users can only view logs for their own company
|
||||
- **Platform Separation**: Platform admin logs tracked separately
|
||||
- **Admin-Only Access**: Only users with `ADMIN` role can view audit logs
|
||||
- **Company Isolation**: Users can only view logs for their own company
|
||||
- **Platform Separation**: Platform admin logs tracked separately
|
||||
|
||||
### Performance
|
||||
|
||||
- **Async Logging**: All logging operations are asynchronous to avoid blocking
|
||||
- **Error Handling**: Logging failures don't affect application functionality
|
||||
- **Indexed Queries**: Database indexes optimize common query patterns
|
||||
- **Batch Operations**: Retention policies use batch operations for efficiency
|
||||
- **Async Logging**: All logging operations are asynchronous to avoid blocking
|
||||
- **Error Handling**: Logging failures don't affect application functionality
|
||||
- **Indexed Queries**: Database indexes optimize common query patterns
|
||||
- **Batch Operations**: Retention policies use batch operations for efficiency
|
||||
|
||||
## Compliance Features
|
||||
|
||||
### Audit Standards
|
||||
|
||||
- **Immutable Records**: Audit logs cannot be modified after creation
|
||||
- **Chronological Ordering**: Precise timestamps for event sequencing
|
||||
- **Non-Repudiation**: User actions clearly attributed and timestamped
|
||||
- **Comprehensive Coverage**: All security-relevant events logged
|
||||
- **Immutable Records**: Audit logs cannot be modified after creation
|
||||
- **Chronological Ordering**: Precise timestamps for event sequencing
|
||||
- **Non-Repudiation**: User actions clearly attributed and timestamped
|
||||
- **Comprehensive Coverage**: All security-relevant events logged
|
||||
|
||||
### Reporting
|
||||
|
||||
- **Event Statistics**: Summary statistics by event type, severity, and time period
|
||||
- **Export Capabilities**: Structured data export for compliance reporting
|
||||
- **Retention Tracking**: Detailed logging of retention policy execution
|
||||
- **Event Statistics**: Summary statistics by event type, severity, and time period
|
||||
- **Export Capabilities**: Structured data export for compliance reporting
|
||||
- **Retention Tracking**: Detailed logging of retention policy execution
|
||||
|
||||
## Monitoring and Alerting
|
||||
|
||||
### System Health
|
||||
|
||||
- **Scheduler Status**: Monitor retention scheduler health
|
||||
- **Error Tracking**: Log retention and audit logging errors
|
||||
- **Performance Metrics**: Track logging performance and database impact
|
||||
- **Scheduler Status**: Monitor retention scheduler health
|
||||
- **Error Tracking**: Log retention and audit logging errors
|
||||
- **Performance Metrics**: Track logging performance and database impact
|
||||
|
||||
### Security Monitoring
|
||||
|
||||
- **Failed Authentication Patterns**: Track repeated login failures
|
||||
- **Privilege Escalation**: Monitor administrative action patterns
|
||||
- **Suspicious Activity**: Identify unusual access patterns
|
||||
- **Failed Authentication Patterns**: Track repeated login failures
|
||||
- **Privilege Escalation**: Monitor administrative action patterns
|
||||
- **Suspicious Activity**: Identify unusual access patterns
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@ -227,9 +227,9 @@ AUDIT_LOG_RETENTION_DRY_RUN=false
|
||||
|
||||
### Debug Information
|
||||
|
||||
- Check application logs for scheduler startup messages
|
||||
- Monitor database query performance for audit log operations
|
||||
- Review retention policy validation warnings
|
||||
- Check application logs for scheduler startup messages
|
||||
- Monitor database query performance for audit log operations
|
||||
- Review retention policy validation warnings
|
||||
|
||||
## Best Practices
|
||||
|
||||
@ -251,13 +251,13 @@ AUDIT_LOG_RETENTION_DRY_RUN=false
|
||||
|
||||
### Planned Features
|
||||
|
||||
- **Real-time Alerting**: Immediate notifications for critical security events
|
||||
- **Advanced Analytics**: ML-based anomaly detection and pattern recognition
|
||||
- **Export Formats**: Additional export formats for compliance reporting
|
||||
- **External Integration**: SIEM and security tool integrations
|
||||
- **Real-time Alerting**: Immediate notifications for critical security events
|
||||
- **Advanced Analytics**: ML-based anomaly detection and pattern recognition
|
||||
- **Export Formats**: Additional export formats for compliance reporting
|
||||
- **External Integration**: SIEM and security tool integrations
|
||||
|
||||
### Performance Optimizations
|
||||
|
||||
- **Log Partitioning**: Database partitioning for improved query performance
|
||||
- **Compression**: Log compression for storage efficiency
|
||||
- **Streaming**: Real-time log streaming for external systems
|
||||
- **Log Partitioning**: Database partitioning for improved query performance
|
||||
- **Compression**: Log compression for storage efficiency
|
||||
- **Streaming**: Real-time log streaming for external systems
|
||||
|
||||
@ -12,33 +12,33 @@ The application implements multiple layers of HTTP security headers to provide d
|
||||
|
||||
#### X-Content-Type-Options: nosniff
|
||||
|
||||
- **Purpose**: Prevents MIME type sniffing attacks
|
||||
- **Protection**: Stops browsers from interpreting files as different MIME types than declared
|
||||
- **Value**: `nosniff`
|
||||
- **Purpose**: Prevents MIME type sniffing attacks
|
||||
- **Protection**: Stops browsers from interpreting files as different MIME types than declared
|
||||
- **Value**: `nosniff`
|
||||
|
||||
#### X-Frame-Options: DENY
|
||||
|
||||
- **Purpose**: Prevents clickjacking attacks
|
||||
- **Protection**: Blocks embedding the site in frames/iframes
|
||||
- **Value**: `DENY`
|
||||
- **Purpose**: Prevents clickjacking attacks
|
||||
- **Protection**: Blocks embedding the site in frames/iframes
|
||||
- **Value**: `DENY`
|
||||
|
||||
#### X-XSS-Protection: 1; mode=block
|
||||
|
||||
- **Purpose**: Enables XSS protection in legacy browsers
|
||||
- **Protection**: Activates built-in XSS filtering (primarily for older browsers)
|
||||
- **Value**: `1; mode=block`
|
||||
- **Purpose**: Enables XSS protection in legacy browsers
|
||||
- **Protection**: Activates built-in XSS filtering (primarily for older browsers)
|
||||
- **Value**: `1; mode=block`
|
||||
|
||||
#### Referrer-Policy: strict-origin-when-cross-origin
|
||||
|
||||
- **Purpose**: Controls referrer information leakage
|
||||
- **Protection**: Limits referrer data sent to external sites
|
||||
- **Value**: `strict-origin-when-cross-origin`
|
||||
- **Purpose**: Controls referrer information leakage
|
||||
- **Protection**: Limits referrer data sent to external sites
|
||||
- **Value**: `strict-origin-when-cross-origin`
|
||||
|
||||
#### X-DNS-Prefetch-Control: off
|
||||
|
||||
- **Purpose**: Prevents DNS rebinding attacks
|
||||
- **Protection**: Disables DNS prefetching to reduce attack surface
|
||||
- **Value**: `off`
|
||||
- **Purpose**: Prevents DNS rebinding attacks
|
||||
- **Protection**: Disables DNS prefetching to reduce attack surface
|
||||
- **Value**: `off`
|
||||
|
||||
### Content Security Policy (CSP)
|
||||
|
||||
@ -50,13 +50,13 @@ Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'un
|
||||
|
||||
#### Key CSP Directives
|
||||
|
||||
- **default-src 'self'**: Restrictive default for all resource types
|
||||
- **script-src 'self' 'unsafe-eval' 'unsafe-inline'**: Allows Next.js dev tools and React functionality
|
||||
- **style-src 'self' 'unsafe-inline'**: Enables TailwindCSS and component styles
|
||||
- **img-src 'self' data: https:**: Allows secure image sources
|
||||
- **frame-ancestors 'none'**: Prevents embedding (reinforces X-Frame-Options)
|
||||
- **object-src 'none'**: Blocks dangerous plugins and embeds
|
||||
- **upgrade-insecure-requests**: Automatically upgrades HTTP to HTTPS
|
||||
- **default-src 'self'**: Restrictive default for all resource types
|
||||
- **script-src 'self' 'unsafe-eval' 'unsafe-inline'**: Allows Next.js dev tools and React functionality
|
||||
- **style-src 'self' 'unsafe-inline'**: Enables TailwindCSS and component styles
|
||||
- **img-src 'self' data: https:**: Allows secure image sources
|
||||
- **frame-ancestors 'none'**: Prevents embedding (reinforces X-Frame-Options)
|
||||
- **object-src 'none'**: Blocks dangerous plugins and embeds
|
||||
- **upgrade-insecure-requests**: Automatically upgrades HTTP to HTTPS
|
||||
|
||||
### Permissions Policy
|
||||
|
||||
@ -66,19 +66,19 @@ Controls browser feature access:
|
||||
Permissions-Policy: camera=(), microphone=(), geolocation=(), interest-cohort=(), browsing-topics=()
|
||||
```
|
||||
|
||||
- **camera=()**: Disables camera access
|
||||
- **microphone=()**: Disables microphone access
|
||||
- **geolocation=()**: Disables location tracking
|
||||
- **interest-cohort=()**: Blocks FLoC (privacy protection)
|
||||
- **browsing-topics=()**: Blocks Topics API (privacy protection)
|
||||
- **camera=()**: Disables camera access
|
||||
- **microphone=()**: Disables microphone access
|
||||
- **geolocation=()**: Disables location tracking
|
||||
- **interest-cohort=()**: Blocks FLoC (privacy protection)
|
||||
- **browsing-topics=()**: Blocks Topics API (privacy protection)
|
||||
|
||||
### Strict Transport Security (HSTS)
|
||||
|
||||
**Production Only**: `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload`
|
||||
|
||||
- **max-age=31536000**: 1 year HSTS policy
|
||||
- **includeSubDomains**: Applies to all subdomains
|
||||
- **preload**: Ready for HSTS preload list inclusion
|
||||
- **max-age=31536000**: 1 year HSTS policy
|
||||
- **includeSubDomains**: Applies to all subdomains
|
||||
- **preload**: Ready for HSTS preload list inclusion
|
||||
|
||||
## Configuration
|
||||
|
||||
@ -110,8 +110,8 @@ headers: async () => {
|
||||
|
||||
### Environment-Specific Behavior
|
||||
|
||||
- **Development**: All headers except HSTS
|
||||
- **Production**: All headers including HSTS
|
||||
- **Development**: All headers except HSTS
|
||||
- **Production**: All headers including HSTS
|
||||
|
||||
## Testing
|
||||
|
||||
@ -121,11 +121,11 @@ Location: `tests/unit/http-security-headers.test.ts`
|
||||
|
||||
Tests cover:
|
||||
|
||||
- Individual header validation
|
||||
- CSP directive verification
|
||||
- Permissions Policy validation
|
||||
- Environment-specific configuration
|
||||
- Next.js compatibility checks
|
||||
- Individual header validation
|
||||
- CSP directive verification
|
||||
- Permissions Policy validation
|
||||
- Environment-specific configuration
|
||||
- Next.js compatibility checks
|
||||
|
||||
### Integration Tests
|
||||
|
||||
@ -133,9 +133,9 @@ Location: `tests/integration/security-headers-basic.test.ts`
|
||||
|
||||
Tests cover:
|
||||
|
||||
- Next.js configuration validation
|
||||
- Header generation verification
|
||||
- Environment-based header differences
|
||||
- Next.js configuration validation
|
||||
- Header generation verification
|
||||
- Environment-based header differences
|
||||
|
||||
### Manual Testing
|
||||
|
||||
@ -160,11 +160,11 @@ pnpm test:security-headers https://your-domain.com
|
||||
|
||||
### Additional Security Benefits
|
||||
|
||||
- **Clickjacking Protection**: X-Frame-Options + CSP frame-ancestors
|
||||
- **MIME Sniffing Prevention**: X-Content-Type-Options
|
||||
- **Information Leakage Reduction**: Referrer-Policy
|
||||
- **Privacy Protection**: Permissions Policy restrictions
|
||||
- **Transport Security**: HSTS enforcement
|
||||
- **Clickjacking Protection**: X-Frame-Options + CSP frame-ancestors
|
||||
- **MIME Sniffing Prevention**: X-Content-Type-Options
|
||||
- **Information Leakage Reduction**: Referrer-Policy
|
||||
- **Privacy Protection**: Permissions Policy restrictions
|
||||
- **Transport Security**: HSTS enforcement
|
||||
|
||||
## Maintenance
|
||||
|
||||
@ -176,9 +176,9 @@ pnpm test:security-headers https://your-domain.com
|
||||
|
||||
### Monitoring
|
||||
|
||||
- Monitor CSP violation reports (when implemented)
|
||||
- Use online tools like securityheaders.com for validation
|
||||
- Include security header tests in CI/CD pipeline
|
||||
- Monitor CSP violation reports (when implemented)
|
||||
- Use online tools like securityheaders.com for validation
|
||||
- Include security header tests in CI/CD pipeline
|
||||
|
||||
### Future Enhancements
|
||||
|
||||
@ -195,18 +195,18 @@ Planned improvements:
|
||||
|
||||
Headers are configured to be compatible with:
|
||||
|
||||
- Next.js 15+ App Router
|
||||
- React 19 development tools
|
||||
- TailwindCSS 4 styling system
|
||||
- Development hot reload functionality
|
||||
- Next.js 15+ App Router
|
||||
- React 19 development tools
|
||||
- TailwindCSS 4 styling system
|
||||
- Development hot reload functionality
|
||||
|
||||
### Browser Support
|
||||
|
||||
Security headers are supported by:
|
||||
|
||||
- All modern browsers (Chrome 60+, Firefox 60+, Safari 12+)
|
||||
- Graceful degradation for older browsers
|
||||
- Progressive enhancement approach
|
||||
- All modern browsers (Chrome 60+, Firefox 60+, Safari 12+)
|
||||
- Graceful degradation for older browsers
|
||||
- Progressive enhancement approach
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@ -219,13 +219,13 @@ Security headers are supported by:
|
||||
|
||||
### Debug Tools
|
||||
|
||||
- Browser DevTools Security tab
|
||||
- CSP Evaluator: <https://csp-evaluator.withgoogle.com/>
|
||||
- Security Headers Scanner: <https://securityheaders.com/>
|
||||
- Browser DevTools Security tab
|
||||
- CSP Evaluator: <https://csp-evaluator.withgoogle.com/>
|
||||
- Security Headers Scanner: <https://securityheaders.com/>
|
||||
|
||||
## References
|
||||
|
||||
- [OWASP Secure Headers Project](https://owasp.org/www-project-secure-headers/)
|
||||
- [MDN Security Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#security)
|
||||
- [Next.js Security Headers](https://nextjs.org/docs/app/api-reference/config/headers)
|
||||
- [Content Security Policy Reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)
|
||||
- [OWASP Secure Headers Project](https://owasp.org/www-project-secure-headers/)
|
||||
- [MDN Security Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#security)
|
||||
- [Next.js Security Headers](https://nextjs.org/docs/app/api-reference/config/headers)
|
||||
- [Content Security Policy Reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)
|
||||
|
||||
@ -9,38 +9,42 @@ The Security Monitoring and Alerting System provides comprehensive real-time sec
|
||||
### Core Components
|
||||
|
||||
1. **Security Monitoring Service** (`lib/securityMonitoring.ts`)
|
||||
- Real-time event processing
|
||||
- Anomaly detection algorithms
|
||||
- Alert generation and management
|
||||
- Security score calculation
|
||||
- Threat level assessment
|
||||
|
||||
- Real-time event processing
|
||||
- Anomaly detection algorithms
|
||||
- Alert generation and management
|
||||
- Security score calculation
|
||||
- Threat level assessment
|
||||
|
||||
2. **Enhanced Security Logging** (`enhancedSecurityLog`)
|
||||
- Integrates with existing audit logger
|
||||
- Processes events through monitoring system
|
||||
- Triggers immediate threat detection
|
||||
|
||||
- Integrates with existing audit logger
|
||||
- Processes events through monitoring system
|
||||
- Triggers immediate threat detection
|
||||
|
||||
3. **API Endpoints** (`app/api/admin/security-monitoring/`)
|
||||
- `/api/admin/security-monitoring` - Main metrics and configuration
|
||||
- `/api/admin/security-monitoring/alerts` - Alert management
|
||||
- `/api/admin/security-monitoring/export` - Data export
|
||||
- `/api/admin/security-monitoring/threat-analysis` - Threat analysis
|
||||
|
||||
- `/api/admin/security-monitoring` - Main metrics and configuration
|
||||
- `/api/admin/security-monitoring/alerts` - Alert management
|
||||
- `/api/admin/security-monitoring/export` - Data export
|
||||
- `/api/admin/security-monitoring/threat-analysis` - Threat analysis
|
||||
|
||||
4. **Dashboard UI** (`app/platform/security/page.tsx`)
|
||||
- Real-time security metrics
|
||||
- Active alerts management
|
||||
- Threat analysis visualization
|
||||
- Configuration management
|
||||
|
||||
- Real-time security metrics
|
||||
- Active alerts management
|
||||
- Threat analysis visualization
|
||||
- Configuration management
|
||||
|
||||
## Features
|
||||
|
||||
### Real-time Monitoring
|
||||
|
||||
- **Authentication Events**: Login attempts, failures, brute force attacks
|
||||
- **Rate Limiting**: Excessive request patterns, API abuse
|
||||
- **Admin Activity**: Unusual administrative actions
|
||||
- **Geographic Anomalies**: Logins from unusual locations
|
||||
- **Temporal Anomalies**: Activity spikes outside normal patterns
|
||||
- **Authentication Events**: Login attempts, failures, brute force attacks
|
||||
- **Rate Limiting**: Excessive request patterns, API abuse
|
||||
- **Admin Activity**: Unusual administrative actions
|
||||
- **Geographic Anomalies**: Logins from unusual locations
|
||||
- **Temporal Anomalies**: Activity spikes outside normal patterns
|
||||
|
||||
### Alert Types
|
||||
|
||||
@ -69,29 +73,32 @@ enum AlertType {
|
||||
The system implements several anomaly detection algorithms:
|
||||
|
||||
1. **Geographic Anomaly Detection**
|
||||
- Detects logins from unusual countries
|
||||
- Compares against historical user patterns
|
||||
- Confidence scoring based on deviation
|
||||
|
||||
- Detects logins from unusual countries
|
||||
- Compares against historical user patterns
|
||||
- Confidence scoring based on deviation
|
||||
|
||||
2. **Temporal Anomaly Detection**
|
||||
- Identifies activity spikes during unusual hours
|
||||
- Compares current activity to historical averages
|
||||
- Configurable thresholds for different event types
|
||||
|
||||
- Identifies activity spikes during unusual hours
|
||||
- Compares current activity to historical averages
|
||||
- Configurable thresholds for different event types
|
||||
|
||||
3. **Behavioral Anomaly Detection**
|
||||
- Multiple failed login attempts
|
||||
- Rapid succession of actions
|
||||
- Pattern deviation analysis
|
||||
|
||||
- Multiple failed login attempts
|
||||
- Rapid succession of actions
|
||||
- Pattern deviation analysis
|
||||
|
||||
### Security Scoring
|
||||
|
||||
The system calculates a real-time security score (0-100) based on:
|
||||
|
||||
- Critical security events (weight: 25)
|
||||
- Active unresolved alerts (weight: 30)
|
||||
- High-severity threats (weight: 20)
|
||||
- Overall event volume (weight: 15)
|
||||
- System stability factors (weight: 10)
|
||||
- Critical security events (weight: 25)
|
||||
- Active unresolved alerts (weight: 30)
|
||||
- High-severity threats (weight: 20)
|
||||
- Overall event volume (weight: 15)
|
||||
- System stability factors (weight: 10)
|
||||
|
||||
### Threat Levels
|
||||
|
||||
@ -255,117 +262,121 @@ await enhancedSecurityLog(
|
||||
|
||||
### Security Overview
|
||||
|
||||
- Real-time security score (0-100)
|
||||
- Current threat level indicator
|
||||
- Active alerts count
|
||||
- Security events summary
|
||||
- Real-time security score (0-100)
|
||||
- Current threat level indicator
|
||||
- Active alerts count
|
||||
- Security events summary
|
||||
|
||||
### Alert Management
|
||||
|
||||
- View active and historical alerts
|
||||
- Filter by severity and type
|
||||
- Acknowledge alerts with tracking
|
||||
- Detailed alert context and metadata
|
||||
- View active and historical alerts
|
||||
- Filter by severity and type
|
||||
- Acknowledge alerts with tracking
|
||||
- Detailed alert context and metadata
|
||||
|
||||
### Threat Analysis
|
||||
|
||||
- Geographic distribution of events
|
||||
- Top threat types and patterns
|
||||
- User risk scoring
|
||||
- IP threat level analysis
|
||||
- Geographic distribution of events
|
||||
- Top threat types and patterns
|
||||
- User risk scoring
|
||||
- IP threat level analysis
|
||||
|
||||
### Configuration Management
|
||||
|
||||
- Adjust detection thresholds
|
||||
- Configure alerting channels
|
||||
- Set data retention policies
|
||||
- Export capabilities
|
||||
- Adjust detection thresholds
|
||||
- Configure alerting channels
|
||||
- Set data retention policies
|
||||
- Export capabilities
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Memory Management
|
||||
|
||||
- Event buffer limited to 1 hour of data
|
||||
- Automatic cleanup of old alerts (configurable)
|
||||
- Efficient in-memory storage for real-time analysis
|
||||
- Event buffer limited to 1 hour of data
|
||||
- Automatic cleanup of old alerts (configurable)
|
||||
- Efficient in-memory storage for real-time analysis
|
||||
|
||||
### Database Impact
|
||||
|
||||
- Leverages existing audit log indexes
|
||||
- Optimized queries for time-range filtering
|
||||
- Background processing to avoid blocking operations
|
||||
- Leverages existing audit log indexes
|
||||
- Optimized queries for time-range filtering
|
||||
- Background processing to avoid blocking operations
|
||||
|
||||
### Scalability
|
||||
|
||||
- Stateless architecture (except for buffering)
|
||||
- Horizontal scaling support
|
||||
- Configurable processing intervals
|
||||
- Stateless architecture (except for buffering)
|
||||
- Horizontal scaling support
|
||||
- Configurable processing intervals
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Access Control
|
||||
|
||||
- Platform admin authentication required
|
||||
- Role-based access to security endpoints
|
||||
- Audit logging of all monitoring activities
|
||||
- Platform admin authentication required
|
||||
- Role-based access to security endpoints
|
||||
- Audit logging of all monitoring activities
|
||||
|
||||
### Data Privacy
|
||||
|
||||
- Sensitive data redaction in logs
|
||||
- IP address anonymization options
|
||||
- Configurable data retention periods
|
||||
- Sensitive data redaction in logs
|
||||
- IP address anonymization options
|
||||
- Configurable data retention periods
|
||||
|
||||
### Alert Suppression
|
||||
|
||||
- Duplicate alert suppression (configurable window)
|
||||
- Rate limiting on alert generation
|
||||
- Escalation policies for critical threats
|
||||
- Duplicate alert suppression (configurable window)
|
||||
- Rate limiting on alert generation
|
||||
- Escalation policies for critical threats
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### Health Checks
|
||||
|
||||
- Monitor service availability
|
||||
- Check alert generation pipeline
|
||||
- Verify data export functionality
|
||||
- Monitor service availability
|
||||
- Check alert generation pipeline
|
||||
- Verify data export functionality
|
||||
|
||||
### Regular Tasks
|
||||
|
||||
- Review and adjust thresholds quarterly
|
||||
- Analyze false positive rates
|
||||
- Update threat detection patterns
|
||||
- Clean up old alert data
|
||||
- Review and adjust thresholds quarterly
|
||||
- Analyze false positive rates
|
||||
- Update threat detection patterns
|
||||
- Clean up old alert data
|
||||
|
||||
### Performance Metrics
|
||||
|
||||
- Alert response time
|
||||
- False positive/negative rates
|
||||
- System resource usage
|
||||
- User engagement with alerts
|
||||
- Alert response time
|
||||
- False positive/negative rates
|
||||
- System resource usage
|
||||
- User engagement with alerts
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
|
||||
1. **Machine Learning Integration**
|
||||
- Behavioral pattern recognition
|
||||
- Adaptive threshold adjustment
|
||||
- Predictive threat modeling
|
||||
|
||||
- Behavioral pattern recognition
|
||||
- Adaptive threshold adjustment
|
||||
- Predictive threat modeling
|
||||
|
||||
2. **Advanced Analytics**
|
||||
- Threat intelligence integration
|
||||
- Cross-correlation analysis
|
||||
- Risk trend analysis
|
||||
|
||||
- Threat intelligence integration
|
||||
- Cross-correlation analysis
|
||||
- Risk trend analysis
|
||||
|
||||
3. **Integration Enhancements**
|
||||
- SIEM system connectors
|
||||
- Webhook customization
|
||||
- Mobile app notifications
|
||||
|
||||
- SIEM system connectors
|
||||
- Webhook customization
|
||||
- Mobile app notifications
|
||||
|
||||
4. **Automated Response**
|
||||
- IP blocking automation
|
||||
- Account suspension workflows
|
||||
- Incident response orchestration
|
||||
|
||||
- IP blocking automation
|
||||
- Account suspension workflows
|
||||
- Incident response orchestration
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@ -373,27 +384,27 @@ await enhancedSecurityLog(
|
||||
|
||||
**High False Positive Rate**
|
||||
|
||||
- Review and adjust detection thresholds
|
||||
- Analyze user behavior patterns
|
||||
- Consider geographical variations
|
||||
- Review and adjust detection thresholds
|
||||
- Analyze user behavior patterns
|
||||
- Consider geographical variations
|
||||
|
||||
**Missing Alerts**
|
||||
|
||||
- Check service configuration
|
||||
- Verify audit log integration
|
||||
- Review threshold settings
|
||||
- Check service configuration
|
||||
- Verify audit log integration
|
||||
- Review threshold settings
|
||||
|
||||
**Performance Issues**
|
||||
|
||||
- Monitor memory usage
|
||||
- Adjust cleanup intervals
|
||||
- Optimize database queries
|
||||
- Monitor memory usage
|
||||
- Adjust cleanup intervals
|
||||
- Optimize database queries
|
||||
|
||||
**Export Failures**
|
||||
|
||||
- Check file permissions
|
||||
- Verify date range validity
|
||||
- Monitor server resources
|
||||
- Check file permissions
|
||||
- Verify date range validity
|
||||
- Monitor server resources
|
||||
|
||||
### Debugging
|
||||
|
||||
@ -419,24 +430,24 @@ console.log("Active alerts:", alerts.length);
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- Alert generation logic
|
||||
- Anomaly detection algorithms
|
||||
- Configuration management
|
||||
- Data export functionality
|
||||
- Alert generation logic
|
||||
- Anomaly detection algorithms
|
||||
- Configuration management
|
||||
- Data export functionality
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- API endpoint security
|
||||
- Database integration
|
||||
- Real-time event processing
|
||||
- Alert acknowledgment flow
|
||||
- API endpoint security
|
||||
- Database integration
|
||||
- Real-time event processing
|
||||
- Alert acknowledgment flow
|
||||
|
||||
### Load Testing
|
||||
|
||||
- High-volume event processing
|
||||
- Concurrent alert generation
|
||||
- Database performance under load
|
||||
- Memory usage patterns
|
||||
- High-volume event processing
|
||||
- Concurrent alert generation
|
||||
- Database performance under load
|
||||
- Memory usage patterns
|
||||
|
||||
Run tests:
|
||||
|
||||
|
||||
@ -15,22 +15,22 @@ The system now includes an automated process for analyzing chat session transcri
|
||||
|
||||
### Session Fetching
|
||||
|
||||
- The system fetches session data from configured CSV URLs for each company
|
||||
- Unlike the previous implementation, it now only adds sessions that don't already exist in the database
|
||||
- This prevents duplicate sessions and allows for incremental updates
|
||||
- The system fetches session data from configured CSV URLs for each company
|
||||
- Unlike the previous implementation, it now only adds sessions that don't already exist in the database
|
||||
- This prevents duplicate sessions and allows for incremental updates
|
||||
|
||||
### Transcript Processing
|
||||
|
||||
- For sessions with transcript content that haven't been processed yet, the system calls OpenAI's API
|
||||
- The API analyzes the transcript and extracts the following information:
|
||||
- Primary language used (ISO 639-1 code)
|
||||
- Number of messages sent by the user
|
||||
- Overall sentiment (positive, neutral, negative)
|
||||
- Whether the conversation was escalated
|
||||
- Whether HR contact was mentioned or provided
|
||||
- Best-fitting category for the conversation
|
||||
- Up to 5 paraphrased questions asked by the user
|
||||
- A brief summary of the conversation
|
||||
- For sessions with transcript content that haven't been processed yet, the system calls OpenAI's API
|
||||
- The API analyzes the transcript and extracts the following information:
|
||||
- Primary language used (ISO 639-1 code)
|
||||
- Number of messages sent by the user
|
||||
- Overall sentiment (positive, neutral, negative)
|
||||
- Whether the conversation was escalated
|
||||
- Whether HR contact was mentioned or provided
|
||||
- Best-fitting category for the conversation
|
||||
- Up to 5 paraphrased questions asked by the user
|
||||
- A brief summary of the conversation
|
||||
|
||||
### Scheduling
|
||||
|
||||
@ -43,10 +43,10 @@ The system includes two schedulers:
|
||||
|
||||
The Session model has been updated with new fields to store the processed data:
|
||||
|
||||
- `processed`: Boolean flag indicating whether the session has been processed
|
||||
- `sentimentCategory`: String value ("positive", "neutral", "negative") from OpenAI
|
||||
- `questions`: JSON array of questions asked by the user
|
||||
- `summary`: Brief summary of the conversation
|
||||
- `processed`: Boolean flag indicating whether the session has been processed
|
||||
- `sentimentCategory`: String value ("positive", "neutral", "negative") from OpenAI
|
||||
- `questions`: JSON array of questions asked by the user
|
||||
- `summary`: Brief summary of the conversation
|
||||
|
||||
## Configuration
|
||||
|
||||
@ -62,9 +62,9 @@ OPENAI_API_KEY=your_api_key_here
|
||||
|
||||
To run the application with schedulers enabled:
|
||||
|
||||
- Development: `npm run dev`
|
||||
- Development (with schedulers disabled): `npm run dev:no-schedulers`
|
||||
- Production: `npm run start`
|
||||
- Development: `npm run dev`
|
||||
- Development (with schedulers disabled): `npm run dev:no-schedulers`
|
||||
- Production: `npm run start`
|
||||
|
||||
Note: These commands will start a custom Next.js server with the schedulers enabled. You'll need to have an OpenAI API key set in your `.env.local` file for the session processing to work.
|
||||
|
||||
@ -82,5 +82,5 @@ This will process all unprocessed sessions that have transcript content.
|
||||
|
||||
The processing logic can be customized by modifying:
|
||||
|
||||
- `lib/processingScheduler.ts`: Contains the OpenAI processing logic
|
||||
- `scripts/process_sessions.ts`: Standalone script for manual processing
|
||||
- `lib/processingScheduler.ts`: Contains the OpenAI processing logic
|
||||
- `scripts/process_sessions.ts`: Standalone script for manual processing
|
||||
|
||||
Reference in New Issue
Block a user