feat: Implement structured message parsing and display in MessageViewer component

- Added MessageViewer component to display parsed messages in a chat-like format. - Introduced new Message table in the database to store individual messages with timestamps, roles, and content. - Updated Session model to include a relation to parsed messages. - Created transcript parsing logic to convert raw transcripts into structured messages. - Enhanced processing scheduler to handle sessions with parsed messages. - Updated API endpoints to return parsed messages alongside session details. - Added manual trigger commands for session refresh, transcript parsing, and processing. - Improved user experience with color-coded message roles and timestamps in the UI. - Documented the new scheduler workflow and transcript parsing implementation.
2026-01-16 10:52:08 +01:00 · 2025-06-25 17:45:08 +02:00
parent 3196dabdf2
commit a9e4145001
20 changed files with 1043 additions and 90 deletions
--- a/docs/scheduler-workflow.md
+++ b/docs/scheduler-workflow.md
@ -0,0 +1,185 @@
+# Scheduler Workflow Documentation
+
+## Overview
+The LiveDash system has two main schedulers that work together to fetch and process session data:
+
+1. **Session Refresh Scheduler** - Fetches new sessions from CSV files
+2. **Processing Scheduler** - Processes session transcripts with AI
+
+## Current Status (as of latest check)
+- **Total sessions**: 107
+- **Processed sessions**: 0  
+- **Sessions with transcript**: 0
+- **Ready for processing**: 0
+
+## How the `processed` Field Works
+
+The ProcessingScheduler picks up sessions where `processed` is **NOT** `true`, which includes:
+- `processed = false` 
+- `processed = null`
+
+**Query used:**
+```javascript
+{ processed: { not: true } } // Either false or null
+```
+
+## Complete Workflow
+
+### Step 1: Session Refresh (CSV Fetching)
+**What it does:**
+- Fetches session data from company CSV URLs
+- Creates session records in database with basic metadata
+- Sets `transcriptContent = null` initially
+- Sets `processed = null` initially
+
+**Runs:** Every 30 minutes (cron: `*/30 * * * *`)
+
+### Step 2: Transcript Fetching
+**What it does:**
+- Downloads full transcript content for sessions
+- Updates `transcriptContent` field with actual conversation data
+- Sessions remain `processed = null` until AI processing
+
+**Runs:** As part of session refresh process
+
+### Step 3: AI Processing
+**What it does:**
+- Finds sessions with transcript content where `processed != true`
+- Sends transcripts to OpenAI for analysis
+- Extracts: sentiment, category, questions, summary, etc.
+- Updates session with processed data
+- Sets `processed = true`
+
+**Runs:** Every hour (cron: `0 * * * *`)
+
+## Manual Trigger Commands
+
+### Check Current Status
+```bash
+node scripts/manual-triggers.js status
+```
+
+### Trigger Session Refresh (Fetch new sessions from CSV)
+```bash
+node scripts/manual-triggers.js refresh
+```
+
+### Trigger AI Processing (Process unprocessed sessions)
+```bash
+node scripts/manual-triggers.js process
+```
+
+### Run Both Schedulers
+```bash
+node scripts/manual-triggers.js both
+```
+
+## Troubleshooting
+
+### No Sessions Being Processed?
+1. **Check if sessions have transcripts:**
+   ```bash
+   node scripts/manual-triggers.js status
+   ```
+
+2. **If "Sessions with transcript" is 0:**
+   - Sessions exist but transcripts haven't been fetched yet
+   - Run session refresh: `node scripts/manual-triggers.js refresh`
+
+3. **If "Ready for processing" is 0 but "Sessions with transcript" > 0:**
+   - All sessions with transcripts have already been processed
+   - Check if `OPENAI_API_KEY` is set in environment
+
+### Common Issues
+
+#### "No sessions found requiring processing"
+- All sessions with transcripts have been processed (`processed = true`)
+- Or no sessions have transcript content yet
+
+#### "OPENAI_API_KEY environment variable is not set"
+- Add OpenAI API key to `.env.development` file
+- Restart the application
+
+#### "Error fetching transcript: Unauthorized"
+- CSV credentials are incorrect or expired
+- Check company CSV username/password in database
+
+## Database Field Mapping
+
+### Before AI Processing
+```javascript
+{
+  id: "session-uuid",
+  transcriptContent: "full conversation text" | null,
+  processed: null,
+  sentimentCategory: null,
+  questions: null,
+  summary: null,
+  // ... other fields
+}
+```
+
+### After AI Processing
+```javascript
+{
+  id: "session-uuid", 
+  transcriptContent: "full conversation text",
+  processed: true,
+  sentimentCategory: "positive" | "neutral" | "negative",
+  questions: '["question 1", "question 2"]', // JSON string
+  summary: "Brief conversation summary",
+  language: "en", // ISO 639-1 code
+  messagesSent: 5,
+  sentiment: 0.8, // Float value (-1 to 1)
+  escalated: false,
+  forwardedHr: false,
+  category: "Schedule & Hours",
+  // ... other fields
+}
+```
+
+## Scheduler Configuration
+
+### Session Refresh Scheduler
+- **File**: `lib/scheduler.js`
+- **Frequency**: Every 30 minutes
+- **Cron**: `*/30 * * * *`
+
+### Processing Scheduler  
+- **File**: `lib/processingScheduler.js`
+- **Frequency**: Every hour
+- **Cron**: `0 * * * *`
+- **Batch size**: 10 sessions per run
+
+## Environment Variables Required
+
+```bash
+# Database
+DATABASE_URL="postgresql://..."
+
+# OpenAI (for processing)
+OPENAI_API_KEY="sk-..."
+
+# NextAuth
+NEXTAUTH_SECRET="..."
+NEXTAUTH_URL="http://localhost:3000"
+```
+
+## Next Steps for Testing
+
+1. **Trigger session refresh** to fetch transcripts:
+   ```bash
+   node scripts/manual-triggers.js refresh
+   ```
+
+2. **Check status** to see if transcripts were fetched:
+   ```bash
+   node scripts/manual-triggers.js status
+   ```
+
+3. **Trigger processing** if transcripts are available:
+   ```bash
+   node scripts/manual-triggers.js process
+   ```
+
+4. **View results** in the dashboard session details pages