feat: Refactor sentiment handling and enhance processing logic for session data

AGENTS.md

@@ -1,128 +0,0 @@
-# LiveDash-Node AGENTS.md
-
-This document provides a comprehensive overview of the LiveDash-Node project, including its architecture, key components, and operational procedures.
-
-## Project Overview
-
-LiveDash-Node is a multi-tenant dashboard system designed for tracking and analyzing chat session metrics. It provides a web-based interface for users to monitor key performance indicators, review chat transcripts, and gain insights into customer interactions. The system is built with a modern technology stack, featuring a Next.js frontend, a Node.js backend, and a Prisma ORM for database interactions.
-
-### Core Features
-
-- **Multi-tenant architecture:** Supports multiple companies, each with its own isolated data and dashboard configurations.
-- **Automated data fetching:** Periodically fetches chat session data from external CSV files.
-- **Data processing and analysis:** Parses and enriches raw session data, calculating metrics such as sentiment, response times, and token usage.
-- **Interactive dashboards:** Visualizes key metrics through a variety of charts and graphs, including geographic maps, donut charts, and time-series data.
-- **Session-level details:** Allows users to drill down into individual chat sessions to view full transcripts and detailed metadata.
-- **User authentication and authorization:** Implements a secure login system with role-based access control.
-
-## Technical Architecture
-
-The application is a full-stack TypeScript project built on the Next.js framework. It uses a custom server to integrate scheduled tasks for data fetching and processing.
-
-### Technology Stack
-
-- **Frontend:**
-  - Next.js (React framework)
-  - TypeScript
-  - Tailwind CSS (styling)
-  - Chart.js, D3.js (data visualization)
-  - Leaflet.js (maps)
-- **Backend:**
-  - Node.js
-  - Next.js API Routes
-  - Prisma (ORM)
-  - SQLite (database)
-- **Authentication:**
-  - NextAuth.js
-- **Testing:**
-  - Playwright (end-to-end testing)
-- **Linting and Formatting:**
-  - ESLint
-  - Prettier
-  - markdownlint
-
-### Project Structure
-
-The project is organized into the following key directories:
-
-- `app/`: Contains the main application code, including pages, layouts, and UI components.
-- `components/`: Reusable React components used throughout the application.
-- `lib/`: Core application logic, including data fetching, processing, and utility functions.
-- `pages/api/`: Next.js API routes for handling backend requests.
-- `prisma/`: Database schema, migrations, and seed scripts.
-- `public/`: Static assets such as images and fonts.
-- `scripts/`: Standalone scripts for various development and operational tasks.
-
-## Key Components
-
-### Data Fetching and Processing
-
-The system uses a two-stage process for handling chat session data:
-
-1.  **Fetching:** The `lib/scheduler.ts` module defines a cron job that periodically fetches new session data from a CSV file specified for each company.
-2.  **Processing:** The `lib/processingScheduler.ts` module defines a second cron job that processes the fetched data. This includes:
-    - Parsing the CSV data.
-    - Enriching the data with additional information (e.g., sentiment analysis, geographic location).
-    - Storing the processed data in the database.
-
-### Database Schema
-
-The database schema is defined in `prisma/schema.prisma` and consists of the following models:
-
-- `Company`: Represents a tenant in the system.
-- `User`: Represents a user with access to the system.
-- `Session`: Represents a single chat session.
-- `Message`: Represents a single message within a chat session.
-
-### API Endpoints
-
-The application exposes a set of API endpoints for handling various client-side requests. These are defined in the `pages/api/` directory and include endpoints for:
-
-- User authentication (login, registration, password reset).
-- Dashboard data (metrics, sessions, users).
-- Administrative tasks (triggering data processing).
-
-## Operational Procedures
-
-### Local Development
-
-To run the application in a local development environment, follow these steps:
-
-1.  Install the dependencies:
-    ```bash
-    npm install
-    ```
-2.  Initialize the database:
-    ```bash
-    npx prisma migrate dev
-    npx prisma db seed
-    ```
-3.  Start the development server:
-    ```bash
-    npm run dev
-    ```
-
-### Running with Schedulers
-
-To run the development server with the data fetching and processing schedulers enabled, use the following command:
-
-```bash
-npm run dev:with-schedulers
-```
-
-### Linting and Formatting
-
-The project uses ESLint and Prettier for code linting and formatting. The following commands are available:
-
-- `npm run lint`: Check for linting errors.
-- `npm run lint:fix`: Automatically fix linting errors.
-- `npm run format`: Format the code using Prettier.
-- `npm run format:check`: Check for formatting errors.
-
-### Testing
-
-The project uses Playwright for end-to-end testing. To run the tests, use the following command:
-
-```bash
-npx playwright test
-```

TODO.md

@@ -1,5 +1,78 @@
 # TODO.md
 
+# Refactor!!!
+
+> Based on my analysis of the codebase, here is a plan with recommendations for improving the project. The focus is on enhancing standardization, abstraction, user experience, and visual
+  design.
+
+## High-Level Recommendations
+
+The project has a solid foundation, but it could be significantly improved by focusing on three key areas:
+
+1. Adopt a UI Component Library: While Tailwind CSS is excellent for styling, using a component library like ShadCN/UI or Headless UI would provide pre-built, accessible, and visually
+   consistent components, saving development time and improving the user experience.
+2. Refactor for Next.js App Router: The project currently uses a mix of the pages and app directories. Migrating fully to the App Router would simplify the project structure, improve
+   performance, and align with the latest Next.js features.
+3. Enhance User Experience: Implementing consistent loading and error states, improving responsiveness, and providing better user feedback would make the application more robust and
+   user-friendly.
+
+## Detailed Improvement Plan
+
+Here is a phased plan to implement these recommendations:
+
+### Phase 1: Foundational Improvements (Standardization & Abstraction)
+
+This phase focuses on cleaning up the codebase, standardizing the project structure, and improving the abstraction of core functionalities.
+
+1. Standardize Project Structure:
+   * Unify Server File: Consolidate server.js, server.mjs, and server.ts into a single server.ts file to remove redundancy.
+   * Migrate to App Router: Move all routes from the pages/api directory to the app/api directory. This will centralize routing logic within the app directory.
+   * Standardize Naming Conventions: Ensure all files and components follow a consistent naming convention (e.g., PascalCase for components, kebab-case for files).
+
+2. Introduce a UI Component Library:
+   * Integrate ShadCN/UI: Add ShadCN/UI to the project to leverage its extensive library of accessible and customizable components.
+   * Replace Custom Components: Gradually replace custom-built components in the components/ directory with their ShadCN/UI equivalents. This will improve visual consistency and reduce
+     maintenance overhead.
+
+3. Refactor Core Logic:
+   * Centralize Data Fetching: Create a dedicated module (e.g., lib/data-service.ts) to handle all data fetching logic, abstracting away the details of using Prisma and external APIs.
+   * Isolate Business Logic: Ensure that business logic (e.g., session processing, metric calculation) is separated from the API routes and UI components.
+
+### Phase 2: UX and Visual Enhancements
+
+This phase focuses on improving the user-facing aspects of the application.
+
+1. Implement Comprehensive Loading and Error States:
+   * Skeleton Loaders: Use skeleton loaders for dashboard components to provide a better loading experience.
+   * Global Error Handling: Implement a global error handling strategy to catch and display user-friendly error messages for API failures or other unexpected issues.
+
+2. Redesign the Dashboard:
+   * Improve Information Hierarchy: Reorganize the dashboard to present the most important information first.
+   * Enhance Visual Appeal: Use the new component library to create a more modern and visually appealing design with a consistent color palette and typography.
+   * Improve Chart Interactivity: Add features like tooltips, zooming, and filtering to the charts to make them more interactive and informative.
+
+3. Ensure Full Responsiveness:
+   * Mobile-First Approach: Review and update all pages and components to ensure they are fully responsive and usable on a wide range of devices.
+
+### Phase 3: Advanced Topics (Security, Performance, and Documentation)
+
+This phase focuses on long-term improvements to the project's stability, performance, and maintainability.
+
+1. Conduct a Security Review:
+   * Input Validation: Ensure that all user inputs are properly validated on both the client and server sides.
+   * Dependency Audit: Regularly audit dependencies for known vulnerabilities.
+
+2. Optimize Performance:
+   * Code Splitting: Leverage Next.js's automatic code splitting to reduce initial load times.
+   * Caching: Implement caching strategies for frequently accessed data to reduce database load and improve API response times.
+
+3. Expand Documentation:
+   * API Documentation: Create detailed documentation for all API endpoints.
+   * Component Library: Document the usage and props of all reusable components.
+   * Update `AGENTS.md`: Keep the AGENTS.md file up-to-date with any architectural changes.
+
+Would you like me to start implementing any part of this plan? I would suggest starting with Phase 1 to build a solid foundation for the other improvements.
+
 ## Dashboard Integration
 
 - [ ] **Resolve `GeographicMap.tsx` and `ResponseTimeDistribution.tsx` data simulation**

app/dashboard/sessions/[id]/page.tsx

@@ -4,7 +4,7 @@ import { useEffect, useState } from "react";
 import { useParams, useRouter } from "next/navigation"; // Import useRouter
 import { useSession } from "next-auth/react"; // Import useSession
 import SessionDetails from "../../../../components/SessionDetails";
-import TranscriptViewer from "../../../../components/TranscriptViewer";
+
 import MessageViewer from "../../../../components/MessageViewer";
 import { ChatSession } from "../../../../lib/types";
 import Link from "next/link";

lib/csvFetcher.ts

@@ -35,10 +35,10 @@ interface SessionData {
   startTime: Date;
   endTime: Date | null;
   ipAddress?: string;
-  country?: string | null; // Will store ISO 3166-1 alpha-2 country code or null/undefined
+  country?: string | null;
-  language?: string | null; // Will store ISO 639-1 language code or null/undefined
+  language?: string | null;
   messagesSent: number;
-  sentiment: number | null;
+  sentiment?: string | null;
   escalated: boolean;
   forwardedHr: boolean;
   fullTranscriptUrl?: string | null;
@@ -142,45 +142,6 @@ function normalizeCategory(categoryStr?: string): string | null {
   return normalized || null;
 }
 
-/**
- * Converts sentiment string values to numeric scores
- * @param sentimentStr The sentiment string from the CSV
- * @returns A numeric score representing the sentiment
- */
-function mapSentimentToScore(sentimentStr?: string): number | null {
-  if (!sentimentStr) return null;
-
-  // Convert to lowercase for case-insensitive matching
-  const sentiment = sentimentStr.toLowerCase();
-
-  // Map sentiment strings to numeric values on a scale from -1 to 2
-  const sentimentMap: Record<string, number> = {
-    happy: 1.0,
-    excited: 1.5,
-    positive: 0.8,
-    neutral: 0.0,
-    playful: 0.7,
-    negative: -0.8,
-    angry: -1.0,
-    sad: -0.7,
-    frustrated: -0.9,
-    positief: 0.8, // Dutch
-    neutraal: 0.0, // Dutch
-    negatief: -0.8, // Dutch
-    positivo: 0.8, // Spanish/Italian
-    neutro: 0.0, // Spanish/Italian
-    negativo: -0.8, // Spanish/Italian
-    yes: 0.5, // For any "yes" sentiment
-    no: -0.5, // For any "no" sentiment
-  };
-
-  return sentimentMap[sentiment] !== undefined
-    ? sentimentMap[sentiment]
-    : isNaN(parseFloat(sentiment))
-      ? null
-      : parseFloat(sentiment);
-}
-
 /**
  * Checks if a string value should be considered as boolean true
  * @param value The string value to check
@@ -314,7 +275,7 @@ export async function fetchAndParseCsv(
     country: getCountryCode(r.country),
     language: getLanguageCode(r.language),
     messagesSent: Number(r.messages_sent) || 0,
-    sentiment: mapSentimentToScore(r.sentiment),
+    sentiment: r.sentiment,
     escalated: isTruthyValue(r.escalated),
     forwardedHr: isTruthyValue(r.forwarded_hr),
     fullTranscriptUrl: r.full_transcript_url,

lib/metrics.ts

@@ -357,7 +357,7 @@ export function sessionMetrics(
   let totalTokens = 0;
   let totalTokensEur = 0;
   const wordCounts: { [key: string]: number } = {};
-  let alerts = 0;
+  const alerts = 0;
 
   // New metrics variables
   const hourlySessionCounts: { [hour: string]: number } = {};
@@ -463,22 +463,15 @@ export function sessionMetrics(
     if (session.forwardedHr) forwardedHrCount++;
 
     // Sentiment
-    if (session.sentiment !== undefined && session.sentiment !== null) {
+    if (session.sentiment === "positive") {
-      // Example thresholds, adjust as needed
+      sentimentPositiveCount++;
-      if (session.sentiment > 0.3) sentimentPositiveCount++;
+    } else if (session.sentiment === "neutral") {
-      else if (session.sentiment < -0.3) sentimentNegativeCount++;
+      sentimentNeutralCount++;
-      else sentimentNeutralCount++;
+    } else if (session.sentiment === "negative") {
+      sentimentNegativeCount++;
     }
 
-    // Sentiment Alert Check
+
-    if (
-      companyConfig.sentimentAlert !== undefined &&
-      session.sentiment !== undefined &&
-      session.sentiment !== null &&
-      session.sentiment < companyConfig.sentimentAlert
-    ) {
-      alerts++;
-    }
 
     // Tokens
     if (session.tokens !== undefined && session.tokens !== null) {

lib/processingScheduler.ts

@@ -38,14 +38,14 @@ const OPENAI_API_URL = "https://api.openai.com/v1/chat/completions";
 
 interface ProcessedData {
   language: string;
-  messages_sent: number;
+  sentiment: "positive" | "neutral" | "negative";
-  sentiment: SentimentCategory;
   escalated: boolean;
   forwarded_hr: boolean;
   category: ValidCategory;
-  questions: string[];
+  questions: string | string[];
   summary: string;
-  session_id: string;
+  tokens: number;
+  tokens_eur: number;
 }
 
 interface ProcessingResult {
@@ -76,30 +76,31 @@ System: You are a JSON-generating assistant. Your task is to analyze raw chat tr
 
 Here is the schema you must follow:
 
-{
+{{
 "language": "ISO 639-1 code, e.g., 'en', 'nl'",
-"messages_sent": "integer, number of messages from the user",
 "sentiment": "'positive', 'neutral', or 'negative'",
 "escalated": "bool: true if the assistant connected or referred to a human agent, otherwise false",
 "forwarded_hr": "bool: true if HR contact info was given, otherwise false",
 "category": "one of: 'Schedule & Hours', 'Leave & Vacation', 'Sick Leave & Recovery', 'Salary & Compensation', 'Contract & Hours', 'Onboarding', 'Offboarding', 'Workwear & Staff Pass', 'Team & Contacts', 'Personal Questions', 'Access & Login', 'Social questions', 'Unrecognized / Other'",
-"questions": array of simplified questions asked by the user formulated in English, try to make a question out of messages,
+"questions": "a single question or an array of simplified questions asked by the user formulated in English, try to make a question out of messages",
 "summary": "Brief summary (1–2 sentences) of the conversation",
-}
+"tokens": "integer, number of tokens used for the API call",
+"tokens_eur": "float, cost of the API call in EUR",
+}}
+
 You must format your output as a JSON value that adheres to a given "JSON Schema" instance.
 
 "JSON Schema" is a declarative language that allows you to annotate and validate JSON documents.
 
-For example, the example "JSON Schema" instance {{"properties": {{"foo": {{"description": "a list of test words", "type": "array", "items": {{"type": "string"}}}}}}, "required": ["foo"]}}}}
+For example, the example "JSON Schema" instance {"properties": {"foo": {"description": "a list of test words", "type": "array", "items": {"type": "string"}}}}, "required": ["foo"]}}
 would match an object with one required property, "foo". The "type" property specifies "foo" must be an "array", and the "description" property semantically describes it as "a list of test words". The items within "foo" must be strings.
-Thus, the object {{"foo": ["bar", "baz"]}} is a well-formatted instance of this example "JSON Schema". The object {{"properties": {{"foo": ["bar", "baz"]}}}} is not well-formatted.
+Thus, the object {"foo": ["bar", "baz"]} is a well-formatted instance of this example "JSON Schema". The object {"properties": {"foo": ["bar", "baz"]}} is not well-formatted.
 
 Your output will be parsed and type-checked according to the provided schema instance, so make sure all fields in your output match the schema exactly and there are no trailing commas!
 
 Here is the JSON Schema instance your output must adhere to. Include the enclosing markdown codeblock:
-\`\`\`json
+
-{"type":"object","properties":{"language":{"type":"string","pattern":"^[a-z]{2}$","description":"ISO 639-1 code for the user's primary language"},"messages_sent":{"type":"integer","minimum":0,"description":"Number of messages sent by the user"},"sentiment":{"type":"string","enum":["positive","neutral","negative"],"description":"Overall tone of the user during the conversation"},"escalated":{"type":"boolean","description":"Whether the assistant indicated it could not help"},"forwarded_hr":{"type":"boolean","description":"Whether HR contact was mentioned or provided"},"category":{"type":"string","enum":["Schedule & Hours","Leave & Vacation","Sick Leave & Recovery","Salary & Compensation","Contract & Hours","Onboarding","Offboarding","Workwear & Staff Pass","Team & Contacts","Personal Questions","Access & Login","Social questions","Unrecognized / Other"],"description":"Best-fitting topic category for the conversation"},"questions":{"type":"array","items":{"type":"string","minLength":5},"minItems":0,"maxItems":5,"description":"List of paraphrased questions asked by the user in English"},"summary":{"type":"string","minLength":10,"maxLength":300,"description":"Brief summary of the conversation"},"session_id":{"type":"string","pattern":"^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$","minLength":36,"maxLength":36,"description":"Unique identifier for the conversation session"}},"required":["language","messages_sent","sentiment","escalated","forwarded_hr","category","questions","summary","session_id"],"additionalProperties":false,"$schema":"http://json-schema.org/draft-07/schema#"}
+{{"type":"object","properties":{"language":{"type":"string","pattern":"^[a-z]{2}$","description":"ISO 639-1 code for the user's primary language"},"sentiment":{"type":"string","enum":["positive","neutral","negative"],"description":"Overall tone of the user during the conversation"},"escalated":{"type":"boolean","description":"Whether the assistant indicated it could not help"},"forwarded_hr":{"type":"boolean","description":"Whether HR contact was mentioned or provided"},"category":{"type":"string","enum":["Schedule & Hours","Leave & Vacation","Sick Leave & Recovery","Salary & Compensation","Contract & Hours","Onboarding","Offboarding","Workwear & Staff Pass","Team & Contacts","Personal Questions","Access & Login","Social questions","Unrecognized / Other"],"description":"Best-fitting topic category for the conversation"},"questions":{"oneOf":[{"type":"string"},{"type":"array","items":{"type":"string"}}],"description":"A single question or a list of paraphrased questions asked by the user in English"},"summary":{"type":"string","minLength":10,"maxLength":300,"description":"Brief summary of the conversation"},"tokens":{"type":"integer","description":"Number of tokens used for the API call"},"tokens_eur":{"type":"number","description":"Cost of the API call in EUR"}},"required":["language","sentiment","escalated","forwarded_hr","category","questions","summary","tokens","tokens_eur"],"additionalProperties":false,"$schema":"http://json-schema.org/draft-07/schema#"}}
-\`\`\`
   `;
 
   try {
@@ -151,13 +152,14 @@ function validateOpenAIResponse(data: any): void {
   // Check required fields
   const requiredFields = [
     "language",
-    "messages_sent",
     "sentiment",
     "escalated",
     "forwarded_hr",
     "category",
     "questions",
     "summary",
+    "tokens",
+    "tokens_eur",
   ];
 
   for (const field of requiredFields) {
@@ -173,10 +175,6 @@ function validateOpenAIResponse(data: any): void {
     );
   }
 
-  if (typeof data.messages_sent !== "number" || data.messages_sent < 0) {
-    throw new Error("Invalid messages_sent. Expected non-negative number");
-  }
-
   if (!["positive", "neutral", "negative"].includes(data.sentiment)) {
     throw new Error(
       "Invalid sentiment. Expected 'positive', 'neutral', or 'negative'"
@@ -197,8 +195,8 @@ function validateOpenAIResponse(data: any): void {
     );
   }
 
-  if (!Array.isArray(data.questions)) {
+  if (typeof data.questions !== "string" && !Array.isArray(data.questions)) {
-    throw new Error("Invalid questions. Expected array of strings");
+    throw new Error("Invalid questions. Expected string or array of strings");
   }
 
   if (
@@ -211,9 +209,12 @@ function validateOpenAIResponse(data: any): void {
     );
   }
 
-  // session_id is optional in the response, we'll use the one we passed in
+  if (typeof data.tokens !== "number" || data.tokens < 0) {
-  if (data.session_id && typeof data.session_id !== "string") {
+    throw new Error("Invalid tokens. Expected non-negative number");
-    throw new Error("Invalid session_id. Expected string");
+  }
+
+  if (typeof data.tokens_eur !== "number" || data.tokens_eur < 0) {
+    throw new Error("Invalid tokens_eur. Expected non-negative number");
   }
 }
 
@@ -275,7 +276,11 @@ async function processSingleSession(session: any): Promise<ProcessingResult> {
     );
 
     // Check if the processed data indicates low quality (empty questions, very short summary, etc.)
-    const hasValidQuestions = processedData.questions && processedData.questions.length > 0;
+    const hasValidQuestions =
+      processedData.questions &&
+      (Array.isArray(processedData.questions)
+        ? processedData.questions.length > 0
+        : typeof processedData.questions === "string");
     const hasValidSummary = processedData.summary && processedData.summary.length >= 10;
     const isValidData = hasValidQuestions && hasValidSummary;
 
@@ -284,14 +289,18 @@ async function processSingleSession(session: any): Promise<ProcessingResult> {
       where: { id: session.id },
       data: {
         language: processedData.language,
-        messagesSent: processedData.messages_sent,
+        sentiment: processedData.sentiment,
-        sentiment: null, // Remove numeric sentiment, use only sentimentCategory
-        sentimentCategory: processedData.sentiment,
         escalated: processedData.escalated,
         forwardedHr: processedData.forwarded_hr,
         category: processedData.category,
-        questions: JSON.stringify(processedData.questions),
+        questions: processedData.questions,
         summary: processedData.summary,
+        tokens: {
+          increment: processedData.tokens,
+        },
+        tokensEur: {
+          increment: processedData.tokens_eur,
+        },
         processed: true,
       },
     });

lib/processingSchedulerNoCron.ts

@@ -36,14 +36,14 @@ const OPENAI_API_URL = "https://api.openai.com/v1/chat/completions";
 
 interface ProcessedData {
   language: string;
-  messages_sent: number;
+  sentiment: "positive" | "neutral" | "negative";
-  sentiment: SentimentCategory;
   escalated: boolean;
   forwarded_hr: boolean;
   category: ValidCategory;
-  questions: string[];
+  questions: string | string[];
   summary: string;
-  session_id: string;
+  tokens: number;
+  tokens_eur: number;
 }
 
 interface ProcessingResult {
@@ -76,14 +76,16 @@ Here is the schema you must follow:
 
 {
 "language": "ISO 639-1 code, e.g., 'en', 'nl'",
-"messages_sent": "integer, number of messages from the user",
 "sentiment": "'positive', 'neutral', or 'negative'",
 "escalated": "bool: true if the assistant connected or referred to a human agent, otherwise false",
 "forwarded_hr": "bool: true if HR contact info was given, otherwise false",
 "category": "one of: 'Schedule & Hours', 'Leave & Vacation', 'Sick Leave & Recovery', 'Salary & Compensation', 'Contract & Hours', 'Onboarding', 'Offboarding', 'Workwear & Staff Pass', 'Team & Contacts', 'Personal Questions', 'Access & Login', 'Social questions', 'Unrecognized / Other'",
-"questions": array of simplified questions asked by the user formulated in English, try to make a question out of messages,
+"questions": "a single question or an array of simplified questions asked by the user formulated in English, try to make a question out of messages",
 "summary": "Brief summary (1–2 sentences) of the conversation",
+"tokens": "integer, number of tokens used for the API call",
+"tokens_eur": "float, cost of the API call in EUR",
 }
+
 You must format your output as a JSON value that adheres to a given "JSON Schema" instance.
 
 "JSON Schema" is a declarative language that allows you to annotate and validate JSON documents.
@@ -95,9 +97,9 @@ Thus, the object {{"foo": ["bar", "baz"]}} is a well-formatted instance of this
 Your output will be parsed and type-checked according to the provided schema instance, so make sure all fields in your output match the schema exactly and there are no trailing commas!
 
 Here is the JSON Schema instance your output must adhere to. Include the enclosing markdown codeblock:
-\`\`\`json
+
-{"type":"object","properties":{"language":{"type":"string","pattern":"^[a-z]{2}$","description":"ISO 639-1 code for the user's primary language"},"messages_sent":{"type":"integer","minimum":0,"description":"Number of messages sent by the user"},"sentiment":{"type":"string","enum":["positive","neutral","negative"],"description":"Overall tone of the user during the conversation"},"escalated":{"type":"boolean","description":"Whether the assistant indicated it could not help"},"forwarded_hr":{"type":"boolean","description":"Whether HR contact was mentioned or provided"},"category":{"type":"string","enum":["Schedule & Hours","Leave & Vacation","Sick Leave & Recovery","Salary & Compensation","Contract & Hours","Onboarding","Offboarding","Workwear & Staff Pass","Team & Contacts","Personal Questions","Access & Login","Social questions","Unrecognized / Other"],"description":"Best-fitting topic category for the conversation"},"questions":{"type":"array","items":{"type":"string","minLength":5},"minItems":0,"maxItems":5,"description":"List of paraphrased questions asked by the user in English"},"summary":{"type":"string","minLength":10,"maxLength":300,"description":"Brief summary of the conversation"},"session_id":{"type":"string","pattern":"^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$","minLength":36,"maxLength":36,"description":"Unique identifier for the conversation session"}},"required":["language","messages_sent","sentiment","escalated","forwarded_hr","category","questions","summary","session_id"],"additionalProperties":false,"$schema":"http://json-schema.org/draft-07/schema#"}
+{{"type":"object","properties":{"language":{"type":"string","pattern":"^[a-z]{2}$","description":"ISO 639-1 code for the user's primary language"},"sentiment":{"type":"string","enum":["positive","neutral","negative"],"description":"Overall tone of the user during the conversation"},"escalated":{"type":"boolean","description":"Whether the assistant indicated it could not help"},"forwarded_hr":{"type":"boolean","description":"Whether HR contact was mentioned or provided"},"category":{"type":"string","enum":["Schedule & Hours","Leave & Vacation","Sick Leave & Recovery","Salary & Compensation","Contract & Hours","Onboarding","Offboarding","Workwear & Staff Pass","Team & Contacts","Personal Questions","Access & Login","Social questions","Unrecognized / Other"],"description":"Best-fitting topic category for the conversation"},"questions":{"oneOf":[{"type":"string"},{"type":"array","items":{"type":"string"}}],"description":"A single question or a list of paraphrased questions asked by the user in English"},"summary":{"type":"string","minLength":10,"maxLength":300,"description":"Brief summary of the conversation"},"tokens":{"type":"integer","description":"Number of tokens used for the API call"},"tokens_eur":{"type":"number","description":"Cost of the API call in EUR"}},"required":["language","sentiment","escalated","forwarded_hr","category","questions","summary","tokens","tokens_eur"],"additionalProperties":false,"$schema":"http://json-schema.org/draft-07/schema#"}}
-\`\`\`
+
   `;
 
   try {
@@ -149,13 +151,14 @@ function validateOpenAIResponse(data: any): void {
   // Check required fields
   const requiredFields = [
     "language",
-    "messages_sent",
     "sentiment",
     "escalated",
     "forwarded_hr",
     "category",
     "questions",
     "summary",
+    "tokens",
+    "tokens_eur",
   ];
 
   for (const field of requiredFields) {
@@ -171,10 +174,6 @@ function validateOpenAIResponse(data: any): void {
     );
   }
 
-  if (typeof data.messages_sent !== "number" || data.messages_sent < 0) {
-    throw new Error("Invalid messages_sent. Expected non-negative number");
-  }
-
   if (!["positive", "neutral", "negative"].includes(data.sentiment)) {
     throw new Error(
       "Invalid sentiment. Expected 'positive', 'neutral', or 'negative'"
@@ -195,8 +194,8 @@ function validateOpenAIResponse(data: any): void {
     );
   }
 
-  if (!Array.isArray(data.questions)) {
+  if (typeof data.questions !== "string" && !Array.isArray(data.questions)) {
-    throw new Error("Invalid questions. Expected array of strings");
+    throw new Error("Invalid questions. Expected string or array of strings");
   }
 
   if (
@@ -209,9 +208,12 @@ function validateOpenAIResponse(data: any): void {
     );
   }
 
-  // session_id is optional in the response, we'll use the one we passed in
+  if (typeof data.tokens !== "number" || data.tokens < 0) {
-  if (data.session_id && typeof data.session_id !== "string") {
+    throw new Error("Invalid tokens. Expected non-negative number");
-    throw new Error("Invalid session_id. Expected string");
+  }
+
+  if (typeof data.tokens_eur !== "number" || data.tokens_eur < 0) {
+    throw new Error("Invalid tokens_eur. Expected non-negative number");
   }
 }
 
@@ -273,7 +275,11 @@ async function processSingleSession(session: any): Promise<ProcessingResult> {
     );
 
     // Check if the processed data indicates low quality (empty questions, very short summary, etc.)
-    const hasValidQuestions = processedData.questions && processedData.questions.length > 0;
+    const hasValidQuestions =
+      processedData.questions &&
+      (Array.isArray(processedData.questions)
+        ? processedData.questions.length > 0
+        : typeof processedData.questions === "string");
     const hasValidSummary = processedData.summary && processedData.summary.length >= 10;
     const isValidData = hasValidQuestions && hasValidSummary;
 
@@ -282,14 +288,18 @@ async function processSingleSession(session: any): Promise<ProcessingResult> {
       where: { id: session.id },
       data: {
         language: processedData.language,
-        messagesSent: processedData.messages_sent,
+        sentiment: processedData.sentiment,
-        sentiment: null, // Remove numeric sentiment, use only sentimentCategory
-        sentimentCategory: processedData.sentiment,
         escalated: processedData.escalated,
         forwardedHr: processedData.forwarded_hr,
         category: processedData.category,
-        questions: JSON.stringify(processedData.questions),
+        questions: processedData.questions,
         summary: processedData.summary,
+        tokens: {
+          increment: processedData.tokens,
+        },
+        tokensEur: {
+          increment: processedData.tokens_eur,
+        },
         processed: true,
       },
     });

lib/types.ts

@@ -75,8 +75,7 @@ export interface ChatSession {
   language?: string | null;
   country?: string | null;
   ipAddress?: string | null;
-  sentiment?: number | null;
+  sentiment?: string | null;
-  sentimentCategory?: string | null; // "positive", "neutral", "negative" from OpenAPI
   messagesSent?: number;
   startTime: Date;
   endTime?: Date | null;

scripts/process_sessions.ts

@@ -8,14 +8,14 @@ const OPENAI_API_URL = "https://api.openai.com/v1/chat/completions";
 // Define the expected response structure from OpenAI
 interface OpenAIProcessedData {
   language: string;
-  messages_sent: number;
   sentiment: "positive" | "neutral" | "negative";
   escalated: boolean;
   forwarded_hr: boolean;
   category: string;
-  questions: string[];
+  questions: string | string[];
   summary: string;
-  session_id: string;
+  tokens: number;
+  tokens_eur: number;
 }
 
 /**
@@ -37,11 +37,10 @@ async function processTranscriptWithOpenAI(
     You are an AI assistant tasked with analyzing chat transcripts.
     Extract the following information from the transcript:
     1. The primary language used by the user (ISO 639-1 code)
-    2. Number of messages sent by the user
+    2. Overall sentiment (positive, neutral, or negative)
-    3. Overall sentiment (positive, neutral, or negative)
+    3. Whether the conversation was escalated
-    4. Whether the conversation was escalated
+    4. Whether HR contact was mentioned or provided
-    5. Whether HR contact was mentioned or provided
+    5. The best-fitting category for the conversation from this list:
-    6. The best-fitting category for the conversation from this list:
        - Schedule & Hours
        - Leave & Vacation
        - Sick Leave & Recovery
@@ -55,20 +54,22 @@ async function processTranscriptWithOpenAI(
        - Access & Login
        - Social questions
        - Unrecognized / Other
-    7. Up to 5 paraphrased questions asked by the user (in English)
+    6. A single question or an array of simplified questions asked by the user formulated in English
-    8. A brief summary of the conversation (10-300 characters)
+    7. A brief summary of the conversation (10-300 characters)
+    8. The number of tokens used for the API call
+    9. The cost of the API call in EUR
     
     Return the data in JSON format matching this schema:
     {
       "language": "ISO 639-1 code",
-      "messages_sent": number,
       "sentiment": "positive|neutral|negative",
       "escalated": boolean,
       "forwarded_hr": boolean,
       "category": "one of the categories listed above",
-      "questions": ["question 1", "question 2", ...],
+      "questions": "a single question or [\"question 1\", \"question 2\", ...]",
       "summary": "brief summary",
-      "session_id": "${sessionId}"
+      "tokens": number,
+      "tokens_eur": number
     }
   `;
 
@@ -124,14 +125,14 @@ function validateOpenAIResponse(
   // Check required fields
   const requiredFields = [
     "language",
-    "messages_sent",
     "sentiment",
     "escalated",
     "forwarded_hr",
     "category",
     "questions",
     "summary",
-    "session_id",
+    "tokens",
+    "tokens_eur",
   ];
 
   for (const field of requiredFields) {
@@ -147,10 +148,6 @@ function validateOpenAIResponse(
     );
   }
 
-  if (typeof data.messages_sent !== "number" || data.messages_sent < 0) {
-    throw new Error("Invalid messages_sent. Expected non-negative number");
-  }
-
   if (!["positive", "neutral", "negative"].includes(data.sentiment)) {
     throw new Error(
       "Invalid sentiment. Expected 'positive', 'neutral', or 'negative'"
@@ -187,8 +184,8 @@ function validateOpenAIResponse(
     );
   }
 
-  if (!Array.isArray(data.questions)) {
+  if (typeof data.questions !== "string" && !Array.isArray(data.questions)) {
-    throw new Error("Invalid questions. Expected array of strings");
+    throw new Error("Invalid questions. Expected string or array of strings");
   }
 
   if (
@@ -201,8 +198,12 @@ function validateOpenAIResponse(
     );
   }
 
-  if (typeof data.session_id !== "string") {
+  if (typeof data.tokens !== "number" || data.tokens < 0) {
-    throw new Error("Invalid session_id. Expected string");
+    throw new Error("Invalid tokens. Expected non-negative number");
+  }
+
+  if (typeof data.tokens_eur !== "number" || data.tokens_eur < 0) {
+    throw new Error("Invalid tokens_eur. Expected non-negative number");
   }
 }
 
@@ -252,26 +253,23 @@ async function processUnprocessedSessions() {
         session.transcriptContent
       );
 
-      // Map sentiment string to float value for compatibility with existing data
-      const sentimentMap: Record<string, number> = {
-        positive: 0.8,
-        neutral: 0.0,
-        negative: -0.8,
-      };
-
       // Update the session with processed data
       await prisma.session.update({
         where: { id: session.id },
         data: {
           language: processedData.language,
-          messagesSent: processedData.messages_sent,
+          sentiment: processedData.sentiment,
-          sentiment: sentimentMap[processedData.sentiment] || 0,
-          sentimentCategory: processedData.sentiment,
           escalated: processedData.escalated,
           forwardedHr: processedData.forwarded_hr,
           category: processedData.category,
-          questions: JSON.stringify(processedData.questions),
+          questions: processedData.questions,
           summary: processedData.summary,
+          tokens: {
+            increment: processedData.tokens,
+          },
+          tokensEur: {
+            increment: processedData.tokens_eur,
+          },
           processed: true,
         },
       });