kjanat/livedash-node
1
0
Fork 0
mirror of https://github.com/kjanat/livedash-node.git synced 2026-01-16 06:32:10 +01:00
Code Issues Packages Projects Releases Wiki Activity

feat: comprehensive security and architecture improvements

Browse Source 
- Add Zod validation schemas with strong password requirements (12+ chars, complexity)
- Implement rate limiting for authentication endpoints (registration, password reset)
- Remove duplicate MetricCard component, consolidate to ui/metric-card.tsx
- Update README.md to use pnpm commands consistently
- Enhance authentication security with 12-round bcrypt hashing
- Add comprehensive input validation for all API endpoints
- Fix security vulnerabilities in user registration and password reset flows

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Kaj Kowalski
2025-06-28 01:52:53 +02:00
parent 192f9497b4
commit 7f48a085bf
68 changed files with 8045 additions and 4542 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.clinerules/pnpm-not-npm.md
View File

@ -1 +1 @@
Use pnpm to manage this project, not npm!
Use pnpm to manage this project, not npm!

18
CLAUDE.md
View File

@ -5,18 +5,21 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Development Commands
**Core Development:**
- `pnpm dev` - Start development server (runs custom server.ts with schedulers)
- `pnpm dev:next-only` - Start Next.js only with Turbopack (no schedulers)
- `pnpm build` - Build production application
- `pnpm start` - Run production server
**Code Quality:**
- `pnpm lint` - Run ESLint
- `pnpm lint:fix` - Fix ESLint issues automatically
- `pnpm format` - Format code with Prettier
- `pnpm format:check` - Check formatting without fixing
**Database:**
- `pnpm prisma:generate` - Generate Prisma client
- `pnpm prisma:migrate` - Run database migrations
- `pnpm prisma:push` - Push schema changes to database
@ -25,11 +28,13 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
- `pnpm prisma:studio` - Open Prisma Studio database viewer
**Testing:**
- `pnpm test` - Run tests once
- `pnpm test:watch` - Run tests in watch mode
- `pnpm test:coverage` - Run tests with coverage report
**Markdown:**
- `pnpm lint:md` - Lint Markdown files
- `pnpm lint:md:fix` - Fix Markdown linting issues
@ -38,6 +43,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
**LiveDash-Node** is a real-time analytics dashboard for monitoring user sessions with AI-powered analysis and processing pipeline.
### Tech Stack
- **Frontend:** Next.js 15 + React 19 + TailwindCSS 4
- **Backend:** Next.js API Routes + Custom Node.js server
- **Database:** PostgreSQL with Prisma ORM
@ -50,6 +56,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
**1. Multi-Stage Processing Pipeline**
The system processes user sessions through distinct stages tracked in `SessionProcessingStatus`:
- `CSV_IMPORT` - Import raw CSV data into `SessionImport`
- `TRANSCRIPT_FETCH` - Fetch transcript content from URLs
- `SESSION_CREATION` - Create normalized `Session` and `Message` records
@ -57,6 +64,7 @@ The system processes user sessions through distinct stages tracked in `SessionPr
- `QUESTION_EXTRACTION` - Extract questions from conversations
**2. Database Architecture**
- **Multi-tenant design** with `Company` as root entity
- **Dual storage pattern**: Raw CSV data in `SessionImport`, processed data in `Session`
- **1-to-1 relationship** between `SessionImport` and `Session` via `importId`
@ -65,11 +73,13 @@ The system processes user sessions through distinct stages tracked in `SessionPr
- **Flexible AI model management** through `AIModel`, `AIModelPricing`, and `CompanyAIModel`
**3. Custom Server Architecture**
- `server.ts` - Custom Next.js server with configurable scheduler initialization
- Three main schedulers: CSV import, import processing, and session processing
- Environment-based configuration via `lib/env.ts`
**4. Key Processing Libraries**
- `lib/scheduler.ts` - CSV import scheduling
- `lib/importProcessor.ts` - Raw data to Session conversion
- `lib/processingScheduler.ts` - AI analysis pipeline
@ -80,18 +90,21 @@ The system processes user sessions through distinct stages tracked in `SessionPr
**Environment Configuration:**
Environment variables are managed through `lib/env.ts` with .env.local file support:
- Database: PostgreSQL via `DATABASE_URL` and `DATABASE_URL_DIRECT`
- Authentication: `NEXTAUTH_SECRET`, `NEXTAUTH_URL`
- AI Processing: `OPENAI_API_KEY`
- Schedulers: `SCHEDULER_ENABLED`, various interval configurations
**Key Files to Understand:**
- `prisma/schema.prisma` - Complete database schema with enums and relationships
- `server.ts` - Custom server entry point
- `lib/env.ts` - Environment variable management and validation
- `app/` - Next.js App Router structure
**Testing:**
- Uses Vitest for unit testing
- Playwright for E2E testing
- Test files in `tests/` directory
@ -99,16 +112,19 @@ Environment variables are managed through `lib/env.ts` with .env.local file supp
### Important Notes
**Scheduler System:**
- Schedulers are optional and controlled by `SCHEDULER_ENABLED` environment variable
- Use `pnpm dev:next-only` to run without schedulers for pure frontend development
- Three separate schedulers handle different pipeline stages
**Database Migrations:**
- Always run `pnpm prisma:generate` after schema changes
- Use `pnpm prisma:migrate` for production-ready migrations
- Use `pnpm prisma:push` for development schema changes
**AI Processing:**
- All AI requests are tracked for cost analysis
- Support for multiple AI models per company
- Time-based pricing management for accurate cost calculation
- Time-based pricing management for accurate cost calculation

106
README.md
View File

@ -2,65 +2,65 @@
A real-time analytics dashboard for monitoring user sessions and interactions with interactive data visualizations and detailed metrics.
![Next.js](https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fkjanat%2Flivedash-node%2Fmaster%2Fpackage.json&search=%22next%22%5Cs*%3A%5Cs*%22%5C%5E(%3F%3Cversion%3E%5Cd%2B%5C.%5Cd*).*%22&replace=%24%3Cversion%3E&logo=nextdotjs&label=Nextjs&color=%23000000)
![React](https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fkjanat%2Flivedash-node%2Fmaster%2Fpackage.json&search=%22react%22%5Cs*%3A%5Cs*%22%5C%5E(%3F%3Cversion%3E%5Cd%2B%5C.%5Cd*).*%22&replace=%24%3Cversion%3E&logo=react&label=React&color=%2361DAFB)
![TypeScript](https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fkjanat%2Flivedash-node%2Fmaster%2Fpackage.json&search=%22typescript%22%5Cs*%3A%5Cs*%22%5C%5E(%3F%3Cversion%3E%5Cd%2B%5C.%5Cd*).*%22&replace=%24%3Cversion%3E&logo=typescript&label=TypeScript&color=%233178C6)
![Prisma](https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fkjanat%2Flivedash-node%2Fmaster%2Fpackage.json&search=%22prisma%22%5Cs*%3A%5Cs*%22%5C%5E(%3F%3Cversion%3E%5Cd%2B%5C.%5Cd*).*%22&replace=%24%3Cversion%3E&logo=prisma&label=Prisma&color=%232D3748)
![TailwindCSS](https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fkjanat%2Flivedash-node%2Fmaster%2Fpackage.json&search=%22tailwindcss%22%5Cs*%3A%5Cs*%22%5C%5E(%3F%3Cversion%3E%5Cd%2B%5C.%5Cd*).*%22&replace=%24%3Cversion%3E&logo=tailwindcss&label=TailwindCSS&color=%2306B6D4)
![Next.js](<https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fkjanat%2Flivedash-node%2Fmaster%2Fpackage.json&search=%22next%22%5Cs*%3A%5Cs*%22%5C%5E(%3F%3Cversion%3E%5Cd%2B%5C.%5Cd*).*%22&replace=%24%3Cversion%3E&logo=nextdotjs&label=Nextjs&color=%23000000>)
![React](<https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fkjanat%2Flivedash-node%2Fmaster%2Fpackage.json&search=%22react%22%5Cs*%3A%5Cs*%22%5C%5E(%3F%3Cversion%3E%5Cd%2B%5C.%5Cd*).*%22&replace=%24%3Cversion%3E&logo=react&label=React&color=%2361DAFB>)
![TypeScript](<https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fkjanat%2Flivedash-node%2Fmaster%2Fpackage.json&search=%22typescript%22%5Cs*%3A%5Cs*%22%5C%5E(%3F%3Cversion%3E%5Cd%2B%5C.%5Cd*).*%22&replace=%24%3Cversion%3E&logo=typescript&label=TypeScript&color=%233178C6>)
![Prisma](<https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fkjanat%2Flivedash-node%2Fmaster%2Fpackage.json&search=%22prisma%22%5Cs*%3A%5Cs*%22%5C%5E(%3F%3Cversion%3E%5Cd%2B%5C.%5Cd*).*%22&replace=%24%3Cversion%3E&logo=prisma&label=Prisma&color=%232D3748>)
![TailwindCSS](<https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fkjanat%2Flivedash-node%2Fmaster%2Fpackage.json&search=%22tailwindcss%22%5Cs*%3A%5Cs*%22%5C%5E(%3F%3Cversion%3E%5Cd%2B%5C.%5Cd*).*%22&replace=%24%3Cversion%3E&logo=tailwindcss&label=TailwindCSS&color=%2306B6D4>)
## Features
- **Real-time Session Monitoring**: Track and analyze user sessions as they happen
- **Interactive Visualizations**: Geographic maps, response time distributions, and more
- **Advanced Analytics**: Detailed metrics and insights about user behavior
- **User Management**: Secure authentication with role-based access control
- **Customizable Dashboard**: Filter and sort data based on your specific needs
- **Session Details**: In-depth analysis of individual user sessions
- **Real-time Session Monitoring**: Track and analyze user sessions as they happen
- **Interactive Visualizations**: Geographic maps, response time distributions, and more
- **Advanced Analytics**: Detailed metrics and insights about user behavior
- **User Management**: Secure authentication with role-based access control
- **Customizable Dashboard**: Filter and sort data based on your specific needs
- **Session Details**: In-depth analysis of individual user sessions
## Tech Stack
- **Frontend**: React 19, Next.js 15, TailwindCSS 4
- **Backend**: Next.js API Routes, Node.js
- **Database**: Prisma ORM with SQLite (default), compatible with PostgreSQL
- **Authentication**: NextAuth.js
- **Visualization**: Chart.js, D3.js, React Leaflet
- **Data Processing**: Node-cron for scheduled tasks
- **Frontend**: React 19, Next.js 15, TailwindCSS 4
- **Backend**: Next.js API Routes, Node.js
- **Database**: Prisma ORM with SQLite (default), compatible with PostgreSQL
- **Authentication**: NextAuth.js
- **Visualization**: Chart.js, D3.js, React Leaflet
- **Data Processing**: Node-cron for scheduled tasks
## Getting Started
### Prerequisites
- Node.js (LTS version recommended)
- npm or yarn
- Node.js (LTS version recommended)
- pnpm (recommended package manager)
### Installation
1. Clone this repository:
```bash
git clone https://github.com/kjanat/livedash-node.git
cd livedash-node
```
```bash
git clone https://github.com/kjanat/livedash-node.git
cd livedash-node
```
2. Install dependencies:
```bash
npm install
```
```bash
pnpm install
```
3. Set up the database:
```bash
npm run prisma:generate
npm run prisma:migrate
npm run prisma:seed
```
```bash
pnpm run prisma:generate
pnpm run prisma:migrate
pnpm run prisma:seed
```
4. Start the development server:
```bash
npm run dev
```
```bash
pnpm run dev
```
5. Open your browser and navigate to <http://localhost:3000>
@ -76,22 +76,22 @@ NEXTAUTH_SECRET=your-secret-here
## Project Structure
- `app/`: Next.js App Router components and pages
- `components/`: Reusable React components
- `lib/`: Utility functions and shared code
- `pages/`: API routes and server-side code
- `prisma/`: Database schema and migrations
- `public/`: Static assets
- `docs/`: Project documentation
- `app/`: Next.js App Router components and pages
- `components/`: Reusable React components
- `lib/`: Utility functions and shared code
- `pages/`: API routes and server-side code
- `prisma/`: Database schema and migrations
- `public/`: Static assets
- `docs/`: Project documentation
## Available Scripts
- `npm run dev`: Start the development server
- `npm run build`: Build the application for production
- `npm run start`: Run the production build
- `npm run lint`: Run ESLint
- `npm run format`: Format code with Prettier
- `npm run prisma:studio`: Open Prisma Studio to view database
- `pnpm run dev`: Start the development server
- `pnpm run build`: Build the application for production
- `pnpm run start`: Run the production build
- `pnpm run lint`: Run ESLint
- `pnpm run format`: Format code with Prettier
- `pnpm run prisma:studio`: Open Prisma Studio to view database
## Contributing
@ -107,9 +107,9 @@ This project is not licensed for commercial use without explicit permission. Fre
## Acknowledgments
- [Next.js](https://nextjs.org/)
- [Prisma](https://prisma.io/)
- [TailwindCSS](https://tailwindcss.com/)
- [Chart.js](https://www.chartjs.org/)
- [D3.js](https://d3js.org/)
- [React Leaflet](https://react-leaflet.js.org/)
- [Next.js](https://nextjs.org/)
- [Prisma](https://prisma.io/)
- [TailwindCSS](https://tailwindcss.com/)
- [Chart.js](https://www.chartjs.org/)
- [D3.js](https://d3js.org/)
- [React Leaflet](https://react-leaflet.js.org/)

15
app/api/admin/refresh-sessions/route.ts
View File

@ -37,12 +37,11 @@ export async function POST(request: NextRequest) {
);
}
const company = await prisma.company.findUnique({ where: { id: companyId } });
const company = await prisma.company.findUnique({
where: { id: companyId },
});
if (!company) {
return NextResponse.json(
{ error: "Company not found" },
{ status: 404 }
);
return NextResponse.json({ error: "Company not found" }, { status: 404 });
}
const rawSessionData = await fetchAndParseCsv(
@ -114,12 +113,12 @@ export async function POST(request: NextRequest) {
}
// Immediately process the queued imports to create Session records
console.log('[Refresh API] Processing queued imports...');
console.log("[Refresh API] Processing queued imports...");
await processQueuedImports(100); // Process up to 100 imports immediately
// Count how many sessions were created
const sessionCount = await prisma.session.count({
where: { companyId: company.id }
where: { companyId: company.id },
});
return NextResponse.json({
@ -127,7 +126,7 @@ export async function POST(request: NextRequest) {
imported: importedCount,
total: rawSessionData.length,
sessions: sessionCount,
message: `Successfully imported ${importedCount} records and processed them into sessions. Total sessions: ${sessionCount}`
message: `Successfully imported ${importedCount} records and processed them into sessions. Total sessions: ${sessionCount}`,
});
} catch (e) {
const error = e instanceof Error ? e.message : "An unknown error occurred";

27
app/api/admin/trigger-processing/route.ts
View File

@ -45,18 +45,21 @@ export async function POST(request: NextRequest) {
const { batchSize, maxConcurrency } = body;
// Validate parameters
const validatedBatchSize = batchSize && batchSize > 0 ? parseInt(batchSize) : null;
const validatedMaxConcurrency = maxConcurrency && maxConcurrency > 0 ? parseInt(maxConcurrency) : 5;
const validatedBatchSize =
batchSize && batchSize > 0 ? parseInt(batchSize) : null;
const validatedMaxConcurrency =
maxConcurrency && maxConcurrency > 0 ? parseInt(maxConcurrency) : 5;
// Check how many sessions need AI processing using the new status system
const sessionsNeedingAI = await ProcessingStatusManager.getSessionsNeedingProcessing(
ProcessingStage.AI_ANALYSIS,
1000 // Get count only
);
const sessionsNeedingAI =
await ProcessingStatusManager.getSessionsNeedingProcessing(
ProcessingStage.AI_ANALYSIS,
1000 // Get count only
);
// Filter to sessions for this company
const companySessionsNeedingAI = sessionsNeedingAI.filter(
statusRecord => statusRecord.session.companyId === user.companyId
(statusRecord) => statusRecord.session.companyId === user.companyId
);
const unprocessedCount = companySessionsNeedingAI.length;
@ -77,10 +80,15 @@ export async function POST(request: NextRequest) {
// The processing will continue in the background
processUnprocessedSessions(validatedBatchSize, validatedMaxConcurrency)
.then(() => {
console.log(`[Manual Trigger] Processing completed for company ${user.companyId}`);
console.log(
`[Manual Trigger] Processing completed for company ${user.companyId}`
);
})
.catch((error) => {
console.error(`[Manual Trigger] Processing failed for company ${user.companyId}:`, error);
console.error(
`[Manual Trigger] Processing failed for company ${user.companyId}:`,
error
);
});
return NextResponse.json({
@ -91,7 +99,6 @@ export async function POST(request: NextRequest) {
maxConcurrency: validatedMaxConcurrency,
startedAt: new Date().toISOString(),
});
} catch (error) {
console.error("[Manual Trigger] Error:", error);
return NextResponse.json(

10
app/api/dashboard/metrics/route.ts
View File

@ -42,7 +42,7 @@ export async function GET(request: NextRequest) {
if (startDate && endDate) {
whereClause.startTime = {
gte: new Date(startDate),
lte: new Date(endDate + 'T23:59:59.999Z'), // Include full end date
lte: new Date(endDate + "T23:59:59.999Z"), // Include full end date
};
}
@ -94,10 +94,12 @@ export async function GET(request: NextRequest) {
// Calculate date range from sessions
let dateRange: { minDate: string; maxDate: string } | null = null;
if (prismaSessions.length > 0) {
const dates = prismaSessions.map(s => new Date(s.startTime)).sort((a, b) => a.getTime() - b.getTime());
const dates = prismaSessions
.map((s) => new Date(s.startTime))
.sort((a, b) => a.getTime() - b.getTime());
dateRange = {
minDate: dates[0].toISOString().split('T')[0], // First session date
maxDate: dates[dates.length - 1].toISOString().split('T')[0] // Last session date
minDate: dates[0].toISOString().split("T")[0], // First session date
maxDate: dates[dates.length - 1].toISOString().split("T")[0], // Last session date
};
}

6
app/api/dashboard/session-filter-options/route.ts
View File

@ -53,9 +53,9 @@ export async function GET(request: NextRequest) {
.map((s) => s.language)
.filter(Boolean) as string[]; // Filter out any nulls and assert as string[]
return NextResponse.json({
categories: distinctCategories,
languages: distinctLanguages
return NextResponse.json({
categories: distinctCategories,
languages: distinctLanguages,
});
} catch (error) {
const errorMessage =

5
app/api/dashboard/session/[id]/route.ts
View File

@ -26,10 +26,7 @@ export async function GET(
});
if (!prismaSession) {
return NextResponse.json(
{ error: "Session not found" },
{ status: 404 }
);
return NextResponse.json({ error: "Session not found" }, { status: 404 });
}
// Map Prisma session object to ChatSession type

6
app/api/dashboard/sessions/route.ts
View File

@ -18,7 +18,7 @@ export async function GET(request: NextRequest) {
const companyId = authSession.user.companyId;
const { searchParams } = new URL(request.url);
const searchTerm = searchParams.get("searchTerm");
const category = searchParams.get("category");
const language = searchParams.get("language");
@ -87,9 +87,7 @@ export async function GET(request: NextRequest) {
| Prisma.SessionOrderByWithRelationInput[];
const primarySortField =
sortKey && validSortKeys[sortKey]
? validSortKeys[sortKey]
: "startTime"; // Default to startTime field if sortKey is invalid/missing
sortKey && validSortKeys[sortKey] ? validSortKeys[sortKey] : "startTime"; // Default to startTime field if sortKey is invalid/missing
const primarySortOrder =
sortOrder === "asc" || sortOrder === "desc" ? sortOrder : "desc"; // Default to desc order

2
app/api/dashboard/users/route.ts
View File

@ -65,7 +65,7 @@ export async function POST(request: NextRequest) {
}
const tempPassword = crypto.randomBytes(12).toString("base64").slice(0, 12); // secure random initial password
await prisma.user.create({
data: {
email,

100
app/api/forgot-password/route.ts
View File

@ -1,28 +1,92 @@
import { NextRequest, NextResponse } from "next/server";
import { prisma } from "../../../lib/prisma";
import { sendEmail } from "../../../lib/sendEmail";
import { forgotPasswordSchema, validateInput } from "../../../lib/validation";
import crypto from "crypto";
export async function POST(request: NextRequest) {
const body = await request.json();
const { email } = body as { email: string };
// In-memory rate limiting for password reset requests
const resetAttempts = new Map<string, { count: number; resetTime: number }>();
const user = await prisma.user.findUnique({ where: { email } });
if (!user) {
// Always return 200 for privacy (don't reveal if email exists)
return NextResponse.json({ success: true }, { status: 200 });
function checkRateLimit(ip: string): boolean {
const now = Date.now();
const attempts = resetAttempts.get(ip);
if (!attempts || now > attempts.resetTime) {
resetAttempts.set(ip, { count: 1, resetTime: now + 15 * 60 * 1000 }); // 15 minute window
return true;
}
const token = crypto.randomBytes(32).toString("hex");
const expiry = new Date(Date.now() + 1000 * 60 * 30); // 30 min expiry
await prisma.user.update({
where: { email },
data: { resetToken: token, resetTokenExpiry: expiry },
});
if (attempts.count >= 5) {
// Max 5 reset requests per 15 minutes per IP
return false;
}
const resetUrl = `${process.env.NEXTAUTH_URL || "http://localhost:3000"}/reset-password?token=${token}`;
await sendEmail(email, "Password Reset", `Reset your password: ${resetUrl}`);
return NextResponse.json({ success: true }, { status: 200 });
attempts.count++;
return true;
}
export async function POST(request: NextRequest) {
try {
// Rate limiting check
const ip =
request.ip || request.headers.get("x-forwarded-for") || "unknown";
if (!checkRateLimit(ip)) {
return NextResponse.json(
{
success: false,
error: "Too many password reset attempts. Please try again later.",
},
{ status: 429 }
);
}
const body = await request.json();
// Validate input
const validation = validateInput(forgotPasswordSchema, body);
if (!validation.success) {
return NextResponse.json(
{
success: false,
error: "Invalid email format",
},
{ status: 400 }
);
}
const { email } = validation.data;
const user = await prisma.user.findUnique({ where: { email } });
// Always return success for privacy (don't reveal if email exists)
// But only send email if user exists
if (user) {
const token = crypto.randomBytes(32).toString("hex");
const tokenHash = crypto.createHash("sha256").update(token).digest("hex");
const expiry = new Date(Date.now() + 1000 * 60 * 30); // 30 min expiry
await prisma.user.update({
where: { email },
data: { resetToken: tokenHash, resetTokenExpiry: expiry },
});
const resetUrl = `${process.env.NEXTAUTH_URL || "http://localhost:3000"}/reset-password?token=${token}`;
await sendEmail(
email,
"Password Reset",
`Reset your password: ${resetUrl}`
);
}
return NextResponse.json({ success: true }, { status: 200 });
} catch (error) {
console.error("Forgot password error:", error);
return NextResponse.json(
{
success: false,
error: "Internal server error",
},
{ status: 500 }
);
}
}

169
app/api/register/route.ts
View File

@ -1,63 +1,136 @@
import { NextRequest, NextResponse } from "next/server";
import { prisma } from "../../../lib/prisma";
import { registerSchema, validateInput } from "../../../lib/validation";
import bcrypt from "bcryptjs";
interface RegisterRequestBody {
email: string;
password: string;
company: string;
csvUrl?: string;
// In-memory rate limiting (for production, use Redis or similar)
const registrationAttempts = new Map<
string,
{ count: number; resetTime: number }
>();
function checkRateLimit(ip: string): boolean {
const now = Date.now();
const attempts = registrationAttempts.get(ip);
if (!attempts || now > attempts.resetTime) {
registrationAttempts.set(ip, { count: 1, resetTime: now + 60 * 60 * 1000 }); // 1 hour window
return true;
}
if (attempts.count >= 3) {
// Max 3 registrations per hour per IP
return false;
}
attempts.count++;
return true;
}
export async function POST(request: NextRequest) {
const body = await request.json();
const { email, password, company, csvUrl } = body as RegisterRequestBody;
try {
// Rate limiting check
const ip =
request.ip || request.headers.get("x-forwarded-for") || "unknown";
if (!checkRateLimit(ip)) {
return NextResponse.json(
{
success: false,
error: "Too many registration attempts. Please try again later.",
},
{ status: 429 }
);
}
if (!email || !password || !company) {
const body = await request.json();
// Validate input with Zod schema
const validation = validateInput(registerSchema, body);
if (!validation.success) {
return NextResponse.json(
{
success: false,
error: "Validation failed",
details: validation.errors,
},
{ status: 400 }
);
}
const { email, password, company } = validation.data;
// Check if email exists
const existingUser = await prisma.user.findUnique({
where: { email },
});
if (existingUser) {
return NextResponse.json(
{
success: false,
error: "Email already exists",
},
{ status: 409 }
);
}
// Check if company name already exists
const existingCompany = await prisma.company.findFirst({
where: { name: company },
});
if (existingCompany) {
return NextResponse.json(
{
success: false,
error: "Company name already exists",
},
{ status: 409 }
);
}
// Create company and user in a transaction
const result = await prisma.$transaction(async (tx) => {
const newCompany = await tx.company.create({
data: {
name: company,
csvUrl: "", // Empty by default, can be set later in settings
},
});
const hashedPassword = await bcrypt.hash(password, 12); // Increased rounds for better security
const newUser = await tx.user.create({
data: {
email,
password: hashedPassword,
companyId: newCompany.id,
role: "USER", // Changed from ADMIN - users should be promoted by existing admins
},
});
return { company: newCompany, user: newUser };
});
return NextResponse.json(
{
success: true,
data: {
message: "Registration successful",
userId: result.user.id,
companyId: result.company.id,
},
},
{ status: 201 }
);
} catch (error) {
console.error("Registration error:", error);
return NextResponse.json(
{
success: false,
error: "Missing required fields",
error: "Internal server error",
},
{ status: 400 }
{ status: 500 }
);
}
// Check if email exists
const exists = await prisma.user.findUnique({
where: { email },
});
if (exists) {
return NextResponse.json(
{
success: false,
error: "Email already exists",
},
{ status: 409 }
);
}
const newCompany = await prisma.company.create({
data: { name: company, csvUrl: csvUrl || "" },
});
const hashed = await bcrypt.hash(password, 10);
await prisma.user.create({
data: {
email,
password: hashed,
companyId: newCompany.id,
role: "ADMIN",
},
});
return NextResponse.json(
{
success: true,
data: { success: true },
},
{ status: 201 }
);
}

57
app/api/reset-password/route.ts
View File

@ -1,29 +1,34 @@
import { NextRequest, NextResponse } from "next/server";
import { prisma } from "../../../lib/prisma";
import { resetPasswordSchema, validateInput } from "../../../lib/validation";
import bcrypt from "bcryptjs";
import crypto from "crypto";
export async function POST(request: NextRequest) {
const body = await request.json();
const { token, password } = body as { token?: string; password?: string };
if (!token || !password) {
return NextResponse.json(
{ error: "Token and password are required." },
{ status: 400 }
);
}
if (password.length < 8) {
return NextResponse.json(
{ error: "Password must be at least 8 characters long." },
{ status: 400 }
);
}
try {
const body = await request.json();
// Validate input with strong password requirements
const validation = validateInput(resetPasswordSchema, body);
if (!validation.success) {
return NextResponse.json(
{
success: false,
error: "Validation failed",
details: validation.errors,
},
{ status: 400 }
);
}
const { token, password } = validation.data;
// Hash the token to compare with stored hash
const tokenHash = crypto.createHash("sha256").update(token).digest("hex");
const user = await prisma.user.findFirst({
where: {
resetToken: token,
resetToken: tokenHash,
resetTokenExpiry: { gte: new Date() },
},
});
@ -31,30 +36,38 @@ export async function POST(request: NextRequest) {
if (!user) {
return NextResponse.json(
{
error: "Invalid or expired token. Please request a new password reset.",
success: false,
error:
"Invalid or expired token. Please request a new password reset.",
},
{ status: 400 }
);
}
const hash = await bcrypt.hash(password, 10);
// Hash password with higher rounds for better security
const hashedPassword = await bcrypt.hash(password, 12);
await prisma.user.update({
where: { id: user.id },
data: {
password: hash,
password: hashedPassword,
resetToken: null,
resetTokenExpiry: null,
},
});
return NextResponse.json(
{ message: "Password has been reset successfully." },
{
success: true,
message: "Password has been reset successfully.",
},
{ status: 200 }
);
} catch (error) {
console.error("Reset password error:", error);
return NextResponse.json(
{
success: false,
error: "An internal server error occurred. Please try again later.",
},
{ status: 500 }

106
app/dashboard/overview/page.tsx
View File

@ -48,7 +48,10 @@ function DashboardContent() {
const [company, setCompany] = useState<Company | null>(null);
const [loading, setLoading] = useState<boolean>(false);
const [refreshing, setRefreshing] = useState<boolean>(false);
const [dateRange, setDateRange] = useState<{ minDate: string; maxDate: string } | null>(null);
const [dateRange, setDateRange] = useState<{
minDate: string;
maxDate: string;
} | null>(null);
const [selectedStartDate, setSelectedStartDate] = useState<string>("");
const [selectedEndDate, setSelectedEndDate] = useState<string>("");
const [isInitialLoad, setIsInitialLoad] = useState<boolean>(true);
@ -56,7 +59,11 @@ function DashboardContent() {
const isAuditor = session?.user?.role === "AUDITOR";
// Function to fetch metrics with optional date range
const fetchMetrics = async (startDate?: string, endDate?: string, isInitial = false) => {
const fetchMetrics = async (
startDate?: string,
endDate?: string,
isInitial = false
) => {
setLoading(true);
try {
let url = "/api/dashboard/metrics";
@ -85,11 +92,14 @@ function DashboardContent() {
};
// Handle date range changes
const handleDateRangeChange = useCallback((startDate: string, endDate: string) => {
setSelectedStartDate(startDate);
setSelectedEndDate(endDate);
fetchMetrics(startDate, endDate);
}, []);
const handleDateRangeChange = useCallback(
(startDate: string, endDate: string) => {
setSelectedStartDate(startDate);
setSelectedEndDate(endDate);
fetchMetrics(startDate, endDate);
},
[]
);
useEffect(() => {
// Redirect if not authenticated
@ -216,33 +226,48 @@ function DashboardContent() {
};
return [
{ name: "Positive", value: sentimentData.positive, color: "hsl(var(--chart-1))" },
{ name: "Neutral", value: sentimentData.neutral, color: "hsl(var(--chart-2))" },
{ name: "Negative", value: sentimentData.negative, color: "hsl(var(--chart-3))" },
{
name: "Positive",
value: sentimentData.positive,
color: "hsl(var(--chart-1))",
},
{
name: "Neutral",
value: sentimentData.neutral,
color: "hsl(var(--chart-2))",
},
{
name: "Negative",
value: sentimentData.negative,
color: "hsl(var(--chart-3))",
},
];
};
const getSessionsOverTimeData = () => {
if (!metrics?.days) return [];
return Object.entries(metrics.days).map(([date, value]) => ({
date: new Date(date).toLocaleDateString('en-US', { month: 'short', day: 'numeric' }),
date: new Date(date).toLocaleDateString("en-US", {
month: "short",
day: "numeric",
}),
value: value as number,
}));
};
const getCategoriesData = () => {
if (!metrics?.categories) return [];
return Object.entries(metrics.categories).map(([name, value]) => ({
name: name.length > 15 ? name.substring(0, 15) + '...' : name,
name: name.length > 15 ? name.substring(0, 15) + "..." : name,
value: value as number,
}));
};
const getLanguagesData = () => {
if (!metrics?.languages) return [];
return Object.entries(metrics.languages).map(([name, value]) => ({
name,
value: value as number,
@ -287,7 +312,9 @@ function DashboardContent() {
<div className="flex flex-col sm:flex-row justify-between items-start sm:items-center gap-4">
<div className="space-y-2">
<div className="flex items-center gap-3">
<h1 className="text-3xl font-bold tracking-tight">{company.name}</h1>
<h1 className="text-3xl font-bold tracking-tight">
{company.name}
</h1>
<Badge variant="secondary" className="text-xs">
Analytics Dashboard
</Badge>
@ -299,7 +326,7 @@ function DashboardContent() {
</span>
</p>
</div>
<div className="flex items-center gap-2">
<Button
onClick={handleRefresh}
@ -307,10 +334,12 @@ function DashboardContent() {
size="sm"
className="gap-2"
>
<RefreshCw className={`h-4 w-4 ${refreshing ? 'animate-spin' : ''}`} />
<RefreshCw
className={`h-4 w-4 ${refreshing ? "animate-spin" : ""}`}
/>
{refreshing ? "Refreshing..." : "Refresh"}
</Button>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="outline" size="sm">
@ -318,7 +347,9 @@ function DashboardContent() {
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuItem onClick={() => signOut({ callbackUrl: "/login" })}>
<DropdownMenuItem
onClick={() => signOut({ callbackUrl: "/login" })}
>
<LogOut className="h-4 w-4 mr-2" />
Sign out
</DropdownMenuItem>
@ -352,7 +383,7 @@ function DashboardContent() {
}}
variant="primary"
/>
<MetricCard
title="Unique Users"
value={metrics.uniqueUsers?.toLocaleString()}
@ -363,7 +394,7 @@ function DashboardContent() {
}}
variant="success"
/>
<MetricCard
title="Avg. Session Time"
value={`${Math.round(metrics.avgSessionLength || 0)}s`}
@ -373,7 +404,7 @@ function DashboardContent() {
isPositive: (metrics.avgSessionTimeTrend ?? 0) >= 0,
}}
/>
<MetricCard
title="Avg. Response Time"
value={`${metrics.avgResponseTime?.toFixed(1) || 0}s`}
@ -384,32 +415,37 @@ function DashboardContent() {
}}
variant="warning"
/>
<MetricCard
title="Daily Costs"
value={`€${metrics.avgDailyCosts?.toFixed(4) || '0.0000'}`}
value={`€${metrics.avgDailyCosts?.toFixed(4) || "0.0000"}`}
icon={<Euro className="h-5 w-5" />}
description="Average per day"
/>
<MetricCard
title="Peak Usage"
value={metrics.peakUsageTime || 'N/A'}
value={metrics.peakUsageTime || "N/A"}
icon={<TrendingUp className="h-5 w-5" />}
description="Busiest hour"
/>
<MetricCard
title="Resolution Rate"
value={`${metrics.resolvedChatsPercentage?.toFixed(1) || '0.0'}%`}
value={`${metrics.resolvedChatsPercentage?.toFixed(1) || "0.0"}%`}
icon={<CheckCircle className="h-5 w-5" />}
trend={{
value: metrics.resolvedChatsPercentage ?? 0,
isPositive: (metrics.resolvedChatsPercentage ?? 0) >= 80,
}}
variant={metrics.resolvedChatsPercentage && metrics.resolvedChatsPercentage >= 80 ? "success" : "warning"}
variant={
metrics.resolvedChatsPercentage &&
metrics.resolvedChatsPercentage >= 80
? "success"
: "warning"
}
/>
<MetricCard
title="Active Languages"
value={Object.keys(metrics.languages || {}).length}
@ -426,7 +462,7 @@ function DashboardContent() {
className="lg:col-span-2"
height={350}
/>
<ModernDonutChart
data={getSentimentData()}
title="Conversation Sentiment"
@ -444,7 +480,7 @@ function DashboardContent() {
title="Sessions by Category"
height={350}
/>
<ModernDonutChart
data={getLanguagesData()}
title="Languages Used"
@ -508,8 +544,8 @@ function DashboardContent() {
{metrics.totalTokens?.toLocaleString() || 0}
</Badge>
<Badge variant="outline" className="gap-1">
<span className="font-semibold">Total Cost:</span>
€{metrics.totalTokensEur?.toFixed(4) || 0}
<span className="font-semibold">Total Cost:</span>€
{metrics.totalTokensEur?.toFixed(4) || 0}
</Badge>
</div>
</div>

31
app/dashboard/page.tsx
View File

@ -46,7 +46,8 @@ const DashboardPage: FC = () => {
const navigationCards = [
{
title: "Analytics Overview",
description: "View comprehensive metrics, charts, and insights from your chat sessions",
description:
"View comprehensive metrics, charts, and insights from your chat sessions",
icon: <BarChart3 className="h-6 w-6" />,
href: "/dashboard/overview",
variant: "primary" as const,
@ -54,7 +55,8 @@ const DashboardPage: FC = () => {
},
{
title: "Session Browser",
description: "Browse, search, and analyze individual conversation sessions",
description:
"Browse, search, and analyze individual conversation sessions",
icon: <MessageSquare className="h-6 w-6" />,
href: "/dashboard/sessions",
variant: "success" as const,
@ -64,16 +66,22 @@ const DashboardPage: FC = () => {
? [
{
title: "Company Settings",
description: "Configure company settings, integrations, and API connections",
description:
"Configure company settings, integrations, and API connections",
icon: <Settings className="h-6 w-6" />,
href: "/dashboard/company",
variant: "warning" as const,
features: ["API configuration", "Integration settings", "Data management"],
features: [
"API configuration",
"Integration settings",
"Data management",
],
adminOnly: true,
},
{
title: "User Management",
description: "Invite team members and manage user accounts and permissions",
description:
"Invite team members and manage user accounts and permissions",
icon: <Users className="h-6 w-6" />,
href: "/dashboard/users",
variant: "default" as const,
@ -129,7 +137,7 @@ const DashboardPage: FC = () => {
Choose a section below to explore your analytics dashboard
</p>
</div>
<div className="flex items-center gap-2">
<div className="flex items-center gap-1 text-sm text-muted-foreground">
<Shield className="h-4 w-4" />
@ -152,7 +160,7 @@ const DashboardPage: FC = () => {
>
{/* Subtle gradient overlay */}
<div className="absolute inset-0 bg-linear-to-br from-white/50 to-transparent dark:from-white/5 pointer-events-none" />
<CardHeader className="relative">
<div className="flex items-start justify-between">
<div className="space-y-3">
@ -186,7 +194,10 @@ const DashboardPage: FC = () => {
{/* Features List */}
<div className="space-y-2">
{card.features.map((feature, featureIndex) => (
<div key={featureIndex} className="flex items-center gap-2 text-sm">
<div
key={featureIndex}
className="flex items-center gap-2 text-sm"
>
<div className="h-1.5 w-1.5 rounded-full bg-current opacity-60" />
<span className="text-muted-foreground">{feature}</span>
</div>
@ -232,7 +243,7 @@ const DashboardPage: FC = () => {
</div>
<p className="text-sm text-muted-foreground">Data updates</p>
</div>
<div className="text-center space-y-2">
<div className="flex items-center justify-center gap-2">
<Shield className="h-5 w-5 text-green-600" />
@ -240,7 +251,7 @@ const DashboardPage: FC = () => {
</div>
<p className="text-sm text-muted-foreground">Data protection</p>
</div>
<div className="text-center space-y-2">
<div className="flex items-center justify-center gap-2">
<BarChart3 className="h-5 w-5 text-blue-600" />

71
check-refactored-pipeline-status.ts
View File

@ -1,75 +1,90 @@
import { PrismaClient } from '@prisma/client';
import { ProcessingStatusManager } from './lib/processingStatusManager';
import { PrismaClient } from "@prisma/client";
import { ProcessingStatusManager } from "./lib/processingStatusManager";
const prisma = new PrismaClient();
async function checkRefactoredPipelineStatus() {
try {
console.log('=== REFACTORED PIPELINE STATUS ===\n');
console.log("=== REFACTORED PIPELINE STATUS ===\n");
// Get pipeline status using the new system
const pipelineStatus = await ProcessingStatusManager.getPipelineStatus();
console.log(`Total Sessions: ${pipelineStatus.totalSessions}\n`);
// Display status for each stage
const stages = ['CSV_IMPORT', 'TRANSCRIPT_FETCH', 'SESSION_CREATION', 'AI_ANALYSIS', 'QUESTION_EXTRACTION'];
const stages = [
"CSV_IMPORT",
"TRANSCRIPT_FETCH",
"SESSION_CREATION",
"AI_ANALYSIS",
"QUESTION_EXTRACTION",
];
for (const stage of stages) {
console.log(`${stage}:`);
const stageData = pipelineStatus.pipeline[stage] || {};
const pending = stageData.PENDING || 0;
const inProgress = stageData.IN_PROGRESS || 0;
const completed = stageData.COMPLETED || 0;
const failed = stageData.FAILED || 0;
const skipped = stageData.SKIPPED || 0;
console.log(` PENDING: ${pending}`);
console.log(` IN_PROGRESS: ${inProgress}`);
console.log(` COMPLETED: ${completed}`);
console.log(` FAILED: ${failed}`);
console.log(` SKIPPED: ${skipped}`);
console.log('');
console.log("");
}
// Show what needs processing
console.log('=== WHAT NEEDS PROCESSING ===');
console.log("=== WHAT NEEDS PROCESSING ===");
for (const stage of stages) {
const stageData = pipelineStatus.pipeline[stage] || {};
const pending = stageData.PENDING || 0;
const failed = stageData.FAILED || 0;
if (pending > 0 || failed > 0) {
console.log(`• ${stage}: ${pending} pending, ${failed} failed`);
}
}
// Show failed sessions if any
const failedSessions = await ProcessingStatusManager.getFailedSessions();
if (failedSessions.length > 0) {
console.log('\n=== FAILED SESSIONS ===');
failedSessions.slice(0, 5).forEach(failure => {
console.log(` ${failure.session.import?.externalSessionId || failure.sessionId}: ${failure.stage} - ${failure.errorMessage}`);
console.log("\n=== FAILED SESSIONS ===");
failedSessions.slice(0, 5).forEach((failure) => {
console.log(
` ${failure.session.import?.externalSessionId || failure.sessionId}: ${failure.stage} - ${failure.errorMessage}`
);
});
if (failedSessions.length > 5) {
console.log(` ... and ${failedSessions.length - 5} more failed sessions`);
console.log(
` ... and ${failedSessions.length - 5} more failed sessions`
);
}
}
// Show sessions ready for AI processing
const readyForAI = await ProcessingStatusManager.getSessionsNeedingProcessing('AI_ANALYSIS', 5);
const readyForAI =
await ProcessingStatusManager.getSessionsNeedingProcessing(
"AI_ANALYSIS",
5
);
if (readyForAI.length > 0) {
console.log('\n=== SESSIONS READY FOR AI PROCESSING ===');
readyForAI.forEach(status => {
console.log(` ${status.session.import?.externalSessionId || status.sessionId} (created: ${status.session.createdAt})`);
console.log("\n=== SESSIONS READY FOR AI PROCESSING ===");
readyForAI.forEach((status) => {
console.log(
` ${status.session.import?.externalSessionId || status.sessionId} (created: ${status.session.createdAt})`
);
});
}
} catch (error) {
console.error('Error checking pipeline status:', error);
console.error("Error checking pipeline status:", error);
} finally {
await prisma.$disconnect();
}

10
components/DateRangePicker.tsx
View File

@ -63,10 +63,11 @@ export default function DateRangePicker({
const setLast30Days = () => {
const thirtyDaysAgo = new Date();
thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30);
const thirtyDaysAgoStr = thirtyDaysAgo.toISOString().split('T')[0];
const thirtyDaysAgoStr = thirtyDaysAgo.toISOString().split("T")[0];
// Use the later of 30 days ago or minDate
const newStartDate = thirtyDaysAgoStr > minDate ? thirtyDaysAgoStr : minDate;
const newStartDate =
thirtyDaysAgoStr > minDate ? thirtyDaysAgoStr : minDate;
setStartDate(newStartDate);
setEndDate(maxDate);
};
@ -74,7 +75,7 @@ export default function DateRangePicker({
const setLast7Days = () => {
const sevenDaysAgo = new Date();
sevenDaysAgo.setDate(sevenDaysAgo.getDate() - 7);
const sevenDaysAgoStr = sevenDaysAgo.toISOString().split('T')[0];
const sevenDaysAgoStr = sevenDaysAgo.toISOString().split("T")[0];
// Use the later of 7 days ago or minDate
const newStartDate = sevenDaysAgoStr > minDate ? sevenDaysAgoStr : minDate;
@ -146,7 +147,8 @@ export default function DateRangePicker({
</div>
<div className="mt-2 text-xs text-gray-500">
Available data: {new Date(minDate).toLocaleDateString()} - {new Date(maxDate).toLocaleDateString()}
Available data: {new Date(minDate).toLocaleDateString()} -{" "}
{new Date(maxDate).toLocaleDateString()}
</div>
</div>
);

2
components/GeographicMap.tsx
View File

@ -48,7 +48,7 @@ const getCountryCoordinates = (): Record<string, [number, number]> => {
BG: [42.7339, 25.4858],
HR: [45.1, 15.2],
SK: [48.669, 19.699],
SI: [46.1512, 14.9955]
SI: [46.1512, 14.9955],
};
// This function now primarily returns fallbacks.
// The actual fetching using @rapideditor/country-coder will be in the component's useEffect.

13
components/MessageViewer.tsx
View File

@ -49,7 +49,9 @@ export default function MessageViewer({ messages }: MessageViewerProps) {
{message.role}
</span>
<span className="text-xs opacity-75 ml-2">
{message.timestamp ? new Date(message.timestamp).toLocaleTimeString() : 'No timestamp'}
{message.timestamp
? new Date(message.timestamp).toLocaleTimeString()
: "No timestamp"}
</span>
</div>
<div className="text-sm whitespace-pre-wrap">
@ -63,13 +65,18 @@ export default function MessageViewer({ messages }: MessageViewerProps) {
<div className="mt-4 pt-3 border-t text-sm text-gray-500">
<div className="flex justify-between">
<span>
First message: {messages[0].timestamp ? new Date(messages[0].timestamp).toLocaleString() : 'No timestamp'}
First message:{" "}
{messages[0].timestamp
? new Date(messages[0].timestamp).toLocaleString()
: "No timestamp"}
</span>
<span>
Last message:{" "}
{(() => {
const lastMessage = messages[messages.length - 1];
return lastMessage.timestamp ? new Date(lastMessage.timestamp).toLocaleString() : 'No timestamp';
return lastMessage.timestamp
? new Date(lastMessage.timestamp).toLocaleString()
: "No timestamp";
})()}
</span>
</div>

88
components/MetricCard.tsx
View File

@ -1,88 +0,0 @@
"use client";
interface MetricCardProps {
title: string;
value: string | number | null | undefined;
description?: string;
icon?: React.ReactNode;
trend?: {
value: number;
label?: string;
isPositive?: boolean;
};
variant?: "default" | "primary" | "success" | "warning" | "danger";
}
export default function MetricCard({
title,
value,
description,
icon,
trend,
variant = "default",
}: MetricCardProps) {
// Determine background and text colors based on variant
const getVariantClasses = () => {
switch (variant) {
case "primary":
return "bg-blue-50 border-blue-200";
case "success":
return "bg-green-50 border-green-200";
case "warning":
return "bg-amber-50 border-amber-200";
case "danger":
return "bg-red-50 border-red-200";
default:
return "bg-white border-gray-200";
}
};
const getIconClasses = () => {
switch (variant) {
case "primary":
return "bg-blue-100 text-blue-600";
case "success":
return "bg-green-100 text-green-600";
case "warning":
return "bg-amber-100 text-amber-600";
case "danger":
return "bg-red-100 text-red-600";
default:
return "bg-gray-100 text-gray-600";
}
};
return (
<div className={`rounded-xl border shadow-sm p-6 ${getVariantClasses()}`}>
<div className="flex items-start justify-between">
<div>
<p className="text-sm font-medium text-gray-500">{title}</p>
<div className="mt-2 flex items-baseline">
<p className="text-2xl font-semibold">{value ?? "-"}</p>
{trend && (
<span
className={`ml-2 text-sm font-medium ${
trend.isPositive !== false ? "text-green-600" : "text-red-600"
}`}
>
{trend.isPositive !== false ? "↑" : "↓"}{" "}
{Math.abs(trend.value).toFixed(1)}%
</span>
)}
</div>
{description && (
<p className="mt-1 text-xs text-gray-500">{description}</p>
)}
</div>
{icon && (
<div
className={`flex h-12 w-12 rounded-full ${getIconClasses()} items-center justify-center`}
>
<span className="text-xl">{icon}</span>
</div>
)}
</div>
</div>
);
}

67
components/ResponseTimeDistribution.tsx
View File

@ -68,8 +68,10 @@ export default function ResponseTimeDistribution({
// Determine color based on response time
let color;
if (i <= 2) color = "hsl(var(--chart-1))"; // Green for fast
else if (i <= 5) color = "hsl(var(--chart-4))"; // Yellow for medium
if (i <= 2)
color = "hsl(var(--chart-1))"; // Green for fast
else if (i <= 5)
color = "hsl(var(--chart-4))"; // Yellow for medium
else color = "hsl(var(--chart-3))"; // Red for slow
return {
@ -82,10 +84,13 @@ export default function ResponseTimeDistribution({
return (
<div className="h-64">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={chartData} margin={{ top: 20, right: 30, left: 20, bottom: 5 }}>
<CartesianGrid
strokeDasharray="3 3"
stroke="hsl(var(--border))"
<BarChart
data={chartData}
margin={{ top: 20, right: 30, left: 20, bottom: 5 }}
>
<CartesianGrid
strokeDasharray="3 3"
stroke="hsl(var(--border))"
strokeOpacity={0.3}
/>
<XAxis
@ -100,57 +105,53 @@ export default function ResponseTimeDistribution({
fontSize={12}
tickLine={false}
axisLine={false}
label={{
value: 'Number of Responses',
angle: -90,
position: 'insideLeft',
style: { textAnchor: 'middle' }
label={{
value: "Number of Responses",
angle: -90,
position: "insideLeft",
style: { textAnchor: "middle" },
}}
/>
<Tooltip content={<CustomTooltip />} />
<Bar
dataKey="value"
radius={[4, 4, 0, 0]}
fill="hsl(var(--chart-1))"
>
<Bar dataKey="value" radius={[4, 4, 0, 0]} fill="hsl(var(--chart-1))">
{chartData.map((entry, index) => (
<Bar key={`cell-${index}`} fill={entry.color} />
))}
</Bar>
{/* Average line */}
<ReferenceLine
x={Math.floor(average)}
stroke="hsl(var(--primary))"
<ReferenceLine
x={Math.floor(average)}
stroke="hsl(var(--primary))"
strokeWidth={2}
strokeDasharray="5 5"
label={{
value: `Avg: ${average.toFixed(1)}s`,
label={{
value: `Avg: ${average.toFixed(1)}s`,
position: "top" as const,
style: {
style: {
fill: "hsl(var(--primary))",
fontSize: "12px",
fontWeight: "500"
}
fontWeight: "500",
},
}}
/>
{/* Target line (if provided) */}
{targetResponseTime && (
<ReferenceLine
x={Math.floor(targetResponseTime)}
stroke="hsl(var(--chart-2))"
<ReferenceLine
x={Math.floor(targetResponseTime)}
stroke="hsl(var(--chart-2))"
strokeWidth={2}
strokeDasharray="3 3"
label={{
value: `Target: ${targetResponseTime}s`,
label={{
value: `Target: ${targetResponseTime}s`,
position: "top" as const,
style: {
style: {
fill: "hsl(var(--chart-2))",
fontSize: "12px",
fontWeight: "500"
}
fontWeight: "500",
},
}}
/>
)}

3
components/SessionDetails.tsx
View File

@ -90,7 +90,6 @@ export default function SessionDetails({ session }: SessionDetailsProps) {
<span className="font-medium">{session.messagesSent || 0}</span>
</div>
{session.avgResponseTime !== null &&
session.avgResponseTime !== undefined && (
<div className="flex justify-between border-b pb-2">
@ -132,7 +131,6 @@ export default function SessionDetails({ session }: SessionDetailsProps) {
</div>
)}
{session.initialMsg && (
<div className="border-b pb-2">
<span className="text-gray-600 block mb-1">Initial Message:</span>
@ -151,7 +149,6 @@ export default function SessionDetails({ session }: SessionDetailsProps) {
</div>
)}
{session.fullTranscriptUrl && (
<div className="flex justify-between pt-2">
<span className="text-gray-600">Transcript:</span>

16
components/TopQuestionsChart.tsx
View File

@ -1,14 +1,17 @@
'use client';
"use client";
import React from 'react';
import { TopQuestion } from '../lib/types';
import React from "react";
import { TopQuestion } from "../lib/types";
interface TopQuestionsChartProps {
data: TopQuestion[];
title?: string;
}
export default function TopQuestionsChart({ data, title = "Top 5 Asked Questions" }: TopQuestionsChartProps) {
export default function TopQuestionsChart({
data,
title = "Top 5 Asked Questions",
}: TopQuestionsChartProps) {
if (!data || data.length === 0) {
return (
<div className="bg-white p-6 rounded-lg shadow-sm border border-gray-200">
@ -21,7 +24,7 @@ export default function TopQuestionsChart({ data, title = "Top 5 Asked Questions
}
// Find the maximum count to calculate relative bar widths
const maxCount = Math.max(...data.map(q => q.count));
const maxCount = Math.max(...data.map((q) => q.count));
return (
<div className="bg-white p-6 rounded-lg shadow-sm border border-gray-200">
@ -29,7 +32,8 @@ export default function TopQuestionsChart({ data, title = "Top 5 Asked Questions
<div className="space-y-4">
{data.map((question, index) => {
const percentage = maxCount > 0 ? (question.count / maxCount) * 100 : 0;
const percentage =
maxCount > 0 ? (question.count / maxCount) * 100 : 0;
return (
<div key={index} className="relative">

19
components/charts/bar-chart.tsx
View File

@ -61,10 +61,13 @@ export default function ModernBarChart({
)}
<CardContent>
<ResponsiveContainer width="100%" height={height}>
<BarChart data={data} margin={{ top: 5, right: 30, left: 20, bottom: 5 }}>
<CartesianGrid
strokeDasharray="3 3"
stroke="hsl(var(--border))"
<BarChart
data={data}
margin={{ top: 5, right: 30, left: 20, bottom: 5 }}
>
<CartesianGrid
strokeDasharray="3 3"
stroke="hsl(var(--border))"
strokeOpacity={0.3}
/>
<XAxis
@ -84,14 +87,14 @@ export default function ModernBarChart({
axisLine={false}
/>
<Tooltip content={<CustomTooltip />} />
<Bar
dataKey={dataKey}
<Bar
dataKey={dataKey}
radius={[4, 4, 0, 0]}
className="transition-all duration-200"
>
{data.map((entry, index) => (
<Cell
key={`cell-${index}`}
<Cell
key={`cell-${index}`}
fill={colors[index % colors.length]}
className="hover:opacity-80"
/>

15
components/charts/donut-chart.tsx
View File

@ -1,6 +1,13 @@
"use client";
import { PieChart, Pie, Cell, ResponsiveContainer, Tooltip, Legend } from "recharts";
import {
PieChart,
Pie,
Cell,
ResponsiveContainer,
Tooltip,
Legend,
} from "recharts";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
interface DonutChartProps {
@ -22,9 +29,7 @@ const CustomTooltip = ({ active, payload }: any) => {
<div className="rounded-lg border bg-background p-3 shadow-md">
<p className="text-sm font-medium">{data.name}</p>
<p className="text-sm text-muted-foreground">
<span className="font-medium text-foreground">
{data.value}
</span>{" "}
<span className="font-medium text-foreground">{data.value}</span>{" "}
sessions ({((data.value / data.payload.total) * 100).toFixed(1)}%)
</p>
</div>
@ -77,7 +82,7 @@ export default function ModernDonutChart({
className,
}: DonutChartProps) {
const total = data.reduce((sum, item) => sum + item.value, 0);
const dataWithTotal = data.map(item => ({ ...item, total }));
const dataWithTotal = data.map((item) => ({ ...item, total }));
return (
<Card className={className}>

13
components/charts/line-chart.tsx
View File

@ -60,7 +60,10 @@ export default function ModernLineChart({
)}
<CardContent>
<ResponsiveContainer width="100%" height={height}>
<ChartComponent data={data} margin={{ top: 5, right: 30, left: 20, bottom: 5 }}>
<ChartComponent
data={data}
margin={{ top: 5, right: 30, left: 20, bottom: 5 }}
>
<defs>
{gradient && (
<linearGradient id="colorGradient" x1="0" y1="0" x2="0" y2="1">
@ -69,9 +72,9 @@ export default function ModernLineChart({
</linearGradient>
)}
</defs>
<CartesianGrid
strokeDasharray="3 3"
stroke="hsl(var(--border))"
<CartesianGrid
strokeDasharray="3 3"
stroke="hsl(var(--border))"
strokeOpacity={0.3}
/>
<XAxis
@ -88,7 +91,7 @@ export default function ModernLineChart({
axisLine={false}
/>
<Tooltip content={<CustomTooltip />} />
{gradient ? (
<Area
type="monotone"

16
components/ui/badge.tsx
View File

@ -1,8 +1,8 @@
import * as React from "react"
import { Slot } from "@radix-ui/react-slot"
import { cva, type VariantProps } from "class-variance-authority"
import * as React from "react";
import { Slot } from "@radix-ui/react-slot";
import { cva, type VariantProps } from "class-variance-authority";
import { cn } from "@/lib/utils"
import { cn } from "@/lib/utils";
const badgeVariants = cva(
"inline-flex items-center justify-center rounded-md border px-2 py-0.5 text-xs font-medium w-fit whitespace-nowrap shrink-0 [&>svg]:size-3 gap-1 [&>svg]:pointer-events-none focus-visible:border-ring focus-visible:ring-ring/50 focus-visible:ring-[3px] aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 aria-invalid:border-destructive transition-[color,box-shadow] overflow-hidden",
@ -23,7 +23,7 @@ const badgeVariants = cva(
variant: "default",
},
}
)
);
function Badge({
className,
@ -32,7 +32,7 @@ function Badge({
...props
}: React.ComponentProps<"span"> &
VariantProps<typeof badgeVariants> & { asChild?: boolean }) {
const Comp = asChild ? Slot : "span"
const Comp = asChild ? Slot : "span";
return (
<Comp
@ -40,7 +40,7 @@ function Badge({
className={cn(badgeVariants({ variant }), className)}
{...props}
/>
)
);
}
export { Badge, badgeVariants }
export { Badge, badgeVariants };

18
components/ui/button.tsx
View File

@ -1,8 +1,8 @@
import * as React from "react"
import { Slot } from "@radix-ui/react-slot"
import { cva, type VariantProps } from "class-variance-authority"
import * as React from "react";
import { Slot } from "@radix-ui/react-slot";
import { cva, type VariantProps } from "class-variance-authority";
import { cn } from "@/lib/utils"
import { cn } from "@/lib/utils";
const buttonVariants = cva(
"inline-flex items-center justify-center gap-2 whitespace-nowrap rounded-md text-sm font-medium transition-all disabled:pointer-events-none disabled:opacity-50 [&_svg]:pointer-events-none [&_svg:not([class*='size-'])]:size-4 shrink-0 [&_svg]:shrink-0 outline-none focus-visible:border-ring focus-visible:ring-ring/50 focus-visible:ring-[3px] aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 aria-invalid:border-destructive",
@ -33,7 +33,7 @@ const buttonVariants = cva(
size: "default",
},
}
)
);
function Button({
className,
@ -43,9 +43,9 @@ function Button({
...props
}: React.ComponentProps<"button"> &
VariantProps<typeof buttonVariants> & {
asChild?: boolean
asChild?: boolean;
}) {
const Comp = asChild ? Slot : "button"
const Comp = asChild ? Slot : "button";
return (
<Comp
@ -53,7 +53,7 @@ function Button({
className={cn(buttonVariants({ variant, size, className }))}
{...props}
/>
)
);
}
export { Button, buttonVariants }
export { Button, buttonVariants };

20
components/ui/card.tsx
View File

@ -1,6 +1,6 @@
import * as React from "react"
import * as React from "react";
import { cn } from "@/lib/utils"
import { cn } from "@/lib/utils";
function Card({ className, ...props }: React.ComponentProps<"div">) {
return (
@ -12,7 +12,7 @@ function Card({ className, ...props }: React.ComponentProps<"div">) {
)}
{...props}
/>
)
);
}
function CardHeader({ className, ...props }: React.ComponentProps<"div">) {
@ -25,7 +25,7 @@ function CardHeader({ className, ...props }: React.ComponentProps<"div">) {
)}
{...props}
/>
)
);
}
function CardTitle({ className, ...props }: React.ComponentProps<"div">) {
@ -35,7 +35,7 @@ function CardTitle({ className, ...props }: React.ComponentProps<"div">) {
className={cn("leading-none font-semibold", className)}
{...props}
/>
)
);
}
function CardDescription({ className, ...props }: React.ComponentProps<"div">) {
@ -45,7 +45,7 @@ function CardDescription({ className, ...props }: React.ComponentProps<"div">) {
className={cn("text-muted-foreground text-sm", className)}
{...props}
/>
)
);
}
function CardAction({ className, ...props }: React.ComponentProps<"div">) {
@ -58,7 +58,7 @@ function CardAction({ className, ...props }: React.ComponentProps<"div">) {
)}
{...props}
/>
)
);
}
function CardContent({ className, ...props }: React.ComponentProps<"div">) {
@ -68,7 +68,7 @@ function CardContent({ className, ...props }: React.ComponentProps<"div">) {
className={cn("px-6", className)}
{...props}
/>
)
);
}
function CardFooter({ className, ...props }: React.ComponentProps<"div">) {
@ -78,7 +78,7 @@ function CardFooter({ className, ...props }: React.ComponentProps<"div">) {
className={cn("flex items-center px-6 [.border-t]:pt-6", className)}
{...props}
/>
)
);
}
export {
@ -89,4 +89,4 @@ export {
CardAction,
CardDescription,
CardContent,
}
};

50
components/ui/dropdown-menu.tsx
View File

@ -1,15 +1,15 @@
"use client"
"use client";
import * as React from "react"
import * as DropdownMenuPrimitive from "@radix-ui/react-dropdown-menu"
import { CheckIcon, ChevronRightIcon, CircleIcon } from "lucide-react"
import * as React from "react";
import * as DropdownMenuPrimitive from "@radix-ui/react-dropdown-menu";
import { CheckIcon, ChevronRightIcon, CircleIcon } from "lucide-react";
import { cn } from "@/lib/utils"
import { cn } from "@/lib/utils";
function DropdownMenu({
...props
}: React.ComponentProps<typeof DropdownMenuPrimitive.Root>) {
return <DropdownMenuPrimitive.Root data-slot="dropdown-menu" {...props} />
return <DropdownMenuPrimitive.Root data-slot="dropdown-menu" {...props} />;
}
function DropdownMenuPortal({
@ -17,7 +17,7 @@ function DropdownMenuPortal({
}: React.ComponentProps<typeof DropdownMenuPrimitive.Portal>) {
return (
<DropdownMenuPrimitive.Portal data-slot="dropdown-menu-portal" {...props} />
)
);
}
function DropdownMenuTrigger({
@ -28,7 +28,7 @@ function DropdownMenuTrigger({
data-slot="dropdown-menu-trigger"
{...props}
/>
)
);
}
function DropdownMenuContent({
@ -48,7 +48,7 @@ function DropdownMenuContent({
{...props}
/>
</DropdownMenuPrimitive.Portal>
)
);
}
function DropdownMenuGroup({
@ -56,7 +56,7 @@ function DropdownMenuGroup({
}: React.ComponentProps<typeof DropdownMenuPrimitive.Group>) {
return (
<DropdownMenuPrimitive.Group data-slot="dropdown-menu-group" {...props} />
)
);
}
function DropdownMenuItem({
@ -65,8 +65,8 @@ function DropdownMenuItem({
variant = "default",
...props
}: React.ComponentProps<typeof DropdownMenuPrimitive.Item> & {
inset?: boolean
variant?: "default" | "destructive"
inset?: boolean;
variant?: "default" | "destructive";
}) {
return (
<DropdownMenuPrimitive.Item
@ -79,7 +79,7 @@ function DropdownMenuItem({
)}
{...props}
/>
)
);
}
function DropdownMenuCheckboxItem({
@ -105,7 +105,7 @@ function DropdownMenuCheckboxItem({
</span>
{children}
</DropdownMenuPrimitive.CheckboxItem>
)
);
}
function DropdownMenuRadioGroup({
@ -116,7 +116,7 @@ function DropdownMenuRadioGroup({
data-slot="dropdown-menu-radio-group"
{...props}
/>
)
);
}
function DropdownMenuRadioItem({
@ -140,7 +140,7 @@ function DropdownMenuRadioItem({
</span>
{children}
</DropdownMenuPrimitive.RadioItem>
)
);
}
function DropdownMenuLabel({
@ -148,7 +148,7 @@ function DropdownMenuLabel({
inset,
...props
}: React.ComponentProps<typeof DropdownMenuPrimitive.Label> & {
inset?: boolean
inset?: boolean;
}) {
return (
<DropdownMenuPrimitive.Label
@ -160,7 +160,7 @@ function DropdownMenuLabel({
)}
{...props}
/>
)
);
}
function DropdownMenuSeparator({
@ -173,7 +173,7 @@ function DropdownMenuSeparator({
className={cn("bg-border -mx-1 my-1 h-px", className)}
{...props}
/>
)
);
}
function DropdownMenuShortcut({
@ -189,13 +189,13 @@ function DropdownMenuShortcut({
)}
{...props}
/>
)
);
}
function DropdownMenuSub({
...props
}: React.ComponentProps<typeof DropdownMenuPrimitive.Sub>) {
return <DropdownMenuPrimitive.Sub data-slot="dropdown-menu-sub" {...props} />
return <DropdownMenuPrimitive.Sub data-slot="dropdown-menu-sub" {...props} />;
}
function DropdownMenuSubTrigger({
@ -204,7 +204,7 @@ function DropdownMenuSubTrigger({
children,
...props
}: React.ComponentProps<typeof DropdownMenuPrimitive.SubTrigger> & {
inset?: boolean
inset?: boolean;
}) {
return (
<DropdownMenuPrimitive.SubTrigger
@ -219,7 +219,7 @@ function DropdownMenuSubTrigger({
{children}
<ChevronRightIcon className="ml-auto size-4" />
</DropdownMenuPrimitive.SubTrigger>
)
);
}
function DropdownMenuSubContent({
@ -235,7 +235,7 @@ function DropdownMenuSubContent({
)}
{...props}
/>
)
);
}
export {
@ -254,4 +254,4 @@ export {
DropdownMenuSub,
DropdownMenuSubTrigger,
DropdownMenuSubContent,
}
};

26
components/ui/metric-card.tsx
View File

@ -80,11 +80,11 @@ export default function MetricCard({
const getTrendIcon = () => {
if (!trend) return null;
if (trend.value === 0) {
return <Minus className="h-3 w-3" />;
}
return trend.isPositive !== false ? (
<TrendingUp className="h-3 w-3" />
) : (
@ -94,11 +94,13 @@ export default function MetricCard({
const getTrendColor = () => {
if (!trend || trend.value === 0) return "text-muted-foreground";
return trend.isPositive !== false ? "text-green-600 dark:text-green-400" : "text-red-600 dark:text-red-400";
return trend.isPositive !== false
? "text-green-600 dark:text-green-400"
: "text-red-600 dark:text-red-400";
};
return (
<Card
<Card
className={cn(
"relative overflow-hidden transition-all duration-200 hover:shadow-lg hover:-translate-y-0.5",
getVariantClasses(),
@ -107,7 +109,7 @@ export default function MetricCard({
>
{/* Subtle gradient overlay */}
<div className="absolute inset-0 bg-linear-to-br from-white/50 to-transparent dark:from-white/5 pointer-events-none" />
<CardHeader className="pb-3 relative">
<div className="flex items-start justify-between">
<div className="space-y-1">
@ -115,9 +117,7 @@ export default function MetricCard({
{title}
</p>
{description && (
<p className="text-xs text-muted-foreground/80">
{description}
</p>
<p className="text-xs text-muted-foreground/80">{description}</p>
)}
</div>
@ -137,13 +137,11 @@ export default function MetricCard({
<CardContent className="relative">
<div className="flex items-end justify-between">
<div className="space-y-1">
<p className="text-2xl font-bold tracking-tight">
{value ?? "—"}
</p>
<p className="text-2xl font-bold tracking-tight">{value ?? "—"}</p>
{trend && (
<Badge
variant="secondary"
<Badge
variant="secondary"
className={cn(
"text-xs font-medium px-2 py-0.5 gap-1",
getTrendColor(),

12
components/ui/separator.tsx
View File

@ -1,9 +1,9 @@
"use client"
"use client";
import * as React from "react"
import * as SeparatorPrimitive from "@radix-ui/react-separator"
import * as React from "react";
import * as SeparatorPrimitive from "@radix-ui/react-separator";
import { cn } from "@/lib/utils"
import { cn } from "@/lib/utils";
function Separator({
className,
@ -22,7 +22,7 @@ function Separator({
)}
{...props}
/>
)
);
}
export { Separator }
export { Separator };

6
components/ui/skeleton.tsx
View File

@ -1,4 +1,4 @@
import { cn } from "@/lib/utils"
import { cn } from "@/lib/utils";
function Skeleton({ className, ...props }: React.ComponentProps<"div">) {
return (
@ -7,7 +7,7 @@ function Skeleton({ className, ...props }: React.ComponentProps<"div">) {
className={cn("bg-accent animate-pulse rounded-md", className)}
{...props}
/>
)
);
}
export { Skeleton }
export { Skeleton };

18
components/ui/tooltip.tsx
View File

@ -1,9 +1,9 @@
"use client"
"use client";
import * as React from "react"
import * as TooltipPrimitive from "@radix-ui/react-tooltip"
import * as React from "react";
import * as TooltipPrimitive from "@radix-ui/react-tooltip";
import { cn } from "@/lib/utils"
import { cn } from "@/lib/utils";
function TooltipProvider({
delayDuration = 0,
@ -15,7 +15,7 @@ function TooltipProvider({
delayDuration={delayDuration}
{...props}
/>
)
);
}
function Tooltip({
@ -25,13 +25,13 @@ function Tooltip({
<TooltipProvider>
<TooltipPrimitive.Root data-slot="tooltip" {...props} />
</TooltipProvider>
)
);
}
function TooltipTrigger({
...props
}: React.ComponentProps<typeof TooltipPrimitive.Trigger>) {
return <TooltipPrimitive.Trigger data-slot="tooltip-trigger" {...props} />
return <TooltipPrimitive.Trigger data-slot="tooltip-trigger" {...props} />;
}
function TooltipContent({
@ -55,7 +55,7 @@ function TooltipContent({
<TooltipPrimitive.Arrow className="bg-primary fill-primary z-50 size-2.5 translate-y-[calc(-50%-2px)] rotate-45 rounded-[2px]" />
</TooltipPrimitive.Content>
</TooltipPrimitive.Portal>
)
);
}
export { Tooltip, TooltipTrigger, TooltipContent, TooltipProvider }
export { Tooltip, TooltipTrigger, TooltipContent, TooltipProvider };

65
debug-import-status.ts
View File

@ -1,78 +1,87 @@
import { PrismaClient } from '@prisma/client';
import { ProcessingStatusManager } from './lib/processingStatusManager';
import { PrismaClient } from "@prisma/client";
import { ProcessingStatusManager } from "./lib/processingStatusManager";
const prisma = new PrismaClient();
async function debugImportStatus() {
try {
console.log('=== DEBUGGING PROCESSING STATUS (REFACTORED SYSTEM) ===\n');
console.log("=== DEBUGGING PROCESSING STATUS (REFACTORED SYSTEM) ===\n");
// Get pipeline status using the new system
const pipelineStatus = await ProcessingStatusManager.getPipelineStatus();
console.log(`Total Sessions: ${pipelineStatus.totalSessions}\n`);
// Display status for each stage
const stages = ['CSV_IMPORT', 'TRANSCRIPT_FETCH', 'SESSION_CREATION', 'AI_ANALYSIS', 'QUESTION_EXTRACTION'];
const stages = [
"CSV_IMPORT",
"TRANSCRIPT_FETCH",
"SESSION_CREATION",
"AI_ANALYSIS",
"QUESTION_EXTRACTION",
];
for (const stage of stages) {
console.log(`${stage}:`);
const stageData = pipelineStatus.pipeline[stage] || {};
const pending = stageData.PENDING || 0;
const inProgress = stageData.IN_PROGRESS || 0;
const completed = stageData.COMPLETED || 0;
const failed = stageData.FAILED || 0;
const skipped = stageData.SKIPPED || 0;
console.log(` PENDING: ${pending}`);
console.log(` IN_PROGRESS: ${inProgress}`);
console.log(` COMPLETED: ${completed}`);
console.log(` FAILED: ${failed}`);
console.log(` SKIPPED: ${skipped}`);
console.log('');
console.log("");
}
// Check Sessions vs SessionImports
console.log('=== SESSION IMPORT RELATIONSHIP ===');
console.log("=== SESSION IMPORT RELATIONSHIP ===");
const sessionsWithImports = await prisma.session.count({
where: { importId: { not: null } }
where: { importId: { not: null } },
});
const totalSessions = await prisma.session.count();
console.log(` Sessions with importId: ${sessionsWithImports}`);
console.log(` Total sessions: ${totalSessions}`);
// Show failed sessions if any
const failedSessions = await ProcessingStatusManager.getFailedSessions();
if (failedSessions.length > 0) {
console.log('\n=== FAILED SESSIONS ===');
failedSessions.slice(0, 10).forEach(failure => {
console.log(` ${failure.session.import?.externalSessionId || failure.sessionId}: ${failure.stage} - ${failure.errorMessage}`);
console.log("\n=== FAILED SESSIONS ===");
failedSessions.slice(0, 10).forEach((failure) => {
console.log(
` ${failure.session.import?.externalSessionId || failure.sessionId}: ${failure.stage} - ${failure.errorMessage}`
);
});
if (failedSessions.length > 10) {
console.log(` ... and ${failedSessions.length - 10} more failed sessions`);
console.log(
` ... and ${failedSessions.length - 10} more failed sessions`
);
}
} else {
console.log('\n✓ No failed sessions found');
console.log("\n✓ No failed sessions found");
}
// Show what needs processing
console.log('\n=== WHAT NEEDS PROCESSING ===');
console.log("\n=== WHAT NEEDS PROCESSING ===");
for (const stage of stages) {
const stageData = pipelineStatus.pipeline[stage] || {};
const pending = stageData.PENDING || 0;
const failed = stageData.FAILED || 0;
if (pending > 0 || failed > 0) {
console.log(`• ${stage}: ${pending} pending, ${failed} failed`);
}
}
} catch (error) {
console.error('Error debugging processing status:', error);
console.error("Error debugging processing status:", error);
} finally {
await prisma.$disconnect();
}

48
docs/dashboard-components.md
View File

@ -12,10 +12,10 @@ The WordCloud component visualizes categories or topics based on their frequency
**Features:**
- Dynamic sizing based on frequency
- Colorful display with a pleasing color palette
- Responsive design
- Interactive hover effects
- Dynamic sizing based on frequency
- Colorful display with a pleasing color palette
- Responsive design
- Interactive hover effects
### 2. GeographicMap
@ -25,10 +25,10 @@ This component displays a world map with circles representing the number of sess
**Features:**
- Interactive map using React Leaflet
- Circle sizes scaled by session count
- Tooltips showing country names and session counts
- Responsive design
- Interactive map using React Leaflet
- Circle sizes scaled by session count
- Tooltips showing country names and session counts
- Responsive design
### 3. MetricCard
@ -38,10 +38,10 @@ A modern, visually appealing card for displaying key metrics.
**Features:**
- Multiple design variants (default, primary, success, warning, danger)
- Support for trend indicators
- Icons and descriptions
- Clean, modern styling
- Multiple design variants (default, primary, success, warning, danger)
- Support for trend indicators
- Icons and descriptions
- Clean, modern styling
### 4. DonutChart
@ -51,10 +51,10 @@ An enhanced donut chart with better styling and a central text display capabilit
**Features:**
- Customizable colors
- Center text area for displaying summaries
- Interactive tooltips with percentages
- Well-balanced legend display
- Customizable colors
- Center text area for displaying summaries
- Interactive tooltips with percentages
- Well-balanced legend display
### 5. ResponseTimeDistribution
@ -64,10 +64,10 @@ Visualizes the distribution of response times as a histogram.
**Features:**
- Color-coded bars (green for fast, yellow for medium, red for slow)
- Target time indicator
- Automatic binning of response times
- Clear labeling and scales
- Color-coded bars (green for fast, yellow for medium, red for slow)
- Target time indicator
- Automatic binning of response times
- Clear labeling and scales
## Dashboard Enhancements
@ -85,7 +85,7 @@ The dashboard has been enhanced with:
## Usage Notes
- The geographic map and response time distribution use simulated data where actual data is not available
- All components are responsive and will adjust to different screen sizes
- The dashboard automatically refreshes data when using the refresh button
- Admin users have access to additional controls at the bottom of the dashboard
- The geographic map and response time distribution use simulated data where actual data is not available
- All components are responsive and will adjust to different screen sizes
- The dashboard automatically refreshes data when using the refresh button
- Admin users have access to additional controls at the bottom of the dashboard

52
docs/postgresql-migration.md
View File

@ -17,48 +17,48 @@ Successfully migrated the livedash-node application from SQLite to PostgreSQL us
#### Production/Development
- **Provider**: PostgreSQL (Neon)
- **Environment Variable**: `DATABASE_URL`
- **Connection**: Neon PostgreSQL cluster
- **Provider**: PostgreSQL (Neon)
- **Environment Variable**: `DATABASE_URL`
- **Connection**: Neon PostgreSQL cluster
#### Testing
- **Provider**: PostgreSQL (Neon - separate database)
- **Environment Variable**: `DATABASE_URL_TEST`
- **Test Setup**: Automatically switches to test database during test runs
- **Provider**: PostgreSQL (Neon - separate database)
- **Environment Variable**: `DATABASE_URL_TEST`
- **Test Setup**: Automatically switches to test database during test runs
### Files Modified
1. **`prisma/schema.prisma`**
- Changed provider from `sqlite` to `postgresql`
- Updated URL to use `env("DATABASE_URL")`
- Changed provider from `sqlite` to `postgresql`
- Updated URL to use `env("DATABASE_URL")`
2. **`tests/setup.ts`**
- Added logic to use `DATABASE_URL_TEST` when available
- Ensures test isolation with separate database
- Added logic to use `DATABASE_URL_TEST` when available
- Ensures test isolation with separate database
3. **`.env`** (created)
- Contains `DATABASE_URL` for Prisma CLI operations
- Contains `DATABASE_URL` for Prisma CLI operations
4. **`.env.local`** (existing)
- Contains both `DATABASE_URL` and `DATABASE_URL_TEST`
- Contains both `DATABASE_URL` and `DATABASE_URL_TEST`
### Database Schema
All existing models and relationships were preserved:
- **Company**: Multi-tenant root entity
- **User**: Authentication and authorization
- **Session**: Processed session data
- **SessionImport**: Raw CSV import data
- **Message**: Individual conversation messages
- **Question**: Normalized question storage
- **SessionQuestion**: Session-question relationships
- **AIProcessingRequest**: AI cost tracking
- **Company**: Multi-tenant root entity
- **User**: Authentication and authorization
- **Session**: Processed session data
- **SessionImport**: Raw CSV import data
- **Message**: Individual conversation messages
- **Question**: Normalized question storage
- **SessionQuestion**: Session-question relationships
- **AIProcessingRequest**: AI cost tracking
### Migration Process
@ -76,7 +76,7 @@ All existing models and relationships were preserved:
✅ **Advanced Features**: Full JSON support, arrays, advanced indexing
✅ **Test Isolation**: Separate test database prevents data conflicts
✅ **Consistency**: Same database engine across all environments
✅ **Cloud-Native**: Neon provides managed PostgreSQL with excellent DX
✅ **Cloud-Native**: Neon provides managed PostgreSQL with excellent DX
### Environment Variables
@ -103,11 +103,11 @@ if (process.env.DATABASE_URL_TEST) {
All tests pass successfully:
- ✅ Environment configuration tests
- ✅ Transcript fetcher tests
- ✅ Database connection tests
- ✅ Schema validation tests
- ✅ CRUD operation tests
- ✅ Environment configuration tests
- ✅ Transcript fetcher tests
- ✅ Database connection tests
- ✅ Schema validation tests
- ✅ CRUD operation tests
### Next Steps

35
docs/scheduler-fixes.md
View File

@ -8,8 +8,8 @@
**Solution**:
- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs
- Removed the invalid company record from the database using `fix_companies.js`
- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs
- Removed the invalid company record from the database using `fix_companies.js`
### 2. Transcript Fetching Errors
@ -17,10 +17,10 @@
**Solution**:
- Improved error handling in `fetchTranscriptContent()` function
- Added probabilistic logging (only ~10% of errors logged) to prevent log spam
- Added timeout (10 seconds) for transcript fetching
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
- Improved error handling in `fetchTranscriptContent()` function
- Added probabilistic logging (only ~10% of errors logged) to prevent log spam
- Added timeout (10 seconds) for transcript fetching
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
### 3. CSV Fetching Errors
@ -28,8 +28,8 @@
**Solution**:
- Added URL validation to skip companies with `example.com` URLs
- Improved error logging to be more descriptive
- Added URL validation to skip companies with `example.com` URLs
- Improved error logging to be more descriptive
## Current Status
@ -42,22 +42,23 @@
After cleanup, only valid companies remain:
- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`)
- CSV URL: `https://proto.notso.ai/jumbo/chats`
- Has valid authentication credentials
- 107 sessions in database
- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`)
- CSV URL: `https://proto.notso.ai/jumbo/chats`
- Has valid authentication credentials
- 107 sessions in database
## Files Modified
1. **lib/csvFetcher.js**
- Added company URL validation
- Improved transcript fetching error handling
- Reduced error log verbosity
- Added company URL validation
- Improved transcript fetching error handling
- Reduced error log verbosity
2. **fix_companies.js** (cleanup script)
- Removes invalid company records
- Can be run again if needed
- Removes invalid company records
- Can be run again if needed
## Monitoring

44
docs/session-processing.md
View File

@ -15,22 +15,22 @@ The system now includes an automated process for analyzing chat session transcri
### Session Fetching
- The system fetches session data from configured CSV URLs for each company
- Unlike the previous implementation, it now only adds sessions that don't already exist in the database
- This prevents duplicate sessions and allows for incremental updates
- The system fetches session data from configured CSV URLs for each company
- Unlike the previous implementation, it now only adds sessions that don't already exist in the database
- This prevents duplicate sessions and allows for incremental updates
### Transcript Processing
- For sessions with transcript content that haven't been processed yet, the system calls OpenAI's API
- The API analyzes the transcript and extracts the following information:
- Primary language used (ISO 639-1 code)
- Number of messages sent by the user
- Overall sentiment (positive, neutral, negative)
- Whether the conversation was escalated
- Whether HR contact was mentioned or provided
- Best-fitting category for the conversation
- Up to 5 paraphrased questions asked by the user
- A brief summary of the conversation
- For sessions with transcript content that haven't been processed yet, the system calls OpenAI's API
- The API analyzes the transcript and extracts the following information:
- Primary language used (ISO 639-1 code)
- Number of messages sent by the user
- Overall sentiment (positive, neutral, negative)
- Whether the conversation was escalated
- Whether HR contact was mentioned or provided
- Best-fitting category for the conversation
- Up to 5 paraphrased questions asked by the user
- A brief summary of the conversation
### Scheduling
@ -43,10 +43,10 @@ The system includes two schedulers:
The Session model has been updated with new fields to store the processed data:
- `processed`: Boolean flag indicating whether the session has been processed
- `sentimentCategory`: String value ("positive", "neutral", "negative") from OpenAI
- `questions`: JSON array of questions asked by the user
- `summary`: Brief summary of the conversation
- `processed`: Boolean flag indicating whether the session has been processed
- `sentimentCategory`: String value ("positive", "neutral", "negative") from OpenAI
- `questions`: JSON array of questions asked by the user
- `summary`: Brief summary of the conversation
## Configuration
@ -62,9 +62,9 @@ OPENAI_API_KEY=your_api_key_here
To run the application with schedulers enabled:
- Development: `npm run dev`
- Development (with schedulers disabled): `npm run dev:no-schedulers`
- Production: `npm run start`
- Development: `npm run dev`
- Development (with schedulers disabled): `npm run dev:no-schedulers`
- Production: `npm run start`
Note: These commands will start a custom Next.js server with the schedulers enabled. You'll need to have an OpenAI API key set in your `.env.local` file for the session processing to work.
@ -82,5 +82,5 @@ This will process all unprocessed sessions that have transcript content.
The processing logic can be customized by modifying:
- `lib/processingScheduler.ts`: Contains the OpenAI processing logic
- `scripts/process_sessions.ts`: Standalone script for manual processing
- `lib/processingScheduler.ts`: Contains the OpenAI processing logic
- `scripts/process_sessions.ts`: Standalone script for manual processing

102
fix-import-status.ts
View File

@ -1,72 +1,93 @@
import { PrismaClient, ProcessingStage, ProcessingStatus } from '@prisma/client';
import { ProcessingStatusManager } from './lib/processingStatusManager';
import {
PrismaClient,
ProcessingStage,
ProcessingStatus,
} from "@prisma/client";
import { ProcessingStatusManager } from "./lib/processingStatusManager";
const prisma = new PrismaClient();
async function fixProcessingStatus() {
try {
console.log('=== FIXING PROCESSING STATUS (REFACTORED SYSTEM) ===\n');
console.log("=== FIXING PROCESSING STATUS (REFACTORED SYSTEM) ===\n");
// Check for any failed processing stages that might need retry
const failedSessions = await ProcessingStatusManager.getFailedSessions();
console.log(`Found ${failedSessions.length} failed processing stages`);
if (failedSessions.length > 0) {
console.log('\nFailed sessions by stage:');
console.log("\nFailed sessions by stage:");
const failuresByStage: Record<string, number> = {};
failedSessions.forEach(failure => {
failuresByStage[failure.stage] = (failuresByStage[failure.stage] || 0) + 1;
failedSessions.forEach((failure) => {
failuresByStage[failure.stage] =
(failuresByStage[failure.stage] || 0) + 1;
});
Object.entries(failuresByStage).forEach(([stage, count]) => {
console.log(` ${stage}: ${count} failures`);
});
// Show sample failed sessions
console.log('\nSample failed sessions:');
failedSessions.slice(0, 5).forEach(failure => {
console.log(` ${failure.session.import?.externalSessionId || failure.sessionId}: ${failure.stage} - ${failure.errorMessage}`);
console.log("\nSample failed sessions:");
failedSessions.slice(0, 5).forEach((failure) => {
console.log(
` ${failure.session.import?.externalSessionId || failure.sessionId}: ${failure.stage} - ${failure.errorMessage}`
);
});
// Ask if user wants to reset failed stages for retry
console.log('\nTo reset failed stages for retry, you can use:');
console.log('ProcessingStatusManager.resetStageForRetry(sessionId, stage)');
console.log("\nTo reset failed stages for retry, you can use:");
console.log(
"ProcessingStatusManager.resetStageForRetry(sessionId, stage)"
);
}
// Check for sessions that might be stuck in IN_PROGRESS
const stuckSessions = await prisma.sessionProcessingStatus.findMany({
where: {
status: ProcessingStatus.IN_PROGRESS,
startedAt: {
lt: new Date(Date.now() - 30 * 60 * 1000) // Started more than 30 minutes ago
}
lt: new Date(Date.now() - 30 * 60 * 1000), // Started more than 30 minutes ago
},
},
include: {
session: {
include: {
import: true
}
}
}
import: true,
},
},
},
});
if (stuckSessions.length > 0) {
console.log(`\nFound ${stuckSessions.length} sessions stuck in IN_PROGRESS state:`);
stuckSessions.forEach(stuck => {
console.log(` ${stuck.session.import?.externalSessionId || stuck.sessionId}: ${stuck.stage} (started: ${stuck.startedAt})`);
console.log(
`\nFound ${stuckSessions.length} sessions stuck in IN_PROGRESS state:`
);
stuckSessions.forEach((stuck) => {
console.log(
` ${stuck.session.import?.externalSessionId || stuck.sessionId}: ${stuck.stage} (started: ${stuck.startedAt})`
);
});
console.log('\nThese sessions may need to be reset to PENDING status for retry.');
console.log(
"\nThese sessions may need to be reset to PENDING status for retry."
);
}
// Show current pipeline status
console.log('\n=== CURRENT PIPELINE STATUS ===');
console.log("\n=== CURRENT PIPELINE STATUS ===");
const pipelineStatus = await ProcessingStatusManager.getPipelineStatus();
const stages = ['CSV_IMPORT', 'TRANSCRIPT_FETCH', 'SESSION_CREATION', 'AI_ANALYSIS', 'QUESTION_EXTRACTION'];
const stages = [
"CSV_IMPORT",
"TRANSCRIPT_FETCH",
"SESSION_CREATION",
"AI_ANALYSIS",
"QUESTION_EXTRACTION",
];
for (const stage of stages) {
const stageData = pipelineStatus.pipeline[stage] || {};
const pending = stageData.PENDING || 0;
@ -74,12 +95,13 @@ async function fixProcessingStatus() {
const completed = stageData.COMPLETED || 0;
const failed = stageData.FAILED || 0;
const skipped = stageData.SKIPPED || 0;
console.log(`${stage}: ${completed} completed, ${pending} pending, ${inProgress} in progress, ${failed} failed, ${skipped} skipped`);
console.log(
`${stage}: ${completed} completed, ${pending} pending, ${inProgress} in progress, ${failed} failed, ${skipped} skipped`
);
}
} catch (error) {
console.error('Error fixing processing status:', error);
console.error("Error fixing processing status:", error);
} finally {
await prisma.$disconnect();
}

74
lib/env.ts
View File

@ -7,20 +7,22 @@ import { dirname, join } from "path";
* Parse environment variable value by removing quotes, comments, and trimming whitespace
*/
function parseEnvValue(value: string | undefined): string {
if (!value) return '';
if (!value) return "";
// Trim whitespace
let cleaned = value.trim();
// Remove inline comments (everything after #)
const commentIndex = cleaned.indexOf('#');
const commentIndex = cleaned.indexOf("#");
if (commentIndex !== -1) {
cleaned = cleaned.substring(0, commentIndex).trim();
}
// Remove surrounding quotes (both single and double)
if ((cleaned.startsWith('"') && cleaned.endsWith('"')) ||
(cleaned.startsWith("'") && cleaned.endsWith("'"))) {
if (
(cleaned.startsWith('"') && cleaned.endsWith('"')) ||
(cleaned.startsWith("'") && cleaned.endsWith("'"))
) {
cleaned = cleaned.slice(1, -1);
}
@ -30,7 +32,10 @@ function parseEnvValue(value: string | undefined): string {
/**
* Parse integer with fallback to default value
*/
function parseIntWithDefault(value: string | undefined, defaultValue: number): number {
function parseIntWithDefault(
value: string | undefined,
defaultValue: number
): number {
const cleaned = parseEnvValue(value);
if (!cleaned) return defaultValue;
@ -41,17 +46,19 @@ function parseIntWithDefault(value: string | undefined, defaultValue: number): n
// Load environment variables from .env.local
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const envPath = join(__dirname, '..', '.env.local');
const envPath = join(__dirname, "..", ".env.local");
// Load .env.local if it exists
try {
const envFile = readFileSync(envPath, 'utf8');
const envVars = envFile.split('\n').filter(line => line.trim() && !line.startsWith('#'));
const envFile = readFileSync(envPath, "utf8");
const envVars = envFile
.split("\n")
.filter((line) => line.trim() && !line.startsWith("#"));
envVars.forEach(line => {
const [key, ...valueParts] = line.split('=');
envVars.forEach((line) => {
const [key, ...valueParts] = line.split("=");
if (key && valueParts.length > 0) {
const rawValue = valueParts.join('=');
const rawValue = valueParts.join("=");
const cleanedValue = parseEnvValue(rawValue);
if (!process.env[key.trim()]) {
process.env[key.trim()] = cleanedValue;
@ -67,21 +74,34 @@ try {
*/
export const env = {
// NextAuth
NEXTAUTH_URL: parseEnvValue(process.env.NEXTAUTH_URL) || 'http://localhost:3000',
NEXTAUTH_SECRET: parseEnvValue(process.env.NEXTAUTH_SECRET) || '',
NODE_ENV: parseEnvValue(process.env.NODE_ENV) || 'development',
NEXTAUTH_URL:
parseEnvValue(process.env.NEXTAUTH_URL) || "http://localhost:3000",
NEXTAUTH_SECRET: parseEnvValue(process.env.NEXTAUTH_SECRET) || "",
NODE_ENV: parseEnvValue(process.env.NODE_ENV) || "development",
// OpenAI
OPENAI_API_KEY: parseEnvValue(process.env.OPENAI_API_KEY) || '',
OPENAI_API_KEY: parseEnvValue(process.env.OPENAI_API_KEY) || "",
// Scheduler Configuration
SCHEDULER_ENABLED: parseEnvValue(process.env.SCHEDULER_ENABLED) === 'true',
CSV_IMPORT_INTERVAL: parseEnvValue(process.env.CSV_IMPORT_INTERVAL) || '*/15 * * * *',
IMPORT_PROCESSING_INTERVAL: parseEnvValue(process.env.IMPORT_PROCESSING_INTERVAL) || '*/5 * * * *',
IMPORT_PROCESSING_BATCH_SIZE: parseIntWithDefault(process.env.IMPORT_PROCESSING_BATCH_SIZE, 50),
SESSION_PROCESSING_INTERVAL: parseEnvValue(process.env.SESSION_PROCESSING_INTERVAL) || '0 * * * *',
SESSION_PROCESSING_BATCH_SIZE: parseIntWithDefault(process.env.SESSION_PROCESSING_BATCH_SIZE, 0),
SESSION_PROCESSING_CONCURRENCY: parseIntWithDefault(process.env.SESSION_PROCESSING_CONCURRENCY, 5),
SCHEDULER_ENABLED: parseEnvValue(process.env.SCHEDULER_ENABLED) === "true",
CSV_IMPORT_INTERVAL:
parseEnvValue(process.env.CSV_IMPORT_INTERVAL) || "*/15 * * * *",
IMPORT_PROCESSING_INTERVAL:
parseEnvValue(process.env.IMPORT_PROCESSING_INTERVAL) || "*/5 * * * *",
IMPORT_PROCESSING_BATCH_SIZE: parseIntWithDefault(
process.env.IMPORT_PROCESSING_BATCH_SIZE,
50
),
SESSION_PROCESSING_INTERVAL:
parseEnvValue(process.env.SESSION_PROCESSING_INTERVAL) || "0 * * * *",
SESSION_PROCESSING_BATCH_SIZE: parseIntWithDefault(
process.env.SESSION_PROCESSING_BATCH_SIZE,
0
),
SESSION_PROCESSING_CONCURRENCY: parseIntWithDefault(
process.env.SESSION_PROCESSING_CONCURRENCY,
5
),
// Server
PORT: parseIntWithDefault(process.env.PORT, 3000),
@ -94,11 +114,11 @@ export function validateEnv(): { valid: boolean; errors: string[] } {
const errors: string[] = [];
if (!env.NEXTAUTH_SECRET) {
errors.push('NEXTAUTH_SECRET is required');
errors.push("NEXTAUTH_SECRET is required");
}
if (!env.OPENAI_API_KEY && env.NODE_ENV === 'production') {
errors.push('OPENAI_API_KEY is required in production');
if (!env.OPENAI_API_KEY && env.NODE_ENV === "production") {
errors.push("OPENAI_API_KEY is required in production");
}
return {
@ -132,14 +152,14 @@ export function getSchedulerConfig() {
* Log environment configuration (safe for production)
*/
export function logEnvConfig(): void {
console.log('[Environment] Configuration:');
console.log("[Environment] Configuration:");
console.log(` NODE_ENV: ${env.NODE_ENV}`);
console.log(` NEXTAUTH_URL: ${env.NEXTAUTH_URL}`);
console.log(` SCHEDULER_ENABLED: ${env.SCHEDULER_ENABLED}`);
console.log(` PORT: ${env.PORT}`);
if (env.SCHEDULER_ENABLED) {
console.log(' Scheduler intervals:');
console.log(" Scheduler intervals:");
console.log(` CSV Import: ${env.CSV_IMPORT_INTERVAL}`);
console.log(` Import Processing: ${env.IMPORT_PROCESSING_INTERVAL}`);
console.log(` Session Processing: ${env.SESSION_PROCESSING_INTERVAL}`);

221
lib/importProcessor.ts
View File

@ -1,7 +1,15 @@
// SessionImport to Session processor
import { PrismaClient, SentimentCategory, SessionCategory, ProcessingStage } from "@prisma/client";
import {
PrismaClient,
SentimentCategory,
SessionCategory,
ProcessingStage,
} from "@prisma/client";
import { getSchedulerConfig } from "./env";
import { fetchTranscriptContent, isValidTranscriptUrl } from "./transcriptFetcher";
import {
fetchTranscriptContent,
isValidTranscriptUrl,
} from "./transcriptFetcher";
import { ProcessingStatusManager } from "./processingStatusManager";
import cron from "node-cron";
@ -11,25 +19,29 @@ const prisma = new PrismaClient();
* Parse European date format (DD.MM.YYYY HH:mm:ss) to JavaScript Date
*/
function parseEuropeanDate(dateStr: string): Date {
if (!dateStr || typeof dateStr !== 'string') {
if (!dateStr || typeof dateStr !== "string") {
throw new Error(`Invalid date string: ${dateStr}`);
}
// Handle format: "DD.MM.YYYY HH:mm:ss"
const [datePart, timePart] = dateStr.trim().split(' ');
const [datePart, timePart] = dateStr.trim().split(" ");
if (!datePart || !timePart) {
throw new Error(`Invalid date format: ${dateStr}. Expected format: DD.MM.YYYY HH:mm:ss`);
throw new Error(
`Invalid date format: ${dateStr}. Expected format: DD.MM.YYYY HH:mm:ss`
);
}
const [day, month, year] = datePart.split('.');
const [day, month, year] = datePart.split(".");
if (!day || !month || !year) {
throw new Error(`Invalid date part: ${datePart}. Expected format: DD.MM.YYYY`);
throw new Error(
`Invalid date part: ${datePart}. Expected format: DD.MM.YYYY`
);
}
// Convert to ISO format: YYYY-MM-DD HH:mm:ss
const isoDateStr = `${year}-${month.padStart(2, '0')}-${day.padStart(2, '0')} ${timePart}`;
const isoDateStr = `${year}-${month.padStart(2, "0")}-${day.padStart(2, "0")} ${timePart}`;
const date = new Date(isoDateStr);
if (isNaN(date.getTime())) {
@ -42,13 +54,15 @@ function parseEuropeanDate(dateStr: string): Date {
/**
* Helper function to parse sentiment from raw string (fallback only)
*/
function parseFallbackSentiment(sentimentRaw: string | null): SentimentCategory | null {
function parseFallbackSentiment(
sentimentRaw: string | null
): SentimentCategory | null {
if (!sentimentRaw) return null;
const sentimentStr = sentimentRaw.toLowerCase();
if (sentimentStr.includes('positive')) {
if (sentimentStr.includes("positive")) {
return SentimentCategory.POSITIVE;
} else if (sentimentStr.includes('negative')) {
} else if (sentimentStr.includes("negative")) {
return SentimentCategory.NEGATIVE;
} else {
return SentimentCategory.NEUTRAL;
@ -60,20 +74,25 @@ function parseFallbackSentiment(sentimentRaw: string | null): SentimentCategory
*/
function parseFallbackBoolean(rawValue: string | null): boolean | null {
if (!rawValue) return null;
return ['true', '1', 'yes', 'escalated', 'forwarded'].includes(rawValue.toLowerCase());
return ["true", "1", "yes", "escalated", "forwarded"].includes(
rawValue.toLowerCase()
);
}
/**
* Parse transcript content into Message records
*/
async function parseTranscriptIntoMessages(sessionId: string, transcriptContent: string): Promise<void> {
async function parseTranscriptIntoMessages(
sessionId: string,
transcriptContent: string
): Promise<void> {
// Clear existing messages for this session
await prisma.message.deleteMany({
where: { sessionId }
where: { sessionId },
});
// Split transcript into lines and parse each message
const lines = transcriptContent.split('\n').filter(line => line.trim());
const lines = transcriptContent.split("\n").filter((line) => line.trim());
let order = 0;
for (const line of lines) {
@ -84,7 +103,7 @@ async function parseTranscriptIntoMessages(sessionId: string, transcriptContent:
// Format 1: "User: message" or "Assistant: message"
// Format 2: "[timestamp] User: message" or "[timestamp] Assistant: message"
let role = 'unknown';
let role = "unknown";
let content = trimmedLine;
let timestamp: Date | null = null;
@ -107,7 +126,7 @@ async function parseTranscriptIntoMessages(sessionId: string, transcriptContent:
content = roleMatch[2].trim();
} else {
// If no role prefix found, try to infer from context or use 'unknown'
role = 'unknown';
role = "unknown";
}
// Skip empty content
@ -127,14 +146,18 @@ async function parseTranscriptIntoMessages(sessionId: string, transcriptContent:
order++;
}
console.log(`[Import Processor] ✓ Parsed ${order} messages for session ${sessionId}`);
console.log(
`[Import Processor] ✓ Parsed ${order} messages for session ${sessionId}`
);
}
/**
* Process a single SessionImport record into a Session record
* Uses new unified processing status tracking
*/
async function processSingleImport(importRecord: any): Promise<{ success: boolean; error?: string }> {
async function processSingleImport(
importRecord: any
): Promise<{ success: boolean; error?: string }> {
let sessionId: string | null = null;
try {
@ -142,7 +165,9 @@ async function processSingleImport(importRecord: any): Promise<{ success: boolea
const startTime = parseEuropeanDate(importRecord.startTimeRaw);
const endTime = parseEuropeanDate(importRecord.endTimeRaw);
console.log(`[Import Processor] Processing ${importRecord.externalSessionId}: ${startTime.toISOString()} - ${endTime.toISOString()}`);
console.log(
`[Import Processor] Processing ${importRecord.externalSessionId}: ${startTime.toISOString()} - ${endTime.toISOString()}`
);
// Create or update Session record with MINIMAL processing
const session = await prisma.session.upsert({
@ -179,15 +204,27 @@ async function processSingleImport(importRecord: any): Promise<{ success: boolea
await ProcessingStatusManager.initializeSession(sessionId);
// Mark CSV_IMPORT as completed
await ProcessingStatusManager.completeStage(sessionId, ProcessingStage.CSV_IMPORT);
await ProcessingStatusManager.completeStage(
sessionId,
ProcessingStage.CSV_IMPORT
);
// Handle transcript fetching
let transcriptContent = importRecord.rawTranscriptContent;
if (!transcriptContent && importRecord.fullTranscriptUrl && isValidTranscriptUrl(importRecord.fullTranscriptUrl)) {
await ProcessingStatusManager.startStage(sessionId, ProcessingStage.TRANSCRIPT_FETCH);
if (
!transcriptContent &&
importRecord.fullTranscriptUrl &&
isValidTranscriptUrl(importRecord.fullTranscriptUrl)
) {
await ProcessingStatusManager.startStage(
sessionId,
ProcessingStage.TRANSCRIPT_FETCH
);
console.log(`[Import Processor] Fetching transcript for ${importRecord.externalSessionId}...`);
console.log(
`[Import Processor] Fetching transcript for ${importRecord.externalSessionId}...`
);
// Get company credentials for transcript fetching
const company = await prisma.company.findUnique({
@ -203,7 +240,9 @@ async function processSingleImport(importRecord: any): Promise<{ success: boolea
if (transcriptResult.success) {
transcriptContent = transcriptResult.content;
console.log(`[Import Processor] ✓ Fetched transcript for ${importRecord.externalSessionId} (${transcriptContent?.length} chars)`);
console.log(
`[Import Processor] ✓ Fetched transcript for ${importRecord.externalSessionId} (${transcriptContent?.length} chars)`
);
// Update the import record with the fetched content
await prisma.sessionImport.update({
@ -211,36 +250,61 @@ async function processSingleImport(importRecord: any): Promise<{ success: boolea
data: { rawTranscriptContent: transcriptContent },
});
await ProcessingStatusManager.completeStage(sessionId, ProcessingStage.TRANSCRIPT_FETCH, {
contentLength: transcriptContent?.length || 0,
url: importRecord.fullTranscriptUrl
});
await ProcessingStatusManager.completeStage(
sessionId,
ProcessingStage.TRANSCRIPT_FETCH,
{
contentLength: transcriptContent?.length || 0,
url: importRecord.fullTranscriptUrl,
}
);
} else {
console.log(`[Import Processor] ⚠️ Failed to fetch transcript for ${importRecord.externalSessionId}: ${transcriptResult.error}`);
await ProcessingStatusManager.failStage(sessionId, ProcessingStage.TRANSCRIPT_FETCH, transcriptResult.error || 'Unknown error');
console.log(
`[Import Processor] ⚠️ Failed to fetch transcript for ${importRecord.externalSessionId}: ${transcriptResult.error}`
);
await ProcessingStatusManager.failStage(
sessionId,
ProcessingStage.TRANSCRIPT_FETCH,
transcriptResult.error || "Unknown error"
);
}
} else if (!importRecord.fullTranscriptUrl) {
// No transcript URL available - skip this stage
await ProcessingStatusManager.skipStage(sessionId, ProcessingStage.TRANSCRIPT_FETCH, 'No transcript URL provided');
await ProcessingStatusManager.skipStage(
sessionId,
ProcessingStage.TRANSCRIPT_FETCH,
"No transcript URL provided"
);
} else {
// Transcript already fetched
await ProcessingStatusManager.completeStage(sessionId, ProcessingStage.TRANSCRIPT_FETCH, {
contentLength: transcriptContent?.length || 0,
source: 'already_fetched'
});
await ProcessingStatusManager.completeStage(
sessionId,
ProcessingStage.TRANSCRIPT_FETCH,
{
contentLength: transcriptContent?.length || 0,
source: "already_fetched",
}
);
}
// Handle session creation (parse messages)
await ProcessingStatusManager.startStage(sessionId, ProcessingStage.SESSION_CREATION);
await ProcessingStatusManager.startStage(
sessionId,
ProcessingStage.SESSION_CREATION
);
if (transcriptContent) {
await parseTranscriptIntoMessages(sessionId, transcriptContent);
}
await ProcessingStatusManager.completeStage(sessionId, ProcessingStage.SESSION_CREATION, {
hasTranscript: !!transcriptContent,
transcriptLength: transcriptContent?.length || 0
});
await ProcessingStatusManager.completeStage(
sessionId,
ProcessingStage.SESSION_CREATION,
{
hasTranscript: !!transcriptContent,
transcriptLength: transcriptContent?.length || 0,
}
);
return { success: true };
} catch (error) {
@ -249,13 +313,31 @@ async function processSingleImport(importRecord: any): Promise<{ success: boolea
// Mark the current stage as failed if we have a sessionId
if (sessionId) {
// Determine which stage failed based on the error
if (errorMessage.includes('transcript') || errorMessage.includes('fetch')) {
await ProcessingStatusManager.failStage(sessionId, ProcessingStage.TRANSCRIPT_FETCH, errorMessage);
} else if (errorMessage.includes('message') || errorMessage.includes('parse')) {
await ProcessingStatusManager.failStage(sessionId, ProcessingStage.SESSION_CREATION, errorMessage);
if (
errorMessage.includes("transcript") ||
errorMessage.includes("fetch")
) {
await ProcessingStatusManager.failStage(
sessionId,
ProcessingStage.TRANSCRIPT_FETCH,
errorMessage
);
} else if (
errorMessage.includes("message") ||
errorMessage.includes("parse")
) {
await ProcessingStatusManager.failStage(
sessionId,
ProcessingStage.SESSION_CREATION,
errorMessage
);
} else {
// General failure - mark CSV_IMPORT as failed
await ProcessingStatusManager.failStage(sessionId, ProcessingStage.CSV_IMPORT, errorMessage);
await ProcessingStatusManager.failStage(
sessionId,
ProcessingStage.CSV_IMPORT,
errorMessage
);
}
}
@ -270,8 +352,10 @@ async function processSingleImport(importRecord: any): Promise<{ success: boolea
* Process unprocessed SessionImport records into Session records
* Uses new processing status system to find imports that need processing
*/
export async function processQueuedImports(batchSize: number = 50): Promise<void> {
console.log('[Import Processor] Starting to process unprocessed imports...');
export async function processQueuedImports(
batchSize: number = 50
): Promise<void> {
console.log("[Import Processor] Starting to process unprocessed imports...");
let totalSuccessCount = 0;
let totalErrorCount = 0;
@ -285,20 +369,24 @@ export async function processQueuedImports(batchSize: number = 50): Promise<void
},
take: batchSize,
orderBy: {
createdAt: 'asc', // Process oldest first
createdAt: "asc", // Process oldest first
},
});
if (unprocessedImports.length === 0) {
if (batchNumber === 1) {
console.log('[Import Processor] No unprocessed imports found');
console.log("[Import Processor] No unprocessed imports found");
} else {
console.log(`[Import Processor] All batches completed. Total: ${totalSuccessCount} successful, ${totalErrorCount} failed`);
console.log(
`[Import Processor] All batches completed. Total: ${totalSuccessCount} successful, ${totalErrorCount} failed`
);
}
return;
}
console.log(`[Import Processor] Processing batch ${batchNumber}: ${unprocessedImports.length} imports...`);
console.log(
`[Import Processor] Processing batch ${batchNumber}: ${unprocessedImports.length} imports...`
);
let batchSuccessCount = 0;
let batchErrorCount = 0;
@ -310,20 +398,28 @@ export async function processQueuedImports(batchSize: number = 50): Promise<void
if (result.success) {
batchSuccessCount++;
totalSuccessCount++;
console.log(`[Import Processor] ✓ Processed import ${importRecord.externalSessionId}`);
console.log(
`[Import Processor] ✓ Processed import ${importRecord.externalSessionId}`
);
} else {
batchErrorCount++;
totalErrorCount++;
console.log(`[Import Processor] ✗ Failed to process import ${importRecord.externalSessionId}: ${result.error}`);
console.log(
`[Import Processor] ✗ Failed to process import ${importRecord.externalSessionId}: ${result.error}`
);
}
}
console.log(`[Import Processor] Batch ${batchNumber} completed: ${batchSuccessCount} successful, ${batchErrorCount} failed`);
console.log(
`[Import Processor] Batch ${batchNumber} completed: ${batchSuccessCount} successful, ${batchErrorCount} failed`
);
batchNumber++;
// If this batch was smaller than the batch size, we're done
if (unprocessedImports.length < batchSize) {
console.log(`[Import Processor] All batches completed. Total: ${totalSuccessCount} successful, ${totalErrorCount} failed`);
console.log(
`[Import Processor] All batches completed. Total: ${totalSuccessCount} successful, ${totalErrorCount} failed`
);
return;
}
}
@ -336,15 +432,20 @@ export function startImportProcessingScheduler(): void {
const config = getSchedulerConfig();
if (!config.enabled) {
console.log('[Import Processing Scheduler] Disabled via configuration');
console.log("[Import Processing Scheduler] Disabled via configuration");
return;
}
// Use a more frequent interval for import processing (every 5 minutes by default)
const interval = process.env.IMPORT_PROCESSING_INTERVAL || '*/5 * * * *';
const batchSize = parseInt(process.env.IMPORT_PROCESSING_BATCH_SIZE || '50', 10);
const interval = process.env.IMPORT_PROCESSING_INTERVAL || "*/5 * * * *";
const batchSize = parseInt(
process.env.IMPORT_PROCESSING_BATCH_SIZE || "50",
10
);
console.log(`[Import Processing Scheduler] Starting with interval: ${interval}`);
console.log(
`[Import Processing Scheduler] Starting with interval: ${interval}`
);
console.log(`[Import Processing Scheduler] Batch size: ${batchSize}`);
cron.schedule(interval, async () => {

45
lib/metrics.ts
View File

@ -350,16 +350,16 @@ export function sessionMetrics(
const wordCounts: { [key: string]: number } = {};
let alerts = 0;
// New metrics variables
const hourlySessionCounts: { [hour: string]: number } = {};
let resolvedChatsCount = 0;
const questionCounts: { [question: string]: number } = {};
// New metrics variables
const hourlySessionCounts: { [hour: string]: number } = {};
let resolvedChatsCount = 0;
const questionCounts: { [question: string]: number } = {};
for (const session of sessions) {
// Track hourly usage for peak time calculation
if (session.startTime) {
const hour = new Date(session.startTime).getHours();
const hourKey = `${hour.toString().padStart(2, '0')}:00`;
const hourKey = `${hour.toString().padStart(2, "0")}:00`;
hourlySessionCounts[hourKey] = (hourlySessionCounts[hourKey] || 0) + 1;
}
@ -493,12 +493,16 @@ export function sessionMetrics(
// 1. Extract questions from user messages (if available)
if (session.messages) {
session.messages
.filter(msg => msg.role === 'User')
.forEach(msg => {
.filter((msg) => msg.role === "User")
.forEach((msg) => {
const content = msg.content.trim();
// Simple heuristic: if message ends with ? or contains question words, treat as question
if (content.endsWith('?') ||
/\b(what|when|where|why|how|who|which|can|could|would|will|is|are|do|does|did)\b/i.test(content)) {
if (
content.endsWith("?") ||
/\b(what|when|where|why|how|who|which|can|could|would|will|is|are|do|does|did)\b/i.test(
content
)
) {
questionCounts[content] = (questionCounts[content] || 0) + 1;
}
});
@ -507,8 +511,12 @@ export function sessionMetrics(
// 3. Extract questions from initial message as fallback
if (session.initialMsg) {
const content = session.initialMsg.trim();
if (content.endsWith('?') ||
/\b(what|when|where|why|how|who|which|can|could|would|will|is|are|do|does|did)\b/i.test(content)) {
if (
content.endsWith("?") ||
/\b(what|when|where|why|how|who|which|can|could|would|will|is|are|do|does|did)\b/i.test(
content
)
) {
questionCounts[content] = (questionCounts[content] || 0) + 1;
}
}
@ -580,20 +588,23 @@ export function sessionMetrics(
// Calculate new metrics
// 1. Average Daily Costs (euros)
const avgDailyCosts = numDaysWithSessions > 0 ? totalTokensEur / numDaysWithSessions : 0;
const avgDailyCosts =
numDaysWithSessions > 0 ? totalTokensEur / numDaysWithSessions : 0;
// 2. Peak Usage Time
let peakUsageTime = "N/A";
if (Object.keys(hourlySessionCounts).length > 0) {
const peakHour = Object.entries(hourlySessionCounts)
.sort(([, a], [, b]) => b - a)[0][0];
const peakHourNum = parseInt(peakHour.split(':')[0]);
const peakHour = Object.entries(hourlySessionCounts).sort(
([, a], [, b]) => b - a
)[0][0];
const peakHourNum = parseInt(peakHour.split(":")[0]);
const endHour = (peakHourNum + 1) % 24;
peakUsageTime = `${peakHour}-${endHour.toString().padStart(2, '0')}:00`;
peakUsageTime = `${peakHour}-${endHour.toString().padStart(2, "0")}:00`;
}
// 3. Resolved Chats Percentage
const resolvedChatsPercentage = totalSessions > 0 ? (resolvedChatsCount / totalSessions) * 100 : 0;
const resolvedChatsPercentage =
totalSessions > 0 ? (resolvedChatsCount / totalSessions) * 100 : 0;
// 4. Top 5 Asked Questions
const topQuestions: TopQuestion[] = Object.entries(questionCounts)

296
lib/processingScheduler.ts
View File

@ -1,6 +1,11 @@
// Enhanced session processing scheduler with AI cost tracking and question management
import cron from "node-cron";
import { PrismaClient, SentimentCategory, SessionCategory, ProcessingStage } from "@prisma/client";
import {
PrismaClient,
SentimentCategory,
SessionCategory,
ProcessingStage,
} from "@prisma/client";
import fetch from "node-fetch";
import { getSchedulerConfig } from "./schedulerConfig";
import { ProcessingStatusManager } from "./processingStatusManager";
@ -44,10 +49,10 @@ async function getCurrentModelPricing(modelName: string): Promise<{
effectiveFrom: { lte: new Date() },
OR: [
{ effectiveUntil: null },
{ effectiveUntil: { gte: new Date() } }
]
{ effectiveUntil: { gte: new Date() } },
],
},
orderBy: { effectiveFrom: 'desc' },
orderBy: { effectiveFrom: "desc" },
take: 1,
},
},
@ -69,7 +74,20 @@ interface ProcessedData {
sentiment: "POSITIVE" | "NEUTRAL" | "NEGATIVE";
escalated: boolean;
forwarded_hr: boolean;
category: "SCHEDULE_HOURS" | "LEAVE_VACATION" | "SICK_LEAVE_RECOVERY" | "SALARY_COMPENSATION" | "CONTRACT_HOURS" | "ONBOARDING" | "OFFBOARDING" | "WORKWEAR_STAFF_PASS" | "TEAM_CONTACTS" | "PERSONAL_QUESTIONS" | "ACCESS_LOGIN" | "SOCIAL_QUESTIONS" | "UNRECOGNIZED_OTHER";
category:
| "SCHEDULE_HOURS"
| "LEAVE_VACATION"
| "SICK_LEAVE_RECOVERY"
| "SALARY_COMPENSATION"
| "CONTRACT_HOURS"
| "ONBOARDING"
| "OFFBOARDING"
| "WORKWEAR_STAFF_PASS"
| "TEAM_CONTACTS"
| "PERSONAL_QUESTIONS"
| "ACCESS_LOGIN"
| "SOCIAL_QUESTIONS"
| "UNRECOGNIZED_OTHER";
questions: string[];
summary: string;
session_id: string;
@ -87,7 +105,7 @@ interface ProcessingResult {
async function recordAIProcessingRequest(
sessionId: string,
openaiResponse: any,
processingType: string = 'session_analysis'
processingType: string = "session_analysis"
): Promise<void> {
const usage = openaiResponse.usage;
const model = openaiResponse.model;
@ -97,14 +115,15 @@ async function recordAIProcessingRequest(
// Fallback pricing if not found in database
const fallbackPricing = {
promptTokenCost: 0.00001, // $10.00 per 1M tokens (gpt-4-turbo rate)
completionTokenCost: 0.00003, // $30.00 per 1M tokens
promptTokenCost: 0.00001, // $10.00 per 1M tokens (gpt-4-turbo rate)
completionTokenCost: 0.00003, // $30.00 per 1M tokens
};
const finalPricing = pricing || fallbackPricing;
const promptCost = usage.prompt_tokens * finalPricing.promptTokenCost;
const completionCost = usage.completion_tokens * finalPricing.completionTokenCost;
const completionCost =
usage.completion_tokens * finalPricing.completionTokenCost;
const totalCostUsd = promptCost + completionCost;
const totalCostEur = totalCostUsd * USD_TO_EUR_RATE;
@ -123,10 +142,14 @@ async function recordAIProcessingRequest(
// Detailed breakdown
cachedTokens: usage.prompt_tokens_details?.cached_tokens || null,
audioTokensPrompt: usage.prompt_tokens_details?.audio_tokens || null,
reasoningTokens: usage.completion_tokens_details?.reasoning_tokens || null,
audioTokensCompletion: usage.completion_tokens_details?.audio_tokens || null,
acceptedPredictionTokens: usage.completion_tokens_details?.accepted_prediction_tokens || null,
rejectedPredictionTokens: usage.completion_tokens_details?.rejected_prediction_tokens || null,
reasoningTokens:
usage.completion_tokens_details?.reasoning_tokens || null,
audioTokensCompletion:
usage.completion_tokens_details?.audio_tokens || null,
acceptedPredictionTokens:
usage.completion_tokens_details?.accepted_prediction_tokens || null,
rejectedPredictionTokens:
usage.completion_tokens_details?.rejected_prediction_tokens || null,
promptTokenCost: finalPricing.promptTokenCost,
completionTokenCost: finalPricing.completionTokenCost,
@ -135,7 +158,7 @@ async function recordAIProcessingRequest(
processingType,
success: true,
completedAt: new Date(),
}
},
});
}
@ -150,7 +173,7 @@ async function recordFailedAIProcessingRequest(
await prisma.aIProcessingRequest.create({
data: {
sessionId,
model: 'unknown',
model: "unknown",
promptTokens: 0,
completionTokens: 0,
totalTokens: 0,
@ -161,17 +184,20 @@ async function recordFailedAIProcessingRequest(
success: false,
errorMessage,
completedAt: new Date(),
}
},
});
}
/**
* Process questions into separate Question and SessionQuestion tables
*/
async function processQuestions(sessionId: string, questions: string[]): Promise<void> {
async function processQuestions(
sessionId: string,
questions: string[]
): Promise<void> {
// Clear existing questions for this session
await prisma.sessionQuestion.deleteMany({
where: { sessionId }
where: { sessionId },
});
// Process each question
@ -183,7 +209,7 @@ async function processQuestions(sessionId: string, questions: string[]): Promise
const question = await prisma.question.upsert({
where: { content: questionText.trim() },
create: { content: questionText.trim() },
update: {}
update: {},
});
// Link to session
@ -191,8 +217,8 @@ async function processQuestions(sessionId: string, questions: string[]): Promise
data: {
sessionId,
questionId: question.id,
order: index
}
order: index,
},
});
}
}
@ -204,8 +230,8 @@ async function calculateMessagesSent(sessionId: string): Promise<number> {
const userMessageCount = await prisma.message.count({
where: {
sessionId,
role: { in: ['user', 'User'] } // Handle both cases
}
role: { in: ["user", "User"] }, // Handle both cases
},
});
return userMessageCount;
}
@ -213,10 +239,13 @@ async function calculateMessagesSent(sessionId: string): Promise<number> {
/**
* Calculate endTime from latest Message timestamp
*/
async function calculateEndTime(sessionId: string, fallbackEndTime: Date): Promise<Date> {
async function calculateEndTime(
sessionId: string,
fallbackEndTime: Date
): Promise<Date> {
const latestMessage = await prisma.message.findFirst({
where: { sessionId },
orderBy: { timestamp: 'desc' }
orderBy: { timestamp: "desc" },
});
return latestMessage?.timestamp || fallbackEndTime;
@ -225,7 +254,11 @@ async function calculateEndTime(sessionId: string, fallbackEndTime: Date): Promi
/**
* Processes a session transcript using OpenAI API
*/
async function processTranscriptWithOpenAI(sessionId: string, transcript: string, companyId: string): Promise<ProcessedData> {
async function processTranscriptWithOpenAI(
sessionId: string,
transcript: string,
companyId: string
): Promise<ProcessedData> {
if (!OPENAI_API_KEY) {
throw new Error("OPENAI_API_KEY environment variable is not set");
}
@ -293,7 +326,11 @@ async function processTranscriptWithOpenAI(sessionId: string, transcript: string
const openaiResponse: any = await response.json();
// Record the AI processing request for cost tracking
await recordAIProcessingRequest(sessionId, openaiResponse, 'session_analysis');
await recordAIProcessingRequest(
sessionId,
openaiResponse,
"session_analysis"
);
const processedData = JSON.parse(openaiResponse.choices[0].message.content);
@ -305,7 +342,7 @@ async function processTranscriptWithOpenAI(sessionId: string, transcript: string
// Record failed request
await recordFailedAIProcessingRequest(
sessionId,
'session_analysis',
"session_analysis",
error instanceof Error ? error.message : String(error)
);
@ -319,8 +356,14 @@ async function processTranscriptWithOpenAI(sessionId: string, transcript: string
*/
function validateOpenAIResponse(data: any): void {
const requiredFields = [
"language", "sentiment", "escalated", "forwarded_hr",
"category", "questions", "summary", "session_id"
"language",
"sentiment",
"escalated",
"forwarded_hr",
"category",
"questions",
"summary",
"session_id",
];
for (const field of requiredFields) {
@ -331,11 +374,15 @@ function validateOpenAIResponse(data: any): void {
// Validate field types and values
if (typeof data.language !== "string" || !/^[a-z]{2}$/.test(data.language)) {
throw new Error("Invalid language format. Expected ISO 639-1 code (e.g., 'en')");
throw new Error(
"Invalid language format. Expected ISO 639-1 code (e.g., 'en')"
);
}
if (!["POSITIVE", "NEUTRAL", "NEGATIVE"].includes(data.sentiment)) {
throw new Error("Invalid sentiment. Expected 'POSITIVE', 'NEUTRAL', or 'NEGATIVE'");
throw new Error(
"Invalid sentiment. Expected 'POSITIVE', 'NEUTRAL', or 'NEGATIVE'"
);
}
if (typeof data.escalated !== "boolean") {
@ -347,22 +394,39 @@ function validateOpenAIResponse(data: any): void {
}
const validCategories = [
"SCHEDULE_HOURS", "LEAVE_VACATION", "SICK_LEAVE_RECOVERY", "SALARY_COMPENSATION",
"CONTRACT_HOURS", "ONBOARDING", "OFFBOARDING", "WORKWEAR_STAFF_PASS",
"TEAM_CONTACTS", "PERSONAL_QUESTIONS", "ACCESS_LOGIN", "SOCIAL_QUESTIONS",
"UNRECOGNIZED_OTHER"
"SCHEDULE_HOURS",
"LEAVE_VACATION",
"SICK_LEAVE_RECOVERY",
"SALARY_COMPENSATION",
"CONTRACT_HOURS",
"ONBOARDING",
"OFFBOARDING",
"WORKWEAR_STAFF_PASS",
"TEAM_CONTACTS",
"PERSONAL_QUESTIONS",
"ACCESS_LOGIN",
"SOCIAL_QUESTIONS",
"UNRECOGNIZED_OTHER",
];
if (!validCategories.includes(data.category)) {
throw new Error(`Invalid category. Expected one of: ${validCategories.join(", ")}`);
throw new Error(
`Invalid category. Expected one of: ${validCategories.join(", ")}`
);
}
if (!Array.isArray(data.questions)) {
throw new Error("Invalid questions. Expected array of strings");
}
if (typeof data.summary !== "string" || data.summary.length < 10 || data.summary.length > 300) {
throw new Error("Invalid summary. Expected string between 10-300 characters");
if (
typeof data.summary !== "string" ||
data.summary.length < 10 ||
data.summary.length > 300
) {
throw new Error(
"Invalid summary. Expected string between 10-300 characters"
);
}
if (typeof data.session_id !== "string") {
@ -384,31 +448,42 @@ async function processSingleSession(session: any): Promise<ProcessingResult> {
try {
// Mark AI analysis as started
await ProcessingStatusManager.startStage(session.id, ProcessingStage.AI_ANALYSIS);
await ProcessingStatusManager.startStage(
session.id,
ProcessingStage.AI_ANALYSIS
);
// Convert messages back to transcript format for OpenAI processing
const transcript = session.messages
.map((msg: any) =>
`[${new Date(msg.timestamp)
.toLocaleString("en-GB", {
day: "2-digit",
month: "2-digit",
year: "numeric",
hour: "2-digit",
minute: "2-digit",
second: "2-digit",
})
.replace(",", "")}] ${msg.role}: ${msg.content}`
.map(
(msg: any) =>
`[${new Date(msg.timestamp)
.toLocaleString("en-GB", {
day: "2-digit",
month: "2-digit",
year: "numeric",
hour: "2-digit",
minute: "2-digit",
second: "2-digit",
})
.replace(",", "")}] ${msg.role}: ${msg.content}`
)
.join("\n");
const processedData = await processTranscriptWithOpenAI(session.id, transcript, session.companyId);
const processedData = await processTranscriptWithOpenAI(
session.id,
transcript,
session.companyId
);
// Calculate messagesSent from actual Message records
const messagesSent = await calculateMessagesSent(session.id);
// Calculate endTime from latest Message timestamp
const calculatedEndTime = await calculateEndTime(session.id, session.endTime);
const calculatedEndTime = await calculateEndTime(
session.id,
session.endTime
);
// Update the session with processed data
await prisma.session.update({
@ -426,23 +501,34 @@ async function processSingleSession(session: any): Promise<ProcessingResult> {
});
// Mark AI analysis as completed
await ProcessingStatusManager.completeStage(session.id, ProcessingStage.AI_ANALYSIS, {
language: processedData.language,
sentiment: processedData.sentiment,
category: processedData.category,
questionsCount: processedData.questions.length
});
await ProcessingStatusManager.completeStage(
session.id,
ProcessingStage.AI_ANALYSIS,
{
language: processedData.language,
sentiment: processedData.sentiment,
category: processedData.category,
questionsCount: processedData.questions.length,
}
);
// Start question extraction stage
await ProcessingStatusManager.startStage(session.id, ProcessingStage.QUESTION_EXTRACTION);
await ProcessingStatusManager.startStage(
session.id,
ProcessingStage.QUESTION_EXTRACTION
);
// Process questions into separate tables
await processQuestions(session.id, processedData.questions);
// Mark question extraction as completed
await ProcessingStatusManager.completeStage(session.id, ProcessingStage.QUESTION_EXTRACTION, {
questionsProcessed: processedData.questions.length
});
await ProcessingStatusManager.completeStage(
session.id,
ProcessingStage.QUESTION_EXTRACTION,
{
questionsProcessed: processedData.questions.length,
}
);
return {
sessionId: session.id,
@ -467,7 +553,10 @@ async function processSingleSession(session: any): Promise<ProcessingResult> {
/**
* Process sessions in parallel with concurrency limit
*/
async function processSessionsInParallel(sessions: any[], maxConcurrency: number = 5): Promise<ProcessingResult[]> {
async function processSessionsInParallel(
sessions: any[],
maxConcurrency: number = 5
): Promise<ProcessingResult[]> {
const results: Promise<ProcessingResult>[] = [];
const executing: Promise<ProcessingResult>[] = [];
@ -486,7 +575,7 @@ async function processSessionsInParallel(sessions: any[], maxConcurrency: number
if (executing.length >= maxConcurrency) {
await Promise.race(executing);
const completedIndex = executing.findIndex(p => p === promise);
const completedIndex = executing.findIndex((p) => p === promise);
if (completedIndex !== -1) {
executing.splice(completedIndex, 1);
}
@ -499,27 +588,37 @@ async function processSessionsInParallel(sessions: any[], maxConcurrency: number
/**
* Process unprocessed sessions using the new processing status system
*/
export async function processUnprocessedSessions(batchSize: number | null = null, maxConcurrency: number = 5): Promise<void> {
process.stdout.write("[ProcessingScheduler] Starting to process sessions needing AI analysis...\n");
// Get sessions that need AI processing using the new status system
const sessionsNeedingAI = await ProcessingStatusManager.getSessionsNeedingProcessing(
ProcessingStage.AI_ANALYSIS,
batchSize || 50
export async function processUnprocessedSessions(
batchSize: number | null = null,
maxConcurrency: number = 5
): Promise<void> {
process.stdout.write(
"[ProcessingScheduler] Starting to process sessions needing AI analysis...\n"
);
// Get sessions that need AI processing using the new status system
const sessionsNeedingAI =
await ProcessingStatusManager.getSessionsNeedingProcessing(
ProcessingStage.AI_ANALYSIS,
batchSize || 50
);
if (sessionsNeedingAI.length === 0) {
process.stdout.write("[ProcessingScheduler] No sessions found requiring AI processing.\n");
process.stdout.write(
"[ProcessingScheduler] No sessions found requiring AI processing.\n"
);
return;
}
// Get session IDs that need processing
const sessionIds = sessionsNeedingAI.map(statusRecord => statusRecord.sessionId);
const sessionIds = sessionsNeedingAI.map(
(statusRecord) => statusRecord.sessionId
);
// Fetch full session data with messages
const sessionsToProcess = await prisma.session.findMany({
where: {
id: { in: sessionIds }
id: { in: sessionIds },
},
include: {
messages: {
@ -534,7 +633,9 @@ export async function processUnprocessedSessions(batchSize: number | null = null
);
if (sessionsWithMessages.length === 0) {
process.stdout.write("[ProcessingScheduler] No sessions with messages found requiring processing.\n");
process.stdout.write(
"[ProcessingScheduler] No sessions with messages found requiring processing.\n"
);
return;
}
@ -543,16 +644,25 @@ export async function processUnprocessedSessions(batchSize: number | null = null
);
const startTime = Date.now();
const results = await processSessionsInParallel(sessionsWithMessages, maxConcurrency);
const results = await processSessionsInParallel(
sessionsWithMessages,
maxConcurrency
);
const endTime = Date.now();
const successCount = results.filter((r) => r.success).length;
const errorCount = results.filter((r) => !r.success).length;
process.stdout.write("[ProcessingScheduler] Session processing complete.\n");
process.stdout.write(`[ProcessingScheduler] Successfully processed: ${successCount} sessions.\n`);
process.stdout.write(`[ProcessingScheduler] Failed to process: ${errorCount} sessions.\n`);
process.stdout.write(`[ProcessingScheduler] Total processing time: ${((endTime - startTime) / 1000).toFixed(2)}s\n`);
process.stdout.write(
`[ProcessingScheduler] Successfully processed: ${successCount} sessions.\n`
);
process.stdout.write(
`[ProcessingScheduler] Failed to process: ${errorCount} sessions.\n`
);
process.stdout.write(
`[ProcessingScheduler] Total processing time: ${((endTime - startTime) / 1000).toFixed(2)}s\n`
);
}
/**
@ -576,11 +686,11 @@ export async function getAIProcessingCosts(): Promise<{
});
const successfulRequests = await prisma.aIProcessingRequest.count({
where: { success: true }
where: { success: true },
});
const failedRequests = await prisma.aIProcessingRequest.count({
where: { success: false }
where: { success: false },
});
return {
@ -599,22 +709,32 @@ export function startProcessingScheduler(): void {
const config = getSchedulerConfig();
if (!config.enabled) {
console.log('[Processing Scheduler] Disabled via configuration');
console.log("[Processing Scheduler] Disabled via configuration");
return;
}
console.log(`[Processing Scheduler] Starting with interval: ${config.sessionProcessing.interval}`);
console.log(`[Processing Scheduler] Batch size: ${config.sessionProcessing.batchSize === 0 ? 'unlimited' : config.sessionProcessing.batchSize}`);
console.log(`[Processing Scheduler] Concurrency: ${config.sessionProcessing.concurrency}`);
console.log(
`[Processing Scheduler] Starting with interval: ${config.sessionProcessing.interval}`
);
console.log(
`[Processing Scheduler] Batch size: ${config.sessionProcessing.batchSize === 0 ? "unlimited" : config.sessionProcessing.batchSize}`
);
console.log(
`[Processing Scheduler] Concurrency: ${config.sessionProcessing.concurrency}`
);
cron.schedule(config.sessionProcessing.interval, async () => {
try {
await processUnprocessedSessions(
config.sessionProcessing.batchSize === 0 ? null : config.sessionProcessing.batchSize,
config.sessionProcessing.batchSize === 0
? null
: config.sessionProcessing.batchSize,
config.sessionProcessing.concurrency
);
} catch (error) {
process.stderr.write(`[ProcessingScheduler] Error in scheduler: ${error}\n`);
process.stderr.write(
`[ProcessingScheduler] Error in scheduler: ${error}\n`
);
}
});
}

44
lib/processingStatusManager.ts
View File

@ -1,4 +1,8 @@
import { PrismaClient, ProcessingStage, ProcessingStatus } from '@prisma/client';
import {
PrismaClient,
ProcessingStage,
ProcessingStatus,
} from "@prisma/client";
const prisma = new PrismaClient();
@ -6,7 +10,6 @@ const prisma = new PrismaClient();
* Centralized processing status management
*/
export class ProcessingStatusManager {
/**
* Initialize processing status for a session with all stages set to PENDING
*/
@ -21,7 +24,7 @@ export class ProcessingStatusManager {
// Create all processing status records for this session
await prisma.sessionProcessingStatus.createMany({
data: stages.map(stage => ({
data: stages.map((stage) => ({
sessionId,
stage,
status: ProcessingStatus.PENDING,
@ -40,7 +43,7 @@ export class ProcessingStatusManager {
): Promise<void> {
await prisma.sessionProcessingStatus.upsert({
where: {
sessionId_stage: { sessionId, stage }
sessionId_stage: { sessionId, stage },
},
update: {
status: ProcessingStatus.IN_PROGRESS,
@ -68,7 +71,7 @@ export class ProcessingStatusManager {
): Promise<void> {
await prisma.sessionProcessingStatus.upsert({
where: {
sessionId_stage: { sessionId, stage }
sessionId_stage: { sessionId, stage },
},
update: {
status: ProcessingStatus.COMPLETED,
@ -98,7 +101,7 @@ export class ProcessingStatusManager {
): Promise<void> {
await prisma.sessionProcessingStatus.upsert({
where: {
sessionId_stage: { sessionId, stage }
sessionId_stage: { sessionId, stage },
},
update: {
status: ProcessingStatus.FAILED,
@ -130,7 +133,7 @@ export class ProcessingStatusManager {
): Promise<void> {
await prisma.sessionProcessingStatus.upsert({
where: {
sessionId_stage: { sessionId, stage }
sessionId_stage: { sessionId, stage },
},
update: {
status: ProcessingStatus.SKIPPED,
@ -154,7 +157,7 @@ export class ProcessingStatusManager {
static async getSessionStatus(sessionId: string) {
return await prisma.sessionProcessingStatus.findMany({
where: { sessionId },
orderBy: { stage: 'asc' },
orderBy: { stage: "asc" },
});
}
@ -179,7 +182,7 @@ export class ProcessingStatusManager {
},
},
take: limit,
orderBy: { session: { createdAt: 'asc' } },
orderBy: { session: { createdAt: "asc" } },
});
}
@ -189,7 +192,7 @@ export class ProcessingStatusManager {
static async getPipelineStatus() {
// Get counts by stage and status
const statusCounts = await prisma.sessionProcessingStatus.groupBy({
by: ['stage', 'status'],
by: ["stage", "status"],
_count: { id: true },
});
@ -233,17 +236,20 @@ export class ProcessingStatusManager {
},
},
},
orderBy: { completedAt: 'desc' },
orderBy: { completedAt: "desc" },
});
}
/**
* Reset a failed stage for retry
*/
static async resetStageForRetry(sessionId: string, stage: ProcessingStage): Promise<void> {
static async resetStageForRetry(
sessionId: string,
stage: ProcessingStage
): Promise<void> {
await prisma.sessionProcessingStatus.update({
where: {
sessionId_stage: { sessionId, stage }
sessionId_stage: { sessionId, stage },
},
data: {
status: ProcessingStatus.PENDING,
@ -257,10 +263,13 @@ export class ProcessingStatusManager {
/**
* Check if a session has completed a specific stage
*/
static async hasCompletedStage(sessionId: string, stage: ProcessingStage): Promise<boolean> {
static async hasCompletedStage(
sessionId: string,
stage: ProcessingStage
): Promise<boolean> {
const status = await prisma.sessionProcessingStatus.findUnique({
where: {
sessionId_stage: { sessionId, stage }
sessionId_stage: { sessionId, stage },
},
});
@ -270,7 +279,10 @@ export class ProcessingStatusManager {
/**
* Check if a session is ready for a specific stage (previous stages completed)
*/
static async isReadyForStage(sessionId: string, stage: ProcessingStage): Promise<boolean> {
static async isReadyForStage(
sessionId: string,
stage: ProcessingStage
): Promise<boolean> {
const stageOrder = [
ProcessingStage.CSV_IMPORT,
ProcessingStage.TRANSCRIPT_FETCH,

6
lib/scheduler.ts
View File

@ -8,11 +8,13 @@ export function startCsvImportScheduler() {
const config = getSchedulerConfig();
if (!config.enabled) {
console.log('[CSV Import Scheduler] Disabled via configuration');
console.log("[CSV Import Scheduler] Disabled via configuration");
return;
}
console.log(`[CSV Import Scheduler] Starting with interval: ${config.csvImport.interval}`);
console.log(
`[CSV Import Scheduler] Starting with interval: ${config.csvImport.interval}`
);
cron.schedule(config.csvImport.interval, async () => {
const companies = await prisma.company.findMany();

5
lib/schedulerConfig.ts
View File

@ -1,7 +1,10 @@
// Legacy scheduler configuration - now uses centralized env management
// This file is kept for backward compatibility but delegates to lib/env.ts
import { getSchedulerConfig as getEnvSchedulerConfig, logEnvConfig } from "./env";
import {
getSchedulerConfig as getEnvSchedulerConfig,
logEnvConfig,
} from "./env";
export interface SchedulerConfig {
enabled: boolean;

25
lib/transcriptFetcher.ts
View File

@ -23,7 +23,7 @@ export async function fetchTranscriptContent(
if (!url || !url.trim()) {
return {
success: false,
error: 'No transcript URL provided',
error: "No transcript URL provided",
};
}
@ -34,7 +34,7 @@ export async function fetchTranscriptContent(
: undefined;
const headers: Record<string, string> = {
'User-Agent': 'LiveDash-Transcript-Fetcher/1.0',
"User-Agent": "LiveDash-Transcript-Fetcher/1.0",
};
if (authHeader) {
@ -46,7 +46,7 @@ export async function fetchTranscriptContent(
const timeoutId = setTimeout(() => controller.abort(), 30000); // 30 second timeout
const response = await fetch(url, {
method: 'GET',
method: "GET",
headers,
signal: controller.signal,
});
@ -65,7 +65,7 @@ export async function fetchTranscriptContent(
if (!content || content.trim().length === 0) {
return {
success: false,
error: 'Empty transcript content',
error: "Empty transcript content",
};
}
@ -73,29 +73,28 @@ export async function fetchTranscriptContent(
success: true,
content: content.trim(),
};
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
// Handle common network errors
if (errorMessage.includes('ENOTFOUND')) {
if (errorMessage.includes("ENOTFOUND")) {
return {
success: false,
error: 'Domain not found',
error: "Domain not found",
};
}
if (errorMessage.includes('ECONNREFUSED')) {
if (errorMessage.includes("ECONNREFUSED")) {
return {
success: false,
error: 'Connection refused',
error: "Connection refused",
};
}
if (errorMessage.includes('timeout')) {
if (errorMessage.includes("timeout")) {
return {
success: false,
error: 'Request timeout',
error: "Request timeout",
};
}
@ -112,13 +111,13 @@ export async function fetchTranscriptContent(
* @returns boolean indicating if URL appears valid
*/
export function isValidTranscriptUrl(url: string): boolean {
if (!url || typeof url !== 'string') {
if (!url || typeof url !== "string") {
return false;
}
try {
const parsedUrl = new URL(url);
return parsedUrl.protocol === 'http:' || parsedUrl.protocol === 'https:';
return parsedUrl.protocol === "http:" || parsedUrl.protocol === "https:";
} catch {
return false;
}

142
lib/transcriptParser.ts
View File

@ -1,5 +1,5 @@
// Transcript parsing utility for converting raw transcript content into structured messages
import { prisma } from './prisma.js';
import { prisma } from "./prisma.js";
export interface ParsedMessage {
sessionId: string;
@ -19,7 +19,9 @@ export interface TranscriptParseResult {
* Parse European date format (DD.MM.YYYY HH:mm:ss) to Date object
*/
function parseEuropeanDate(dateStr: string): Date {
const match = dateStr.match(/(\d{2})\.(\d{2})\.(\d{4}) (\d{2}):(\d{2}):(\d{2})/);
const match = dateStr.match(
/(\d{2})\.(\d{2})\.(\d{4}) (\d{2}):(\d{2}):(\d{2})/
);
if (!match) {
throw new Error(`Invalid date format: ${dateStr}`);
}
@ -51,13 +53,17 @@ export function parseTranscriptToMessages(
if (!content || !content.trim()) {
return {
success: false,
error: 'Empty transcript content'
error: "Empty transcript content",
};
}
const messages: ParsedMessage[] = [];
const lines = content.split('\n');
let currentMessage: { role: string; content: string; timestamp?: string } | null = null;
const lines = content.split("\n");
let currentMessage: {
role: string;
content: string;
timestamp?: string;
} | null = null;
let order = 0;
for (const line of lines) {
@ -69,56 +75,64 @@ export function parseTranscriptToMessages(
}
// Check if line starts with a timestamp and role [DD.MM.YYYY HH:MM:SS] Role: content
const timestampRoleMatch = trimmedLine.match(/^\[(\d{2}\.\d{2}\.\d{4} \d{2}:\d{2}:\d{2})\]\s+(User|Assistant|System|user|assistant|system):\s*(.*)$/i);
const timestampRoleMatch = trimmedLine.match(
/^\[(\d{2}\.\d{2}\.\d{4} \d{2}:\d{2}:\d{2})\]\s+(User|Assistant|System|user|assistant|system):\s*(.*)$/i
);
// Check if line starts with just a role (User:, Assistant:, System:, etc.)
const roleMatch = trimmedLine.match(/^(User|Assistant|System|user|assistant|system):\s*(.*)$/i);
const roleMatch = trimmedLine.match(
/^(User|Assistant|System|user|assistant|system):\s*(.*)$/i
);
if (timestampRoleMatch) {
// Save previous message if exists
if (currentMessage) {
messages.push({
sessionId: '', // Will be set by caller
sessionId: "", // Will be set by caller
timestamp: new Date(), // Will be calculated below
role: currentMessage.role,
content: currentMessage.content.trim(),
order: order++
order: order++,
});
}
// Start new message with timestamp
const timestamp = timestampRoleMatch[1];
const role = timestampRoleMatch[2].charAt(0).toUpperCase() + timestampRoleMatch[2].slice(1).toLowerCase();
const content = timestampRoleMatch[3] || '';
const role =
timestampRoleMatch[2].charAt(0).toUpperCase() +
timestampRoleMatch[2].slice(1).toLowerCase();
const content = timestampRoleMatch[3] || "";
currentMessage = {
role,
content,
timestamp // Store the timestamp for later parsing
timestamp, // Store the timestamp for later parsing
};
} else if (roleMatch) {
// Save previous message if exists
if (currentMessage) {
messages.push({
sessionId: '', // Will be set by caller
sessionId: "", // Will be set by caller
timestamp: new Date(), // Will be calculated below
role: currentMessage.role,
content: currentMessage.content.trim(),
order: order++
order: order++,
});
}
// Start new message without timestamp
const role = roleMatch[1].charAt(0).toUpperCase() + roleMatch[1].slice(1).toLowerCase();
const content = roleMatch[2] || '';
const role =
roleMatch[1].charAt(0).toUpperCase() +
roleMatch[1].slice(1).toLowerCase();
const content = roleMatch[2] || "";
currentMessage = {
role,
content
content,
};
} else if (currentMessage) {
// Continue previous message (multi-line)
currentMessage.content += '\n' + trimmedLine;
currentMessage.content += "\n" + trimmedLine;
}
// If no current message and no role match, skip the line (orphaned content)
}
@ -126,23 +140,23 @@ export function parseTranscriptToMessages(
// Save the last message
if (currentMessage) {
messages.push({
sessionId: '', // Will be set by caller
sessionId: "", // Will be set by caller
timestamp: new Date(), // Will be calculated below
role: currentMessage.role,
content: currentMessage.content.trim(),
order: order++
order: order++,
});
}
if (messages.length === 0) {
return {
success: false,
error: 'No messages found in transcript'
error: "No messages found in transcript",
};
}
// Calculate timestamps - use parsed timestamps if available, otherwise distribute across session duration
const hasTimestamps = messages.some(msg => (msg as any).timestamp);
const hasTimestamps = messages.some((msg) => (msg as any).timestamp);
if (hasTimestamps) {
// Use parsed timestamps from the transcript
@ -154,35 +168,45 @@ export function parseTranscriptToMessages(
} catch (error) {
// Fallback to distributed timestamp if parsing fails
const sessionDurationMs = endTime.getTime() - startTime.getTime();
const messageInterval = messages.length > 1 ? sessionDurationMs / (messages.length - 1) : 0;
message.timestamp = new Date(startTime.getTime() + (index * messageInterval));
const messageInterval =
messages.length > 1
? sessionDurationMs / (messages.length - 1)
: 0;
message.timestamp = new Date(
startTime.getTime() + index * messageInterval
);
}
} else {
// Fallback to distributed timestamp
const sessionDurationMs = endTime.getTime() - startTime.getTime();
const messageInterval = messages.length > 1 ? sessionDurationMs / (messages.length - 1) : 0;
message.timestamp = new Date(startTime.getTime() + (index * messageInterval));
const messageInterval =
messages.length > 1 ? sessionDurationMs / (messages.length - 1) : 0;
message.timestamp = new Date(
startTime.getTime() + index * messageInterval
);
}
});
} else {
// Distribute messages across session duration
const sessionDurationMs = endTime.getTime() - startTime.getTime();
const messageInterval = messages.length > 1 ? sessionDurationMs / (messages.length - 1) : 0;
const messageInterval =
messages.length > 1 ? sessionDurationMs / (messages.length - 1) : 0;
messages.forEach((message, index) => {
message.timestamp = new Date(startTime.getTime() + (index * messageInterval));
message.timestamp = new Date(
startTime.getTime() + index * messageInterval
);
});
}
return {
success: true,
messages
messages,
};
} catch (error) {
return {
success: false,
error: error instanceof Error ? error.message : String(error)
error: error instanceof Error ? error.message : String(error),
};
}
}
@ -198,17 +222,17 @@ export async function storeMessagesForSession(
): Promise<void> {
// Delete existing messages for this session (in case of re-processing)
await prisma.message.deleteMany({
where: { sessionId }
where: { sessionId },
});
// Create new messages
const messagesWithSessionId = messages.map(msg => ({
const messagesWithSessionId = messages.map((msg) => ({
...msg,
sessionId
sessionId,
}));
await prisma.message.createMany({
data: messagesWithSessionId
data: messagesWithSessionId,
});
}
@ -216,13 +240,15 @@ export async function storeMessagesForSession(
* Process transcript for a single session
* @param sessionId The session ID to process
*/
export async function processSessionTranscript(sessionId: string): Promise<void> {
export async function processSessionTranscript(
sessionId: string
): Promise<void> {
// Get the session and its import data
const session = await prisma.session.findUnique({
where: { id: sessionId },
include: {
import: true
}
import: true,
},
});
if (!session) {
@ -255,35 +281,37 @@ export async function processSessionTranscript(sessionId: string): Promise<void>
// Store the messages
await storeMessagesForSession(sessionId, parseResult.messages!);
console.log(`✅ Processed ${parseResult.messages!.length} messages for session ${sessionId}`);
console.log(
`✅ Processed ${parseResult.messages!.length} messages for session ${sessionId}`
);
}
/**
* Process all sessions that have transcript content but no messages
*/
export async function processAllUnparsedTranscripts(): Promise<void> {
console.log('🔍 Finding sessions with unparsed transcripts...');
console.log("🔍 Finding sessions with unparsed transcripts...");
// Find sessions that have transcript content but no messages
const sessionsToProcess = await prisma.session.findMany({
where: {
import: {
rawTranscriptContent: {
not: null
}
not: null,
},
},
messages: {
none: {}
}
none: {},
},
},
include: {
import: true,
_count: {
select: {
messages: true
}
}
}
messages: true,
},
},
},
});
console.log(`📋 Found ${sessionsToProcess.length} sessions to process`);
@ -323,7 +351,7 @@ export async function getTotalMessageCount(): Promise<number> {
export async function getMessagesForSession(sessionId: string) {
return await prisma.message.findMany({
where: { sessionId },
orderBy: { order: 'asc' }
orderBy: { order: "asc" },
});
}
@ -336,17 +364,17 @@ export async function getParsingStats() {
where: {
import: {
rawTranscriptContent: {
not: null
}
}
}
not: null,
},
},
},
});
const sessionsWithMessages = await prisma.session.count({
where: {
messages: {
some: {}
}
}
some: {},
},
},
});
const totalMessages = await getTotalMessageCount();
@ -355,6 +383,6 @@ export async function getParsingStats() {
sessionsWithTranscripts,
sessionsWithMessages,
unparsedSessions: sessionsWithTranscripts - sessionsWithMessages,
totalMessages
totalMessages,
};
}

6
lib/utils.ts
View File

@ -1,6 +1,6 @@
import { clsx, type ClassValue } from "clsx"
import { twMerge } from "tailwind-merge"
import { clsx, type ClassValue } from "clsx";
import { twMerge } from "tailwind-merge";
export function cn(...inputs: ClassValue[]) {
return twMerge(clsx(inputs))
return twMerge(clsx(inputs));
}

134
lib/validation.ts Normal file
View File

@ -0,0 +1,134 @@
import { z } from "zod";
// Password validation with strong requirements
const passwordSchema = z
.string()
.min(12, "Password must be at least 12 characters long")
.regex(/^(?=.*[a-z])/, "Password must contain at least one lowercase letter")
.regex(/^(?=.*[A-Z])/, "Password must contain at least one uppercase letter")
.regex(/^(?=.*\d)/, "Password must contain at least one number")
.regex(
/^(?=.*[@$!%*?&])/,
"Password must contain at least one special character (@$!%*?&)"
);
// Email validation
const emailSchema = z
.string()
.email("Invalid email format")
.max(255, "Email must be less than 255 characters")
.toLowerCase();
// Company name validation
const companyNameSchema = z
.string()
.min(1, "Company name is required")
.max(100, "Company name must be less than 100 characters")
.regex(/^[a-zA-Z0-9\s\-_.]+$/, "Company name contains invalid characters");
// User registration schema
export const registerSchema = z.object({
email: emailSchema,
password: passwordSchema,
company: companyNameSchema,
});
// User login schema
export const loginSchema = z.object({
email: emailSchema,
password: z.string().min(1, "Password is required"),
});
// Password reset request schema
export const forgotPasswordSchema = z.object({
email: emailSchema,
});
// Password reset schema
export const resetPasswordSchema = z.object({
token: z.string().min(1, "Reset token is required"),
password: passwordSchema,
});
// Session filter schema
export const sessionFilterSchema = z.object({
search: z.string().max(100).optional(),
sentiment: z.enum(["POSITIVE", "NEUTRAL", "NEGATIVE"]).optional(),
category: z
.enum([
"SCHEDULE_HOURS",
"LEAVE_VACATION",
"SICK_LEAVE_RECOVERY",
"SALARY_COMPENSATION",
"CONTRACT_HOURS",
"ONBOARDING",
"OFFBOARDING",
"WORKWEAR_STAFF_PASS",
"TEAM_CONTACTS",
"PERSONAL_QUESTIONS",
"ACCESS_LOGIN",
"SOCIAL_QUESTIONS",
"UNRECOGNIZED_OTHER",
])
.optional(),
startDate: z.string().datetime().optional(),
endDate: z.string().datetime().optional(),
page: z.number().int().min(1).default(1),
limit: z.number().int().min(1).max(100).default(20),
});
// Company settings schema
export const companySettingsSchema = z.object({
name: companyNameSchema,
csvUrl: z.string().url("Invalid CSV URL"),
csvUsername: z.string().max(100).optional(),
csvPassword: z.string().max(100).optional(),
sentimentAlert: z.number().min(0).max(1).optional(),
dashboardOpts: z.object({}).passthrough().optional(),
});
// User management schema
export const userUpdateSchema = z.object({
email: emailSchema.optional(),
role: z.enum(["ADMIN", "USER", "AUDITOR"]).optional(),
password: passwordSchema.optional(),
});
// Metrics query schema
export const metricsQuerySchema = z.object({
startDate: z.string().datetime().optional(),
endDate: z.string().datetime().optional(),
companyId: z.string().uuid().optional(),
});
// Helper function to validate and sanitize input
export function validateInput<T>(
schema: z.ZodSchema<T>,
data: unknown
): { success: true; data: T } | { success: false; errors: string[] } {
try {
const result = schema.parse(data);
return { success: true, data: result };
} catch (error) {
if (error instanceof z.ZodError) {
const errors = error.errors.map(
(err) => `${err.path.join(".")}: ${err.message}`
);
return { success: false, errors };
}
return { success: false, errors: ["Invalid input"] };
}
}
// Rate limiting helper types
export interface RateLimitConfig {
windowMs: number;
maxRequests: number;
skipSuccessfulRequests?: boolean;
}
export const rateLimitConfigs = {
auth: { windowMs: 15 * 60 * 1000, maxRequests: 5 }, // 5 requests per 15 minutes
registration: { windowMs: 60 * 60 * 1000, maxRequests: 3 }, // 3 registrations per hour
api: { windowMs: 15 * 60 * 1000, maxRequests: 100 }, // 100 API requests per 15 minutes
} as const;

177
migrate-to-refactored-system.ts
View File

@ -1,12 +1,16 @@
import { PrismaClient, ProcessingStage, ProcessingStatus } from '@prisma/client';
import { ProcessingStatusManager } from './lib/processingStatusManager';
import {
PrismaClient,
ProcessingStage,
ProcessingStatus,
} from "@prisma/client";
import { ProcessingStatusManager } from "./lib/processingStatusManager";
const prisma = new PrismaClient();
async function migrateToRefactoredSystem() {
try {
console.log('=== MIGRATING TO REFACTORED PROCESSING SYSTEM ===\n');
console.log("=== MIGRATING TO REFACTORED PROCESSING SYSTEM ===\n");
// Get all existing sessions
const sessions = await prisma.session.findMany({
include: {
@ -14,113 +18,166 @@ async function migrateToRefactoredSystem() {
messages: true,
sessionQuestions: true,
},
orderBy: { createdAt: 'asc' }
orderBy: { createdAt: "asc" },
});
console.log(`Found ${sessions.length} sessions to migrate...\n`);
let migratedCount = 0;
for (const session of sessions) {
console.log(`Migrating session ${session.import?.externalSessionId || session.id}...`);
console.log(
`Migrating session ${session.import?.externalSessionId || session.id}...`
);
// Initialize processing status for this session
await ProcessingStatusManager.initializeSession(session.id);
// Determine the current state of each stage based on existing data
// 1. CSV_IMPORT - Always completed if session exists
await ProcessingStatusManager.completeStage(session.id, ProcessingStage.CSV_IMPORT, {
migratedFrom: 'existing_session',
importId: session.importId
});
await ProcessingStatusManager.completeStage(
session.id,
ProcessingStage.CSV_IMPORT,
{
migratedFrom: "existing_session",
importId: session.importId,
}
);
// 2. TRANSCRIPT_FETCH - Check if transcript content exists
if (session.import?.rawTranscriptContent) {
await ProcessingStatusManager.completeStage(session.id, ProcessingStage.TRANSCRIPT_FETCH, {
migratedFrom: 'existing_transcript',
contentLength: session.import.rawTranscriptContent.length
});
await ProcessingStatusManager.completeStage(
session.id,
ProcessingStage.TRANSCRIPT_FETCH,
{
migratedFrom: "existing_transcript",
contentLength: session.import.rawTranscriptContent.length,
}
);
} else if (!session.import?.fullTranscriptUrl) {
// No transcript URL - skip this stage
await ProcessingStatusManager.skipStage(session.id, ProcessingStage.TRANSCRIPT_FETCH, 'No transcript URL in original import');
await ProcessingStatusManager.skipStage(
session.id,
ProcessingStage.TRANSCRIPT_FETCH,
"No transcript URL in original import"
);
} else {
// Has URL but no content - mark as pending for retry
console.log(` - Transcript fetch pending for ${session.import.externalSessionId}`);
console.log(
` - Transcript fetch pending for ${session.import.externalSessionId}`
);
}
// 3. SESSION_CREATION - Check if messages exist
if (session.messages.length > 0) {
await ProcessingStatusManager.completeStage(session.id, ProcessingStage.SESSION_CREATION, {
migratedFrom: 'existing_messages',
messageCount: session.messages.length
});
await ProcessingStatusManager.completeStage(
session.id,
ProcessingStage.SESSION_CREATION,
{
migratedFrom: "existing_messages",
messageCount: session.messages.length,
}
);
} else if (session.import?.rawTranscriptContent) {
// Has transcript but no messages - needs reprocessing
console.log(` - Session creation pending for ${session.import.externalSessionId} (has transcript but no messages)`);
console.log(
` - Session creation pending for ${session.import.externalSessionId} (has transcript but no messages)`
);
} else {
// No transcript content - skip or mark as pending based on transcript fetch status
if (!session.import?.fullTranscriptUrl) {
await ProcessingStatusManager.skipStage(session.id, ProcessingStage.SESSION_CREATION, 'No transcript content available');
await ProcessingStatusManager.skipStage(
session.id,
ProcessingStage.SESSION_CREATION,
"No transcript content available"
);
}
}
// 4. AI_ANALYSIS - Check if AI fields are populated
const hasAIAnalysis = session.summary || session.sentiment || session.category || session.language;
const hasAIAnalysis =
session.summary ||
session.sentiment ||
session.category ||
session.language;
if (hasAIAnalysis) {
await ProcessingStatusManager.completeStage(session.id, ProcessingStage.AI_ANALYSIS, {
migratedFrom: 'existing_ai_analysis',
hasSummary: !!session.summary,
hasSentiment: !!session.sentiment,
hasCategory: !!session.category,
hasLanguage: !!session.language
});
await ProcessingStatusManager.completeStage(
session.id,
ProcessingStage.AI_ANALYSIS,
{
migratedFrom: "existing_ai_analysis",
hasSummary: !!session.summary,
hasSentiment: !!session.sentiment,
hasCategory: !!session.category,
hasLanguage: !!session.language,
}
);
} else {
// No AI analysis - mark as pending if session creation is complete
if (session.messages.length > 0) {
console.log(` - AI analysis pending for ${session.import?.externalSessionId}`);
console.log(
` - AI analysis pending for ${session.import?.externalSessionId}`
);
}
}
// 5. QUESTION_EXTRACTION - Check if questions exist
if (session.sessionQuestions.length > 0) {
await ProcessingStatusManager.completeStage(session.id, ProcessingStage.QUESTION_EXTRACTION, {
migratedFrom: 'existing_questions',
questionCount: session.sessionQuestions.length
});
await ProcessingStatusManager.completeStage(
session.id,
ProcessingStage.QUESTION_EXTRACTION,
{
migratedFrom: "existing_questions",
questionCount: session.sessionQuestions.length,
}
);
} else {
// No questions - mark as pending if AI analysis is complete
if (hasAIAnalysis) {
console.log(` - Question extraction pending for ${session.import?.externalSessionId}`);
console.log(
` - Question extraction pending for ${session.import?.externalSessionId}`
);
}
}
migratedCount++;
if (migratedCount % 10 === 0) {
console.log(` Migrated ${migratedCount}/${sessions.length} sessions...`);
console.log(
` Migrated ${migratedCount}/${sessions.length} sessions...`
);
}
}
console.log(`\n✓ Successfully migrated ${migratedCount} sessions to the new processing system`);
console.log(
`\n✓ Successfully migrated ${migratedCount} sessions to the new processing system`
);
// Show final status
console.log('\n=== MIGRATION COMPLETE - FINAL STATUS ===');
console.log("\n=== MIGRATION COMPLETE - FINAL STATUS ===");
const pipelineStatus = await ProcessingStatusManager.getPipelineStatus();
const stages = ['CSV_IMPORT', 'TRANSCRIPT_FETCH', 'SESSION_CREATION', 'AI_ANALYSIS', 'QUESTION_EXTRACTION'];
const stages = [
"CSV_IMPORT",
"TRANSCRIPT_FETCH",
"SESSION_CREATION",
"AI_ANALYSIS",
"QUESTION_EXTRACTION",
];
for (const stage of stages) {
const stageData = pipelineStatus.pipeline[stage] || {};
const pending = stageData.PENDING || 0;
const completed = stageData.COMPLETED || 0;
const skipped = stageData.SKIPPED || 0;
console.log(`${stage}: ${completed} completed, ${pending} pending, ${skipped} skipped`);
console.log(
`${stage}: ${completed} completed, ${pending} pending, ${skipped} skipped`
);
}
} catch (error) {
console.error('Error migrating to refactored system:', error);
console.error("Error migrating to refactored system:", error);
} finally {
await prisma.$disconnect();
}

5
next.config.js
View File

@ -4,10 +4,7 @@
const nextConfig = {
reactStrictMode: true,
// Allow cross-origin requests from specific origins in development
allowedDevOrigins: [
"localhost",
"127.0.0.1"
],
allowedDevOrigins: ["localhost", "127.0.0.1"],
};
export default nextConfig;

4
package.json
View File

@ -59,7 +59,8 @@
"react-markdown": "^10.1.0",
"recharts": "^3.0.2",
"rehype-raw": "^7.0.0",
"tailwind-merge": "^3.3.1"
"tailwind-merge": "^3.3.1",
"zod": "^3.25.67"
},
"devDependencies": {
"@eslint/eslintrc": "^3.3.1",
@ -68,7 +69,6 @@
"@tailwindcss/postcss": "^4.1.11",
"@testing-library/dom": "^10.4.0",
"@testing-library/react": "^16.3.0",
"@types/bcryptjs": "^3.0.0",
"@types/node": "^24.0.6",
"@types/node-cron": "^3.0.11",
"@types/react": "^19.1.8",

8605
pnpm-lock.yaml generated
View File

File diff suppressed because it is too large Load Diff

570
prisma/schema.prisma
View File

@ -1,5 +1,5 @@
generator client {
provider = "prisma-client-js"
provider = "prisma-client-js"
previewFeatures = ["driverAdapters"]
}
@ -9,9 +9,242 @@ datasource db {
directUrl = env("DATABASE_URL_DIRECT")
}
/**
* ENUMS – fewer magic strings
*/
/// *
/// * COMPANY (multi-tenant root)
model Company {
id String @id @default(uuid())
name String
csvUrl String
csvUsername String?
csvPassword String?
sentimentAlert Float?
dashboardOpts Json?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
companyAiModels CompanyAIModel[]
sessions Session[]
imports SessionImport[]
users User[] @relation("CompanyUsers")
}
/// *
/// * USER (auth accounts)
model User {
id String @id @default(uuid())
email String @unique
password String
role UserRole @default(USER)
companyId String
resetToken String?
resetTokenExpiry DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
company Company @relation("CompanyUsers", fields: [companyId], references: [id], onDelete: Cascade)
}
/// *
/// * 1. Normalised session ---------------------------
model Session {
id String @id @default(uuid())
companyId String
importId String? @unique
/// *
/// * session-level data (processed from SessionImport)
startTime DateTime
endTime DateTime
ipAddress String?
country String?
fullTranscriptUrl String?
avgResponseTime Float?
initialMsg String?
language String?
messagesSent Int?
sentiment SentimentCategory?
escalated Boolean?
forwardedHr Boolean?
category SessionCategory?
summary String?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
aiProcessingRequests AIProcessingRequest[]
messages Message[]
company Company @relation(fields: [companyId], references: [id], onDelete: Cascade)
import SessionImport? @relation("ImportToSession", fields: [importId], references: [id])
processingStatus SessionProcessingStatus[]
sessionQuestions SessionQuestion[]
@@index([companyId, startTime])
}
/// *
/// * 2. Raw CSV row (pure data storage) ----------
model SessionImport {
id String @id @default(uuid())
companyId String
externalSessionId String @unique
startTimeRaw String
endTimeRaw String
ipAddress String?
countryCode String?
language String?
messagesSent Int?
sentimentRaw String?
escalatedRaw String?
forwardedHrRaw String?
fullTranscriptUrl String?
avgResponseTimeSeconds Float?
tokens Int?
tokensEur Float?
category String?
initialMessage String?
rawTranscriptContent String?
createdAt DateTime @default(now())
session Session? @relation("ImportToSession")
company Company @relation(fields: [companyId], references: [id], onDelete: Cascade)
@@unique([companyId, externalSessionId])
}
/// *
/// * MESSAGE (individual lines)
model Message {
id String @id @default(uuid())
sessionId String
timestamp DateTime?
role String
content String
order Int
createdAt DateTime @default(now())
session Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
@@unique([sessionId, order])
@@index([sessionId, order])
}
/// *
/// * UNIFIED PROCESSING STATUS TRACKING
model SessionProcessingStatus {
id String @id @default(uuid())
sessionId String
stage ProcessingStage
status ProcessingStatus @default(PENDING)
startedAt DateTime?
completedAt DateTime?
errorMessage String?
retryCount Int @default(0)
metadata Json?
session Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
@@unique([sessionId, stage])
@@index([stage, status])
@@index([sessionId])
}
/// *
/// * QUESTION MANAGEMENT (separate from Session for better analytics)
model Question {
id String @id @default(uuid())
content String @unique
createdAt DateTime @default(now())
sessionQuestions SessionQuestion[]
}
model SessionQuestion {
id String @id @default(uuid())
sessionId String
questionId String
order Int
createdAt DateTime @default(now())
question Question @relation(fields: [questionId], references: [id])
session Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
@@unique([sessionId, questionId])
@@unique([sessionId, order])
@@index([sessionId])
}
/// *
/// * AI PROCESSING COST TRACKING
model AIProcessingRequest {
id String @id @default(uuid())
sessionId String
openaiRequestId String?
model String
serviceTier String?
systemFingerprint String?
promptTokens Int
completionTokens Int
totalTokens Int
cachedTokens Int?
audioTokensPrompt Int?
reasoningTokens Int?
audioTokensCompletion Int?
acceptedPredictionTokens Int?
rejectedPredictionTokens Int?
promptTokenCost Float
completionTokenCost Float
totalCostEur Float
processingType String
success Boolean
errorMessage String?
requestedAt DateTime @default(now())
completedAt DateTime?
session Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
@@index([sessionId])
@@index([requestedAt])
@@index([model])
}
/// *
/// * AI Model definitions (without pricing)
model AIModel {
id String @id @default(uuid())
name String @unique
provider String
maxTokens Int?
isActive Boolean @default(true)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
pricing AIModelPricing[]
companyModels CompanyAIModel[]
@@index([provider, isActive])
}
/// *
/// * Time-based pricing for AI models
model AIModelPricing {
id String @id @default(uuid())
aiModelId String
promptTokenCost Float
completionTokenCost Float
effectiveFrom DateTime
effectiveUntil DateTime?
createdAt DateTime @default(now())
aiModel AIModel @relation(fields: [aiModelId], references: [id], onDelete: Cascade)
@@index([aiModelId, effectiveFrom])
@@index([effectiveFrom, effectiveUntil])
}
/// *
/// * Company-specific AI model assignments
model CompanyAIModel {
id String @id @default(uuid())
companyId String
aiModelId String
isDefault Boolean @default(false)
createdAt DateTime @default(now())
aiModel AIModel @relation(fields: [aiModelId], references: [id], onDelete: Cascade)
company Company @relation(fields: [companyId], references: [id], onDelete: Cascade)
@@unique([companyId, aiModelId])
@@index([companyId, isDefault])
}
/// *
/// * ENUMS – fewer magic strings
enum UserRole {
ADMIN
USER
@ -41,11 +274,11 @@ enum SessionCategory {
}
enum ProcessingStage {
CSV_IMPORT // SessionImport created
TRANSCRIPT_FETCH // Transcript content fetched
SESSION_CREATION // Session + Messages created
AI_ANALYSIS // AI processing completed
QUESTION_EXTRACTION // Questions extracted
CSV_IMPORT
TRANSCRIPT_FETCH
SESSION_CREATION
AI_ANALYSIS
QUESTION_EXTRACTION
}
enum ProcessingStatus {
@ -55,322 +288,3 @@ enum ProcessingStatus {
FAILED
SKIPPED
}
/**
* COMPANY (multi-tenant root)
*/
model Company {
id String @id @default(uuid())
name String
csvUrl String
csvUsername String?
csvPassword String?
sentimentAlert Float?
dashboardOpts Json? // JSON column instead of opaque string
users User[] @relation("CompanyUsers")
sessions Session[]
imports SessionImport[]
companyAiModels CompanyAIModel[]
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
}
/**
* USER (auth accounts)
*/
model User {
id String @id @default(uuid())
email String @unique
password String
role UserRole @default(USER)
company Company @relation("CompanyUsers", fields: [companyId], references: [id], onDelete: Cascade)
companyId String
resetToken String?
resetTokenExpiry DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
}
/**
* SESSION ↔ SESSIONIMPORT (1-to-1)
*/
/**
* 1. Normalised session ---------------------------
*/
model Session {
id String @id @default(uuid())
company Company @relation(fields: [companyId], references: [id], onDelete: Cascade)
companyId String
/**
* 1-to-1 link back to the import row
*/
import SessionImport? @relation("ImportToSession", fields: [importId], references: [id])
importId String? @unique
/**
* session-level data (processed from SessionImport)
*/
startTime DateTime
endTime DateTime
// Direct copies from SessionImport (minimal processing)
ipAddress String?
country String? // from countryCode
fullTranscriptUrl String?
avgResponseTime Float? // from avgResponseTimeSeconds
initialMsg String? // from initialMessage
// AI-processed fields (calculated from Messages or AI analysis)
language String? // AI-detected from Messages
messagesSent Int? // Calculated from Message count
sentiment SentimentCategory? // AI-analyzed (changed from Float to enum)
escalated Boolean? // AI-detected
forwardedHr Boolean? // AI-detected
category SessionCategory? // AI-categorized (changed to enum)
// AI-generated fields
summary String? // AI-generated summary
/**
* Relationships
*/
messages Message[] // Individual conversation messages
sessionQuestions SessionQuestion[] // Questions asked in this session
aiProcessingRequests AIProcessingRequest[] // AI processing cost tracking
processingStatus SessionProcessingStatus[] // Processing pipeline status
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@index([companyId, startTime])
}
/**
* 2. Raw CSV row (pure data storage) ----------
*/
model SessionImport {
id String @id @default(uuid())
company Company @relation(fields: [companyId], references: [id], onDelete: Cascade)
companyId String
/**
* 1-to-1 back-relation; NO fields/references here
*/
session Session? @relation("ImportToSession")
// ─── 16 CSV columns 1-to-1 ────────────────────────
externalSessionId String @unique // value from CSV column 1
startTimeRaw String
endTimeRaw String
ipAddress String?
countryCode String?
language String?
messagesSent Int?
sentimentRaw String?
escalatedRaw String?
forwardedHrRaw String?
fullTranscriptUrl String?
avgResponseTimeSeconds Float?
tokens Int?
tokensEur Float?
category String?
initialMessage String?
// ─── Raw transcript content ─────────────────────────
rawTranscriptContent String? // Fetched content from fullTranscriptUrl
// ─── bookkeeping ─────────────────────────────────
createdAt DateTime @default(now())
@@unique([companyId, externalSessionId]) // idempotent re-imports
}
/**
* MESSAGE (individual lines)
*/
model Message {
id String @id @default(uuid())
session Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
sessionId String
timestamp DateTime?
role String // "user" | "assistant" | "system" – free-form keeps migration easy
content String
order Int
createdAt DateTime @default(now())
@@unique([sessionId, order]) // guards against duplicate order values
@@index([sessionId, order])
}
/**
* UNIFIED PROCESSING STATUS TRACKING
*/
model SessionProcessingStatus {
id String @id @default(uuid())
sessionId String
stage ProcessingStage
status ProcessingStatus @default(PENDING)
startedAt DateTime?
completedAt DateTime?
errorMessage String?
retryCount Int @default(0)
// Stage-specific metadata (e.g., AI costs, token usage, fetch details)
metadata Json?
session Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
@@unique([sessionId, stage])
@@index([stage, status])
@@index([sessionId])
}
/**
* QUESTION MANAGEMENT (separate from Session for better analytics)
*/
model Question {
id String @id @default(uuid())
content String @unique // The actual question text
createdAt DateTime @default(now())
// Relationships
sessionQuestions SessionQuestion[]
}
model SessionQuestion {
id String @id @default(uuid())
sessionId String
questionId String
order Int // Order within the session
createdAt DateTime @default(now())
// Relationships
session Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
question Question @relation(fields: [questionId], references: [id])
@@unique([sessionId, questionId]) // Prevent duplicate questions per session
@@unique([sessionId, order]) // Ensure unique ordering
@@index([sessionId])
}
/**
* AI PROCESSING COST TRACKING
*/
model AIProcessingRequest {
id String @id @default(uuid())
sessionId String
// OpenAI Request Details
openaiRequestId String? // "chatcmpl-Bn8IH9UM8t7luZVWnwZG7CVJ0kjPo"
model String // "gpt-4o-2024-08-06"
serviceTier String? // "default"
systemFingerprint String? // "fp_07871e2ad8"
// Token Usage (from usage object)
promptTokens Int // 11
completionTokens Int // 9
totalTokens Int // 20
// Detailed Token Breakdown
cachedTokens Int? // prompt_tokens_details.cached_tokens
audioTokensPrompt Int? // prompt_tokens_details.audio_tokens
reasoningTokens Int? // completion_tokens_details.reasoning_tokens
audioTokensCompletion Int? // completion_tokens_details.audio_tokens
acceptedPredictionTokens Int? // completion_tokens_details.accepted_prediction_tokens
rejectedPredictionTokens Int? // completion_tokens_details.rejected_prediction_tokens
// Cost Calculation
promptTokenCost Float // Cost per prompt token (varies by model)
completionTokenCost Float // Cost per completion token (varies by model)
totalCostEur Float // Calculated total cost in EUR
// Processing Context
processingType String // "session_analysis", "reprocessing", etc.
success Boolean // Whether the request succeeded
errorMessage String? // If failed, what went wrong
// Timestamps
requestedAt DateTime @default(now())
completedAt DateTime?
// Relationships
session Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
@@index([sessionId])
@@index([requestedAt])
@@index([model])
}
/**
* AI MODEL MANAGEMENT SYSTEM
*/
/**
* AI Model definitions (without pricing)
*/
model AIModel {
id String @id @default(uuid())
name String @unique // "gpt-4o", "gpt-4-turbo", etc.
provider String // "openai", "anthropic", etc.
maxTokens Int? // Maximum tokens for this model
isActive Boolean @default(true)
// Relationships
pricing AIModelPricing[]
companyModels CompanyAIModel[]
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@index([provider, isActive])
}
/**
* Time-based pricing for AI models
*/
model AIModelPricing {
id String @id @default(uuid())
aiModelId String
promptTokenCost Float // Cost per prompt token in USD
completionTokenCost Float // Cost per completion token in USD
effectiveFrom DateTime // When this pricing becomes effective
effectiveUntil DateTime? // When this pricing expires (null = current)
// Relationships
aiModel AIModel @relation(fields: [aiModelId], references: [id], onDelete: Cascade)
createdAt DateTime @default(now())
@@index([aiModelId, effectiveFrom])
@@index([effectiveFrom, effectiveUntil])
}
/**
* Company-specific AI model assignments
*/
model CompanyAIModel {
id String @id @default(uuid())
companyId String
aiModelId String
isDefault Boolean @default(false) // Is this the default model for the company?
// Relationships
company Company @relation(fields: [companyId], references: [id], onDelete: Cascade)
aiModel AIModel @relation(fields: [aiModelId], references: [id], onDelete: Cascade)
createdAt DateTime @default(now())
@@unique([companyId, aiModelId]) // Prevent duplicate assignments
@@index([companyId, isDefault])
}

23
prisma/seed.ts
View File

@ -73,28 +73,28 @@ async function main() {
const pricingData = [
{
modelName: "gpt-4o",
promptTokenCost: 0.0000025, // $2.50 per 1M tokens
completionTokenCost: 0.00001, // $10.00 per 1M tokens
promptTokenCost: 0.0000025, // $2.50 per 1M tokens
completionTokenCost: 0.00001, // $10.00 per 1M tokens
},
{
modelName: "gpt-4o-2024-08-06",
promptTokenCost: 0.0000025, // $2.50 per 1M tokens
completionTokenCost: 0.00001, // $10.00 per 1M tokens
promptTokenCost: 0.0000025, // $2.50 per 1M tokens
completionTokenCost: 0.00001, // $10.00 per 1M tokens
},
{
modelName: "gpt-4-turbo",
promptTokenCost: 0.00001, // $10.00 per 1M tokens
completionTokenCost: 0.00003, // $30.00 per 1M tokens
promptTokenCost: 0.00001, // $10.00 per 1M tokens
completionTokenCost: 0.00003, // $30.00 per 1M tokens
},
{
modelName: "gpt-4o-mini",
promptTokenCost: 0.00000015, // $0.15 per 1M tokens
promptTokenCost: 0.00000015, // $0.15 per 1M tokens
completionTokenCost: 0.0000006, // $0.60 per 1M tokens
},
];
for (const pricing of pricingData) {
const model = createdModels.find(m => m.name === pricing.modelName);
const model = createdModels.find((m) => m.name === pricing.modelName);
if (model) {
await prisma.aIModelPricing.create({
data: {
@ -110,7 +110,7 @@ async function main() {
}
// Assign default AI model to company (gpt-4o)
const defaultModel = createdModels.find(m => m.name === "gpt-4o");
const defaultModel = createdModels.find((m) => m.name === "gpt-4o");
if (defaultModel) {
await prisma.companyAIModel.create({
data: {
@ -127,10 +127,11 @@ async function main() {
console.log(`Company: ${company.name}`);
console.log(`Admin user: ${adminUser.email}`);
console.log(`Password: 8QbL26tB7fWS`);
console.log(`AI Models: ${createdModels.length} models created with current pricing`);
console.log(
`AI Models: ${createdModels.length} models created with current pricing`
);
console.log(`Default model: ${defaultModel?.name}`);
console.log("\n🚀 Ready to start importing CSV data!");
} catch (error) {
console.error("❌ Error seeding database:", error);
process.exit(1);

31
scripts/fetch_transcripts.ts
View File

@ -22,7 +22,9 @@ async function fetchTranscriptContent(
});
if (!response.ok) {
console.warn(`Failed to fetch transcript from ${url}: ${response.statusText}`);
console.warn(
`Failed to fetch transcript from ${url}: ${response.statusText}`
);
return null;
}
@ -42,7 +44,7 @@ function parseTranscriptToMessages(transcriptContent: string): Array<{
content: string;
order: number;
}> {
const lines = transcriptContent.split('\n').filter(line => line.trim());
const lines = transcriptContent.split("\n").filter((line) => line.trim());
const messages: Array<{
timestamp: Date | null;
role: string;
@ -55,10 +57,10 @@ function parseTranscriptToMessages(transcriptContent: string): Array<{
for (const line of lines) {
// Try to parse lines in format: [timestamp] role: content
const match = line.match(/^\[([^\]]+)\]\s*([^:]+):\s*(.+)$/);
if (match) {
const [, timestampStr, role, content] = match;
// Try to parse the timestamp
let timestamp: Date | null = null;
try {
@ -79,12 +81,12 @@ function parseTranscriptToMessages(transcriptContent: string): Array<{
} else {
// If line doesn't match expected format, treat as content continuation
if (messages.length > 0) {
messages[messages.length - 1].content += '\n' + line;
messages[messages.length - 1].content += "\n" + line;
} else {
// First line doesn't match format, create a generic message
messages.push({
timestamp: null,
role: 'unknown',
role: "unknown",
content: line,
order: order++,
});
@ -120,7 +122,9 @@ async function fetchTranscriptsForSessions() {
return;
}
console.log(`Found ${sessionsNeedingTranscripts.length} sessions that need transcript fetching.`);
console.log(
`Found ${sessionsNeedingTranscripts.length} sessions that need transcript fetching.`
);
let successCount = 0;
let errorCount = 0;
@ -131,7 +135,7 @@ async function fetchTranscriptsForSessions() {
}
console.log(`Fetching transcript for session ${session.id}...`);
try {
// Fetch transcript content
const transcriptContent = await fetchTranscriptContent(
@ -153,7 +157,7 @@ async function fetchTranscriptsForSessions() {
// Create messages in database
await prisma.message.createMany({
data: messages.map(msg => ({
data: messages.map((msg) => ({
sessionId: session.id,
timestamp: msg.timestamp,
role: msg.role,
@ -162,10 +166,15 @@ async function fetchTranscriptsForSessions() {
})),
});
console.log(`Successfully fetched transcript for session ${session.id} (${messages.length} messages)`);
console.log(
`Successfully fetched transcript for session ${session.id} (${messages.length} messages)`
);
successCount++;
} catch (error) {
console.error(`Error fetching transcript for session ${session.id}:`, error);
console.error(
`Error fetching transcript for session ${session.id}:`,
error
);
errorCount++;
}
}

103
scripts/process_sessions.ts
View File

@ -37,7 +37,9 @@ async function fetchTranscriptContent(
});
if (!response.ok) {
console.warn(`Failed to fetch transcript from ${url}: ${response.statusText}`);
console.warn(
`Failed to fetch transcript from ${url}: ${response.statusText}`
);
return null;
}
return await response.text();
@ -140,10 +142,19 @@ async function processTranscriptWithOpenAI(
/**
* Validates the OpenAI response against our expected schema
*/
function validateOpenAIResponse(data: any): asserts data is OpenAIProcessedData {
function validateOpenAIResponse(
data: any
): asserts data is OpenAIProcessedData {
const requiredFields = [
"language", "messages_sent", "sentiment", "escalated",
"forwarded_hr", "category", "questions", "summary", "session_id"
"language",
"messages_sent",
"sentiment",
"escalated",
"forwarded_hr",
"category",
"questions",
"summary",
"session_id",
];
for (const field of requiredFields) {
@ -153,7 +164,9 @@ function validateOpenAIResponse(data: any): asserts data is OpenAIProcessedData
}
if (typeof data.language !== "string" || !/^[a-z]{2}$/.test(data.language)) {
throw new Error("Invalid language format. Expected ISO 639-1 code (e.g., 'en')");
throw new Error(
"Invalid language format. Expected ISO 639-1 code (e.g., 'en')"
);
}
if (typeof data.messages_sent !== "number" || data.messages_sent < 0) {
@ -161,7 +174,9 @@ function validateOpenAIResponse(data: any): asserts data is OpenAIProcessedData
}
if (!["positive", "neutral", "negative"].includes(data.sentiment)) {
throw new Error("Invalid sentiment. Expected 'positive', 'neutral', or 'negative'");
throw new Error(
"Invalid sentiment. Expected 'positive', 'neutral', or 'negative'"
);
}
if (typeof data.escalated !== "boolean") {
@ -173,22 +188,39 @@ function validateOpenAIResponse(data: any): asserts data is OpenAIProcessedData
}
const validCategories = [
"Schedule & Hours", "Leave & Vacation", "Sick Leave & Recovery",
"Salary & Compensation", "Contract & Hours", "Onboarding", "Offboarding",
"Workwear & Staff Pass", "Team & Contacts", "Personal Questions",
"Access & Login", "Social questions", "Unrecognized / Other"
"Schedule & Hours",
"Leave & Vacation",
"Sick Leave & Recovery",
"Salary & Compensation",
"Contract & Hours",
"Onboarding",
"Offboarding",
"Workwear & Staff Pass",
"Team & Contacts",
"Personal Questions",
"Access & Login",
"Social questions",
"Unrecognized / Other",
];
if (!validCategories.includes(data.category)) {
throw new Error(`Invalid category. Expected one of: ${validCategories.join(", ")}`);
throw new Error(
`Invalid category. Expected one of: ${validCategories.join(", ")}`
);
}
if (!Array.isArray(data.questions)) {
throw new Error("Invalid questions. Expected array of strings");
}
if (typeof data.summary !== "string" || data.summary.length < 10 || data.summary.length > 300) {
throw new Error("Invalid summary. Expected string between 10-300 characters");
if (
typeof data.summary !== "string" ||
data.summary.length < 10 ||
data.summary.length > 300
) {
throw new Error(
"Invalid summary. Expected string between 10-300 characters"
);
}
if (typeof data.session_id !== "string") {
@ -218,18 +250,24 @@ async function processUnprocessedSessions() {
return;
}
console.log(`Found ${importsToProcess.length} SessionImport records to process.`);
console.log(
`Found ${importsToProcess.length} SessionImport records to process.`
);
let successCount = 0;
let errorCount = 0;
for (const importRecord of importsToProcess) {
if (!importRecord.fullTranscriptUrl) {
console.warn(`SessionImport ${importRecord.id} has no transcript URL, skipping.`);
console.warn(
`SessionImport ${importRecord.id} has no transcript URL, skipping.`
);
continue;
}
console.log(`Processing transcript for SessionImport ${importRecord.id}...`);
console.log(
`Processing transcript for SessionImport ${importRecord.id}...`
);
try {
// Mark as processing (status field doesn't exist in new schema)
console.log(`Processing SessionImport ${importRecord.id}...`);
@ -265,7 +303,10 @@ async function processUnprocessedSessions() {
country: importRecord.countryCode,
language: processedData.language,
messagesSent: processedData.messages_sent,
sentiment: processedData.sentiment.toUpperCase() as "POSITIVE" | "NEUTRAL" | "NEGATIVE",
sentiment: processedData.sentiment.toUpperCase() as
| "POSITIVE"
| "NEUTRAL"
| "NEGATIVE",
escalated: processedData.escalated,
forwardedHr: processedData.forwarded_hr,
fullTranscriptUrl: importRecord.fullTranscriptUrl,
@ -284,7 +325,10 @@ async function processUnprocessedSessions() {
country: importRecord.countryCode,
language: processedData.language,
messagesSent: processedData.messages_sent,
sentiment: processedData.sentiment.toUpperCase() as "POSITIVE" | "NEUTRAL" | "NEGATIVE",
sentiment: processedData.sentiment.toUpperCase() as
| "POSITIVE"
| "NEUTRAL"
| "NEGATIVE",
escalated: processedData.escalated,
forwardedHr: processedData.forwarded_hr,
fullTranscriptUrl: importRecord.fullTranscriptUrl,
@ -296,16 +340,25 @@ async function processUnprocessedSessions() {
});
// Mark SessionImport as processed (processedAt field doesn't exist in new schema)
console.log(`Successfully processed SessionImport ${importRecord.id} -> Session ${session.id}`);
console.log(
`Successfully processed SessionImport ${importRecord.id} -> Session ${session.id}`
);
console.log(`Successfully processed SessionImport ${importRecord.id} -> Session ${session.id}`);
console.log(
`Successfully processed SessionImport ${importRecord.id} -> Session ${session.id}`
);
successCount++;
} catch (error) {
console.error(`Error processing SessionImport ${importRecord.id}:`, error);
console.error(
`Error processing SessionImport ${importRecord.id}:`,
error
);
// Log error (status and errorMsg fields don't exist in new schema)
console.error(`Failed to process SessionImport ${importRecord.id}: ${error instanceof Error ? error.message : String(error)}`);
console.error(
`Failed to process SessionImport ${importRecord.id}: ${error instanceof Error ? error.message : String(error)}`
);
errorCount++;
}
}

6
server.ts
View File

@ -19,11 +19,11 @@ app.prepare().then(() => {
// Validate and log environment configuration
const envValidation = validateEnv();
if (!envValidation.valid) {
console.error('[Environment] Validation errors:', envValidation.errors);
console.error("[Environment] Validation errors:", envValidation.errors);
}
logEnvConfig();
// Get scheduler configuration
const config = getSchedulerConfig();

13
test-ai-processing.ts
View File

@ -1,16 +1,15 @@
import { processUnprocessedSessions } from './lib/processingScheduler';
import { processUnprocessedSessions } from "./lib/processingScheduler";
async function testAIProcessing() {
console.log('=== TESTING AI PROCESSING ===\n');
console.log("=== TESTING AI PROCESSING ===\n");
try {
// Process with batch size of 10 to test multiple batches (since we have 109 sessions)
await processUnprocessedSessions(10, 3); // batch size 10, max concurrency 3
console.log('\n=== AI PROCESSING COMPLETED ===');
console.log("\n=== AI PROCESSING COMPLETED ===");
} catch (error) {
console.error('Error during AI processing:', error);
console.error("Error during AI processing:", error);
}
}

13
test-import-processing.ts
View File

@ -1,16 +1,15 @@
import { processQueuedImports } from './lib/importProcessor';
import { processQueuedImports } from "./lib/importProcessor";
async function testImportProcessing() {
console.log('=== TESTING IMPORT PROCESSING ===\n');
console.log("=== TESTING IMPORT PROCESSING ===\n");
try {
// Process with batch size of 50 to test multiple batches
await processQueuedImports(50);
console.log('\n=== IMPORT PROCESSING COMPLETED ===');
console.log("\n=== IMPORT PROCESSING COMPLETED ===");
} catch (error) {
console.error('Error during import processing:', error);
console.error("Error during import processing:", error);
}
}

75
test-refactored-pipeline.js
View File

@ -1,49 +1,52 @@
// Test script for the refactored data processing pipeline
import { PrismaClient } from '@prisma/client';
import { processQueuedImports } from './lib/importProcessor.ts';
import { processAllUnparsedTranscripts } from './lib/transcriptParser.ts';
import { processUnprocessedSessions, getAIProcessingCosts } from './lib/processingScheduler.ts';
import { PrismaClient } from "@prisma/client";
import { processQueuedImports } from "./lib/importProcessor.ts";
import { processAllUnparsedTranscripts } from "./lib/transcriptParser.ts";
import {
processUnprocessedSessions,
getAIProcessingCosts,
} from "./lib/processingScheduler.ts";
const prisma = new PrismaClient();
async function testRefactoredPipeline() {
console.log('🧪 Testing Refactored Data Processing Pipeline\n');
console.log("🧪 Testing Refactored Data Processing Pipeline\n");
// Step 1: Check current state
console.log('📊 Current Database State:');
console.log("📊 Current Database State:");
const stats = await getDatabaseStats();
console.log(stats);
console.log('');
console.log("");
// Step 2: Test import processing (minimal fields only)
console.log('🔄 Testing Import Processing (Phase 1)...');
console.log("🔄 Testing Import Processing (Phase 1)...");
await processQueuedImports(5); // Process 5 imports
console.log('');
console.log("");
// Step 3: Test transcript parsing
console.log('📝 Testing Transcript Parsing (Phase 2)...');
console.log("📝 Testing Transcript Parsing (Phase 2)...");
await processAllUnparsedTranscripts();
console.log('');
console.log("");
// Step 4: Test AI processing with cost tracking
console.log('🤖 Testing AI Processing with Cost Tracking (Phase 3)...');
console.log("🤖 Testing AI Processing with Cost Tracking (Phase 3)...");
await processUnprocessedSessions(3, 2); // Process 3 sessions with concurrency 2
console.log('');
console.log("");
// Step 5: Show final results
console.log('📈 Final Results:');
console.log("📈 Final Results:");
const finalStats = await getDatabaseStats();
console.log(finalStats);
console.log('');
console.log("");
// Step 6: Show AI processing costs
console.log('💰 AI Processing Costs:');
console.log("💰 AI Processing Costs:");
const costs = await getAIProcessingCosts();
console.log(costs);
console.log('');
console.log("");
// Step 7: Show sample processed session
console.log('🔍 Sample Processed Session:');
console.log("🔍 Sample Processed Session:");
const sampleSession = await getSampleProcessedSession();
if (sampleSession) {
console.log(`Session ID: ${sampleSession.id}`);
@ -54,19 +57,23 @@ async function testRefactoredPipeline() {
console.log(`Escalated: ${sampleSession.escalated}`);
console.log(`Forwarded HR: ${sampleSession.forwardedHr}`);
console.log(`Summary: ${sampleSession.summary}`);
console.log(`Questions: ${sampleSession.sessionQuestions.length} questions`);
console.log(`AI Requests: ${sampleSession.aiProcessingRequests.length} requests`);
console.log(
`Questions: ${sampleSession.sessionQuestions.length} questions`
);
console.log(
`AI Requests: ${sampleSession.aiProcessingRequests.length} requests`
);
if (sampleSession.sessionQuestions.length > 0) {
console.log('Sample Questions:');
console.log("Sample Questions:");
sampleSession.sessionQuestions.slice(0, 3).forEach((sq, i) => {
console.log(` ${i + 1}. ${sq.question.content}`);
});
}
}
console.log('');
console.log("");
console.log('✅ Pipeline test completed!');
console.log("✅ Pipeline test completed!");
}
async function getDatabaseStats() {
@ -78,7 +85,7 @@ async function getDatabaseStats() {
totalMessages,
totalQuestions,
totalSessionQuestions,
totalAIRequests
totalAIRequests,
] = await Promise.all([
prisma.session.count(),
prisma.session.count({ where: { importId: { not: null } } }),
@ -87,7 +94,7 @@ async function getDatabaseStats() {
prisma.message.count(),
prisma.question.count(),
prisma.sessionQuestion.count(),
prisma.aIProcessingRequest.count()
prisma.aIProcessingRequest.count(),
]);
return {
@ -99,27 +106,27 @@ async function getDatabaseStats() {
totalMessages,
totalQuestions,
totalSessionQuestions,
totalAIRequests
totalAIRequests,
};
}
async function getSampleProcessedSession() {
return await prisma.session.findFirst({
where: {
where: {
processed: true,
messages: { some: {} }
messages: { some: {} },
},
include: {
sessionQuestions: {
include: {
question: true
question: true,
},
orderBy: { order: 'asc' }
orderBy: { order: "asc" },
},
aiProcessingRequests: {
orderBy: { requestedAt: 'desc' }
}
}
orderBy: { requestedAt: "desc" },
},
},
});
}

8
tests/setup.ts
View File

@ -1,5 +1,5 @@
// Vitest test setup
import { vi } from 'vitest';
import { vi } from "vitest";
// Mock console methods to reduce noise in tests
global.console = {
@ -10,8 +10,8 @@ global.console = {
};
// Set test environment variables
process.env.NEXTAUTH_SECRET = 'test-secret';
process.env.NEXTAUTH_URL = 'http://localhost:3000';
process.env.NEXTAUTH_SECRET = "test-secret";
process.env.NEXTAUTH_URL = "http://localhost:3000";
// Use test database for all database operations during tests
if (process.env.DATABASE_URL_TEST) {
@ -19,6 +19,6 @@ if (process.env.DATABASE_URL_TEST) {
}
// Mock node-fetch for transcript fetcher tests
vi.mock('node-fetch', () => ({
vi.mock("node-fetch", () => ({
default: vi.fn(),
}));

54
tests/unit/database.test.ts
View File

@ -1,7 +1,7 @@
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { PrismaClient } from '@prisma/client';
import { describe, it, expect, beforeAll, afterAll } from "vitest";
import { PrismaClient } from "@prisma/client";
describe('Database Configuration', () => {
describe("Database Configuration", () => {
let prisma: PrismaClient;
beforeAll(() => {
@ -12,62 +12,62 @@ describe('Database Configuration', () => {
await prisma.$disconnect();
});
it('should connect to the test database', async () => {
it("should connect to the test database", async () => {
// Verify we can connect to the database
const result = await prisma.$queryRaw`SELECT 1 as test`;
expect(result).toBeDefined();
});
it('should use PostgreSQL as the database provider', async () => {
it("should use PostgreSQL as the database provider", async () => {
// Query the database to verify it's PostgreSQL
const result = await prisma.$queryRaw`SELECT version()` as any[];
expect(result[0].version).toContain('PostgreSQL');
const result = (await prisma.$queryRaw`SELECT version()`) as any[];
expect(result[0].version).toContain("PostgreSQL");
});
it('should be using the test database URL', () => {
it("should be using the test database URL", () => {
// Verify that DATABASE_URL is set to the test database
expect(process.env.DATABASE_URL).toBeDefined();
expect(process.env.DATABASE_URL).toContain('postgresql://');
expect(process.env.DATABASE_URL).toContain("postgresql://");
// If DATABASE_URL_TEST is set, DATABASE_URL should match it (from our test setup)
if (process.env.DATABASE_URL_TEST) {
expect(process.env.DATABASE_URL).toBe(process.env.DATABASE_URL_TEST);
}
});
it('should have all required tables', async () => {
it("should have all required tables", async () => {
// Verify all our tables exist
const tables = await prisma.$queryRaw`
const tables = (await prisma.$queryRaw`
SELECT table_name
FROM information_schema.tables
WHERE table_schema = 'public'
AND table_type = 'BASE TABLE'
ORDER BY table_name
` as any[];
`) as any[];
const tableNames = tables.map(t => t.table_name);
expect(tableNames).toContain('Company');
expect(tableNames).toContain('User');
expect(tableNames).toContain('Session');
expect(tableNames).toContain('SessionImport');
expect(tableNames).toContain('Message');
expect(tableNames).toContain('Question');
expect(tableNames).toContain('SessionQuestion');
expect(tableNames).toContain('AIProcessingRequest');
const tableNames = tables.map((t) => t.table_name);
expect(tableNames).toContain("Company");
expect(tableNames).toContain("User");
expect(tableNames).toContain("Session");
expect(tableNames).toContain("SessionImport");
expect(tableNames).toContain("Message");
expect(tableNames).toContain("Question");
expect(tableNames).toContain("SessionQuestion");
expect(tableNames).toContain("AIProcessingRequest");
});
it('should be able to create and query data', async () => {
it("should be able to create and query data", async () => {
// Test basic CRUD operations
const company = await prisma.company.create({
data: {
name: 'Test Company',
csvUrl: 'https://example.com/test.csv',
name: "Test Company",
csvUrl: "https://example.com/test.csv",
},
});
expect(company.id).toBeDefined();
expect(company.name).toBe('Test Company');
expect(company.name).toBe("Test Company");
// Clean up
await prisma.company.delete({

164
tests/unit/env.test.ts
View File

@ -1,7 +1,7 @@
// Unit tests for environment management
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
describe('Environment Management', () => {
describe("Environment Management", () => {
let originalEnv: NodeJS.ProcessEnv;
beforeEach(() => {
@ -15,8 +15,8 @@ describe('Environment Management', () => {
vi.resetModules();
});
describe('env object', () => {
it('should have default values when environment variables are not set', async () => {
describe("env object", () => {
it("should have default values when environment variables are not set", async () => {
// Clear relevant env vars
delete process.env.NEXTAUTH_URL;
delete process.env.SCHEDULER_ENABLED;
@ -24,151 +24,153 @@ describe('Environment Management', () => {
// Re-import to get fresh env object
vi.resetModules();
const { env: freshEnv } = await import('../../lib/env');
const { env: freshEnv } = await import("../../lib/env");
expect(freshEnv.NEXTAUTH_URL).toBe('http://localhost:3000');
expect(freshEnv.NEXTAUTH_URL).toBe("http://localhost:3000");
// Note: SCHEDULER_ENABLED will be true because .env.local sets it to "true"
expect(freshEnv.SCHEDULER_ENABLED).toBe(true);
expect(freshEnv.PORT).toBe(3000);
});
it('should use environment variables when set', async () => {
process.env.NEXTAUTH_URL = 'https://example.com';
process.env.SCHEDULER_ENABLED = 'true';
process.env.PORT = '8080';
it("should use environment variables when set", async () => {
process.env.NEXTAUTH_URL = "https://example.com";
process.env.SCHEDULER_ENABLED = "true";
process.env.PORT = "8080";
vi.resetModules();
const { env: freshEnv } = await import('../../lib/env');
const { env: freshEnv } = await import("../../lib/env");
expect(freshEnv.NEXTAUTH_URL).toBe('https://example.com');
expect(freshEnv.NEXTAUTH_URL).toBe("https://example.com");
expect(freshEnv.SCHEDULER_ENABLED).toBe(true);
expect(freshEnv.PORT).toBe(8080);
});
it('should parse numeric environment variables correctly', async () => {
process.env.IMPORT_PROCESSING_BATCH_SIZE = '100';
process.env.SESSION_PROCESSING_CONCURRENCY = '10';
it("should parse numeric environment variables correctly", async () => {
process.env.IMPORT_PROCESSING_BATCH_SIZE = "100";
process.env.SESSION_PROCESSING_CONCURRENCY = "10";
vi.resetModules();
const { env: freshEnv } = await import('../../lib/env');
const { env: freshEnv } = await import("../../lib/env");
expect(freshEnv.IMPORT_PROCESSING_BATCH_SIZE).toBe(100);
expect(freshEnv.SESSION_PROCESSING_CONCURRENCY).toBe(10);
});
it('should handle invalid numeric values gracefully', async () => {
process.env.IMPORT_PROCESSING_BATCH_SIZE = 'invalid';
process.env.SESSION_PROCESSING_CONCURRENCY = '';
it("should handle invalid numeric values gracefully", async () => {
process.env.IMPORT_PROCESSING_BATCH_SIZE = "invalid";
process.env.SESSION_PROCESSING_CONCURRENCY = "";
vi.resetModules();
const { env: freshEnv } = await import('../../lib/env');
const { env: freshEnv } = await import("../../lib/env");
expect(freshEnv.IMPORT_PROCESSING_BATCH_SIZE).toBe(50); // Falls back to default value
expect(freshEnv.SESSION_PROCESSING_CONCURRENCY).toBe(5); // Falls back to default value
});
it('should parse quoted environment variables correctly', async () => {
it("should parse quoted environment variables correctly", async () => {
process.env.NEXTAUTH_URL = '"https://quoted.example.com"';
process.env.NEXTAUTH_SECRET = "'single-quoted-secret'";
vi.resetModules();
const { env: freshEnv } = await import('../../lib/env');
const { env: freshEnv } = await import("../../lib/env");
expect(freshEnv.NEXTAUTH_URL).toBe('https://quoted.example.com');
expect(freshEnv.NEXTAUTH_SECRET).toBe('single-quoted-secret');
expect(freshEnv.NEXTAUTH_URL).toBe("https://quoted.example.com");
expect(freshEnv.NEXTAUTH_SECRET).toBe("single-quoted-secret");
});
it('should strip inline comments from environment variables', async () => {
process.env.CSV_IMPORT_INTERVAL = '*/10 * * * * # Custom comment';
process.env.IMPORT_PROCESSING_INTERVAL = '*/3 * * * * # Another comment';
it("should strip inline comments from environment variables", async () => {
process.env.CSV_IMPORT_INTERVAL = "*/10 * * * * # Custom comment";
process.env.IMPORT_PROCESSING_INTERVAL =
"*/3 * * * * # Another comment";
vi.resetModules();
const { env: freshEnv } = await import('../../lib/env');
const { env: freshEnv } = await import("../../lib/env");
expect(freshEnv.CSV_IMPORT_INTERVAL).toBe('*/10 * * * *');
expect(freshEnv.IMPORT_PROCESSING_INTERVAL).toBe('*/3 * * * *');
expect(freshEnv.CSV_IMPORT_INTERVAL).toBe("*/10 * * * *");
expect(freshEnv.IMPORT_PROCESSING_INTERVAL).toBe("*/3 * * * *");
});
it('should handle whitespace around environment variables', async () => {
process.env.NEXTAUTH_URL = ' https://spaced.example.com ';
process.env.PORT = ' 8080 ';
it("should handle whitespace around environment variables", async () => {
process.env.NEXTAUTH_URL = " https://spaced.example.com ";
process.env.PORT = " 8080 ";
vi.resetModules();
const { env: freshEnv } = await import('../../lib/env');
const { env: freshEnv } = await import("../../lib/env");
expect(freshEnv.NEXTAUTH_URL).toBe('https://spaced.example.com');
expect(freshEnv.NEXTAUTH_URL).toBe("https://spaced.example.com");
expect(freshEnv.PORT).toBe(8080);
});
it('should handle complex combinations of quotes, comments, and whitespace', async () => {
process.env.NEXTAUTH_URL = ' "https://complex.example.com" # Production URL';
it("should handle complex combinations of quotes, comments, and whitespace", async () => {
process.env.NEXTAUTH_URL =
' "https://complex.example.com" # Production URL';
process.env.IMPORT_PROCESSING_BATCH_SIZE = " '100' # Batch size";
vi.resetModules();
const { env: freshEnv } = await import('../../lib/env');
const { env: freshEnv } = await import("../../lib/env");
expect(freshEnv.NEXTAUTH_URL).toBe('https://complex.example.com');
expect(freshEnv.NEXTAUTH_URL).toBe("https://complex.example.com");
expect(freshEnv.IMPORT_PROCESSING_BATCH_SIZE).toBe(100);
});
});
describe('validateEnv', () => {
it('should return valid when all required variables are set', async () => {
vi.stubEnv('NEXTAUTH_SECRET', 'test-secret');
vi.stubEnv('OPENAI_API_KEY', 'test-key');
vi.stubEnv('NODE_ENV', 'production');
describe("validateEnv", () => {
it("should return valid when all required variables are set", async () => {
vi.stubEnv("NEXTAUTH_SECRET", "test-secret");
vi.stubEnv("OPENAI_API_KEY", "test-key");
vi.stubEnv("NODE_ENV", "production");
vi.resetModules();
const { validateEnv: freshValidateEnv } = await import('../../lib/env');
const { validateEnv: freshValidateEnv } = await import("../../lib/env");
const result = freshValidateEnv();
expect(result.valid).toBe(true);
expect(result.errors).toHaveLength(0);
});
it('should return invalid when NEXTAUTH_SECRET is missing', async () => {
it("should return invalid when NEXTAUTH_SECRET is missing", async () => {
// Test the validation logic by checking what happens with the current environment
// Since .env.local provides values, we'll test the validation function directly
const { validateEnv } = await import('../../lib/env');
const { validateEnv } = await import("../../lib/env");
// Mock the env object to simulate missing NEXTAUTH_SECRET
const originalEnv = process.env.NEXTAUTH_SECRET;
delete process.env.NEXTAUTH_SECRET;
vi.resetModules();
const { validateEnv: freshValidateEnv } = await import('../../lib/env');
const { validateEnv: freshValidateEnv } = await import("../../lib/env");
const result = freshValidateEnv();
// Restore the original value
if (originalEnv) {
process.env.NEXTAUTH_SECRET = originalEnv;
}
// Since .env.local loads values, this test validates the current setup is working
// We expect it to be valid because .env.local provides the secret
expect(result.valid).toBe(true);
});
it('should require OPENAI_API_KEY in production', async () => {
it("should require OPENAI_API_KEY in production", async () => {
// Test the validation logic with production environment
// Since .env.local provides values, this test validates the current behavior
const { validateEnv } = await import('../../lib/env');
const { validateEnv } = await import("../../lib/env");
const result = validateEnv();
// Since .env.local provides both NEXTAUTH_SECRET and OPENAI_API_KEY,
// and NODE_ENV is 'development' by default, this should be valid
expect(result.valid).toBe(true);
});
it('should not require OPENAI_API_KEY in development', async () => {
vi.stubEnv('NEXTAUTH_SECRET', 'test-secret');
vi.stubEnv('OPENAI_API_KEY', '');
vi.stubEnv('NODE_ENV', 'development');
it("should not require OPENAI_API_KEY in development", async () => {
vi.stubEnv("NEXTAUTH_SECRET", "test-secret");
vi.stubEnv("OPENAI_API_KEY", "");
vi.stubEnv("NODE_ENV", "development");
vi.resetModules();
const { validateEnv: freshValidateEnv } = await import('../../lib/env');
const { validateEnv: freshValidateEnv } = await import("../../lib/env");
const result = freshValidateEnv();
expect(result.valid).toBe(true);
@ -176,45 +178,49 @@ describe('Environment Management', () => {
});
});
describe('getSchedulerConfig', () => {
it('should return correct scheduler configuration', async () => {
process.env.SCHEDULER_ENABLED = 'true';
process.env.CSV_IMPORT_INTERVAL = '*/10 * * * *';
process.env.IMPORT_PROCESSING_INTERVAL = '*/3 * * * *';
process.env.IMPORT_PROCESSING_BATCH_SIZE = '25';
process.env.SESSION_PROCESSING_INTERVAL = '0 2 * * *';
process.env.SESSION_PROCESSING_BATCH_SIZE = '100';
process.env.SESSION_PROCESSING_CONCURRENCY = '8';
describe("getSchedulerConfig", () => {
it("should return correct scheduler configuration", async () => {
process.env.SCHEDULER_ENABLED = "true";
process.env.CSV_IMPORT_INTERVAL = "*/10 * * * *";
process.env.IMPORT_PROCESSING_INTERVAL = "*/3 * * * *";
process.env.IMPORT_PROCESSING_BATCH_SIZE = "25";
process.env.SESSION_PROCESSING_INTERVAL = "0 2 * * *";
process.env.SESSION_PROCESSING_BATCH_SIZE = "100";
process.env.SESSION_PROCESSING_CONCURRENCY = "8";
vi.resetModules();
const { getSchedulerConfig: freshGetSchedulerConfig } = await import('../../lib/env');
const { getSchedulerConfig: freshGetSchedulerConfig } = await import(
"../../lib/env"
);
const config = freshGetSchedulerConfig();
expect(config.enabled).toBe(true);
expect(config.csvImport.interval).toBe('*/10 * * * *');
expect(config.importProcessing.interval).toBe('*/3 * * * *');
expect(config.csvImport.interval).toBe("*/10 * * * *");
expect(config.importProcessing.interval).toBe("*/3 * * * *");
expect(config.importProcessing.batchSize).toBe(25);
expect(config.sessionProcessing.interval).toBe('0 2 * * *');
expect(config.sessionProcessing.interval).toBe("0 2 * * *");
expect(config.sessionProcessing.batchSize).toBe(100);
expect(config.sessionProcessing.concurrency).toBe(8);
});
it('should use defaults when environment variables are not set', async () => {
it("should use defaults when environment variables are not set", async () => {
delete process.env.SCHEDULER_ENABLED;
delete process.env.CSV_IMPORT_INTERVAL;
delete process.env.IMPORT_PROCESSING_INTERVAL;
vi.resetModules();
const { getSchedulerConfig: freshGetSchedulerConfig } = await import('../../lib/env');
const { getSchedulerConfig: freshGetSchedulerConfig } = await import(
"../../lib/env"
);
const config = freshGetSchedulerConfig();
// Note: SCHEDULER_ENABLED will be true because .env.local sets it to "true"
expect(config.enabled).toBe(true);
// The .env.local file is loaded and comments are now stripped, so we expect clean values
expect(config.csvImport.interval).toBe('*/15 * * * *');
expect(config.importProcessing.interval).toBe('*/5 * * * *');
expect(config.csvImport.interval).toBe("*/15 * * * *");
expect(config.importProcessing.interval).toBe("*/5 * * * *");
expect(config.importProcessing.batchSize).toBe(50);
});
});

225
tests/unit/transcriptFetcher.test.ts
View File

@ -1,222 +1,237 @@
// Unit tests for transcript fetcher
import { describe, it, expect, beforeEach, vi } from 'vitest';
import fetch from 'node-fetch';
import {
fetchTranscriptContent,
isValidTranscriptUrl,
extractSessionIdFromTranscript
} from '../../lib/transcriptFetcher';
import { describe, it, expect, beforeEach, vi } from "vitest";
import fetch from "node-fetch";
import {
fetchTranscriptContent,
isValidTranscriptUrl,
extractSessionIdFromTranscript,
} from "../../lib/transcriptFetcher";
// Mock node-fetch
const mockFetch = fetch as any;
describe('Transcript Fetcher', () => {
describe("Transcript Fetcher", () => {
beforeEach(() => {
vi.clearAllMocks();
});
describe('fetchTranscriptContent', () => {
it('should successfully fetch transcript content', async () => {
describe("fetchTranscriptContent", () => {
it("should successfully fetch transcript content", async () => {
const mockResponse = {
ok: true,
text: vi.fn().mockResolvedValue('Session transcript content'),
};
mockFetch.mockResolvedValue(mockResponse as any);
const result = await fetchTranscriptContent('https://example.com/transcript');
expect(result.success).toBe(true);
expect(result.content).toBe('Session transcript content');
expect(result.error).toBeUndefined();
expect(mockFetch).toHaveBeenCalledWith('https://example.com/transcript', {
method: 'GET',
headers: {
'User-Agent': 'LiveDash-Transcript-Fetcher/1.0',
},
signal: expect.any(AbortSignal),
});
});
it('should handle authentication with username and password', async () => {
const mockResponse = {
ok: true,
text: vi.fn().mockResolvedValue('Authenticated transcript'),
text: vi.fn().mockResolvedValue("Session transcript content"),
};
mockFetch.mockResolvedValue(mockResponse as any);
const result = await fetchTranscriptContent(
'https://example.com/transcript',
'user123',
'pass456'
"https://example.com/transcript"
);
expect(result.success).toBe(true);
expect(result.content).toBe('Authenticated transcript');
const expectedAuth = 'Basic ' + Buffer.from('user123:pass456').toString('base64');
expect(mockFetch).toHaveBeenCalledWith('https://example.com/transcript', {
method: 'GET',
expect(result.content).toBe("Session transcript content");
expect(result.error).toBeUndefined();
expect(mockFetch).toHaveBeenCalledWith("https://example.com/transcript", {
method: "GET",
headers: {
'User-Agent': 'LiveDash-Transcript-Fetcher/1.0',
'Authorization': expectedAuth,
"User-Agent": "LiveDash-Transcript-Fetcher/1.0",
},
signal: expect.any(AbortSignal),
});
});
it('should handle HTTP errors', async () => {
it("should handle authentication with username and password", async () => {
const mockResponse = {
ok: true,
text: vi.fn().mockResolvedValue("Authenticated transcript"),
};
mockFetch.mockResolvedValue(mockResponse as any);
const result = await fetchTranscriptContent(
"https://example.com/transcript",
"user123",
"pass456"
);
expect(result.success).toBe(true);
expect(result.content).toBe("Authenticated transcript");
const expectedAuth =
"Basic " + Buffer.from("user123:pass456").toString("base64");
expect(mockFetch).toHaveBeenCalledWith("https://example.com/transcript", {
method: "GET",
headers: {
"User-Agent": "LiveDash-Transcript-Fetcher/1.0",
Authorization: expectedAuth,
},
signal: expect.any(AbortSignal),
});
});
it("should handle HTTP errors", async () => {
const mockResponse = {
ok: false,
status: 404,
statusText: 'Not Found',
statusText: "Not Found",
};
mockFetch.mockResolvedValue(mockResponse as any);
const result = await fetchTranscriptContent('https://example.com/transcript');
const result = await fetchTranscriptContent(
"https://example.com/transcript"
);
expect(result.success).toBe(false);
expect(result.error).toBe('HTTP 404: Not Found');
expect(result.error).toBe("HTTP 404: Not Found");
expect(result.content).toBeUndefined();
});
it('should handle empty transcript content', async () => {
it("should handle empty transcript content", async () => {
const mockResponse = {
ok: true,
text: vi.fn().mockResolvedValue(' '),
text: vi.fn().mockResolvedValue(" "),
};
mockFetch.mockResolvedValue(mockResponse as any);
const result = await fetchTranscriptContent('https://example.com/transcript');
const result = await fetchTranscriptContent(
"https://example.com/transcript"
);
expect(result.success).toBe(false);
expect(result.error).toBe('Empty transcript content');
expect(result.error).toBe("Empty transcript content");
expect(result.content).toBeUndefined();
});
it('should handle network errors', async () => {
mockFetch.mockRejectedValue(new Error('ENOTFOUND example.com'));
it("should handle network errors", async () => {
mockFetch.mockRejectedValue(new Error("ENOTFOUND example.com"));
const result = await fetchTranscriptContent('https://example.com/transcript');
const result = await fetchTranscriptContent(
"https://example.com/transcript"
);
expect(result.success).toBe(false);
expect(result.error).toBe('Domain not found');
expect(result.error).toBe("Domain not found");
expect(result.content).toBeUndefined();
});
it('should handle connection refused errors', async () => {
mockFetch.mockRejectedValue(new Error('ECONNREFUSED'));
it("should handle connection refused errors", async () => {
mockFetch.mockRejectedValue(new Error("ECONNREFUSED"));
const result = await fetchTranscriptContent('https://example.com/transcript');
const result = await fetchTranscriptContent(
"https://example.com/transcript"
);
expect(result.success).toBe(false);
expect(result.error).toBe('Connection refused');
expect(result.error).toBe("Connection refused");
expect(result.content).toBeUndefined();
});
it('should handle timeout errors', async () => {
mockFetch.mockRejectedValue(new Error('Request timeout'));
it("should handle timeout errors", async () => {
mockFetch.mockRejectedValue(new Error("Request timeout"));
const result = await fetchTranscriptContent('https://example.com/transcript');
const result = await fetchTranscriptContent(
"https://example.com/transcript"
);
expect(result.success).toBe(false);
expect(result.error).toBe('Request timeout');
expect(result.error).toBe("Request timeout");
expect(result.content).toBeUndefined();
});
it('should handle empty URL', async () => {
const result = await fetchTranscriptContent('');
it("should handle empty URL", async () => {
const result = await fetchTranscriptContent("");
expect(result.success).toBe(false);
expect(result.error).toBe('No transcript URL provided');
expect(result.error).toBe("No transcript URL provided");
expect(result.content).toBeUndefined();
expect(mockFetch).not.toHaveBeenCalled();
});
it('should trim whitespace from content', async () => {
it("should trim whitespace from content", async () => {
const mockResponse = {
ok: true,
text: vi.fn().mockResolvedValue(' \n Session content \n '),
text: vi.fn().mockResolvedValue(" \n Session content \n "),
};
mockFetch.mockResolvedValue(mockResponse as any);
const result = await fetchTranscriptContent('https://example.com/transcript');
const result = await fetchTranscriptContent(
"https://example.com/transcript"
);
expect(result.success).toBe(true);
expect(result.content).toBe('Session content');
expect(result.content).toBe("Session content");
});
});
describe('isValidTranscriptUrl', () => {
it('should validate HTTP URLs', () => {
expect(isValidTranscriptUrl('http://example.com/transcript')).toBe(true);
describe("isValidTranscriptUrl", () => {
it("should validate HTTP URLs", () => {
expect(isValidTranscriptUrl("http://example.com/transcript")).toBe(true);
});
it('should validate HTTPS URLs', () => {
expect(isValidTranscriptUrl('https://example.com/transcript')).toBe(true);
it("should validate HTTPS URLs", () => {
expect(isValidTranscriptUrl("https://example.com/transcript")).toBe(true);
});
it('should reject invalid URLs', () => {
expect(isValidTranscriptUrl('not-a-url')).toBe(false);
expect(isValidTranscriptUrl('ftp://example.com')).toBe(false);
expect(isValidTranscriptUrl('')).toBe(false);
it("should reject invalid URLs", () => {
expect(isValidTranscriptUrl("not-a-url")).toBe(false);
expect(isValidTranscriptUrl("ftp://example.com")).toBe(false);
expect(isValidTranscriptUrl("")).toBe(false);
expect(isValidTranscriptUrl(null as any)).toBe(false);
expect(isValidTranscriptUrl(undefined as any)).toBe(false);
});
it('should handle malformed URLs', () => {
expect(isValidTranscriptUrl('http://')).toBe(false);
expect(isValidTranscriptUrl('https://')).toBe(false);
expect(isValidTranscriptUrl('://example.com')).toBe(false);
it("should handle malformed URLs", () => {
expect(isValidTranscriptUrl("http://")).toBe(false);
expect(isValidTranscriptUrl("https://")).toBe(false);
expect(isValidTranscriptUrl("://example.com")).toBe(false);
});
});
describe('extractSessionIdFromTranscript', () => {
it('should extract session ID from session_id pattern', () => {
const content = 'session_id: abc123def456\nOther content...';
describe("extractSessionIdFromTranscript", () => {
it("should extract session ID from session_id pattern", () => {
const content = "session_id: abc123def456\nOther content...";
const result = extractSessionIdFromTranscript(content);
expect(result).toBe('abc123def456');
expect(result).toBe("abc123def456");
});
it('should extract session ID from sessionId pattern', () => {
const content = 'sessionId: xyz789\nTranscript data...';
it("should extract session ID from sessionId pattern", () => {
const content = "sessionId: xyz789\nTranscript data...";
const result = extractSessionIdFromTranscript(content);
expect(result).toBe('xyz789');
expect(result).toBe("xyz789");
});
it('should extract session ID from id pattern', () => {
const content = 'id: session-12345678\nChat log...';
it("should extract session ID from id pattern", () => {
const content = "id: session-12345678\nChat log...";
const result = extractSessionIdFromTranscript(content);
expect(result).toBe('session-12345678');
expect(result).toBe("session-12345678");
});
it('should extract session ID from first line', () => {
const content = 'abc123def456\nUser: Hello\nBot: Hi there';
it("should extract session ID from first line", () => {
const content = "abc123def456\nUser: Hello\nBot: Hi there";
const result = extractSessionIdFromTranscript(content);
expect(result).toBe('abc123def456');
expect(result).toBe("abc123def456");
});
it('should return null for content without session ID', () => {
const content = 'User: Hello\nBot: Hi there\nUser: How are you?';
it("should return null for content without session ID", () => {
const content = "User: Hello\nBot: Hi there\nUser: How are you?";
const result = extractSessionIdFromTranscript(content);
expect(result).toBe(null);
});
it('should return null for empty content', () => {
expect(extractSessionIdFromTranscript('')).toBe(null);
it("should return null for empty content", () => {
expect(extractSessionIdFromTranscript("")).toBe(null);
expect(extractSessionIdFromTranscript(null as any)).toBe(null);
expect(extractSessionIdFromTranscript(undefined as any)).toBe(null);
});
it('should handle case-insensitive patterns', () => {
const content = 'SESSION_ID: ABC123\nContent...';
it("should handle case-insensitive patterns", () => {
const content = "SESSION_ID: ABC123\nContent...";
const result = extractSessionIdFromTranscript(content);
expect(result).toBe('ABC123');
expect(result).toBe("ABC123");
});
it('should extract the first matching pattern', () => {
const content = 'session_id: first123\nid: second456\nMore content...';
it("should extract the first matching pattern", () => {
const content = "session_id: first123\nid: second456\nMore content...";
const result = extractSessionIdFromTranscript(content);
expect(result).toBe('first123');
expect(result).toBe("first123");
});
});
});

24
vitest.config.mts
View File

@ -1,23 +1,23 @@
import { defineConfig } from 'vitest/config'
import react from '@vitejs/plugin-react'
import tsconfigPaths from 'vite-tsconfig-paths'
import { defineConfig } from "vitest/config";
import react from "@vitejs/plugin-react";
import tsconfigPaths from "vite-tsconfig-paths";
export default defineConfig({
plugins: [tsconfigPaths(), react()],
test: {
environment: 'node',
environment: "node",
globals: true,
setupFiles: ['./tests/setup.ts'],
include: ['tests/**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}'],
setupFiles: ["./tests/setup.ts"],
include: ["tests/**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}"],
env: {
NODE_ENV: 'test',
NODE_ENV: "test",
},
coverage: {
provider: 'v8',
reporter: ['text', 'lcov', 'html'],
include: ['lib/**/*.ts'],
exclude: ['lib/**/*.d.ts', 'lib/**/*.test.ts'],
provider: "v8",
reporter: ["text", "lcov", "html"],
include: ["lib/**/*.ts"],
exclude: ["lib/**/*.d.ts", "lib/**/*.test.ts"],
},
testTimeout: 10000,
},
})
});