mirror of
https://github.com/kjanat/livedash-node.git
synced 2026-01-16 15:32:10 +01:00
3196dabdf2
- Added processingScheduler.js and processingScheduler.ts to handle session transcript processing using OpenAI API. - Implemented a new scheduler (scheduler.js and schedulers.ts) for refreshing sessions every 15 minutes. - Updated Prisma migrations to add new fields for processed sessions, including questions, sentimentCategory, and summary. - Created scripts (process_sessions.mjs and process_sessions.ts) for manual processing of unprocessed sessions. - Enhanced server.js and server.mjs to initialize schedulers on server start.
2.3 KiB
2.3 KiB
Scheduler Error Fixes
Issues Identified and Resolved
1. Invalid Company Configuration
Problem: Company 26fc3d34-c074-4556-85bd-9a66fafc0e08 had an invalid CSV URL (https://example.com/data.csv) with no authentication credentials.
Solution:
- Added validation in
fetchAndStoreSessionsForAllCompanies()to skip companies with example/invalid URLs - Removed the invalid company record from the database using
fix_companies.js
2. Transcript Fetching Errors
Problem: Multiple "Error fetching transcript: Unauthorized" messages were flooding the logs when individual transcript files couldn't be accessed.
Solution:
- Improved error handling in
fetchTranscriptContent()function - Added probabilistic logging (only ~10% of errors logged) to prevent log spam
- Added timeout (10 seconds) for transcript fetching
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
3. CSV Fetching Errors
Problem: "Failed to fetch CSV: Not Found" errors for companies with invalid URLs.
Solution:
- Added URL validation to skip companies with
example.comURLs - Improved error logging to be more descriptive
Current Status
✅ Fixed: No more "Unauthorized" error spam
✅ Fixed: No more "Not Found" CSV errors
✅ Fixed: Scheduler runs cleanly without errors
✅ Improved: Better error handling and logging
Remaining Companies
After cleanup, only valid companies remain:
- Demo Company (
790b9233-d369-451f-b92c-f4dceb42b649)- CSV URL:
https://proto.notso.ai/jumbo/chats - Has valid authentication credentials
- 107 sessions in database
- CSV URL:
Files Modified
-
lib/csvFetcher.js
- Added company URL validation
- Improved transcript fetching error handling
- Reduced error log verbosity
-
fix_companies.js (cleanup script)
- Removes invalid company records
- Can be run again if needed
Monitoring
The scheduler now runs cleanly every 15 minutes. To monitor:
# Check scheduler logs
node debug_db.js
# Test manual refresh
node -e "import('./lib/csvFetcher.js').then(m => m.fetchAndStoreSessionsForAllCompanies())"
Future Improvements
- Add health check endpoint for scheduler status
- Add metrics for successful/failed fetches
- Consider retry logic for temporary failures
- Add alerting for persistent failures