kjanat/livedash-node
1
0
Fork 0
mirror of https://github.com/kjanat/livedash-node.git synced 2026-01-16 08:32:09 +01:00
Code Issues Packages Projects Releases Wiki Activity

DB refactor

Browse Source
This commit is contained in:
Max Kowalski
2025-06-27 23:05:46 +02:00
parent 185bb6da58
commit 2dfc49f840
20 changed files with 1607 additions and 339 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
docs/scheduler-fixes.md
View File

@ -8,8 +8,8 @@
**Solution**:
- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs
- Removed the invalid company record from the database using `fix_companies.js`
- Added validation in `fetchAndStoreSessionsForAllCompanies()` to skip companies with example/invalid URLs
- Removed the invalid company record from the database using `fix_companies.js`
### 2. Transcript Fetching Errors
@ -17,10 +17,10 @@
**Solution**:
- Improved error handling in `fetchTranscriptContent()` function
- Added probabilistic logging (only ~10% of errors logged) to prevent log spam
- Added timeout (10 seconds) for transcript fetching
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
- Improved error handling in `fetchTranscriptContent()` function
- Added probabilistic logging (only ~10% of errors logged) to prevent log spam
- Added timeout (10 seconds) for transcript fetching
- Made transcript fetching failures non-blocking (sessions are still created without transcript content)
### 3. CSV Fetching Errors
@ -28,8 +28,8 @@
**Solution**:
- Added URL validation to skip companies with `example.com` URLs
- Improved error logging to be more descriptive
- Added URL validation to skip companies with `example.com` URLs
- Improved error logging to be more descriptive
## Current Status
@ -42,22 +42,22 @@
After cleanup, only valid companies remain:
- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`)
- CSV URL: `https://proto.notso.ai/jumbo/chats`
- Has valid authentication credentials
- 107 sessions in database
- **Demo Company** (`790b9233-d369-451f-b92c-f4dceb42b649`)
- CSV URL: `https://proto.notso.ai/jumbo/chats`
- Has valid authentication credentials
- 107 sessions in database
## Files Modified
1. **lib/csvFetcher.js**
1. **lib/csvFetcher.js**
- Added company URL validation
- Improved transcript fetching error handling
- Reduced error log verbosity
- Added company URL validation
- Improved transcript fetching error handling
- Reduced error log verbosity
2. **fix_companies.js** (cleanup script)
- Removes invalid company records
- Can be run again if needed
2. **fix_companies.js** (cleanup script)
- Removes invalid company records
- Can be run again if needed
## Monitoring
@ -73,7 +73,7 @@ node -e "import('./lib/csvFetcher.js').then(m => m.fetchAndStoreSessionsForAllCo
## Future Improvements
1. Add health check endpoint for scheduler status
2. Add metrics for successful/failed fetches
3. Consider retry logic for temporary failures
4. Add alerting for persistent failures
1. Add health check endpoint for scheduler status
2. Add metrics for successful/failed fetches
3. Consider retry logic for temporary failures
4. Add alerting for persistent failures