Kaj Kowalski
f0ae061fa7
- Improved date parsing in fetch_and_store_chat_data to support multiple formats and added error logging for unparseable dates. - Enhanced parse_and_store_transcript_messages to handle empty transcripts and expanded message pattern recognition for both User and Assistant. - Implemented intelligent splitting of transcripts based on detected patterns and timestamps, with fallback mechanisms for unrecognized formats. - Updated documentation for Celery and Redis setup, troubleshooting, and project structure. - Added markdown linting configuration and scripts for code formatting. - Updated Nginx configuration to change the web server port. - Added xlsxwriter dependency for Excel file handling in project requirements.
3.5 KiB
Redis and Celery Configuration
This document explains how to set up and use Redis and Celery for background task processing in the LiveGraphs application.
Overview
The data integration module uses Celery to handle:
- Periodic data fetching from external APIs
- Processing and storing CSV data
- Downloading and parsing transcript files
- Manual data refresh triggered by users
Installation
Redis (Recommended)
Redis is the recommended message broker for Celery due to its performance and reliability:
Ubuntu/Debian
sudo apt update
sudo apt install redis-server
sudo systemctl start redis-server
sudo systemctl enable redis-server
# Verify that Redis is running
redis-cli ping # Should output PONG
After installation, check if Redis is properly configured:
-
Open Redis configuration file:
sudo nano /etc/redis/redis.conf -
Ensure the following settings:
# For development (localhost only) bind 127.0.0.1 # For production (accept connections from specific IP) # bind 127.0.0.1 your.server.ip.address # Protected mode (recommended) protected-mode yes # Port port 6379 -
Restart Redis after any changes:
sudo systemctl restart redis-server
macOS
brew install redis
brew services start redis
Windows
Download and install from microsoftarchive/redis
SQLite Fallback
If Redis is not available, the application will automatically fall back to using SQLite for Celery tasks. This works well for development but is not recommended for production.
Configuration
Environment Variables
Set these environment variables in your .env file or deployment environment:
# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0
# Task Scheduling
CHAT_DATA_FETCH_INTERVAL=3600 # In seconds (1 hour)
FETCH_DATA_TIMEOUT=300 # In seconds (5 minutes)
Testing Redis Connection
To test if Redis is properly configured:
cd dashboard_project
python manage.py test_redis
Testing Celery
To test if Celery is working correctly:
# Start a Celery worker in one terminal
make celery
# In another terminal, run the test task
cd dashboard_project
python manage.py test_celery
Running with Docker
The included docker-compose.yml file sets up Redis, Celery worker, and Celery beat for you:
docker-compose up -d
Running in Development
Development requires multiple terminal windows:
-
Django Development Server:
make run -
Redis Server (if needed):
make run-redis -
Celery Worker:
make celery -
Celery Beat (for scheduled tasks):
make celery-beat
Or use the combined command:
make run-all
Common Issues
Redis Connection Failures
If you see connection errors:
- Check that Redis is running:
redis-cli pingshould returnPONG - Verify firewall settings are not blocking port 6379
- Check Redis binding in
/etc/redis/redis.conf(should bebind 127.0.0.1for local dev)
Celery Workers Not Processing Tasks
- Ensure the worker is running with the correct app name:
celery -A dashboard_project worker - Check the Celery logs for errors
- Verify broker URL settings in both code and environment variables