livegraphs-django/docs/CELERY_REDIS.md

Redis and Celery Configuration

This document explains how to set up and use Redis and Celery for background task processing in the LiveGraphs application.

Overview

The data integration module uses Celery to handle:

• Periodic data fetching from external APIs
• Processing and storing CSV data
• Downloading and parsing transcript files
• Manual data refresh triggered by users

Installation

Redis (Recommended)

Redis is the recommended message broker for Celery due to its performance and reliability:

Ubuntu/Debian

sudo apt update
sudo apt install redis-server
sudo systemctl start redis-server
sudo systemctl enable redis-server

# Verify that Redis is running
redis-cli ping  # Should output PONG

After installation, check if Redis is properly configured:

1. Open Redis configuration file:

   sudo nano /etc/redis/redis.conf
   

2. Ensure the following settings:

   # For development (localhost only)
   bind 127.0.0.1
   
   # For production (accept connections from specific IP)
   # bind 127.0.0.1 your.server.ip.address
   
   # Protected mode (recommended)
   protected-mode yes
   
   # Port
   port 6379
   

3. Restart Redis after any changes:

   sudo systemctl restart redis-server
   

macOS

brew install redis
brew services start redis

Windows

Download and install from microsoftarchive/redis

SQLite Fallback

If Redis is not available, the application will automatically fall back to using SQLite for Celery tasks. This works well for development but is not recommended for production.

Configuration

Environment Variables

Set these environment variables in your .env file or deployment environment:

# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0

# Task Scheduling
CHAT_DATA_FETCH_INTERVAL=3600  # In seconds (1 hour)
FETCH_DATA_TIMEOUT=300         # In seconds (5 minutes)

Testing Redis Connection

To test if Redis is properly configured:

cd dashboard_project
python manage.py test_redis

Testing Celery

To test if Celery is working correctly:

# Start a Celery worker in one terminal
make celery

# In another terminal, run the test task
cd dashboard_project
python manage.py test_celery

Running with Docker

The included docker-compose.yml file sets up Redis, Celery worker, and Celery beat for you:

docker-compose up -d

Running in Development

Development requires multiple terminal windows:

1. Django Development Server:

   make run
   

2. Redis Server (if needed):

   make run-redis
   

3. Celery Worker:

   make celery
   

4. Celery Beat (for scheduled tasks):

   make celery-beat
   

Or use the combined command:

make run-all

Common Issues

Redis Connection Failures

If you see connection errors:

1. Check that Redis is running: redis-cli ping should return PONG
2. Verify firewall settings are not blocking port 6379
3. Check Redis binding in /etc/redis/redis.conf (should be bind 127.0.0.1 for local dev)

Celery Workers Not Processing Tasks

1. Ensure the worker is running with the correct app name: celery -A dashboard_project worker
2. Check the Celery logs for errors
3. Verify broker URL settings in both code and environment variables