kjanat/livegraphs-django
1
0
Fork 0
mirror of https://github.com/kjanat/livegraphs-django.git synced 2026-01-16 09:02:11 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Enhance data integration and transcript parsing

Browse Source 
- Improved date parsing in fetch_and_store_chat_data to support multiple formats and added error logging for unparseable dates.
- Enhanced parse_and_store_transcript_messages to handle empty transcripts and expanded message pattern recognition for both User and Assistant.
- Implemented intelligent splitting of transcripts based on detected patterns and timestamps, with fallback mechanisms for unrecognized formats.
- Updated documentation for Celery and Redis setup, troubleshooting, and project structure.
- Added markdown linting configuration and scripts for code formatting.
- Updated Nginx configuration to change the web server port.
- Added xlsxwriter dependency for Excel file handling in project requirements.
This commit is contained in:
Kaj Kowalski
2025-05-18 19:18:31 +00:00
parent 8bbbb109bd
commit f0ae061fa7
24 changed files with 1672 additions and 931 deletions
Download Patch File Download Diff File Expand all files Collapse all files

653
README.md
View File

@ -2,203 +2,433 @@
A Django application that creates an analytics dashboard for chat session data. The application allows different companies to have their own dashboards and view their own data.
## Features
## Project Overview
- Multi-company support with user authentication
- CSV file upload and processing
- Interactive dashboard with charts and visualizations
- Detailed data views for chat sessions
- Search functionality to find specific chat sessions
- Admin interface for managing users and companies
- Responsive design using Bootstrap 5
This Django project creates a multi-tenant dashboard application for analyzing chat session data. Companies can upload their chat data (in CSV format) and view analytics and metrics through an interactive dashboard. The application supports user authentication, role-based access control, and separate data isolation for different companies.
### Project Structure
The project consists of two main Django apps:
1. **accounts**: Handles user authentication, company management, and user roles
2. **dashboard**: Manages data sources, chat sessions, and dashboard visualization
3. **data_integration**: Handles external API data integration
### Key Features
- **Multi-company Support**: Each company has their own private dashboards and data
- **User Management**: Different user roles (admin, company admin, regular user)
- **CSV File Upload**: Upload and process CSV files containing chat session data
- **Interactive Dashboard**: Visualize chat data with charts and metrics
- **Search Functionality**: Find specific chat sessions based on various criteria
- **Data Export**: Export data in CSV, JSON, and Excel formats
- **Data Exploration**: Drill down into individual chat sessions for detailed analysis
- **Responsive Design**: Mobile-friendly interface using Bootstrap 5
## Requirements
- Python 3.13+
- Django 5.2+
- UV package manager (recommended)
- Other dependencies listed in [`pyproject.toml`](./pyproject.toml)
- Python 3.13+
- Django 5.2+
- UV package manager (recommended)
- Other dependencies listed in [`pyproject.toml`](./pyproject.toml)
## Setup
### Local Development
1. Clone the repository:
1. Clone the repository:
```sh
git clone <repository-url>
cd LiveGraphsDjango
```
```sh
git clone <repository-url>
cd LiveGraphsDjango
```
2. Install uv if you don't have it yet:
2. Install uv if you don't have it yet:
```sh
# Install using pip
pip install uv
```sh
# Install using pip
pip install uv
# Or with curl (Unix/macOS)
curl -sSf https://install.pypa.io/get-uv.py | python3 -
# Or with curl (Unix/macOS)
curl -sSf https://install.pypa.io/get-uv.py | python3 -
# Or on Windows with PowerShell
irm https://install.pypa.io/get-uv.ps1 | iex
```
# Or on Windows with PowerShell
irm https://install.pypa.io/get-uv.ps1 | iex
```
3. Create a virtual environment and activate it:
3. Create a virtual environment and activate it:
```sh
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
```
```sh
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
```
4. Install dependencies:
4. Install dependencies:
```sh
# Install all dependencies including dev dependencies
uv pip install -e ".[dev]"
```sh
# Install all dependencies including dev dependencies
uv pip install -e ".[dev]"
# Or just runtime dependencies
uv pip install -e .
```
# Or just runtime dependencies
uv pip install -e .
```
5. Run migrations:
5. Run migrations:
```sh
cd dashboard_project
python manage.py makemigrations
python manage.py migrate
```
```sh
cd dashboard_project
python manage.py makemigrations
python manage.py migrate
```
6. Create a superuser:
6. Create a superuser:
```sh
python manage.py createsuperuser
```
```sh
python manage.py createsuperuser
```
7. Set up environment variables:
7. Set up environment variables:
```sh
# Copy the sample .env file
cp .env.sample .env
```sh
# Copy the sample .env file
cp .env.sample .env
# Edit the .env file with your credentials
nano .env
```
# Edit the .env file with your credentials
nano .env
```
Be sure to update:
Be sure to update:
- `EXTERNAL_API_USERNAME` and `EXTERNAL_API_PASSWORD` for the data integration API
- `DJANGO_SECRET_KEY` for production environments
- Redis URL if using a different configuration for Celery
- `EXTERNAL_API_USERNAME` and `EXTERNAL_API_PASSWORD` for the data integration API
- `DJANGO_SECRET_KEY` for production environments
- Redis URL if using a different configuration for Celery
8. Start Celery for background tasks:
8. Start Celery for background tasks:
```sh
# In a separate terminal
cd dashboard_project
celery -A dashboard_project worker --loglevel=info
```sh
# In a separate terminal
cd dashboard_project
celery -A dashboard_project worker --loglevel=info
# Start the Celery Beat scheduler in another terminal
cd dashboard_project
celery -A dashboard_project beat --scheduler django_celery_beat.schedulers:DatabaseScheduler
```
# Start the Celery Beat scheduler in another terminal
cd dashboard_project
celery -A dashboard_project beat --scheduler django_celery_beat.schedulers:DatabaseScheduler
```
Alternative without Redis (using SQLite):
Alternative without Redis (using SQLite):
```sh
# Set environment variables to use SQLite instead of Redis
export CELERY_BROKER_URL=sqla+sqlite:///celery.sqlite
export CELERY_RESULT_BACKEND=db+sqlite:///results.sqlite
```sh
# Set environment variables to use SQLite instead of Redis
export CELERY_BROKER_URL=sqla+sqlite:///celery.sqlite
export CELERY_RESULT_BACKEND=db+sqlite:///results.sqlite
# In a separate terminal
cd dashboard_project
celery -A dashboard_project worker --loglevel=info
# In a separate terminal
cd dashboard_project
celery -A dashboard_project worker --loglevel=info
# Start the Celery Beat scheduler in another terminal with the same env vars
cd dashboard_project
celery -A dashboard_project beat --scheduler django_celery_beat.schedulers:DatabaseScheduler
```
# Start the Celery Beat scheduler in another terminal with the same env vars
cd dashboard_project
celery -A dashboard_project beat --scheduler django_celery_beat.schedulers:DatabaseScheduler
```
9. Run the development server:
9. Run the development server:
```sh
python manage.py runserver
```
```sh
python manage.py runserver
```
10. Access the application at <http://127.0.0.1:8000/>
10. Access the application at <http://127.0.0.1:8000/>
### Development Workflow with UV
UV offers several advantages over traditional pip, including faster dependency resolution and installation:
1. Running linting and formatting:
1. Running linting and formatting:
```sh
# Using the convenience script
./.scripts/lint.sh
```sh
# Using the convenience script
./.scripts/lint.sh
# Or directly
uv run -m ruff check dashboard_project
uv run -m ruff format dashboard_project
uv run -m black dashboard_project
```
# Or directly
uv run -m ruff check dashboard_project
uv run -m ruff format dashboard_project
uv run -m black dashboard_project
```
2. Running tests:
2. Running tests:
```sh
# Using the convenience script
./.scripts/test.sh
```sh
# Using the convenience script
./.scripts/test.sh
# Or directly
uv run -m pytest
```
# Or directly
uv run -m pytest
```
3. Adding new dependencies:
3. Adding new dependencies:
```sh
# Add to project
uv pip install package_name
```sh
# Add to project
uv pip install package_name
# Then update pyproject.toml manually
# And update the lockfile
uv pip freeze > requirements.lock
```
# Then update pyproject.toml manually
# And update the lockfile
uv pip freeze > requirements.lock
```
4. Updating the lockfile:
4. Updating the lockfile:
```sh
uv pip compile pyproject.toml -o uv.lock
```
```sh
uv pip compile pyproject.toml -o uv.lock
```
### Using Docker
1. Clone the repository:
1. Clone the repository:
```sh
git clone <repository-url>
cd dashboard_project
```
```sh
git clone <repository-url>
cd dashboard_project
```
2. Build and run with Docker Compose:
2. Build and run with Docker Compose:
```sh
docker-compose up -d --build
```
```sh
docker-compose up -d --build
```
3. Create a superuser:
3. Create a superuser:
```sh
docker-compose exec web python manage.py createsuperuser
```
```sh
docker-compose exec web python manage.py createsuperuser
```
4. Access the application at <http://localhost/>
4. Access the application at <http://localhost/>
## Usage
## Development Tools
1. Login as the superuser you created.
2. Go to the admin interface (<http://localhost/admin/>) and create companies and users.
3. Assign users to companies.
4. Upload CSV files for each company.
5. View the analytics dashboard.
### Prettier for Django Templates
This project uses Prettier with the `prettier-plugin-django-annotations` plugin to format HTML templates with Django template syntax.
#### Prettier Configuration
The project is already configured with Prettier integration in pre-commit hooks. The configuration includes:
1. `.prettierrc` - Configuration file with Django HTML support
2. `.prettierignore` - Files to exclude from formatting
3. Pre-commit hook for automatic formatting on commits
#### Manual Installation
To use Prettier locally (outside of pre-commit hooks), you'll need to install the dependencies:
```bash
# Using npm
npm install
# Or install just the required packages
npm install --save-dev prettier prettier-plugin-django-annotations
```
#### Usage
##### With Pre-commit
Prettier will automatically run as part of the pre-commit hooks when you commit changes.
To manually run the pre-commit hooks on all files:
```bash
pre-commit run prettier --all-files
```
##### Using npm Scripts
The package.json includes npm scripts for formatting:
```bash
# Format all static files
npm run format
# Check formatting without modifying files
npm run format:check
```
##### Command Line
You can also run Prettier directly:
```bash
# Format a specific file
npx prettier --write path/to/template.html
# Format all HTML files
npx prettier --write "dashboard_project/templates/**/*.html"
```
#### VSCode Integration
For VSCode users, install the Prettier extension and add these settings to your `.vscode/settings.json`:
```json
{
"editor.defaultFormatter": "esbenp.prettier-vscode",
"[html]": {
"editor.defaultFormatter": "esbenp.prettier-vscode",
"editor.formatOnSave": true
},
"prettier.requireConfig": true
}
```
#### Ignoring Parts of Templates
If you need to prevent Prettier from formatting a section of your template:
```html
{# prettier-ignore #}
<div>This section will not be formatted by Prettier.</div>
<!-- prettier-ignore -->
<div>
This works too.
</div>
```
#### Django Template Support
The `prettier-plugin-django-annotations` plugin provides special handling for Django templates, including:
- Proper formatting of Django template tags (`{% %}`)
- Support for Django template comments (`{# #}`)
- Preservation of Django template variable output (`{{ }}`)
- Special handling for Django template syntax inside HTML attributes
## Basic Usage Instructions
1. Login as the superuser you created.
2. Go to the admin interface (<http://localhost/admin/>) and create companies and users.
3. Assign users to companies.
4. Upload CSV files for each company.
5. View the analytics dashboard.
## Quick Start Guide
### Creating Sample Data (Optional)
To quickly populate the system with sample data:
```sh
python manage.py create_sample_data
```
This will create:
- Admin user (username: admin, password: admin123)
- Three companies with users
- Sample chat data and dashboards
### Admin Tasks
1. **Access Admin Panel**:
- Go to <http://localhost/admin/>
- Login with your admin credentials
2. **Create a Company**:
- Go to Companies > Add Company
- Fill in the company details and save
3. **Create Users**:
- Go to Users > Add User
- Fill in user details
- Assign the user to a company
- Set appropriate permissions (staff status, company admin)
### Company Admin Tasks
1. **Login to Dashboard**:
- Go to <http://localhost/>
- Login with your company admin credentials
2. **Upload Chat Data**:
- Click on "Upload Data" in the sidebar
- Fill in the data source details
- Select a CSV file containing chat data
- Click "Upload"
3. **Create a Dashboard**:
- Click on "New Dashboard" in the sidebar
- Fill in the dashboard details
- Select data sources to include
- Click "Create Dashboard"
### Regular User Tasks
1. **View Dashboard**:
- Login with your user credentials
- The dashboard will show automatically
- Select different dashboards from the sidebar
2. **Search Chat Sessions**:
- Click on "Search" in the top navigation
- Enter search terms
- Use filters to refine results
3. **View Session Details**:
- In search results, click the eye icon for a session
- View complete session information and transcript
### Dashboard Features
The dashboard includes:
- **Sessions Over Time**: Line chart showing chat volume trends
- **Sentiment Analysis**: Pie chart of positive/negative/neutral chats
- **Top Countries**: Bar chart of user countries
- **Categories**: Distribution of chat categories
### Data Source Details
View details for each data source:
- Upload date and time
- Total sessions
- Source description
- List of all chat sessions from the source
### Troubleshooting
#### CSV Upload Issues
If your CSV upload fails:
- Ensure all required columns are present
- Check date formats (should be YYYY-MM-DD HH:MM:SS)
- Verify boolean values (TRUE/FALSE, Yes/No, 1/0)
- Check for special characters in text fields
#### Access Issues
If you can't access certain features:
- Verify your user role (admin, company admin, or regular user)
- Ensure you're assigned to the correct company
- Check if you're trying to access another company's data
#### Empty Dashboard
If your dashboard is empty:
- Verify that data sources have been uploaded
- Check that the dashboard is configured to use those data sources
- Ensure the CSV was processed successfully
## CSV File Format
@ -224,14 +454,159 @@ The CSV file should contain the following columns:
| `initial_msg` | First message from the user (text) |
| `user_rating` | User rating of the conversation (string) |
Example CSV row:
```csv
acme_1,2023-05-01 10:30:00,2023-05-01 10:45:00,192.168.1.1,USA,English,10,Positive,FALSE,FALSE,"User: Hello\nAgent: Hi there!",2.5,500,0.01,Support,Hello I need help,Good
```
## Implementation Details
### Core Features Implemented
1. **Multi-Tenant Architecture**:
- Companies have isolated data and user access
- Users belong to specific companies
- Role-based permissions (admin, company admin, regular user)
2. **Data Management**:
- CSV file upload and processing
- Data source management
- Chat session records with comprehensive metadata
3. **Dashboard Visualization**:
- Interactive charts using Plotly.js
- Key metrics and KPIs
- Time-series analysis
- Geographic distribution
- Sentiment analysis
- Category distribution
4. **Search and Analysis**:
- Full-text search across chat sessions
- Filtering by various attributes
- Detailed view of individual chat sessions
- Transcript viewing
5. **User Management**:
- User registration and authentication
- Profile management
- Password change functionality
- Role assignment
6. **Admin Interface**:
- Company management
- User administration
- Data source oversight
- System-wide configuration
7. **Responsive Design**:
- Mobile-friendly interface using Bootstrap 5
- Consistent layout and navigation
- Accessible UI components
### Technical Implementation
#### Backend (Django)
- **Custom User Model**: Extended for company association and roles
- **Database Models**: Structured for efficient data storage and queries
- **View Logic**: Separation of concerns with dedicated view functions
- **Form Handling**: Validated data input and file uploads
- **Data Processing**: CSV parsing and structured storage
- **Template Context**: Prepared data for frontend rendering
- **URL Routing**: Clean URL structure
- **Access Control**: Permission checks throughout
#### Frontend
- **Bootstrap 5**: For responsive layout and UI components
- **Plotly.js**: For interactive charts and visualizations
- **jQuery**: For AJAX functionality
- **Font Awesome**: For icons
- **Custom CSS**: For styling enhancements
#### Data Flow
1. **Upload Process**:
- File validation
- CSV parsing
- Data normalization
- Record creation
- Association with company
2. **Dashboard Generation**:
- Data aggregation
- Statistical calculations
- Chart data preparation
- JSON serialization for frontend
3. **User Authentication**:
- Login/registration handling
- Session management
- Permission checks
- Access control based on company
#### Deployment Configuration
- **Docker**: Containerization for consistent deployment
- **Docker Compose**: Multi-container orchestration
- **Nginx**: Web server and static file serving
- **PostgreSQL**: Production-ready database
- **Gunicorn**: WSGI HTTP server
### Models
#### Accounts App
- **CustomUser**: Extends Django's User model with company association and role
- **Company**: Represents a company with users and data sources
#### Dashboard App
- **DataSource**: Represents an uploaded CSV file with chat data
- **ChatSession**: Stores individual chat session data parsed from CSV
- **Dashboard**: Allows configuration of custom dashboards with selected data sources
### Usage Flow
1. **Admin Setup**:
- Admin creates companies
- Admin creates users and assigns them to companies
2. **Company Admin**:
- Uploads CSV files with chat data
- Creates and configures dashboards
- Manages company users
3. **Regular Users**:
- View dashboards
- Search and explore chat data
- Analyze chat metrics
## Future Enhancements
- API integration for real-time data
- More advanced visualizations
- Custom reports
- Export functionality
- Theme customization
- User access control with more granular permissions
- API integration for real-time data
- More advanced visualizations
- Custom reports
- Export to additional formats (XML, HTML, PDF)
- Theme customization
- User access control with more granular permissions
- Direct integration with chat platforms via API
- Real-time dashboard updates using WebSockets
- Advanced analytics with machine learning
- Customizable reports and scheduling
- Enhanced visualization options
## License