README.md

---
title: Participatory Planning Application
emoji: 🏙️
colorFrom: blue
colorTo: purple
sdk: docker
pinned: false
license: mit
---

# Participatory Planning Application

An AI-powered collaborative urban planning platform for multi-stakeholder engagement sessions with advanced sentence-level categorization and fine-tuning capabilities.

## Author & Copyright

**Created by: Marcos Thadeu Queiroz Magalhães**
📧 [email protected]

Copyright © 2024-2025 Marcos Thadeu Queiroz Magalhães. All rights reserved.

This software must be cited and credited to Marcos Thadeu Queiroz Magalhães whenever used or employed in any capacity.

*Developed using [Claude Code](https://claude.com/claude-code) for AI-enhanced development.*

## Features

### Core Features
- 🎯 **Token-based access** - Self-service registration for participants
- 🤖 **AI categorization** - Automatic classification using BART zero-shot models (free & offline)
- 📝 **Sentence-level analysis** - Each sentence categorized independently for multi-topic submissions
- 🗺️ **Geographic mapping** - Interactive visualization of geotagged contributions
- 📊 **Analytics dashboard** - Real-time charts with submission and sentence-level aggregation
- 💾 **Session management** - Export/import for pause/resume workflows
- 👥 **Multi-stakeholder** - Government, Community, Industry, NGO, Academic, Other

### Advanced AI Features
- 🧠 **Model Fine-tuning** - Train custom models with LoRA or head-only methods
- 📈 **Real-time training progress** - Detailed epoch/step/loss tracking during training
- 🔄 **Training data management** - Export, import, and clear training examples
- 🎛️ **Multiple training modes** - Head-only (fast, <100 examples) or LoRA (better, >100 examples)
- 📦 **Model deployment** - Deploy fine-tuned models with one click
- 🗑️ **Force delete** - Remove stuck or problematic training runs

### Sentence-Level Categorization
- ✂️ **Smart segmentation** - Handles abbreviations, bullet points, and complex punctuation
- 🎯 **Independent classification** - Each sentence gets its own category
- 📊 **Category distribution** - View breakdown of categories within submissions
- 🔄 **Backward compatible** - Falls back to submission-level for legacy data
- ✏️ **Sentence editing** - Edit individual sentence categories in UI

## Categories

The system classifies text into six strategic planning categories:

1. **Vision** - Long-term aspirational goals and ideal future states
2. **Problem** - Current issues, challenges, and gaps
3. **Objectives** - Specific, measurable goals and targets
4. **Directives** - High-level mandates and policy directions
5. **Values** - Guiding principles and community priorities
6. **Actions** - Concrete implementation steps and projects

## Quick Start

### Basic Setup
1. Access the application at `http://localhost:5000`
2. Login with admin token: `<your-admin-token-from-startup>`
3. Go to **Registration** to get the participant signup link
4. Share the link with stakeholders
5. Collect submissions and analyze with AI

### Sentence-Level Analysis Workflow
1. **Collect Submissions** - Participants submit via web form
2. **Run Analysis** - Click "Analyze All" in Admin → Submissions
3. **Review Sentences** - Click "View Sentences" on any submission
4. **Correct Categories** - Edit sentence categories as needed (creates training data)
5. **Train Model** - Once you have 20+ sentence corrections, train a custom model
6. **Deploy Model** - Activate your fine-tuned model for better accuracy

## Default Login

- **Admin Token**: `<your-admin-token-from-startup>`
- **Admin Access**: Full dashboard, analytics, moderation, AI training

## Tech Stack

- **Backend**: Flask (Python web framework)
- **Database**: SQLite with sentence-level schema
- **AI Models**:
  - BART-large-MNLI (default, 400M parameters)
  - DeBERTa-v3-base-MNLI (fast, 86M parameters)
  - DistilBART-MNLI (balanced, 134M parameters)
- **Fine-tuning**: LoRA (Low-Rank Adaptation) with PEFT
- **Frontend**: Bootstrap 5, Leaflet.js, Chart.js
- **Deployment**: Docker support

## AI Training

### Training Data Management

**Export Training Examples**
- Download all training data as JSON
- Option to export only sentence-level examples
- Use for backups or sharing datasets

**Import Training Examples**
- Load training data from JSON files
- Automatically skips duplicates
- Useful for migrating between environments

**Clear Training Examples**
- Remove unused examples to clean up
- Option to clear only sentence-level data
- Safe defaults prevent accidental deletion

### Training Modes

**Head-Only Training** (Recommended for <100 examples)
- Faster training (2-5 minutes)
- Lower memory usage
- Good for small datasets
- Only trains classification layer

**LoRA Fine-tuning** (Recommended for >100 examples)
- Better accuracy on larger datasets
- Parameter-efficient (trains adapter layers)
- Configurable rank, alpha, dropout
- Takes 5-15 minutes depending on data size

### Progress Tracking

During training, you'll see:
- Current epoch / total epochs
- Current step / total steps
- Real-time loss values
- Precise progress percentage
- Estimated time remaining

### Model Management

- Deploy models with one click
- Rollback to base model anytime
- Export trained models as ZIP files
- Force delete stuck or failed runs
- View detailed training metrics

## Demo Data

The app starts empty. You can:
1. Generate tokens for test users
2. Submit sample contributions (multi-sentence for best results)
3. Run AI sentence-level analysis
4. Correct sentence categories to build training data
5. Train a custom fine-tuned model
6. View analytics in submission or sentence mode

## File Structure

```
participatory_planner/
├── app/
│   ├── analyzer.py              # AI classification engine
│   ├── sentence_segmenter.py    # Sentence splitting logic
│   ├── models/
│   │   └── models.py            # Database models (Submission, SubmissionSentence, etc.)
│   ├── routes/
│   │   ├── admin.py             # Admin dashboard and API endpoints
│   │   └── main.py              # Public submission forms
│   ├── fine_tuning/
│   │   ├── trainer.py           # LoRA fine-tuning engine
│   │   └── model_manager.py     # Model deployment/rollback
│   └── templates/
│       ├── admin/
│       │   ├── submissions.html # Sentence-level UI
│       │   ├── dashboard.html   # Analytics with dual modes
│       │   └── training.html    # Fine-tuning interface
│       └── submit.html          # Public submission form
├── migrations/
│   └── migrate_to_sentence_level.py
├── models/
│   ├── finetuned/              # Trained model checkpoints
│   └── zero_shot/              # Base BART models
├── data/
│   └── app.db                  # SQLite database
└── README.md
```

## Environment Variables

```bash
SECRET_KEY=your-secret-key-here
MODELS_DIR=models/finetuned
ZERO_SHOT_MODELS_DIR=models/zero_shot
```

## API Endpoints

### Public
- `POST /submit` - Submit new contribution
- `GET /register/:token` - Participant registration

### Admin (requires auth)
- `POST /admin/api/analyze` - Analyze submissions with sentences
- `POST /admin/api/update-sentence-category/:id` - Edit sentence category
- `GET /admin/api/export-training-examples` - Export training data
- `POST /admin/api/import-training-examples` - Import training data
- `POST /admin/api/clear-training-examples` - Clear training data
- `POST /admin/api/start-fine-tuning` - Start model training
- `GET /admin/api/training-status/:id` - Get training progress
- `POST /admin/api/deploy-model/:id` - Deploy fine-tuned model
- `DELETE /admin/api/force-delete-training-run/:id` - Force delete run

## Database Schema

### Key Tables

**submissions**
- Core submission data
- `sentence_analysis_done` flag for tracking
- Backward compatible with old category field

**submission_sentences**
- Individual sentences from submissions
- Each sentence has its own category
- Linked to parent submission via foreign key

**training_examples**
- Admin corrections for fine-tuning
- Supports both sentence and submission-level
- Tracks usage in training runs

**fine_tuning_runs**
- Training job metadata and results
- Real-time progress tracking fields
- Model paths and deployment status

## Troubleshooting

**Training stuck at 0% progress?**
- Check if CUDA is available or forcing CPU mode
- Reduce batch size if out of memory
- Check training logs for errors

**Sentences not being categorized?**
- Run database migration: `python migrations/migrate_to_sentence_level.py`
- Ensure `sentence_analysis_done` column exists
- Check that sentence segmenter is working

**Can't delete training run?**
- Use "Force Delete" button for active/training runs
- Type "DELETE" to confirm force deletion
- Check model files aren't locked

## License

MIT License - See [LICENSE](LICENSE) file for full details.

Copyright © 2024-2025 Marcos Thadeu Queiroz Magalhães ([email protected])

### Citation Requirement

Any use, modification, or distribution of this software must include proper attribution to:

**Marcos Thadeu Queiroz Magalhães ([email protected])**

This includes:
- Source code and documentation
- Academic publications or presentations
- Derivative works and commercial applications
- User interfaces and credits sections

## Contributing

Contributions welcome! Please:
1. Fork the repository
2. Create a feature branch
3. Maintain attribution to the original author in all contributions
4. Submit a pull request with clear description

## Support

For issues or questions:
1. Check existing documentation files
2. Review troubleshooting section above
3. Contact: [email protected]
4. Open an issue with detailed description