From f629321dca7832794a0184227cbd6f2845f99789 Mon Sep 17 00:00:00 2001 From: Christopher Odoom Date: Sun, 29 Jun 2025 18:14:14 -0400 Subject: [PATCH] Implement model database optimization --- .DS_Store | Bin 14340 -> 14340 bytes docs/DATABASE_OPTIMIZATION_PLAN.md | 659 ++++++++++++++++++ docs/IMPROVED_RECOMMENDATION_ALGORITHM.md | 808 ++++++++++++++++++++++ docs/PRACTICAL_IMPLEMENTATION_PLAN.md | 353 ++++++++++ docs/QUICK_IMPLEMENTATION_REFERENCE.md | 116 ++++ docs/README.md | 47 ++ docs/SYNTHETIC_DATA_RATIONALE.md | 216 ++++++ 7 files changed, 2199 insertions(+) create mode 100644 docs/DATABASE_OPTIMIZATION_PLAN.md create mode 100644 docs/IMPROVED_RECOMMENDATION_ALGORITHM.md create mode 100644 docs/PRACTICAL_IMPLEMENTATION_PLAN.md create mode 100644 docs/QUICK_IMPLEMENTATION_REFERENCE.md create mode 100644 docs/README.md create mode 100644 docs/SYNTHETIC_DATA_RATIONALE.md diff --git a/.DS_Store b/.DS_Store index a3cde8d3e1942387b4e040cec3af62e2d58ea2d9..853705795fbf5e186ba9dd724954ebf126bc1e3a 100644 GIT binary patch delta 633 zcmZoEXepQ=&DcIs##l6tfq{XUfkA+QA)ldyA)XkcfoxlPOR#f?d48!2${M-VdJOe}Ag^d?k*f+Ct zuy8PH0}WG}Tq5Yhy2CE}yZYp#g7R!&7VG9$f=-MaM&_nE3dZIZlhY*ZCl?D#G3zD0 zn>;~SMtkM1#sH`-Yz)OfmnH&nCPOl^OQ32dUl*2RGEdn2UD%(AZDxPQcXiguK4K~y z5aw~m$^S&fCijUIv30KmI)QaEo45jusUz;dBxg_H|Cq=4d!&_fE@ z{X~T1=4ORTMy4XM$^R79IL`UK`k?|0c_2q-vWL>~$`LPX0$mCz5 zI*PJ#K)YFi{z(P0E)F51kOAW$>pg>_@n9Qww6Bx)0bCOslE2yYU=2PJTg?SDz z%oS9YfD*%HS&*aqRCI(8re`Q!TLKlkU4!#l1Jzu{1ea0NoGDg#!<2+u_Y-_S-a;b{{unUxz0PJrRLjV8( diff --git a/docs/DATABASE_OPTIMIZATION_PLAN.md b/docs/DATABASE_OPTIMIZATION_PLAN.md new file mode 100644 index 0000000..9ad35df --- /dev/null +++ b/docs/DATABASE_OPTIMIZATION_PLAN.md @@ -0,0 +1,659 @@ +# Model Database Optimization Implementation Plan + +## Overview + +This document outlines the implementation strategy for optimizing the Statistical Model Suggester's database architecture and template system for better performance, scalability, and maintainability. + +## Current Issues Summary + +### Database Problems + +- **File Size**: 204KB JSON file loaded entirely at startup +- **Memory Usage**: All 396 models + implementation code kept in memory +- **Search Performance**: Linear scan through all models for recommendations +- **Code Duplication**: Massive repetition in R code, Python implementations +- **No Lazy Loading**: Everything pre-loaded regardless of usage + +### Critical Issue: Synthetic Data Storage Anti-Pattern + +**Problem Identified**: Synthetic data is currently stored as R code strings within JSON, creating severe inefficiencies: + +1. **Code Duplication**: Same R scripts exist both in JSON strings AND as separate `.R` files in `synthetic_data_examples/` +2. **Maintainability Crisis**: R code embedded in JSON loses all IDE benefits (syntax highlighting, linting, debugging) +3. **Performance Impact**: Large code strings bloat JSON files (3000+ lines) and slow parsing +4. **Version Control Issues**: Code changes require JSON manipulation instead of direct file editing +5. **Testing Difficulties**: Cannot independently test or validate synthetic data scripts + +**Root Cause**: Storing executable code as JSON strings violates separation of concerns and creates a maintenance nightmare. + +### Template Problems + +- **Heavy Data Dependencies**: Templates expect fully-populated complex objects +- **Synchronous Loading**: All content loaded at once, including large plots +- **Static File Coupling**: Hardcoded paths to diagnostic plots +- **No Error Handling**: Missing fallbacks for unavailable content +- **Monolithic Structure**: Single large template instead of modular components + +## Phase 1: Database Architecture Refactoring + +### 1.1 Data Separation Strategy + +```text +data/ +├── models/ +│ ├── metadata.json # Core searchable fields only (~20KB) +│ ├── descriptions.json # Model descriptions and use cases +│ ├── implementations/ # Language-specific code +│ │ ├── python.json +│ │ ├── r.json +│ │ ├── spss.json +│ │ ├── sas.json +│ │ └── stata.json +│ └── interpretations.json # All interpretation guides in one file +│ +├── synthetic_data/ # EXECUTABLE SCRIPTS (not JSON strings!) +│ ├── scripts/ # Organized by statistical model category +│ │ ├── regression/ +│ │ │ ├── linear_regression.R +│ │ │ ├── logistic_regression.R +│ │ │ ├── poisson_regression.R +│ │ │ ├── multiple_regression.py +│ │ │ └── polynomial_regression.R +│ │ ├── time_series/ +│ │ │ ├── arima_example.R +│ │ │ ├── var_model.R +│ │ │ ├── garch_volatility.R +│ │ │ ├── prophet_forecasting.py +│ │ │ └── seasonal_decomposition.R +│ │ ├── survival/ +│ │ │ ├── cox_regression.R +│ │ │ ├── kaplan_meier.R +│ │ │ └── accelerated_failure_time.R +│ │ ├── machine_learning/ +│ │ │ ├── random_forest.R +│ │ │ ├── svm_classification.py +│ │ │ ├── xgboost_example.py +│ │ │ ├── neural_network.py +│ │ │ └── gradient_boosting.R +│ │ ├── clustering/ +│ │ │ ├── kmeans_example.R +│ │ │ ├── hierarchical_clustering.R +│ │ │ ├── dbscan_clustering.py +│ │ │ └── gaussian_mixture.py +│ │ ├── hypothesis_testing/ +│ │ │ ├── t_test_examples.R +│ │ │ ├── anova_designs.R +│ │ │ ├── chi_square_tests.R +│ │ │ └── nonparametric_tests.R +│ │ └── shared/ +│ │ ├── data_generators.R # Reusable data generation functions +│ │ ├── plot_helpers.R # Standard plotting functions +│ │ ├── validation_utils.py # Data validation and checking +│ │ └── common_parameters.json # Standard configurations +│ ├── registry.json # Maps model names to script file paths +│ ├── execution_config.json # Runtime parameters, dependencies, R packages +│ └── results_cache/ # Optional: pre-computed outputs for speed +│ ├── outputs/ # Cached script execution results +│ ├── plots/ # Generated plot files +│ └── datasets/ # Reusable generated datasets +│ +└── templates/ # Reusable code and interpretation templates + ├── sklearn_template.py + ├── r_glm_template.R + └── interpretation_templates.json +``` + +### Key Innovation: Script-Based Synthetic Data + +Instead of storing R code as JSON strings (current anti-pattern), we use executable script files: + +**Script Registry** (`synthetic_data/registry.json`): + +```json +{ + "Linear Regression": { + "script_path": "scripts/regression/linear_regression.R", + "language": "R", + "dependencies": ["base", "stats"], + "estimated_runtime": "5s", + "generates_plots": true, + "dataset_size": "small" + }, + "ARIMA": { + "script_path": "scripts/time_series/arima_example.R", + "language": "R", + "dependencies": ["forecast", "tseries"], + "estimated_runtime": "15s", + "generates_plots": true, + "dataset_size": "medium" + } +} +``` + +**Benefits:** + +- ✅ **Maintainability**: Full IDE support (syntax highlighting, debugging) +- ✅ **Performance**: No JSON parsing of large code strings +- ✅ **Reusability**: Shared functions eliminate duplication +- ✅ **Testability**: Scripts can be independently tested +- ✅ **Version Control**: Proper diff tracking for code changes +- ✅ **Modularity**: Clear separation by model category + +### 1.2 Model Service Layer + +```python +# utils/model_service.py +class ModelService: + """Efficient model data access with caching and script execution""" + + def __init__(self): + self.metadata_cache = {} + self.script_registry = {} + self.results_cache = {} + + # Core methods: + def get_model_metadata(self, name: str) -> Dict[str, Any] + def search_models(self, **criteria) -> List[str] + def get_implementation(self, name: str, language: str) -> Optional[Dict] + def get_interpretation_guide(self, name: str) -> Optional[Dict] + + # NEW: Script-based synthetic data methods + def get_synthetic_data_info(self, name: str) -> Optional[Dict]: + """Get script metadata without executing""" + return self.script_registry.get(name) + + def execute_synthetic_data_script(self, name: str, cache=True) -> Dict: + """Execute R/Python script and return results""" + script_info = self.script_registry.get(name) + if not script_info: + raise ValueError(f"No synthetic data script for {name}") + + # Check cache first + if cache and name in self.results_cache: + return self.results_cache[name] + + # Execute script based on language + if script_info['language'] == 'R': + result = self._execute_r_script(script_info['script_path']) + elif script_info['language'] == 'python': + result = self._execute_python_script(script_info['script_path']) + else: + raise ValueError(f"Unsupported language: {script_info['language']}") + + # Cache results if requested + if cache: + self.results_cache[name] = result + + return result + + def _execute_r_script(self, script_path: str) -> Dict: + """Execute R script and capture output, plots, warnings""" + # Implementation details for R script execution + pass + + def _execute_python_script(self, script_path: str) -> Dict: + """Execute Python script and capture output""" + # Implementation details for Python script execution + pass + + # Caching strategy: + # - Metadata always in memory (lightweight) + # - Script registry loaded once at startup + # - Script execution results cached with LRU eviction + # - Background cache warming for popular models +``` + +### 1.3 Database Migration Strategy + +```python +# scripts/migrate_database.py +def migrate_current_database(): + """Split monolithic JSON into optimized structure with script extraction""" + + # Step 1: Extract metadata + extract_metadata() # -> models/metadata.json + + # Step 2: Separate implementations + extract_implementations() # -> implementations/*.json + + # Step 3: CRITICAL: Extract synthetic data code to script files + extract_synthetic_data_scripts() + + # Step 4: Create script registry + create_script_registry() + + # Step 5: Create interpretation guides + extract_interpretations() + + # Step 6: Validate migration + validate_migration() + +def extract_synthetic_data_scripts(): + """Convert JSON-embedded R code to executable script files""" + + # Load current model database + with open('data/model_database.json', 'r') as f: + models = json.load(f) + + script_registry = {} + + for model_name, model_data in models.items(): + if 'synthetic_data' in model_data and 'r_code' in model_data['synthetic_data']: + # Extract R code from JSON string + r_code = model_data['synthetic_data']['r_code'] + + # Determine category and script path + category = determine_model_category(model_name) + script_filename = f"{model_name.lower().replace(' ', '_')}.R" + script_path = f"synthetic_data/scripts/{category}/{script_filename}" + + # Write R code to file + os.makedirs(os.path.dirname(script_path), exist_ok=True) + with open(script_path, 'w') as script_file: + script_file.write(add_script_header(model_name)) + script_file.write(r_code) + script_file.write(add_script_footer()) + + # Create registry entry + script_registry[model_name] = { + "script_path": script_path, + "language": "R", + "dependencies": extract_r_dependencies(r_code), + "estimated_runtime": estimate_runtime(r_code), + "generates_plots": check_for_plots(r_code), + "dataset_size": estimate_dataset_size(r_code) + } + + # Save script registry + with open('synthetic_data/registry.json', 'w') as f: + json.dump(script_registry, f, indent=2) + +def determine_model_category(model_name: str) -> str: + """Categorize model for script organization""" + category_map = { + 'regression': ['Linear Regression', 'Logistic Regression', 'Poisson Regression'], + 'time_series': ['ARIMA', 'VAR', 'GARCH', 'Prophet'], + 'survival': ['Cox Regression', 'Kaplan-Meier'], + 'machine_learning': ['Random Forest', 'SVM', 'XGBoost', 'Neural Network'], + 'clustering': ['K-Means', 'Hierarchical Clustering', 'DBSCAN'], + 'hypothesis_testing': ['T-Test', 'ANOVA', 'Chi-Square'] + } + + for category, models in category_map.items(): + if any(model in model_name for model in models): + return category + + return 'other' # Default category + +def add_script_header(model_name: str) -> str: + """Add standardized header to script files""" + return f"""#!/usr/bin/env Rscript +# Synthetic Data Generation Script: {model_name} +# Generated by Statistical Model Suggester migration +# Date: {datetime.now().strftime('%Y-%m-%d')} + +# Load required libraries +library(base) +library(stats) + +""" + +def extract_r_dependencies(r_code: str) -> List[str]: + """Extract R package dependencies from code""" + import re + + # Find library() and require() calls + library_pattern = r'library\(([^)]+)\)' + require_pattern = r'require\(([^)]+)\)' + + libraries = re.findall(library_pattern, r_code) + requires = re.findall(require_pattern, r_code) + + # Clean up package names (remove quotes) + dependencies = [lib.strip('"\'') for lib in libraries + requires] + + # Add base dependencies + base_deps = ['base', 'stats'] + for dep in base_deps: + if dep not in dependencies: + dependencies.append(dep) + + return dependencies +``` + +## Phase 2: Service Integration + +### 2.1 App Integration Points + +```python +# app.py modifications +def create_app(): + # Replace current model loading + # OLD: Load entire 204KB JSON + # NEW: Initialize service with metadata only + + from utils.model_service import ModelService + model_service = ModelService() + model_service.init_app(app) + app.extensions['model_service'] = model_service +``` + +### 2.2 Route Handler Updates + +```python +# routes/main_routes.py modifications + +# Replace direct MODEL_DATABASE access: +# OLD: MODEL_DATABASE = current_app.config.get('MODEL_DATABASE', {}) +# NEW: model_service = current_app.extensions['model_service'] + +@main.route('/model/') +def model_details(name): + # OLD: Load full model data + # NEW: Load only needed sections + metadata = model_service.get_model_metadata(name) + # Implementation loaded on-demand via AJAX + +@main.route('/api/model//implementation/') +def get_implementation(name, language): + # New endpoint for lazy loading + return jsonify(model_service.get_implementation(name, language)) +``` + +### 2.3 Backwards Compatibility + +```python +# utils/compatibility.py +class ModelDatabaseCompat: + """Maintains existing API while using new backend""" + + def __init__(self, model_service): + self.service = model_service + + def __getitem__(self, key): + # Lazy load full model on access + return self.service.get_model(key) + + def keys(self): + return self.service.get_all_model_names() + + def items(self): + # Generator to avoid loading everything + for name in self.service.get_all_model_names(): + yield name, self.service.get_model(name) +``` + +## Phase 3: Template System Optimization + +### 3.1 Modular Template Structure + +```text +templates/ +├── model_interpretation/ +│ ├── base.html # Layout and navigation +│ ├── sections/ +│ │ ├── introduction.html # Basic model info +│ │ ├── data_description.html +│ │ ├── model_output.html +│ │ ├── coefficients.html # Loaded via AJAX +│ │ ├── diagnostic_plots.html # Progressive loading +│ │ ├── assumptions.html +│ │ ├── predictions.html +│ │ └── pitfalls.html +│ └── components/ +│ ├── loading_spinner.html +│ ├── error_fallback.html +│ └── plot_placeholder.html +``` + +### 3.2 Progressive Loading Template + +```html + +
+ + {% include 'model_interpretation/sections/introduction.html' %} + + +
+

Diagnostic Plots

+
+ +
+
+ + +
+ {% include 'model_interpretation/components/loading_spinner.html' %} +
+
+``` + +### 3.3 AJAX Loading System + +```javascript +// static/js/model_interpretation.js +class ModelInterpretationLoader { + constructor(modelName) { + this.modelName = modelName; + this.loadedSections = new Set(); + this.initializeEventListeners(); + this.initializeLazyLoading(); + } + + async loadSection(sectionName) { + if (this.loadedSections.has(sectionName)) return; + + try { + const response = await fetch(`/api/model/${this.modelName}/${sectionName}`); + const html = await response.text(); + document.getElementById(`${sectionName}-container`).innerHTML = html; + this.loadedSections.add(sectionName); + } catch (error) { + this.showErrorFallback(sectionName, error); + } + } + + initializeLazyLoading() { + const observer = new IntersectionObserver((entries) => { + entries.forEach(entry => { + if (entry.isIntersecting) { + const section = entry.target.dataset.endpoint; + this.loadSection(section); + observer.unobserve(entry.target); + } + }); + }); + + document.querySelectorAll('.lazy-load').forEach(el => { + observer.observe(el); + }); + } +} +``` + +## Phase 4: API Endpoints for Lazy Loading + +### 4.1 New API Routes + +```python +# routes/api_routes.py (new file) +@api.route('/model//coefficients') +def get_model_coefficients(name): + """Load coefficient interpretation on demand""" + interpretation = model_service.get_interpretation_guide(name) + return render_template('model_interpretation/sections/coefficients.html', + interpretation=interpretation) + +@api.route('/model//plots') +def get_diagnostic_plots(name): + """Load diagnostic plots progressively""" + plots = model_service.get_diagnostic_plots(name) + return render_template('model_interpretation/sections/diagnostic_plots.html', + plots=plots) + +@api.route('/model//implementation/') +def get_implementation_code(name, language): + """Load implementation code on demand""" + code = model_service.get_implementation(name, language) + return jsonify(code) +``` + +### 4.2 Caching Strategy + +```python +# utils/cache_manager.py +class CacheManager: + """Multi-level caching for model data""" + + def __init__(self): + self.memory_cache = {} # Frequently accessed + self.disk_cache = {} # Recently accessed + self.metrics = {} # Usage tracking + + def get_with_fallback(self, key, loader_func): + # Memory -> Disk -> Database -> Generate + pass + + def warm_cache(self, popular_models): + # Background task to pre-load popular models + pass +``` + +## Phase 5: Performance Optimizations + +### 5.1 Database Indexing (Future SQLite Migration) + +```sql +-- When migrating to SQLite +CREATE TABLE models ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + description TEXT, + category TEXT +); + +CREATE TABLE model_criteria ( + model_id TEXT, + criterion_type TEXT, -- 'analysis_goal', 'dependent_variable', etc. + criterion_value TEXT, + FOREIGN KEY (model_id) REFERENCES models(id) +); + +CREATE INDEX idx_criteria_type_value ON model_criteria(criterion_type, criterion_value); +CREATE INDEX idx_model_category ON models(category); +``` + +### 5.2 Search Optimization + +```python +# utils/search_engine.py +class ModelSearchEngine: + """Optimized model search with indexing""" + + def __init__(self, metadata): + self.build_indexes(metadata) + + def build_indexes(self, metadata): + # Create lookup tables for fast searching + self.by_analysis_goal = defaultdict(list) + self.by_dependent_var = defaultdict(list) + self.by_sample_size = defaultdict(list) + + for model_name, data in metadata.items(): + for goal in data.get('analysis_goals', []): + self.by_analysis_goal[goal].append(model_name) + # ... build other indexes + + def search(self, **criteria): + # Intersection of index lookups instead of linear scan + candidate_sets = [] + + if 'analysis_goal' in criteria: + candidates = set(self.by_analysis_goal[criteria['analysis_goal']]) + candidate_sets.append(candidates) + + # Intersect all candidate sets + if candidate_sets: + return list(set.intersection(*candidate_sets)) + return [] +``` + +## Implementation Timeline + +### Week 1-2: Database Refactoring + +- [ ] Create migration script +- [ ] Split monolithic JSON +- [ ] Implement ModelService class +- [ ] Add unit tests + +### Week 3: Service Integration + +- [ ] Update app.py initialization +- [ ] Modify route handlers +- [ ] Add compatibility layer +- [ ] Test existing functionality + +### Week 4-5: Template Optimization + +- [ ] Create modular template structure +- [ ] Implement AJAX loading system +- [ ] Add error handling and fallbacks +- [ ] Update existing templates + +### Week 6: API and Caching + +- [ ] Create new API endpoints +- [ ] Implement caching strategy +- [ ] Add performance monitoring +- [ ] Load testing + +### Week 7: Testing and Optimization + +- [ ] Performance benchmarking +- [ ] Bug fixes and optimizations +- [ ] Documentation updates +- [ ] Deployment preparation + +## Success Metrics + +### Performance Improvements + +- **Startup Time**: 204KB → 20KB initial load (90% reduction) +- **Memory Usage**: Only active models in memory (80% reduction) +- **Search Speed**: O(n) → O(1) lookup (100x improvement for large datasets) +- **Page Load Time**: Progressive loading (50% faster perceived performance) + +### Maintainability Improvements + +- **Code Duplication**: Template-based implementation generation +- **Separation of Concerns**: Clear boundaries between data layers +- **Scalability**: Easy to add new models without performance penalty +- **Testing**: Modular components easier to unit test + +## Risk Mitigation + +### Backwards Compatibility + +- Maintain existing API surface +- Gradual migration path +- Rollback capability + +### Data Integrity + +- Validation scripts for migrated data +- Automated tests for data consistency +- Backup and recovery procedures + +### Performance Regression + +- Benchmarking before/after +- Monitoring and alerting +- Load testing with realistic data + +This implementation plan provides a structured approach to modernizing the model database architecture while maintaining system stability and improving performance. diff --git a/docs/IMPROVED_RECOMMENDATION_ALGORITHM.md b/docs/IMPROVED_RECOMMENDATION_ALGORITHM.md new file mode 100644 index 0000000..f19a099 --- /dev/null +++ b/docs/IMPROVED_RECOMMENDATION_ALGORITHM.md @@ -0,0 +1,808 @@ +# Improved Recommendation Algorithm Proposals + +## Current Algorithm Limitations + +### 1. Hard-coded Scoring System +```python +# Current approach - inflexible +if analysis_goal in model.get('analysis_goals', []): + score += 3 # Why 3? Why not 2.8 or 3.2? + +if variables_correlated == 'yes' and model_name in regularization_models: + score += 3.5 # Magic number with no justification +``` + +### 2. No Learning or Adaptation +- Algorithm never improves from user feedback +- No mechanism to track recommendation success +- Static rules can't adapt to new patterns + +### 3. Limited Context Awareness +- Doesn't consider user expertise level +- No domain-specific recommendations +- Ignores computational constraints +- No data quality assessment + +## Proposed Improvements + +### Option 1: Machine Learning-Based System + +```python +# utils/ml_recommender.py +import pandas as pd +from sklearn.ensemble import RandomForestClassifier +from sklearn.feature_extraction import DictVectorizer +from sklearn.model_selection import train_test_split +import joblib + +class MLModelRecommender: + """Machine learning-based model recommendation system""" + + def __init__(self): + self.vectorizer = DictVectorizer() + self.classifier = RandomForestClassifier(n_estimators=100, random_state=42) + self.is_trained = False + + def extract_features(self, user_input): + """Convert user input to feature vector""" + features = { + 'analysis_goal': user_input['analysis_goal'], + 'dependent_variable': user_input['dependent_variable'], + 'sample_size_category': self._categorize_sample_size(user_input['sample_size']), + 'missing_data': user_input['missing_data'], + 'data_distribution': user_input['data_distribution'], + 'relationship_type': user_input['relationship_type'], + 'variables_correlated': user_input['variables_correlated'], + 'num_independent_vars': len(user_input['independent_variables']), + 'has_continuous_vars': 'continuous' in user_input['independent_variables'], + 'has_categorical_vars': 'categorical' in user_input['independent_variables'], + 'has_binary_vars': 'binary' in user_input['independent_variables'], + } + + # Add domain context if available + if 'domain' in user_input: + features['domain'] = user_input['domain'] + + # Add user expertise level + if 'user_expertise' in user_input: + features['user_expertise'] = user_input['user_expertise'] + + return features + + def train_from_historical_data(self, historical_analyses): + """Train the model from historical user analyses""" + features = [] + labels = [] + + for analysis in historical_analyses: + feature_dict = self.extract_features(analysis) + features.append(feature_dict) + labels.append(analysis['recommended_model']) + + # Convert to feature matrix + X = self.vectorizer.fit_transform(features) + y = labels + + # Train the classifier + self.classifier.fit(X, y) + self.is_trained = True + + # Save the trained model + self._save_model() + + def predict(self, user_input, top_k=5): + """Predict top-k model recommendations""" + if not self.is_trained: + self._load_model() + + features = self.extract_features(user_input) + X = self.vectorizer.transform([features]) + + # Get prediction probabilities + probabilities = self.classifier.predict_proba(X)[0] + classes = self.classifier.classes_ + + # Sort by probability + model_scores = list(zip(classes, probabilities)) + model_scores.sort(key=lambda x: x[1], reverse=True) + + return model_scores[:top_k] + + def update_with_feedback(self, user_input, chosen_model, rating): + """Update model with user feedback (online learning)""" + # Implementation for incremental learning + pass +``` + +### Option 2: Hybrid Scoring System with Dynamic Weights + +```python +# utils/hybrid_recommender.py +import numpy as np +from typing import Dict, List, Tuple +import json + +class HybridRecommender: + """Combines rule-based and ML approaches with dynamic weight learning""" + + def __init__(self): + self.base_weights = self._load_base_weights() + self.user_feedback_weights = {} + self.success_rates = {} + + def recommend(self, user_input: Dict) -> List[Tuple[str, float, str]]: + """Generate recommendations with explanations""" + # 1. Rule-based scoring (current system improved) + rule_scores = self._calculate_rule_based_scores(user_input) + + # 2. Similarity-based scoring + similarity_scores = self._calculate_similarity_scores(user_input) + + # 3. Success rate scoring (based on historical performance) + success_scores = self._calculate_success_scores(user_input) + + # 4. User preference scoring (personalized) + preference_scores = self._calculate_preference_scores(user_input) + + # 5. Combine scores with learned weights + final_scores = self._combine_scores( + rule_scores, similarity_scores, success_scores, preference_scores + ) + + # 6. Generate explanations + recommendations = [] + for model, score in final_scores: + explanation = self._generate_explanation(model, user_input, score) + recommendations.append((model, score, explanation)) + + return recommendations + + def _calculate_rule_based_scores(self, user_input: Dict) -> Dict[str, float]: + """Improved version of current rule-based system""" + MODEL_DATABASE = self._get_model_database() + scores = {} + + for model_name, model_info in MODEL_DATABASE.items(): + score = 0.0 + + # Core compatibility (weighted by importance and confidence) + compatibility_scores = { + 'analysis_goal': self._score_compatibility( + user_input['analysis_goal'], + model_info.get('analysis_goals', []), + weight=3.0, confidence=0.9 + ), + 'dependent_variable': self._score_compatibility( + user_input['dependent_variable'], + model_info.get('dependent_variable', []), + weight=3.0, confidence=0.9 + ), + 'relationship_type': self._score_compatibility( + user_input['relationship_type'], + model_info.get('relationship_type', []), + weight=2.0, confidence=0.7 + ), + # ... other factors + } + + # Apply uncertainty-aware scoring + for factor, (base_score, confidence) in compatibility_scores.items(): + adjusted_score = base_score * confidence + score += adjusted_score + + scores[model_name] = score + + return scores + + def _calculate_similarity_scores(self, user_input: Dict) -> Dict[str, float]: + """Score based on similarity to successful past analyses""" + # Find similar historical analyses + similar_analyses = self._find_similar_analyses(user_input) + + scores = {} + for analysis in similar_analyses: + similarity = self._calculate_similarity(user_input, analysis) + model = analysis['recommended_model'] + success_rating = analysis.get('user_rating', 3.0) # Default neutral + + if model not in scores: + scores[model] = 0.0 + + scores[model] += similarity * success_rating + + return scores + + def learn_from_feedback(self, user_input: Dict, recommendations: List, + chosen_model: str, rating: float): + """Update system based on user feedback""" + # Update success rates + context_key = self._create_context_key(user_input) + + if context_key not in self.success_rates: + self.success_rates[context_key] = {} + + if chosen_model not in self.success_rates[context_key]: + self.success_rates[context_key][chosen_model] = [] + + self.success_rates[context_key][chosen_model].append(rating) + + # Update weights based on performance + self._update_weights(user_input, recommendations, chosen_model, rating) + + # Save updated weights + self._save_weights() +``` + +### Option 3: Multi-Armed Bandit Approach + +```python +# utils/bandit_recommender.py +import numpy as np +from collections import defaultdict + +class BanditRecommender: + """Multi-armed bandit for recommendation with exploration/exploitation""" + + def __init__(self, epsilon=0.1): + self.epsilon = epsilon # Exploration rate + self.arm_counts = defaultdict(int) # Number of times each model was recommended + self.arm_rewards = defaultdict(list) # Rewards for each model + self.context_arms = defaultdict(dict) # Context-specific arm performance + + def recommend(self, user_input: Dict, available_models: List[str]) -> str: + """Select model using epsilon-greedy strategy""" + context = self._create_context(user_input) + + # Exploration: random selection + if np.random.random() < self.epsilon: + return np.random.choice(available_models) + + # Exploitation: select best performing model for this context + best_model = None + best_score = -float('inf') + + for model in available_models: + score = self._get_expected_reward(model, context) + if score > best_score: + best_score = score + best_model = model + + return best_model or np.random.choice(available_models) + + def update_reward(self, model: str, context: str, reward: float): + """Update model performance based on user feedback""" + self.arm_counts[model] += 1 + self.arm_rewards[model].append(reward) + + if context not in self.context_arms: + self.context_arms[context] = defaultdict(list) + + self.context_arms[context][model].append(reward) + + def _get_expected_reward(self, model: str, context: str) -> float: + """Calculate expected reward with confidence bounds""" + # Global performance + if model in self.arm_rewards and self.arm_rewards[model]: + global_mean = np.mean(self.arm_rewards[model]) + global_confidence = self._calculate_confidence_bound( + self.arm_rewards[model], self.arm_counts[model] + ) + else: + global_mean = 0.0 + global_confidence = float('inf') # High uncertainty + + # Context-specific performance + if (context in self.context_arms and + model in self.context_arms[context] and + self.context_arms[context][model]): + + context_rewards = self.context_arms[context][model] + context_mean = np.mean(context_rewards) + context_confidence = self._calculate_confidence_bound( + context_rewards, len(context_rewards) + ) + + # Weighted combination of global and context-specific performance + weight = min(len(context_rewards) / 10, 0.8) # More weight to context with more data + expected_reward = weight * context_mean + (1 - weight) * global_mean + confidence_bound = weight * context_confidence + (1 - weight) * global_confidence + + else: + expected_reward = global_mean + confidence_bound = global_confidence + + # Upper confidence bound for exploration + return expected_reward + confidence_bound +``` + +### Option 4: Deep Learning Content-Based Filtering + +```python +# utils/deep_recommender.py +import tensorflow as tf +from tensorflow.keras.models import Model +from tensorflow.keras.layers import Dense, Embedding, Concatenate, Input + +class DeepModelRecommender: + """Neural network-based recommendation system""" + + def __init__(self): + self.model = None + self.feature_encoders = {} + + def build_model(self, feature_dims: Dict): + """Build neural network architecture""" + # Input layers for different feature types + inputs = {} + encoded_features = [] + + # Categorical features (embedded) + for feature, vocab_size in feature_dims['categorical'].items(): + input_layer = Input(shape=(1,), name=f'{feature}_input') + embedding = Embedding(vocab_size, 50, name=f'{feature}_embedding')(input_layer) + inputs[feature] = input_layer + encoded_features.append(tf.keras.layers.Flatten()(embedding)) + + # Numerical features + numerical_input = Input(shape=(len(feature_dims['numerical']),), name='numerical_input') + inputs['numerical'] = numerical_input + encoded_features.append(numerical_input) + + # Combine all features + combined = Concatenate()(encoded_features) + + # Deep layers + x = Dense(512, activation='relu')(combined) + x = tf.keras.layers.Dropout(0.3)(x) + x = Dense(256, activation='relu')(x) + x = tf.keras.layers.Dropout(0.3)(x) + x = Dense(128, activation='relu')(x) + + # Output layer (model recommendations) + output = Dense(len(self._get_all_models()), activation='softmax', name='model_probs')(x) + + # Create and compile model + self.model = Model(inputs=list(inputs.values()), outputs=output) + self.model.compile( + optimizer='adam', + loss='categorical_crossentropy', + metrics=['accuracy', 'top_k_categorical_accuracy'] + ) + + def train(self, training_data: List[Dict]): + """Train the deep learning model""" + # Prepare training data + X, y = self._prepare_training_data(training_data) + + # Train with validation split + history = self.model.fit( + X, y, + epochs=100, + batch_size=32, + validation_split=0.2, + callbacks=[ + tf.keras.callbacks.EarlyStopping(patience=10), + tf.keras.callbacks.ReduceLROnPlateau(patience=5) + ] + ) + + return history + + def predict(self, user_input: Dict, top_k: int = 5) -> List[Tuple[str, float]]: + """Predict top-k model recommendations""" + X = self._prepare_input(user_input) + predictions = self.model.predict(X) + + model_names = self._get_all_models() + model_probs = list(zip(model_names, predictions[0])) + model_probs.sort(key=lambda x: x[1], reverse=True) + + return model_probs[:top_k] +``` + +### Option 5: AI-Assisted Recommendation System + +```python +# utils/ai_recommender.py +import openai +import json +from typing import Dict, List, Tuple, Optional +import re + +class AIModelRecommender: + """AI-powered statistical model recommendation system using LLMs""" + + def __init__(self, api_key: Optional[str] = None, model: str = "gpt-4"): + self.client = openai.OpenAI(api_key=api_key) if api_key else None + self.model = model + self.available_models = self._load_available_models() + + def get_recommendation(self, user_input: Dict) -> Tuple[List[str], str, float]: + """Get AI-powered model recommendations with explanations""" + + # Create a comprehensive prompt with user's scenario + prompt = self._create_analysis_prompt(user_input) + + try: + # Get AI response + response = self._query_ai(prompt) + + # Parse the AI response + recommendations, explanation, confidence = self._parse_ai_response(response) + + # Validate recommendations against our available models + validated_recommendations = self._validate_recommendations(recommendations) + + return validated_recommendations, explanation, confidence + + except Exception as e: + # Fallback to rule-based system if AI fails + fallback_rec = self._get_fallback_recommendation(user_input) + return fallback_rec, "AI unavailable, using rule-based fallback", 0.6 + + def _create_analysis_prompt(self, user_input: Dict) -> str: + """Create a detailed prompt for the AI statistical consultant""" + + available_models_list = "\n".join([f"- {model}" for model in self.available_models]) + + prompt = f""" +You are an expert statistical consultant. A researcher has described their data analysis needs, and you need to recommend the most appropriate statistical models from the available options. + +RESEARCHER'S SCENARIO: +- Analysis Goal: {user_input.get('analysis_goal', 'Not specified')} +- Dependent Variable Type: {user_input.get('dependent_variable', 'Not specified')} +- Independent Variables: {', '.join(user_input.get('independent_variables', []))} +- Sample Size: {user_input.get('sample_size', 'Not specified')} +- Data Distribution: {user_input.get('data_distribution', 'Not specified')} +- Missing Data: {user_input.get('missing_data', 'Not specified')} +- Variables Correlated: {user_input.get('variables_correlated', 'Not specified')} +- Relationship Type: {user_input.get('relationship_type', 'Not specified')} + +AVAILABLE STATISTICAL MODELS: +{available_models_list} + +TASK: +1. Analyze the researcher's scenario carefully +2. Consider statistical assumptions, sample size requirements, and appropriateness +3. Recommend the top 3-5 most suitable models from the available list +4. Provide clear reasoning for each recommendation +5. Mention any important assumptions or limitations +6. Suggest data preparation steps if needed + +RESPONSE FORMAT: +Please structure your response as follows: + +RECOMMENDED MODELS: +1. [Model Name] - [Brief reason] +2. [Model Name] - [Brief reason] +3. [Model Name] - [Brief reason] + +DETAILED EXPLANATION: +[Comprehensive explanation of why these models are appropriate, what assumptions need to be checked, potential limitations, and any data preparation recommendations] + +CONFIDENCE LEVEL: [High/Medium/Low] + """ + + return prompt + + def _query_ai(self, prompt: str) -> str: + """Send prompt to AI and get response""" + if not self.client: + raise Exception("AI client not configured") + + response = self.client.chat.completions.create( + model=self.model, + messages=[ + { + "role": "system", + "content": "You are an expert statistician and data analyst with deep knowledge of statistical methods, their assumptions, and appropriate applications." + }, + {"role": "user", "content": prompt} + ], + temperature=0.3, # Lower temperature for more consistent, focused responses + max_tokens=1500 + ) + + return response.choices[0].message.content + + def _parse_ai_response(self, response: str) -> Tuple[List[str], str, float]: + """Parse the AI response to extract recommendations and metadata""" + recommendations = [] + explanation = "" + confidence = 0.5 # Default medium confidence + + # Extract recommended models using regex + model_pattern = r'\d+\.\s*([^-\n]+?)\s*-' + matches = re.findall(model_pattern, response) + + for match in matches: + model_name = match.strip() + # Try to match with our available models (fuzzy matching) + best_match = self._find_best_model_match(model_name) + if best_match: + recommendations.append(best_match) + + # Extract detailed explanation + explanation_match = re.search(r'DETAILED EXPLANATION:\s*(.*?)(?=CONFIDENCE LEVEL:|$)', + response, re.DOTALL) + if explanation_match: + explanation = explanation_match.group(1).strip() + else: + explanation = response # Use full response if structure isn't followed + + # Extract confidence level + confidence_match = re.search(r'CONFIDENCE LEVEL:\s*(High|Medium|Low)', response, re.IGNORECASE) + if confidence_match: + confidence_level = confidence_match.group(1).lower() + confidence_map = {'high': 0.9, 'medium': 0.7, 'low': 0.5} + confidence = confidence_map.get(confidence_level, 0.7) + + return recommendations, explanation, confidence + + def _find_best_model_match(self, ai_suggested_model: str) -> Optional[str]: + """Find the best match between AI suggestion and available models""" + ai_model_lower = ai_suggested_model.lower() + + # Exact match first + for available_model in self.available_models: + if available_model.lower() == ai_model_lower: + return available_model + + # Partial match (AI might say "linear regression" for "Linear Regression") + for available_model in self.available_models: + if ai_model_lower in available_model.lower() or available_model.lower() in ai_model_lower: + return available_model + + # Common AI name variations + name_mappings = { + 'multiple regression': 'Linear Regression', + 'ordinary least squares': 'Linear Regression', + 'ols': 'Linear Regression', + 'binary logistic regression': 'Logistic Regression', + 'multinomial logistic regression': 'Multinomial Logistic Regression', + 'random forest': 'Random Forest', + 'decision tree': 'Decision Trees', + 'neural network': 'Neural Networks', + 'support vector machine': 'SVM', + 'k-means': 'K-Means Clustering', + 'hierarchical clustering': 'Hierarchical Clustering', + 'principal component analysis': 'PCA', + 'factor analysis': 'Factor Analysis' + } + + for ai_name, our_name in name_mappings.items(): + if ai_name in ai_model_lower and our_name in self.available_models: + return our_name + + return None + + def _validate_recommendations(self, recommendations: List[str]) -> List[str]: + """Ensure recommended models are available and appropriate""" + validated = [] + for model in recommendations: + if model in self.available_models: + validated.append(model) + + # If no valid recommendations, add some safe defaults + if not validated: + safe_defaults = ['Linear Regression', 'Logistic Regression', 'Decision Trees'] + for default in safe_defaults: + if default in self.available_models: + validated.append(default) + if len(validated) >= 3: + break + + return validated[:5] # Return top 5 at most + +class HybridAIRecommender: + """Combines AI recommendations with rule-based validation and fallback""" + + def __init__(self): + self.ai_recommender = AIModelRecommender() + self.rule_based_recommender = EnhancedRuleBasedRecommender() + + def get_recommendation(self, user_input: Dict) -> Dict: + """Get hybrid recommendation combining AI and rules""" + + # Get AI recommendation + try: + ai_models, ai_explanation, ai_confidence = self.ai_recommender.get_recommendation(user_input) + ai_available = True + except Exception as e: + ai_models, ai_explanation, ai_confidence = [], f"AI Error: {str(e)}", 0.0 + ai_available = False + + # Get rule-based recommendation + rule_models, rule_explanation, rule_confidence = self.rule_based_recommender.get_recommendation(user_input) + + # Combine and rank recommendations + final_recommendations = self._combine_recommendations( + ai_models, rule_models, ai_confidence, rule_confidence, user_input + ) + + # Generate comprehensive explanation + combined_explanation = self._create_combined_explanation( + ai_explanation, rule_explanation, ai_available, ai_confidence, rule_confidence + ) + + return { + 'primary_recommendation': final_recommendations[0] if final_recommendations else None, + 'alternative_recommendations': final_recommendations[1:4], + 'explanation': combined_explanation, + 'confidence': max(ai_confidence, rule_confidence), + 'ai_available': ai_available, + 'ai_models': ai_models, + 'rule_models': rule_models + } + + def _combine_recommendations(self, ai_models: List[str], rule_models: List[str], + ai_conf: float, rule_conf: float, user_input: Dict) -> List[str]: + """Intelligently combine AI and rule-based recommendations""" + + # If AI confidence is high and rule confidence is low, prefer AI + if ai_conf > 0.8 and rule_conf < 0.6: + primary_source = ai_models + secondary_source = rule_models + # If rule confidence is high and AI confidence is low, prefer rules + elif rule_conf > 0.8 and ai_conf < 0.6: + primary_source = rule_models + secondary_source = ai_models + # If both are confident, combine with AI first (more context-aware) + elif ai_conf > 0.7 and rule_conf > 0.7: + primary_source = ai_models + secondary_source = rule_models + # If both have low confidence, be more conservative with rule-based + else: + primary_source = rule_models + secondary_source = ai_models + + # Combine while avoiding duplicates and maintaining order + combined = [] + seen = set() + + # Add primary source models first + for model in primary_source: + if model not in seen: + combined.append(model) + seen.add(model) + + # Add secondary source models + for model in secondary_source: + if model not in seen and len(combined) < 5: + combined.append(model) + seen.add(model) + + return combined + + def _create_combined_explanation(self, ai_explanation: str, rule_explanation: str, + ai_available: bool, ai_conf: float, rule_conf: float) -> str: + """Create a comprehensive explanation combining both approaches""" + + if not ai_available: + return f""" +**Rule-Based Recommendation** (Confidence: {rule_conf:.0%}) + +{rule_explanation} + +*Note: AI recommendation was unavailable for this analysis.* + """ + + return f""" +**Hybrid AI + Rule-Based Recommendation** + +**AI Analysis** (Confidence: {ai_conf:.0%}): +{ai_explanation} + +**Statistical Validation**: +{rule_explanation} + +**Final Assessment**: +This recommendation combines AI-powered analysis of your scenario with traditional statistical validation rules. +The AI provides context-aware insights while rule-based validation ensures statistical appropriateness. + """ +``` + +### Integration with Flask App + +```python +# routes/ai_routes.py +from flask import Blueprint, request, jsonify +from utils.ai_recommender import HybridAIRecommender + +ai_bp = Blueprint('ai', __name__) +hybrid_recommender = HybridAIRecommender() + +@ai_bp.route('/api/ai-recommend', methods=['POST']) +def ai_recommend(): + """AI-powered recommendation endpoint""" + try: + user_input = request.json + + # Get hybrid recommendation + result = hybrid_recommender.get_recommendation(user_input) + + return jsonify({ + 'success': True, + 'primary_model': result['primary_recommendation'], + 'alternatives': result['alternative_recommendations'], + 'explanation': result['explanation'], + 'confidence': result['confidence'], + 'ai_available': result['ai_available'] + }) + + except Exception as e: + return jsonify({ + 'success': False, + 'error': str(e), + 'fallback_used': True + }), 500 + +@ai_bp.route('/api/compare-recommendations', methods=['POST']) +def compare_recommendations(): + """Compare AI vs rule-based recommendations side by side""" + try: + user_input = request.json + result = hybrid_recommender.get_recommendation(user_input) + + return jsonify({ + 'success': True, + 'ai_recommendations': result['ai_models'], + 'rule_recommendations': result['rule_models'], + 'hybrid_final': [result['primary_recommendation']] + result['alternative_recommendations'], + 'explanation': result['explanation'] + }) + + except Exception as e: + return jsonify({'success': False, 'error': str(e)}), 500 +``` + +### Advantages of AI-Assisted Approach + +✅ **Rich Context Understanding**: AI can understand nuanced scenarios and edge cases +✅ **Natural Language Processing**: Can handle complex, descriptive user inputs +✅ **Comprehensive Knowledge**: Leverages vast statistical knowledge from training +✅ **Adaptive Reasoning**: Can consider multiple factors simultaneously +✅ **Educational Value**: Provides detailed explanations and reasoning +✅ **Handles Novel Scenarios**: Better at unusual or complex analysis situations + +### Challenges and Mitigation Strategies + +⚠️ **Potential Issues**: + +- AI might hallucinate non-existent models +- Could provide statistically inappropriate advice +- API costs and latency concerns +- Dependence on external services + +✅ **Mitigation Strategies**: + +- Always validate AI suggestions against available models +- Use rule-based fallback when AI fails +- Implement confidence scoring to detect uncertain responses +- Cache common scenarios to reduce API calls +- Use local/open-source models for privacy and cost control + +### Implementation Strategy + +#### Phase 1: Basic AI Integration + +1. Set up AI recommendation endpoint +2. Create prompt templates for different scenarios +3. Implement response parsing and validation +4. Add fallback to existing rule-based system + +#### Phase 2: Hybrid Intelligence + +1. Combine AI and rule-based recommendations +2. Add confidence scoring and explanation generation +3. Implement user feedback collection for AI recommendations +4. Create A/B testing framework + +#### Phase 3: Advanced Features + +1. Add domain-specific prompts (medical, business, social science) +2. Implement conversation-based clarification +3. Add AI-powered assumption checking +4. Create personalized recommendation learning + +This AI approach could provide significantly more intelligent and context-aware recommendations while maintaining the reliability of rule-based validation! diff --git a/docs/PRACTICAL_IMPLEMENTATION_PLAN.md b/docs/PRACTICAL_IMPLEMENTATION_PLAN.md new file mode 100644 index 0000000..0f615e5 --- /dev/null +++ b/docs/PRACTICAL_IMPLEMENTATION_PLAN.md @@ -0,0 +1,353 @@ +# Practical Implementation: Enhanced Rule-Based Recommender + +## Step-by-Step Implementation Plan + +This document provides a concrete implementation plan for improving the current recommendation system without requiring ML or large datasets. + +## Phase 1: Immediate Improvements (Week 1-2) + +### 1. Add Input Validation and Preprocessing + +```python +# utils/validation.py +class InputValidator: + """Validates and preprocesses user input for better recommendations""" + + def __init__(self): + self.valid_combinations = self._load_valid_combinations() + self.common_issues = self._load_common_issues() + + def validate_input(self, user_input: Dict) -> ValidationResult: + """Validate input and provide helpful feedback""" + result = ValidationResult() + + # Check for required fields + required_fields = ['analysis_goal', 'research_question'] + for field in required_fields: + if not user_input.get(field): + result.add_error(f"Missing required field: {field}") + + # Check for logical inconsistencies + if user_input.get('analysis_goal') == 'predict' and not user_input.get('dependent_variable'): + result.add_warning("Prediction requires specifying what you want to predict (dependent variable)") + + # Check sample size appropriateness + sample_size = user_input.get('sample_size', 0) + if isinstance(sample_size, str): + try: + sample_size = int(sample_size) + except ValueError: + result.add_error("Sample size must be a number") + + if sample_size < 10: + result.add_warning("Very small sample size - consider descriptive analysis instead") + elif sample_size > 10000: + result.add_suggestion("Large dataset - consider computational efficiency in model choice") + + # Check for problematic combinations + if (user_input.get('variables_correlated') == 'yes' and + user_input.get('analysis_goal') == 'predict' and + 'continuous' in user_input.get('independent_variables', [])): + result.add_suggestion("Consider regularization methods for correlated predictors") + + return result + +class ValidationResult: + def __init__(self): + self.errors = [] + self.warnings = [] + self.suggestions = [] + self.is_valid = True + + def add_error(self, message: str): + self.errors.append(message) + self.is_valid = False + + def add_warning(self, message: str): + self.warnings.append(message) + + def add_suggestion(self, message: str): + self.suggestions.append(message) +``` + +### 2. Enhanced Scoring with Confidence + +```python +# utils/enhanced_scoring.py +class EnhancedScorer: + """Improved scoring system with confidence tracking""" + + def calculate_score_with_confidence(self, model_name: str, model_info: Dict, + user_input: Dict) -> Tuple[float, float]: + """Calculate score and confidence level""" + + score_components = {} + confidence_factors = {} + + # Core compatibility scoring + analysis_goal_match = self._score_analysis_goal(user_input, model_info) + score_components['analysis_goal'] = analysis_goal_match['score'] + confidence_factors['analysis_goal'] = analysis_goal_match['confidence'] + + dependent_var_match = self._score_dependent_variable(user_input, model_info) + score_components['dependent_variable'] = dependent_var_match['score'] + confidence_factors['dependent_variable'] = dependent_var_match['confidence'] + + # Additional factors + relationship_match = self._score_relationship_type(user_input, model_info) + score_components['relationship'] = relationship_match['score'] + confidence_factors['relationship'] = relationship_match['confidence'] + + sample_size_match = self._score_sample_size(user_input, model_info) + score_components['sample_size'] = sample_size_match['score'] + confidence_factors['sample_size'] = sample_size_match['confidence'] + + # Calculate weighted score + total_score = sum(score_components.values()) + + # Calculate overall confidence (how certain we are about this recommendation) + overall_confidence = sum(confidence_factors.values()) / len(confidence_factors) + + return total_score, overall_confidence + + def _score_analysis_goal(self, user_input: Dict, model_info: Dict) -> Dict: + """Score analysis goal compatibility with confidence""" + analysis_goal = user_input.get('analysis_goal') + model_goals = model_info.get('analysis_goals', []) + + if analysis_goal in model_goals: + return {'score': 3.0, 'confidence': 0.9} # High confidence for exact match + elif self._is_compatible_goal(analysis_goal, model_goals): + return {'score': 1.5, 'confidence': 0.6} # Medium confidence for compatible + else: + return {'score': 0.0, 'confidence': 0.9} # High confidence it's wrong + + def _is_compatible_goal(self, user_goal: str, model_goals: List[str]) -> bool: + """Check if goals are compatible even if not exact match""" + compatibility_map = { + 'predict': ['explore', 'describe'], + 'explore': ['predict', 'describe'], + 'classify': ['predict'] + } + + compatible_goals = compatibility_map.get(user_goal, []) + return any(goal in model_goals for goal in compatible_goals) +``` + +### 3. Statistical Knowledge Integration + +```python +# utils/statistical_knowledge.py +class StatisticalKnowledgeBase: + """Expert statistical knowledge for better recommendations""" + + def __init__(self): + self.model_assumptions = self._load_model_assumptions() + self.problem_patterns = self._load_problem_patterns() + self.complexity_levels = self._load_complexity_levels() + + def check_assumptions(self, model_name: str, user_input: Dict) -> List[str]: + """Check if user's data likely violates model assumptions""" + violations = [] + assumptions = self.model_assumptions.get(model_name, []) + + for assumption in assumptions: + if self._is_assumption_violated(assumption, user_input): + violations.append(assumption) + + return violations + + def _is_assumption_violated(self, assumption: str, user_input: Dict) -> bool: + """Check if specific assumption is likely violated""" + + if assumption == 'normal_distribution' and user_input.get('data_distribution') == 'non_normal': + return True + + if assumption == 'no_multicollinearity' and user_input.get('variables_correlated') == 'yes': + return True + + if assumption == 'large_sample_size': + sample_size = int(user_input.get('sample_size', 0)) + return sample_size < 30 + + if assumption == 'linear_relationship' and user_input.get('relationship_type') == 'non_linear': + return True + + return False + + def identify_problem_type(self, user_input: Dict) -> str: + """Identify the type of statistical problem""" + + analysis_goal = user_input.get('analysis_goal') + dependent_var = user_input.get('dependent_variable') + variables_correlated = user_input.get('variables_correlated') + sample_size = int(user_input.get('sample_size', 0)) + + # Simple decision tree for problem identification + if analysis_goal == 'predict': + if dependent_var == 'continuous': + if variables_correlated == 'yes': + return 'multicollinearity_regression' + elif sample_size < 50: + return 'small_sample_regression' + else: + return 'standard_regression' + elif dependent_var == 'binary': + return 'binary_classification' + elif dependent_var == 'count': + return 'count_data_analysis' + + elif analysis_goal == 'explore': + if len(user_input.get('independent_variables', [])) > 5: + return 'dimensionality_reduction' + else: + return 'exploratory_analysis' + + elif analysis_goal == 'cluster': + return 'clustering_analysis' + + return 'general_analysis' + + def get_optimal_models(self, problem_type: str) -> List[str]: + """Get the best models for a specific problem type""" + optimal_models = { + 'multicollinearity_regression': ['Ridge Regression', 'Lasso Regression', 'Elastic Net'], + 'small_sample_regression': ['Bayesian Linear Regression', 'Ridge Regression'], + 'standard_regression': ['Linear Regression', 'Multiple Regression'], + 'binary_classification': ['Logistic Regression', 'Decision Trees'], + 'count_data_analysis': ['Poisson Regression', 'Negative Binomial'], + 'dimensionality_reduction': ['PCA', 'Factor Analysis'], + 'exploratory_analysis': ['Descriptive Statistics', 'Correlation Analysis'], + 'clustering_analysis': ['K-Means', 'Hierarchical Clustering'] + } + + return optimal_models.get(problem_type, []) +``` + +## Phase 2: Enhanced Explanations (Week 2-3) + +### Educational Explanation Generator + +```python +# utils/explanation_generator.py +class EducationalExplanationGenerator: + """Generate educational explanations for recommendations""" + + def generate_explanation(self, model_name: str, user_input: Dict, + score_breakdown: Dict) -> str: + """Generate comprehensive explanation""" + + model_info = self._get_model_info(model_name) + problem_type = self._identify_problem_type(user_input) + + explanation = f""" +📊 **Recommended Statistical Method: {model_name}** + +**Why this method fits your needs:** +{self._explain_fit_reasoning(model_name, user_input, score_breakdown)} + +**What this method does:** +{model_info.get('description', 'Performs statistical analysis on your data')} + +**Key advantages for your scenario:** +{self._list_advantages(model_name, user_input)} + +**Important assumptions to check:** +{self._list_assumptions(model_name)} + +**Implementation considerations:** +{self._provide_implementation_tips(model_name, user_input)} + +**Confidence in this recommendation:** {score_breakdown.get('confidence', 0.8):.0%} +{self._explain_confidence_level(score_breakdown.get('confidence', 0.8))} + +**Alternative approaches to consider:** +{self._suggest_alternatives(model_name, user_input)} + """ + + return explanation.strip() + + def _explain_confidence_level(self, confidence: float) -> str: + """Explain what the confidence level means""" + if confidence > 0.8: + return "High confidence - This method strongly matches your requirements." + elif confidence > 0.6: + return "Moderate confidence - Good match, but consider alternatives." + else: + return "Lower confidence - Multiple methods could work, expert consultation recommended." +``` + +## Phase 3: Gradual Enhancement (Week 3-4) + +### Smart Defaults and Suggestions + +```python +# utils/smart_defaults.py +class SmartDefaultProvider: + """Provide intelligent defaults and suggestions""" + + def suggest_improvements(self, user_input: Dict) -> List[str]: + """Suggest improvements to user's analysis approach""" + suggestions = [] + + # Sample size suggestions + sample_size = int(user_input.get('sample_size', 0)) + if sample_size < 30: + suggestions.append( + "Consider collecting more data if possible. Small samples limit method choices." + ) + + # Missing data suggestions + if user_input.get('missing_data') == 'systematic': + suggestions.append( + "Systematic missing data is concerning. Consider why data is missing and if it affects results." + ) + + # Correlation suggestions + if user_input.get('variables_correlated') == 'unknown': + suggestions.append( + "Consider checking correlation between your variables before analysis." + ) + + return suggestions + + def provide_next_steps(self, model_name: str, user_input: Dict) -> List[str]: + """Provide actionable next steps""" + steps = [] + + # Always start with data exploration + steps.append("1. Explore your data with descriptive statistics and visualizations") + + # Model-specific steps + if 'Regression' in model_name: + steps.append("2. Check for outliers and influential observations") + steps.append("3. Examine residual plots to validate assumptions") + + if 'Logistic' in model_name: + steps.append("2. Check for class imbalance in your outcome variable") + + # General steps + steps.append(f"4. Implement {model_name} and interpret results carefully") + steps.append("5. Consider validation techniques if sample size permits") + + return steps +``` + +## Implementation Priority + +### Immediate (This Week) +1. ✅ **Input Validation** - Catch problems early +2. ✅ **Confidence Scoring** - Show uncertainty levels +3. ✅ **Basic Assumption Checking** - Warn about violations + +### Short-term (Next 2 Weeks) +1. ✅ **Enhanced Explanations** - Educational value +2. ✅ **Problem Type Detection** - Better matching +3. ✅ **Smart Suggestions** - Help users improve + +### Medium-term (Month 2) +1. **Expert Review Interface** - Let statisticians improve rules +2. **A/B Testing** - Compare recommendation strategies +3. **Domain-Specific Rules** - Field-specific recommendations + +This approach provides immediate, measurable improvements without the risks and requirements of ML-based systems. Each enhancement builds on expert statistical knowledge rather than trying to learn patterns from limited data. diff --git a/docs/QUICK_IMPLEMENTATION_REFERENCE.md b/docs/QUICK_IMPLEMENTATION_REFERENCE.md new file mode 100644 index 0000000..6431c64 --- /dev/null +++ b/docs/QUICK_IMPLEMENTATION_REFERENCE.md @@ -0,0 +1,116 @@ +# Quick Implementation Reference + +## Current Problems Summary + +### Database Issues + +- 204KB JSON loaded at startup (inefficient) +- All 396 models in memory always +- Linear search through models +- Massive code duplication in implementations +- No lazy loading + +### Template Issues + +- Heavy data dependencies (full objects required) +- All content loaded synchronously +- Hardcoded static file paths +- No progressive loading +- Monolithic template structure + +## Key Implementation Ideas + +### 1. Database Restructuring + +```text +BEFORE: Single 204KB model_database.json +AFTER: Separated structure: + - metadata.json (20KB - searchable fields only) + - implementations/ (by language) + - synthetic_data/ (by model type) + - interpretations/ (by model) + - templates/ (reusable code patterns) +``` + +### 2. Service Layer Pattern + +```python +# Replace direct JSON access with service +model_service = ModelService() +model_service.get_metadata(name) # Fast +model_service.get_implementation(name, lang) # On-demand +model_service.search_models(**criteria) # Indexed +``` + +### 3. Template Optimization + +```html + +
+ +
+ + + +``` + +### 4. API Endpoints for Lazy Loading + +```python +/api/model/{name}/implementation/{language} +/api/model/{name}/plots +/api/model/{name}/coefficients +/api/model/{name}/interpretation +``` + +## Performance Improvements Expected + +- **Startup**: 204KB → 20KB (90% reduction) +- **Memory**: Only active models loaded (80% reduction) +- **Search**: O(n) → O(1) with indexing (100x faster) +- **Page Load**: Progressive loading (50% faster perceived) + +## Implementation Priority + +1. **Phase 1**: Split database structure +2. **Phase 2**: Create service layer +3. **Phase 3**: Update route handlers +4. **Phase 4**: Optimize templates with AJAX +5. **Phase 5**: Add caching and indexing + +## Backwards Compatibility Strategy + +- Keep existing API surface unchanged +- Add compatibility wrapper for old access patterns +- Gradual migration path +- Rollback capability + +## Files to Create/Modify + +### New Files + +- `utils/model_service.py` - Service layer +- `routes/api_routes.py` - AJAX endpoints +- `static/js/model_interpretation.js` - Progressive loading +- `utils/cache_manager.py` - Caching strategy + +### Modified Files + +- `app.py` - Service initialization +- `routes/main_routes.py` - Use service instead of direct access +- `templates/model_interpretation.html` - Modular sections +- `models.py` - Add get_model_details optimization + +## Quick Wins (Easy to implement) + +1. **Split metadata extraction** - Immediate 90% startup improvement +2. **Add LRU cache** to existing get_model_details() +3. **Lazy load diagnostic plots** with simple AJAX +4. **Template error fallbacks** for missing content + +## Risk Mitigation + +- Maintain exact same user experience +- Add comprehensive tests +- Benchmark before/after changes +- Have rollback plan ready diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..2b0092a --- /dev/null +++ b/docs/README.md @@ -0,0 +1,47 @@ +# Documentation + +This directory contains implementation plans and documentation for the Statistical Model Suggester optimization project. + +## Database Optimization Plans + +### [DATABASE_OPTIMIZATION_PLAN.md](./DATABASE_OPTIMIZATION_PLAN.md) + +Comprehensive implementation plan for optimizing the model database architecture: + +- **Current Issues**: 204KB monolithic JSON, linear search, memory inefficiency +- **5-Phase Implementation**: Database refactoring → Service integration → Template optimization → API endpoints → Performance tuning +- **Timeline**: 7-week structured approach +- **Success Metrics**: 90% startup reduction, 80% memory savings, 100x search improvement + +### [QUICK_IMPLEMENTATION_REFERENCE.md](./QUICK_IMPLEMENTATION_REFERENCE.md) + +Concise reference guide for developers: + +- **Problem Summary**: Key inefficiencies identified +- **Solution Overview**: Service layer, progressive loading, indexed search +- **Implementation Priority**: Phases and quick wins +- **File Changes**: New files to create and existing files to modify + +## Implementation Context + +These plans address performance and scalability issues identified in: + +- **Database Storage**: 204KB JSON file loaded entirely at startup +- **Template System**: Heavy synchronous loading without progressive enhancement +- **Search Performance**: O(n) linear scans instead of indexed lookups +- **Memory Usage**: All 396 models kept in memory regardless of usage + +## Expected Improvements + +- **90% reduction** in application startup time +- **80% reduction** in memory usage +- **100x improvement** in search performance +- **50% improvement** in perceived page load times + +## Next Steps + +1. Review implementation plans +2. Create feature branch for development +3. Begin with Phase 1: Database refactoring +4. Implement backwards compatibility layer +5. Add comprehensive testing and benchmarking diff --git a/docs/SYNTHETIC_DATA_RATIONALE.md b/docs/SYNTHETIC_DATA_RATIONALE.md new file mode 100644 index 0000000..87e3bf3 --- /dev/null +++ b/docs/SYNTHETIC_DATA_RATIONALE.md @@ -0,0 +1,216 @@ +# Synthetic Data Storage Architecture: Rationale and Decision Analysis + +## Executive Summary + +This document explains the critical decision to migrate from JSON-embedded code strings to executable script files for synthetic data storage in the Statistical Model Suggester application. + +## Current State Analysis + +### The Anti-Pattern: Code-in-JSON + +The current implementation stores R code as JSON strings within the model database: + +```json +{ + "Linear Regression": { + "synthetic_data": { + "r_code": "# Generate synthetic data for linear regression\nset.seed(123)\nn <- 100...", + "results": { "text_output": "...", "plots": [] } + } + } +} +``` + +### Critical Issues Identified + +1. **Code Duplication Crisis** + - Same R scripts exist in both JSON strings AND separate `.R` files in `synthetic_data_examples/` + - No single source of truth for synthetic data generation + - Manual synchronization required between JSON and script files + +2. **Maintainability Nightmare** + - R code embedded in JSON loses all IDE benefits: + - No syntax highlighting + - No linting or error checking + - No debugging capabilities + - No code completion + - Editing requires JSON manipulation instead of direct code editing + - Risk of JSON syntax errors when modifying code strings + +3. **Performance Impact** + - Large JSON files (3000+ lines) due to embedded code strings + - Slow JSON parsing at application startup + - All code loaded into memory regardless of usage + - No lazy loading or on-demand execution + +4. **Version Control Problems** + - Code changes appear as JSON string modifications in diffs + - Difficult to track actual code logic changes + - No blame tracking for specific lines of R code + - Merge conflicts in JSON are harder to resolve + +5. **Testing and Validation Issues** + - Cannot independently test synthetic data scripts + - No way to validate R syntax without full JSON parsing + - Difficult to run scripts in isolation for debugging + +## Alternative Architectures Considered + +### Option 1: Keep JSON, Improve Structure ❌ + +**Rejected:** Still maintains the fundamental anti-pattern of storing code as strings. + +### Option 2: Hybrid Approach ❌ + +**Rejected:** Would create even more complexity with multiple sources of truth. + +### Option 3: CSV/Pre-generated Data ❌ + +**Rejected:** Loses the educational value and flexibility of generating synthetic data with parameters. + +### Option 4: Script-Based Architecture ✅ **CHOSEN** + +**Selected:** Executable script files with JSON registry for metadata. + +## Recommended Architecture + +### Structure + +```text +synthetic_data/ +├── scripts/ # Executable R/Python scripts +│ ├── regression/ +│ │ ├── linear_regression.R +│ │ ├── logistic_regression.R +│ │ └── poisson_regression.R +│ ├── time_series/ +│ │ ├── arima_example.R +│ │ ├── var_model.R +│ │ └── garch_volatility.R +│ ├── machine_learning/ +│ │ ├── random_forest.R +│ │ ├── svm_classification.py +│ │ └── neural_network.py +│ └── shared/ +│ ├── data_generators.R # Reusable functions +│ └── plot_helpers.R # Common plotting utilities +├── registry.json # Script metadata and mappings +├── execution_config.json # Runtime parameters +└── results_cache/ # Optional performance optimization + ├── outputs/ + └── plots/ +``` + +### Script Registry Design + +```json +{ + "Linear Regression": { + "script_path": "scripts/regression/linear_regression.R", + "language": "R", + "dependencies": ["base", "stats"], + "estimated_runtime": "5s", + "generates_plots": true, + "dataset_size": "small", + "parameters": { + "sample_size": {"default": 100, "range": [50, 1000]}, + "noise_level": {"default": 1, "range": [0.1, 5]} + } + } +} +``` + +## Benefits Analysis + +### Immediate Benefits + +1. **Developer Experience** + - Full IDE support for R/Python scripts + - Syntax highlighting, linting, debugging + - Code completion and error detection + - Easy editing without JSON manipulation + +2. **Performance Gains** + - Smaller JSON files (metadata only) + - Faster application startup + - Lazy script execution (on-demand) + - Efficient caching of results + +3. **Maintainability** + - Single source of truth for each script + - Clear separation of concerns + - Independent testing of scripts + - Better version control and collaboration + +### Long-term Benefits + +1. **Scalability** + - Easy addition of new model types + - Support for multiple programming languages + - Modular script organization + - Reusable component libraries + +2. **Educational Value** + - Students can download and run scripts independently + - Scripts serve as learning resources + - Easy customization of parameters + - Clear progression from basic to advanced examples + +3. **Research Applications** + - Researchers can modify scripts for their needs + - Easy integration with external tools + - Reproducible research examples + - Version tracking of methodological changes + +## Implementation Strategy + +### Phase 1: Migration Script +1. Extract R code from JSON strings +2. Create organized script files +3. Generate script registry with metadata +4. Validate extracted scripts + +### Phase 2: Service Layer Updates +1. Implement script execution service +2. Add caching mechanism +3. Create error handling and logging +4. Build performance monitoring + +### Phase 3: Frontend Integration +1. Update templates to use script service +2. Add script download functionality +3. Implement real-time execution progress +4. Create script customization interface + +## Risk Mitigation + +### Technical Risks +- **Script Execution Security**: Sandboxed execution environment +- **Dependency Management**: Automated R package installation +- **Performance**: Caching and background execution +- **Error Handling**: Graceful failure with meaningful messages + +### Migration Risks +- **Data Loss**: Comprehensive backup and validation +- **Downtime**: Phased rollout with rollback capability +- **User Impact**: Maintain backward compatibility during transition + +## Conclusion + +The migration from JSON-embedded code to executable script files addresses fundamental architectural flaws in the current system. This change will significantly improve: + +- **Developer productivity** through better tooling +- **Application performance** through optimized data loading +- **System maintainability** through proper separation of concerns +- **Educational value** through accessible, runnable examples + +The script-based architecture aligns with software engineering best practices and provides a foundation for future enhancements to the Statistical Model Suggester platform. + +## Next Steps + +1. **Immediate**: Begin migration script development +2. **Short-term**: Implement script execution service +3. **Medium-term**: Update frontend and templates +4. **Long-term**: Add advanced features (parameterization, multi-language support) + +This architectural change represents a critical investment in the long-term success and scalability of the Statistical Model Suggester application.