This project aims to automatically detect the relationship between markdown cells and code cells in Jupyter Notebooks.
1. Data preprocessing Advanced text cleaning and normalization, NLP-based feature extraction, code analysis features, and feature engineering.
2. Model architecture TF-IDF based baseline model, an advanced Transformer-based model (sentence-transformers/all-MiniLM-L6-v2), XGBoost-based classification, and weighted ensemble learning.
3. Model optimization Automatic hyperparameter optimization with Optuna, 5-fold cross-validation, ensemble weight optimization, early stopping, and model selection.
4. Model evaluation Comprehensive performance metrics, detailed visualizations, per-notebook analysis, and error analysis and reporting.
5. Analysis tools Interactive performance charts, error pattern analysis, detailed model comparisons, and feature importance analysis.
Install the required Python packages:
pip install -r requirements.txt
Download the spaCy model:
python -m spacy download en_core_web_sm
Download the NLTK data:
import nltk
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')
1. Data preprocessing
python data_preprocessing.py
This command cleans and normalizes text, performs feature extraction, and saves the processed data.
2. Model training
python ai4code_model.py
This command trains the TF-IDF model, trains the Transformer model, trains the XGBoost model, and generates ensemble predictions.
3. Model optimization
python model_optimization.py
This command optimizes hyperparameters, performs cross-validation, and saves the best parameters.
4. Model evaluation
python model_evaluation.py
This command computes performance metrics, generates visualizations, and produces an evaluation report.
5. Detailed analysis
python model_analysis.py
This command performs detailed performance analysis, generates interactive visualizations, and analyzes error patterns.
Model predictions submission_tfidf.csv (TF-IDF model predictions), submission_transformer.csv (Transformer model predictions), submission_xgboost.csv (XGBoost model predictions), submission_ensemble.csv (ensemble model predictions).
Evaluation outputs evaluation_outputs/metrics.json (detailed performance metrics), evaluation_outputs/model_comparison.png (model comparison chart), evaluation_outputs/rank_distribution.png (rank distribution chart), evaluation_outputs/notebook_performance.png (per-notebook performance chart), evaluation_outputs/error_analysis.png (error analysis chart), evaluation_outputs/evaluation_report.md (comprehensive evaluation report).
Analysis outputs analysis_outputs/model_performance.json (detailed performance analysis), analysis_outputs/error_analysis.json (error pattern analysis), analysis_outputs/interactive_performance.html (interactive performance chart), analysis_outputs/analysis_report.md (comprehensive analysis report).
Optimization outputs best_params.json (optimized model parameters), processed_features.csv (processed features).
TF-IDF model: Top-1 accuracy ~0.65, Top-3 accuracy ~0.85, average rank ~1.8
Transformer model: Top-1 accuracy ~0.75, Top-3 accuracy ~0.90, average rank ~1.5
XGBoost model: Top-1 accuracy ~0.70, Top-3 accuracy ~0.88, average rank ~1.6
Ensemble model: Top-1 accuracy ~0.78, Top-3 accuracy ~0.92, average rank ~1.4
Fork this repository, create a new branch (git checkout -b feature/newFeature), commit your changes (git commit -am 'Add new feature: X'), push your branch (git push origin feature/newFeature), and open a pull request.
This project is licensed under the MIT License. See the LICENSE file for details.
For questions about this project, feel free to reach out via GitHub.