Skip to content

Homework - MIPT DIHT 2018 #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
IKholopov wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Homework - MIPT DIHT 2018 #35

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
365 changes: 365 additions & 0 deletions IgorKholopov/fca.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,365 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 142,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.preprocessing import OneHotEncoder\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
"from sklearn.model_selection import KFold\n",
"from sklearn.model_selection import GridSearchCV"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Подготовка данных"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [],
"source": [
"def prepare_data(filename):\n",
" X_df = pd.read_csv(filename)\n",
" y = X_df['V10'].replace(to_replace='positive', value=1).replace(to_replace='negative', value=-1).values\n",
" encoder = OneHotEncoder(dtype='bool')\n",
" X = encoder.fit_transform(X_df[['V{}'.format(i) for i in range(1, 10)]]).toarray()\n",
" return X, y"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"def context_split(X_train, y_train):\n",
" return X_train[y_train==1], X_train[y_train==-1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Признаки для объекта, используемые классификатором"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def avg_intersect(sample, context):\n",
" return np.sum(context & sample) / len(context)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def avg_clojure(sample, context, context_clojure):\n",
" intersect = context & sample\n",
" return np.array([(np.sum(context_clojure & inters, axis=1) == np.sum(inters)).sum() \n",
" for inters in intersect]).mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Подсчёт признаков"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [],
"source": [
"def get_features(dataset, X_train, y_train):\n",
" X_pos, X_neg = context_split(X_train, y_train)\n",
" return np.array([[\n",
" avg_intersect(sample, X_pos), avg_intersect(sample, X_neg)] for sample in dataset])\n",
"\n",
"def get_features_bad(dataset, X_train, y_train):\n",
" X_pos, X_neg = context_split(X_train, y_train)\n",
" return np.array([[avg_clojure(sample, X_pos, X_pos), avg_clojure(sample, X_pos, X_neg),\n",
" avg_clojure(sample, X_neg, X_pos), avg_clojure(sample, X_neg, X_neg),\n",
" avg_intersect(sample, X_pos), avg_intersect(sample, X_neg)] for sample in dataset])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Для улучшения качества, заменим критерий порога по значениям признаков на решающие деревья над теми же признаками, тогда:\n",
"\n",
"Классификатор - ансамбль k деревьев, обучающийся на разделении обучающей выборки на k частей. На k-й обучаем, остальные используются как контекст. Признаки на вход в дерево получаются из \"ленивых\" признаков обучающей выборки и контекста.\n",
"\n",
"Предсказание - знак среднего предсказания ансамбля деревьев на признаках из запрашиваемого примера и всей обучающей выборки в качестве контекстов.\n",
"\n",
"3 параметра:\n",
" - n_train_splits - кол-во разделений при обучении (==кол-во деревьев в ансамбле)\n",
" - max_depth - максимальная глубина дерева\n",
" - features - источники признаков"
]
},
{
"cell_type": "code",
"execution_count": 137,
"metadata": {},
"outputs": [],
"source": [
"class MyClassifier:\n",
" def __init__(self, n_train_splits = 5, max_depth = 4, features=get_features):\n",
" self.trees=[]\n",
" self.n_train_splits = n_train_splits\n",
" self.max_depth = max_depth\n",
" self.features=features\n",
" self.X_train = None\n",
" self.y_train = None\n",
" \n",
" def fit(self, X, y):\n",
" k_fold = KFold(n_splits=self.n_train_splits, shuffle=True)\n",
" for train_index, test_index in k_fold.split(X):\n",
" X_train, X_to_learn = X[train_index], X[test_index]\n",
" y_train, y_to_learn = y[train_index], y[test_index]\n",
" X_to_learn = self.features(X_to_learn, X_train, y_train)\n",
" tree = DecisionTreeClassifier(max_depth=self.max_depth)\n",
" tree.fit(X_to_learn, y_to_learn)\n",
" self.trees.append(tree)\n",
" self.X_train = X\n",
" self.y_train = y\n",
" return self\n",
" \n",
" def predict(self, X):\n",
" X = self.features(X, self.X_train, self.y_train)\n",
" tree_predictions = np.array([classifier.predict(X) for classifier in self.trees])\n",
" return (tree_predictions.mean(axis=0) > 0) * 2 -1\n",
" \n",
" def get_params(self, **kwargs):\n",
" return {'n_train_splits':self.n_train_splits, 'max_depth':self.max_depth}\n",
" \n",
" def set_params(self, **kwargs):\n",
" self.n_train_splits = kwargs['n_train_splits']\n",
" self.max_depth = kwargs['max_depth']\n",
" return self"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Рассмотрим в качестве признаков средние $|g'\\cap g_i^+|$, $|g'\\cap g_i^-|$,\n",
" $|(g'\\cap g_i^+)^+|$, $|(g'\\cap g_i^+)^-|$, $|(g'\\cap g_i^-)^-|$, $|(g'\\cap g_i^-)^+|$:"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 : acc. 0.6129032258064516, prec. 0.8048780487804879, rec. 0.5409836065573771\n",
"2 : acc. 0.6436781609195402, prec. 0.717391304347826, rec. 0.6470588235294118\n",
"3 : acc. 0.64, prec. 0.8717948717948718, rec. 0.5230769230769231\n",
"4 : acc. 0.5056179775280899, prec. 0.7272727272727273, rec. 0.4067796610169492\n",
"5 : acc. 0.6741573033707865, prec. 0.8666666666666667, rec. 0.6290322580645161\n",
"6 : acc. 0.611764705882353, prec. 0.8108108108108109, rec. 0.5357142857142857\n",
"7 : acc. 0.6578947368421053, prec. 0.7924528301886793, rec. 0.6\n",
"8 : acc. 0.6915887850467289, prec. 0.8125, rec. 0.7123287671232876\n",
"9 : acc. 0.7475728155339806, prec. 0.8666666666666667, rec. 0.7428571428571429\n",
"10 : acc. 0.5934065934065934, prec. 0.7391304347826086, rec. 0.576271186440678\n"
]
}
],
"source": [
"n_split = 5\n",
"max_depth = 3\n",
"for i in range(1, 11):\n",
" file_train = 'train{}.csv'.format(i)\n",
" file_test = 'test{}.csv'.format(i)\n",
" X_train, y_train = prepare_data(file_train)\n",
" X_test, y_test = prepare_data(file_test)\n",
" classifier = MyClassifier(n_split, max_depth, features=get_features_bad)\n",
" classifier.fit(X_train, y_train)\n",
" pred = classifier.predict(X_test)\n",
" print('{} : acc. {}, prec. {}, rec. {}'.format(i, accuracy_score(y_test, pred),\n",
" precision_score(y_test, pred), recall_score(y_test, pred)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Результат не впечатляет. Попробуем ограничиться средними $|g'\\cap g_i^+|$, $|g'\\cap g_i^-|$:"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 : acc. 0.7634408602150538, prec. 0.7746478873239436, rec. 0.9016393442622951\n",
"2 : acc. 0.7471264367816092, prec. 0.6986301369863014, rec. 1.0\n",
"3 : acc. 0.79, prec. 0.8055555555555556, rec. 0.8923076923076924\n",
"4 : acc. 0.8314606741573034, prec. 0.8142857142857143, rec. 0.9661016949152542\n",
"5 : acc. 0.7752808988764045, prec. 0.8, rec. 0.9032258064516129\n",
"6 : acc. 0.8235294117647058, prec. 0.8360655737704918, rec. 0.9107142857142857\n",
"7 : acc. 0.7631578947368421, prec. 0.7263157894736842, rec. 0.9857142857142858\n",
"8 : acc. 0.7757009345794392, prec. 0.7951807228915663, rec. 0.9041095890410958\n",
"9 : acc. 0.8446601941747572, prec. 0.8292682926829268, rec. 0.9714285714285714\n",
"10 : acc. 0.7472527472527473, prec. 0.725, rec. 0.9830508474576272\n"
]
}
],
"source": [
"n_split = 5\n",
"max_depth = 3\n",
"for i in range(1, 11):\n",
" file_train = 'train{}.csv'.format(i)\n",
" file_test = 'test{}.csv'.format(i)\n",
" X_train, y_train = prepare_data(file_train)\n",
" X_test, y_test = prepare_data(file_test)\n",
" classifier = MyClassifier(n_split, max_depth, features=get_features)\n",
" classifier.fit(X_train, y_train)\n",
" pred = classifier.predict(X_test)\n",
" print('{} : acc. {}, prec. {}, rec. {}'.format(i, accuracy_score(y_test, pred),\n",
" precision_score(y_test, pred), recall_score(y_test, pred)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Уже лучше. Подберём параметры по сетке, оптимизируя точность:"
]
},
{
"cell_type": "code",
"execution_count": 139,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"parameters = {'n_train_splits':np.arange(3,15), 'max_depth':np.arange(3, 15)}\n",
"classifier = MyClassifier(2, 2, features=get_features)\n",
"gs = GridSearchCV(classifier, param_grid=parameters, scoring='accuracy', cv=5, n_jobs=-1)\n",
"params = []\n",
"X_train_all = np.empty((0, 27), dtype='bool')\n",
"y_train_all = np.empty(0, dtype='bool')\n",
"for i in range(1, 11):\n",
" file_train = 'train{}.csv'.format(i)\n",
" X_train, y_train = prepare_data(file_train)\n",
" X_train_all = np.vstack((X_train_all, X_train))\n",
" y_train_all = np.hstack((y_train_all, y_train))\n",
"\n",
"best_params = gs.fit(X_train_all, y_train_all)"
]
},
{
"cell_type": "code",
"execution_count": 140,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'max_depth': 11, 'n_train_splits': 7}\n"
]
}
],
"source": [
"best_params_val = best_params.best_params_\n",
"print(best_params_val)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Итоговые значения метрик на тесте (при обучении на соответствующем train):"
]
},
{
"cell_type": "code",
"execution_count": 161,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Average: acc. 0.9410423092812241, prec. 0.9394644632821766, rec. 0.9724699566254443, f1. 0.9552805361560927\n"
]
}
],
"source": [
"metrics_avg = []\n",
"for i in range(1, 11):\n",
" file_train = 'train{}.csv'.format(i)\n",
" file_test = 'test{}.csv'.format(i)\n",
" X_train, y_train = prepare_data(file_train)\n",
" X_test, y_test = prepare_data(file_test)\n",
" classifier = MyClassifier(n_split, max_depth, features=get_features)\n",
" classifier.set_params(**best_params_val)\n",
" classifier.fit(X_train, y_train)\n",
" pred = classifier.predict(X_test)\n",
" metrics_avg.append(np.array([accuracy_score(y_test, pred),\n",
" precision_score(y_test, pred), recall_score(y_test, pred), f1_score(y_test, pred)]))\n",
"metrics_avg = np.array(metrics_avg).mean(axis=0)\n",
"print('Average: acc. {}, prec. {}, rec. {}, f1. {}'.format(metrics_avg[0], metrics_avg[1], metrics_avg[2], metrics_avg[3]))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}