Note
Click here to download the full example code
Basic usage example of DupleBalanceClassifier
This example shows the basic usage of duplebalance.DupleBalanceClassifier.
print(__doc__)
RANDOM_STATE = 42
Preparation
First, we will import necessary packages and generate an example multi-class imbalanced dataset.
from duplebalance import DupleBalanceClassifier
from duplebalance.base import sort_dict_by_key
from duplebalance.utils._plot import plot_2Dprojection_and_cardinality
from collections import Counter
import matplotlib.pyplot as plt
from sklearn.decomposition import KernelPCA
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
Make a 5-class imbalanced classification task
X, y = make_classification(n_classes=5, class_sep=1, # 5-class
weights=[0.05, 0.05, 0.15, 0.25, 0.5], n_informative=10, n_redundant=1, flip_y=0,
n_features=20, n_clusters_per_class=1, n_samples=2000, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)
origin_distr = sort_dict_by_key(Counter(y_train))
test_distr = sort_dict_by_key(Counter(y_test))
print('Original training dataset shape %s' % origin_distr)
print('Original test dataset shape %s' % test_distr)
# Visualize the dataset
projection = KernelPCA(n_components=2).fit(X, y)
fig = plot_2Dprojection_and_cardinality(X, y, projection=projection)
plt.show()
Out:
Original training dataset shape {0: 60, 1: 42, 2: 131, 3: 257, 4: 510}
Original test dataset shape {0: 40, 1: 58, 2: 169, 3: 243, 4: 490}
Train a DupleBalance Classifier
Basic usage of DupleBalanceClassifier
# Train a DupleBalanceClassifier
clf = DupleBalanceClassifier(
n_estimators=5,
random_state=RANDOM_STATE,
).fit(X_train, y_train)
# Predict & Evaluate
score = clf.score(X_test, y_test)
print ("DupleBalance {} | Balanced AUROC: {:.3f} | #Training Samples: {:d}".format(
len(clf.estimators_), score, sum(clf.estimators_n_training_samples_)
))
Out:
DupleBalance 5 | Balanced AUROC: 0.915 | #Training Samples: 5000
Train DupleBalanceClassifier with automatic parameter tuning
# Train a DupleBalanceClassifier
clf = DupleBalanceClassifier(
n_estimators=5,
random_state=RANDOM_STATE,
).fit(
X_train, y_train,
perturb_alpha='auto',
)
# Predict & Evaluate
score = clf.score(X_test, y_test)
print ("DupleBalance {} | Balanced AUROC: {:.3f} | #Training Samples: {:d}".format(
len(clf.estimators_), score, sum(clf.estimators_n_training_samples_)
))
Out:
0%| | 0/21 [00:00<?, ?it/s]
'perturb_alpha' == 'auto', auto tuning: 0%| | 0/21 [00:00<?, ?it/s]
'perturb_alpha' == 'auto', auto tuning: 5%|## | 1/21 [00:00<00:05, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 10%|#### | 2/21 [00:00<00:04, 3.89it/s]
'perturb_alpha' == 'auto', auto tuning: 14%|######1 | 3/21 [00:00<00:04, 3.89it/s]
'perturb_alpha' == 'auto', auto tuning: 19%|########1 | 4/21 [00:01<00:04, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 24%|##########2 | 5/21 [00:01<00:04, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 29%|############2 | 6/21 [00:01<00:03, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 33%|##############3 | 7/21 [00:01<00:03, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 38%|################3 | 8/21 [00:02<00:03, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 43%|##################4 | 9/21 [00:02<00:03, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 48%|#################### | 10/21 [00:02<00:02, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 52%|###################### | 11/21 [00:02<00:02, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 57%|######################## | 12/21 [00:03<00:02, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 62%|########################## | 13/21 [00:03<00:02, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 67%|############################ | 14/21 [00:03<00:01, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 71%|############################## | 15/21 [00:03<00:01, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 76%|################################ | 16/21 [00:04<00:01, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 81%|################################## | 17/21 [00:04<00:01, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 86%|#################################### | 18/21 [00:04<00:00, 3.89it/s]
'perturb_alpha' == 'auto', auto tuning: 90%|###################################### | 19/21 [00:04<00:00, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 95%|######################################## | 20/21 [00:05<00:00, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 100%|##########################################| 21/21 [00:05<00:00, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 100%|##########################################| 21/21 [00:05<00:00, 3.90it/s]
The perturb_alpha will be set to 0.025 with 0.930 balanced AUROC (validation score)
DupleBalance 5 | Balanced AUROC: 0.921 | #Training Samples: 5000
Train DupleBalanceClassifier with advanced training log
# Train a DupleBalanceClassifier
clf = DupleBalanceClassifier(
n_estimators=5,
random_state=RANDOM_STATE,
).fit(
X_train, y_train,
perturb_alpha='auto',
eval_datasets={'test': (X_test, y_test)},
train_verbose={
'granularity': 1,
'print_distribution': True,
'print_metrics': True,
},
)
# Predict & Evaluate
score = clf.score(X_test, y_test)
print ("DupleBalance {} | Balanced AUROC: {:.3f} | #Training Samples: {:d}".format(
len(clf.estimators_), score, sum(clf.estimators_n_training_samples_)
))
Out:
0%| | 0/21 [00:00<?, ?it/s]
'perturb_alpha' == 'auto', auto tuning: 0%| | 0/21 [00:00<?, ?it/s]
'perturb_alpha' == 'auto', auto tuning: 5%|## | 1/21 [00:00<00:05, 3.87it/s]
'perturb_alpha' == 'auto', auto tuning: 10%|#### | 2/21 [00:00<00:04, 3.88it/s]
'perturb_alpha' == 'auto', auto tuning: 14%|######1 | 3/21 [00:00<00:04, 3.89it/s]
'perturb_alpha' == 'auto', auto tuning: 19%|########1 | 4/21 [00:01<00:04, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 24%|##########2 | 5/21 [00:01<00:04, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 29%|############2 | 6/21 [00:01<00:03, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 33%|##############3 | 7/21 [00:01<00:03, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 38%|################3 | 8/21 [00:02<00:03, 3.91it/s]
'perturb_alpha' == 'auto', auto tuning: 43%|##################4 | 9/21 [00:02<00:03, 3.89it/s]
'perturb_alpha' == 'auto', auto tuning: 48%|#################### | 10/21 [00:02<00:02, 3.86it/s]
'perturb_alpha' == 'auto', auto tuning: 52%|###################### | 11/21 [00:02<00:02, 3.86it/s]
'perturb_alpha' == 'auto', auto tuning: 57%|######################## | 12/21 [00:03<00:02, 3.88it/s]
'perturb_alpha' == 'auto', auto tuning: 62%|########################## | 13/21 [00:03<00:02, 3.89it/s]
'perturb_alpha' == 'auto', auto tuning: 67%|############################ | 14/21 [00:03<00:01, 3.88it/s]
'perturb_alpha' == 'auto', auto tuning: 71%|############################## | 15/21 [00:03<00:01, 3.89it/s]
'perturb_alpha' == 'auto', auto tuning: 76%|################################ | 16/21 [00:04<00:01, 3.89it/s]
'perturb_alpha' == 'auto', auto tuning: 81%|################################## | 17/21 [00:04<00:01, 3.89it/s]
'perturb_alpha' == 'auto', auto tuning: 86%|#################################### | 18/21 [00:04<00:00, 3.87it/s]
'perturb_alpha' == 'auto', auto tuning: 90%|###################################### | 19/21 [00:04<00:00, 3.88it/s]
'perturb_alpha' == 'auto', auto tuning: 95%|######################################## | 20/21 [00:05<00:00, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 100%|##########################################| 21/21 [00:05<00:00, 3.90it/s]
'perturb_alpha' == 'auto', auto tuning: 100%|##########################################| 21/21 [00:05<00:00, 3.89it/s]
The perturb_alpha will be set to 0.025 with 0.930 balanced AUROC (validation score)
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ ┃ ┃ Data: train ┃ Data: test ┃
┃ #Estimators ┃ Class Distribution ┃ Metric ┃ Metric ┃
┃ ┃ ┃ balance-roc-auc ┃ balance-roc-auc ┃
┣━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━┫
┃ 1 ┃ {0: 200, 1: 200, 2: 200, 3: 200, 4: 200} ┃ 0.926 ┃ 0.773 ┃
┃ 2 ┃ {0: 200, 1: 200, 2: 200, 3: 200, 4: 200} ┃ 0.981 ┃ 0.866 ┃
┃ 3 ┃ {0: 200, 1: 200, 2: 200, 3: 200, 4: 200} ┃ 0.996 ┃ 0.897 ┃
┃ 4 ┃ {0: 200, 1: 200, 2: 200, 3: 200, 4: 200} ┃ 0.998 ┃ 0.924 ┃
┃ 5 ┃ {0: 200, 1: 200, 2: 200, 3: 200, 4: 200} ┃ 0.999 ┃ 0.930 ┃
┣━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━┫
┃ final ┃ {0: 200, 1: 200, 2: 200, 3: 200, 4: 200} ┃ 0.999 ┃ 0.930 ┃
┗━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━┛
DupleBalance 5 | Balanced AUROC: 0.921 | #Training Samples: 5000
Total running time of the script: ( 1 minutes 1.194 seconds)
Estimated memory usage: 67 MB