hyperparameter optimization with Model class

from ai4water.functional import Model
from ai4water.datasets import busan_beach
from ai4water.utils.utils import get_version_info
from ai4water.hyperopt import Categorical, Real, Integer

for k,v in get_version_info().items():
    print(f"{k} version: {v}")

python version: 3.7.9 (default, Oct 19 2020, 15:13:17)
[GCC 7.5.0]
os version: posix
ai4water version: 1.06
xgboost version: 1.6.2
easy_mpl version: 0.21.2
SeqMetrics version: 1.3.4
tensorflow version: 2.7.0
keras.api._v2.keras version: 2.7.0
numpy version: 1.21.6
pandas version: 1.3.5
matplotlib version: 3.5.3
h5py version: 3.7.0
joblib version: 1.2.0

# prepare the data
data = busan_beach()
print(data.shape)

(1446, 14)

input_features = data.columns.tolist()[0:-1]
print(input_features)

['tide_cm', 'wat_temp_c', 'sal_psu', 'air_temp_c', 'pcp_mm', 'pcp3_mm', 'pcp6_mm', 'pcp12_mm', 'wind_dir_deg', 'wind_speed_mps', 'air_p_hpa', 'mslp_hpa', 'rel_hum']

output_features = data.columns.tolist()[-1:]
print(output_features)

['tetx_coppml']

build the model

model = Model(
    model = {"XGBRegressor": {
        "iterations": Integer(low=10, high=30, name='iterations', num_samples=10),
        "learning_rate": Real(low=0.09, high=0.3, prior='log', name='learning_rate', num_samples=10),
        "l2_leaf_reg": Real(low=0.5, high=5.0, name='l2_leaf_reg', num_samples=10),
        "model_size_reg": Real(low=0.1, high=10, name='model_size_reg', num_samples=10),
        "rsm": Real(low=0.1, high=0.5, name='rsm', num_samples=10),
        "border_count": Integer(low=32, high=50, name='border_count', num_samples=10),
        "feature_border_type": Categorical(categories=['Median', 'Uniform', 'UniformAndQuantiles',
                        'MaxLogSum', 'MinEntropy', 'GreedyLogSum'], name='feature_border_type'),
        "n_jobs": 0,
    }},
    train_fraction=1.0,
    split_random=True,
    input_features=input_features,
    output_features=output_features,
    x_transformation="zscore",
    y_transformation={"method": "log", "replace_zeros": True, "treat_negatives": True},
)

building ML model for
regression problem using XGBRegressor

optimizer = model.optimize_hyperparameters(
    data=data,
    num_iterations=25,
    process_results=False  # we can set it to True if we want post-processing of results
)

Iteration No.   Validation Score
[01:53:00] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


0               0.76321              0.23679
[01:53:00] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


1               0.65969              0.34031
[01:53:01] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


2               0.85585              0.14415
[01:53:01] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


3               0.29644              0.70356
[01:53:02] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


4               1.30347              -0.30347
[01:53:02] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


5               0.59253              0.40747
[01:53:03] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


6               0.60976              0.39024
[01:53:03] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


7               0.78775              0.21225
[01:53:04] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


8               0.71816              0.28184
[01:53:04] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


9               0.96965              0.03035
[01:53:06] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


10              1.17726              -0.17726
[01:53:07] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


11              1.32955              -0.32955
[01:53:08] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


12              0.98610              0.01390
[01:53:09] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


13              0.76427              0.23573
[01:53:10] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


14              0.29686              0.70314
[01:53:12] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


15              1.00601              -0.00601
[01:53:13] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


16              1.38017              -0.38017
[01:53:15] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


17              0.28935              0.71065
[01:53:16] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


18              0.24302              0.75698
[01:53:18] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


19              0.26081              0.73919
[01:53:20] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


20              0.29716              0.70284
[01:53:21] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


21              0.31398              0.68602
[01:53:23] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


22              1.15949              -0.15949
[01:53:25] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


23              0.33662              0.66338
[01:53:26] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.


24              0.60774              0.39226
/home/docs/checkouts/readthedocs.org/user_builds/hyperopt-examples/envs/latest/lib/python3.7/site-packages/ai4water/utils/utils.py:818: UserWarning: 0 is not equal to 25 so can not perform ranking
  warnings.warn(f"{num_folders} is not equal to {len(results)} so can not perform ranking")

********** Removing Examples with nan in labels  **********

***** Training *****
input_x shape:  (174, 13)
target shape:  (174, 1)

********** Removing Examples with nan in labels  **********

***** Validation *****
input_x shape:  (44, 13)
target shape:  (44, 1)
[01:53:28] WARNING: ../src/learner.cc:627:
Parameters: { "border_count", "feature_border_type", "iterations", "l2_leaf_reg", "model_size_reg", "rsm" } might not be used.

  This could be a false alarm, with some parameters getting used by language bindings but
  then being mistakenly passed down to XGBoost core, or some parameter actually being used
  but getting flagged wrongly here. Please open an issue if you find any such cases.

optimizer._plot_convergence()

<AxesSubplot:title={'center':'Convergence plot'}, xlabel='Number of calls $n$', ylabel='$\\min f(x)$ after $n$ calls'>

optimizer.best_iter()

18

optimizer.best_paras()

{'iterations': 12, 'learning_rate': 0.16261580301200218, 'l2_leaf_reg': 4.300786492314255, 'model_size_reg': 0.5512921764936062, 'rsm': 0.44119249704395946, 'border_count': 37, 'feature_border_type': 'MinEntropy'}

print(model.config['model'])

{'XGBRegressor': {'iterations': 12, 'learning_rate': 0.16261580301200218, 'l2_leaf_reg': 4.300786492314255, 'model_size_reg': 0.5512921764936062, 'rsm': 0.44119249704395946, 'border_count': 37, 'feature_border_type': 'MinEntropy', 'n_jobs': 0, 'random_state': 313}}

print(model._model)

XGBRegressor(base_score=0.5, booster='gbtree', border_count=37, callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_border_type='MinEntropy', gamma=0,
             gpu_id=-1, grow_policy='depthwise', importance_type=None,
             interaction_constraints='', iterations=12,
             l2_leaf_reg=4.300786492314255, learning_rate=0.16261580301200218,
             max_bin=256, max_cat_to_onehot=4, max_delta_step=0, max_depth=6,
             max_leaves=0, min_child_weight=1, missing=nan,
             model_size_reg=0.5512921764936062, monotone_constraints='()',
             n_estimators=100, n_jobs=0, ...)