hyperparameter optimization with Model class for NNs

from ai4water.functional import Model
from ai4water.datasets import busan_beach
from ai4water.utils.utils import get_version_info
from ai4water.hyperopt import Categorical, Real, Integer

for k,v in get_version_info().items():
    print(f"{k} version: {v}")

/home/docs/checkouts/readthedocs.org/user_builds/hyperopt-examples/envs/latest/lib/python3.7/site-packages/sklearn/experimental/enable_hist_gradient_boosting.py:17: UserWarning: Since version 1.0, it is not needed to import enable_hist_gradient_boosting anymore. HistGradientBoostingClassifier and HistGradientBoostingRegressor are now stable and can be normally imported from sklearn.ensemble.
  "Since version 1.0, "
python version: 3.7.9 (default, Oct 19 2020, 15:13:17)
[GCC 7.5.0]
os version: posix
ai4water version: 1.06
xgboost version: 1.6.2
easy_mpl version: 0.21.2
SeqMetrics version: 1.3.4
tensorflow version: 2.7.0
keras.api._v2.keras version: 2.7.0
numpy version: 1.21.6
pandas version: 1.3.5
matplotlib version: 3.5.3
h5py version: 3.7.0
joblib version: 1.2.0

data = busan_beach()
print(data.shape)

(1446, 14)

input_features = data.columns.tolist()[0:-1]
print(input_features)

['tide_cm', 'wat_temp_c', 'sal_psu', 'air_temp_c', 'pcp_mm', 'pcp3_mm', 'pcp6_mm', 'pcp12_mm', 'wind_dir_deg', 'wind_speed_mps', 'air_p_hpa', 'mslp_hpa', 'rel_hum']

output_features = data.columns.tolist()[-1:]
print(output_features)

['tetx_coppml']

build the model

lookback = 14

model = Model(
    model = {"layers": {
        "Input": {"shape": (lookback, len(input_features))},
        "LSTM": {"units": Integer(10, 20, name="units"),
                 "activation": Categorical(["relu", "elu", "tanh"], name="activation")},
        "Dense": 1
    }},
    lr=Real(0.00001, 0.01, name="lr"),
    batch_size=Categorical([4, 8, 12, 16, 24], name="batch_size"),
    train_fraction=1.0,
    split_random=True,
    epochs=50,
    ts_args={"lookback": lookback},
    input_features=input_features,
    output_features=output_features,
    x_transformation="zscore",
    y_transformation={"method": "log", "replace_zeros": True, "treat_negatives": True},
)

            building DL model for
            regression problem using Model
Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 Input (InputLayer)          [(None, 14, 13)]          0

 LSTM (LSTM)                 (None, 17)                2108

 Dense (Dense)               (None, 1)                 18

=================================================================
Total params: 2,126
Trainable params: 2,126
Non-trainable params: 0
_________________________________________________________________
dot plot of model could not be plotted due to ('You must install pydot (`pip install pydot`) and install graphviz (see instructions at https://graphviz.gitlab.io/download/) ', 'for plot_model/model_to_dot to work.')

optimizer = model.optimize_hyperparameters(
    data=data,
    num_iterations=25,
    process_results=False,
    refit=False,
)

Iteration No.   Validation Score
0               726462473540633624576.00000 -726462473540633624576.00000
1               2.67351              -1.67351
2               689.21594            -688.21594
3               527.67830            -526.67830
WARNING:tensorflow:5 out of the last 9 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7fecd54dedd0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
4               1.08681              -0.08681
WARNING:tensorflow:6 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7fecd45c4560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
5               5.97746              -4.97746
6               1.04858              -0.04858
7               165132267.56762      -165132266.56762
8               102.12408            -101.12408
9               22085.72568          -22084.72568
10              1.59101              -0.59101
11              29.25840             -28.25840
12              276.89809            -275.89809
13              41.22146             -40.22146
14              27.15423             -26.15423
15              38.06921             -37.06921
16              26.47467             -25.47467
17              45.46555             -44.46555
18              38.64113             -37.64113
19              27.20578             -26.20578
20              19.77687             -18.77687
21              29.88066             -28.88066
22              37.45994             -36.45994
23              26.60292             -25.60292
24              38.55547             -37.55547
/home/docs/checkouts/readthedocs.org/user_builds/hyperopt-examples/envs/latest/lib/python3.7/site-packages/ai4water/utils/utils.py:818: UserWarning: 0 is not equal to 25 so can not perform ranking
  warnings.warn(f"{num_folders} is not equal to {len(results)} so can not perform ranking")

_ = optimizer.plot_importance(save=False)