HPO Problem Formulation

HPs in NNs must be determined before training starts since they govern the learning process and cannot be adjusted during training. As such, HPO is conducted during the NN construction phase. The goal of HPO is to minimize the loss function of NN predictions on the validation set by tuning HPs \(\mathbf{x} = (x_1, x_2, ..., x_6)\), where each \(x_i\) corresponds to a specific HP. The optimization is expressed as:

\[\mathbf{x}^* = \arg\min_{\mathbf{x} \in \chi} \mathcal{L}(\mathbf{x}),\]

where \(\mathbf{x}^*\) are the optimal HP values, \(\chi\) is the HP domain, and \(\mathcal{L}(\mathbf{x})\) is the loss function. In this work, the root mean squared error (RMSE) is used as the loss function, defined as:

\[\mathcal{L}(\mathbf{x}) = \sqrt{ \frac{\sum_{i=1}^{n_v} \left[f(\mathbf{x}_d^{(i)}) - \hat{f}(\mathbf{x}, \mathbf{x}_d^{(i)})\right]^2}{n_v} },\]

where \(n_v\) is the number of validation data points, \(\mathbf{x}_d\) represents design variables, \(f(\mathbf{x}_d)\) is the actual observation, and \(\hat{f}(\mathbf{x}, \mathbf{x}_d)\) is the predicted output using HPs \(\mathbf{x}\). The HPO concludes once the RMSE is minimized to meet a convergence criterion, such as a maximum number of evaluations or achieving a target RMSE threshold.