Intelligent Design of Hairpin Filters Based on Artificial Neural Network and Proximal Policy Optimization

Abstract

Microstrip filters are widely used in high-frequency circuit design for signal frequency selection. However, designing these filters often requires extensive trial and error to achieve the desired performance metrics, leading to significant time costs. In this work, we propose an automated design flow for hairpin filters, a specific type of microstrip filter. We employ artificial neural network (ANN) modeling techniques to predict the circuit performance of hairpin filters, and leverage the efficiency of low-cost models to deploy reinforcement learning agents. Specifically, we use the proximal policy optimization (PPO) reinforcement learning algorithm to learn abstract design actions for the filters, allowing us to achieve automated optimization design. Through simulation results, we demonstrate the effectiveness of the proposed approach. By optimizing the geometric dimensions, we significantly improve the performance metrics of hairpin filters, and the trained agent successfully meets our specified design goals within 5 to 15 design steps. This work serves as a conceptual validation attempt to apply reinforcement learning techniques and pre-trained ANN models to automate MMIC filter design. It exhibits clear advantages in terms of time-saving and performance efficiency when compared to other optimization algorithms.

1. Introduction

In recent years, the rapid development of wireless communication technology has driven the rapid growth in demand for various custom microwave circuits. As a core component of wireless communication systems, microwave circuits not only need to provide high-frequency signal processing and transmission functions but also face higher performance and integration requirements. However, the design and manufacturing of custom microwave circuits is not an easy task. It requires engineers to possess deep expertise and experience, as well as advanced simulation and testing equipment. Additionally, the development cycle of custom microwave circuits is long and the cost is high, requiring a trade-off between performance and cost. Therefore, in different application scenarios, engineers need to carefully analyze the requirements and select suitable microwave circuit solutions to meet the requirements of various scenarios. The design process of microwave filters serves as a typical example.

Microstrip filters are devices that utilize microstrip line structures to select signal frequencies. They are known for their compact structure, small size, low cost, and ease of integration and manufacturing [1], making them a critical component in modern wireless communication systems [2]. The hairpin filter is a classic design of microstrip filters, which enhances the filter’s compactness and further reduces its size by using a folded resonator structure [3].

To ensure an effective design, microstrip filter designers typically need to go through several crucial steps [4,5,6]: 1. Determine the design specifications of the circuit and represent them using transfer functions; 2. Synthesize the transfer functions using an ideal lumped element network; 3. Due to the comparable wavelength of circuit elements in the microwave frequency range, it is necessary to transform the lumped circuit implementation into a distributed circuit implementation and confirm the equivalence between the lumped synthesis network and the desired distributed circuit structure; 4. In the optimization stage, adjust the dimensions of the actual filter structure to achieve the desired frequency response.

The final step often requires engineers to iterate and modify the physical dimensions of the filter to meet the specified design requirements. This iterative process heavily relies on the engineer’s expertise and design experience [7], which can be tedious and time-consuming. It also limits engineers from engaging in more innovative circuit structure development. Therefore, research on automating this design process continues to grow. Additionally, this process is constrained by the computational resources and time costs associated with electromagnetic simulation algorithms [8]. Depending on the complexity of the electromagnetic model, the computation time can range from minutes to hours [9]. These two factors are the main reasons for the longer development cycle of MMIC compared to other IC designs.

Developing more efficient Electronic Design Automation (EDA) software to meet the growing market demands is one of the research directions that most researchers are focusing on [10]. To overcome the challenges of CPU-intensive electromagnetic (EM) simulation calculations and automated synthesis of microwave circuits, researchers have begun exploring the potential applications of machine learning techniques in design automation and computer-aided design, such as artificial neural networks (ANN) and heuristic optimization algorithms.

The main contributions of this paper are as follows:

(1)

For the design of hairpin filters, we transform the design process into a continuous reinforcement learning (RL) problem and propose an RL framework based on the PPO algorithm.

(2)

With guidance from practical design experience, we appropriately design a reward function to stabilize and accelerate the training process of RL.

(3)

To reduce the time cost of repetitive success or failure in the RL training process, we introduce a forward neural network model trained through supervised learning to replace the CPU-intensive EM simulation.

The remaining structure of the article is as follows. Section 2 introduces the related work on neural networks and heuristic algorithms. In Section 3, we present the algorithm framework, design a method for hairpin filter design based on PPO, and provide detailed explanations for the training settings of the forward network model for hairpin filters, as well as the design of RL states, actions, and reward functions, with a specific focus on the reward function design. To validate the effectiveness of the proposed design method, we conducted a series of experiments, and the experimental results are presented and discussed in Section 4. Finally, Section 5 concludes the article, discussing the achievements, limitations, and future research plans.

2. Related Work

2.1. Modeling and Applications of Artificial Neural Networks

Artificial neural networks (ANNs) are a type of machine learning algorithm inspired by biological neurons. They simulate the mechanism of brain structure and external stimulus response, can capture any nonlinear relationship with high precision, and fit and estimate functions, thus forming mathematical mapping between any input–output relationship. Once the ANN model is trained, it can quickly and accurately complete calculations with almost negligible overhead. Therefore, artificial neural network technology is considered a possible solution, training the ANN model with the simulation results of relevant electromagnetic simulation software in order to reduce the cost of simulation calculations by replacing expensive EM models with ANN models [11,12].

Artificial neural network technology has been widely applied in the modeling and design of microwave components or circuits, including the modeling of on-chip square spiral inductors [13], the development of efficient and accurate models of CPW-related components [14], and the modeling of small-signal and large-signal models of field-effect transistors combined with equivalent circuit modeling and ANN modeling technology [15]. It also involves research directions such as the optimization of filter design [16,17,18], power amplifiers [19], and microstrip antennas [20]. Another design idea based on ANN modeling is called “neural network inverse modeling” [21], from which there have been some research results [22,23,24]. Unlike the forward model of a microwave device, where the input is the geometric design parameters, the input of the neural network inverse model is electrical parameters and the output is the geometric design parameters of the device. Although the inverse model of the neural network can produce design solutions faster than traditional optimization methods, the multivalued problem that the input and output are not uniquely mapped makes the inverse modeling much more difficult than the forward one [25].

Neural network modeling technology is not only used in the field of microwave device modeling; ANN modeling technology is also employed to predict the boundary conditions of the concentration distribution of a microfluidic concentration gradient generator (a device used to generate gradient concentrations in microfluidic chips [26]), which is widely used in chemistry, biology, medicine, and other fields. Compared to traditional methods, this approach achieved an acceleration of over 300 times while maintaining an accuracy rate of 93.71%.

In this work, we utilize neural network modeling techniques to establish an ANN surrogate model for hairpin filters, replacing expensive EM simulations.

2.2. Application of Nature-Inspired Heuristic Algorithms

In recent years, heuristic intelligent algorithms have been widely applied in various fields. These algorithms utilize optimization methods such as particle swarm optimization algorithm [27], genetic algorithm [28], and reinforcement learning [29] for the automatic design and optimization of neural network parameters.

Ref. [30] proposes a hybrid algorithm, MPA-SCA, for parameter selection of hybrid active power filters (HAPF) in power plants. This algorithm combines the Marine Predator Algorithm (MPA) and Sine Cosine Algorithm (SCA) to optimize the performance of HAPF. The proposed algorithm is tested in various industrial power plants and the results show that it outperforms other algorithms in mitigating power harmonics. The study concludes that the proposed algorithm has the potential to be a valuable tool for designing and optimizing HAPFs in the power industry.

Ref. [31] introduces a novel metaheuristic approach based on cuckoo search for deep learning-based depression prediction. This method is tested on a Twitter post dataset, achieving an accuracy of 87.5%. The results demonstrate the potential of this approach in utilizing social media data for predicting depression.

Heuristic algorithms are widely applied in the field of microwave/RF as well. Due to the lack of analytical solutions or well-defined mathematical formulas in microwave circuit design optimization methods and the involvement of a highly complex optimization environment, these heuristic algorithms provide a flexible exploratory approach and have the ability to escape local optima. Therefore, they are well-suited to global optimization design of microwave circuits.

Ref. [32] applied the particle swarm optimization algorithm to microwave circuit design, specifically optimizing microstrip couplers and single parallel short-circuit matching circuits. Simulation tests confirmed the effectiveness of this method. Ref. [33] proposed a universal precise design method for RF circuits based on the NSGA-II genetic algorithm, addressing the problem of multiple conflicting objectives in RF circuit design using a low noise amplifier as an example. Refs. [34,35] proposed an automatic tuning framework for cavity filters based on reinforcement learning algorithms. This framework was tested in a simulation environment and verified its applicability under various custom tuning tasks. Ref. [36] utilized the deep deterministic policy gradient algorithm to automatically adjust and design the topology structure of high-sensitivity microwave microfluidic sensors, resulting in a significant improvement in sensor sensitivity.

However, these heuristic intelligent search algorithms still require a large number of iterative optimizations to explore the entire design space. In microwave circuit design, each iteration of optimization requires evaluation based on electromagnetic simulations, resulting in significant computational overhead. In this work, we address this issue by utilizing neural network modeling techniques. Additionally, building upon previous research, this study explores the extension and application of the PPO reinforcement learning algorithm in microstrip filter design tasks. Furthermore, we accelerate the training process by incorporating a reward function based on engineer’s expertise, thereby improving learning efficiency.

3. Overview of the Design Method

The object of our study is a fifth-order hairpin filter, as shown in Figure 1. The hairpin filter achieves a more compact design by folding parallel-coupled line resonators into a “U” shape [37], essentially functioning as a coupled bandpass filter.

Its operation is based on the characteristics of microstrip lines and the propagation principles of electromagnetic waves. A microstrip line is a conductive wire fabricated on a dielectric substrate, possessing certain inductance and capacitance. When multiple microstrip lines are arranged in parallel, electromagnetic coupling occurs between them. This coupling causes a phase difference in the propagation of electromagnetic waves between the microstrip lines, resulting in filtering effects. Specifically, when an input signal passes through the hairpin filter, it is influenced by the coupling between the microstrip lines. Signals of different frequencies propagate at different speeds between the microstrip lines, leading to changes in the phase difference at the output end. By properly designing the parameters and arrangement of the microstrip lines, the phase difference within specific frequency ranges can be selectively increased or decreased, achieving filtering effects on the signal. Additionally, the inductance and capacitance characteristics of the microstrip lines also contribute to signal filtering. The inductance and capacitance of microstrip lines are frequency-dependent, and the propagation speed and impedance of signals on microstrip lines vary with different frequencies. By appropriately designing the inductance and capacitance of the microstrip lines, signals within specific frequency ranges can be selectively impeded or transmitted, achieving filtering effects.

By adjusting the geometric parameters and spacing of the microstrip lines, the resonant frequency and bandwidth of the hairpin filter can be controlled, selectively impeding or transmitting signals within specific frequency ranges, achieving filtering effects. As shown in Figure 1, the arm lengths $(L_1,L_2,Lt)$, gaps $(d_1,d_2)$, and line width $\left(W\right)$ are all related to the frequency response of the filter, affecting its stopband attenuation and selectivity.

3.1. Framework

Figure 2 illustrates the design flowchart of the optimization strategy proposed for the hairpin filter. We propose a method for designing and optimizing the hairpin filter using neural network modeling and reinforcement learning techniques. In this method, we train an ANN model using EM simulation data of the hairpin filter. The trained ANN model is then used as a substitute for EM simulation and an RL agent is established to interact with the design environment of the hairpin filter. The design solutions are evaluated using the ANN model and reasonable rewards are given to the agent, allowing it to learn the strategies for designing the hairpin filter during this process. Therefore, the entire design process can be divided into two stages.

The initial stage, known as the ANN modeling stage, involves simulating and modeling the electrical behavior of the hairpin filter. To acquire different physical characteristics of the filter under various designs, we scan the geometric parameters that might influence its performance within a chosen design space and conduct simulations. We then create a database that maps the relationship between the geometric and performance parameters, serving as the foundation for training the ANN model. This enables us to develop a cost-effective model that provides the circuit performance, specifically the S-parameters in this instance, with significantly lower computational and time costs compared to traditional electromagnetic simulation software.

In the second stage, namely the reinforcement learning phase, we employ the PPO algorithm. PPO retains the concepts of the actor network and critic network within the Actor–Critic (AC) architecture: the actor network is a policy function that determines what action to take based on the current state of the environment to achieve the maximum reward; the critic network’s task is to estimate the value of the current state, minimizing the error between the predicted state value function and the actual reward. The actor makes decisions based on the critic’s evaluation and optimizes the policy through feedback from the environment. The critic updates the value function based on the actual reward and the action chosen by the actor, providing a more accurate value assessment. Through this mechanism, the agent continuously interacts with the environment by taking actions, and optimizes its own strategy by observing the current state and feedback from the environment, ultimately leading the agent to adopt a strategy that aligns with our expectations and completes the optimization of the hairpin filter.

Our proposed method is not limited to hairpin filters. This method can be extended to the sizing design of other microwave circuits as well.

3.2. Training an ANN for Predicting the S-Parameters of Hairpin Filters

In this section, we delve into the specifics of the training phase for the artificial neural network applied to the hairpin filter. Given that the S-parameters of the microstrip filter precisely delineate its passband, stopband, insertion loss, and return loss frequency response characteristics, we designated the S-parameters as our research objective. We established an artificial neural network model to decipher the nonlinear correlation between the geometric design parameters of the hairpin filter and its S-parameters (amplitude). By varying the geometric design parameters of the hairpin filter, we investigated a potential design space, as illustrated in

Table 1. Simulation parameter space for the hairpin filter in HFSS.

Design Parameters	Lower Bound/μm	Upper Bound/μm	Step Size/μm
W	25	30	0.1
$Lt$	350	380	1
$L_1$	500	600	1
$L_2$	70	100	0.1
$d_1$	10	20	0.1
$d_2$	10	25	0.1
$tw$	30	60	0.1
$scpwg$	6	20	0.1

. We employed the electromagnetic simulation software, HFSS, to acquire the corresponding return loss and insertion loss of the filter. In this phase, we simulated 1759 distinct combinations of geometric parameters. Our goal was to uniformly distribute the sampled design solutions across the design space to ensure our samples effectively encapsulate the frequency characteristics of the hairpin filter within this design space.

The quality of the dataset significantly influences the performance of the final neural network model. Thus, we initially cleanse the acquired simulation data by eliminating outliers and duplicate entries. Subsequently, each simulation dataset is shuffled to form pairs of design-frequency-S parameter data, amounting to a total of 118,698 data pairs. Ultimately, the acquired dataset is standardized. We have engineered an artificial neural network model to predict the S parameters of the hairpin filter. The model’s input layer comprises eight geometric design parameters of the hairpin filter and simulated frequency values. Four hidden layers are constructed, each containing 128 neurons, with the ReLU function serving as their activation function. Furthermore, the amplitude (expressed in dB) of S11  and S21 of the designed filter at the simulated frequency point functions as the output of the artificial neural network, as depicted in Figure 3.

During the training phase, we define the accuracy using the coefficient of determination, $R^2$, as illustrated in Formula (1):

$$R^2=1-\frac{{\sum }_{i=1}^m{(f\left(x_i\right)-y_i)}^2}{{\sum }_{i=1}^m{(f\left(x_i\right)-\overline{y_i})}^2},$$

(1)

where m refers to the total number of items in the current training or testing batch, while i stands for the index of each item. $f\left(x_i\right)$ is the neural network’s predicted value when the input is $x_i$ and $y_i$ is the actual observed value (i.e., the result of the simulation) for $x_i$. $\overline{y_i}$ represents the average of the actual observed values for the current batch. A larger $R^2$ value indicates that the model is more successful at explaining the observed data.

The loss, on the other hand, is defined using the mean squared error (MSE), as depicted in Formula (2):

$$Loss=\frac{1}{m}\sum _m^{i=1}{(f\left(x_i\right)-y_i)}^2,$$

(2)

where m represents the total number of items in the current training or testing batch, while i represents the index of each item within the batch. $x_i$ refers to the input of the neural network, which is the geometric dimensions of the hairpin filter. The functions $f\left(x_i\right)$ and $y_i$ correspond to the predicted value associated with $x_i$ and the actual label value, respectively.

3.3. The PPO in the Reinforcement Learning Algorithm

This section primarily elucidates the foundational concepts of reinforcement learning and the rudimentary principles of the PPO algorithm. Reinforcement learning can be conceptualized as a Markov Decision Process (MDP), wherein the current decision is exclusively contingent upon the present state, devoid of any influence from previous states. Thus, in the reinforcement learning process, we define a sequence of states, actions, and rewards at each time step, akin to $\tau =\{s_1,a_1,r_1,s_2,a_2,r_2,\dots ,s_T,a_T,r_T\}$, where T denotes the termination time step of the learning process. The objective of reinforcement learning is to maximize future rewards by discounting them using a discount factor $\gamma$. We define the discounted return as $R_t={\sum }_T^{t^{{}^{\prime }}}{\gamma }^{t^{{}^{\prime }}-t}{r}_{t^{{}^{\prime }}}$. PPO is a type of policy gradient algorithm. Its goal is to maximize the discounted return by directly enhancing the parameters, denoted as $\theta$, of the policy denoted as ${\pi }_{\theta }$. Presently, the estimators that are widely used for computing policy gradients can be expressed using the following Equation (3):

$$\widehat{g}={\widehat{\mathbb{E}}}_t\left[{\nabla }_{\theta }{log}_{}{\pi }_{\theta }(a_t\mid s_t){\widehat{A}}_t\right],$$

(3)

where ${\widehat{A}}_t$ represents the estimate of the advantage function at the current time step, $E_t[\cdots ]$ signifies the empirical average of a finite batch of samples, and $\widehat{g}$ can be obtained by differentiating the loss function $L^{PG}\left(\theta \right)$ with respect to the parameter $\theta$:

$$L^{PG}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[{log}_{}{\pi }_{\theta }(a_t\mid s_t){\widehat{A}}_t\right],$$

(4)

where the meanings of the various symbolic variables are the same as in Equation (3): ${\pi }_{\theta }(a_t\mid s_t)$ represents the possibility of taking action $a_t$ under state $s_t$, and ${\widehat{A}}_t$ represents the advantage function at the current time step.

However, this updating method of policy gradients is susceptible to excessively updating the policy parameters, which disrupts the stability of the training process. To address this issue, PPO introduces a clipping mechanism to limit the magnitude of updates between the new and old policies, enabling a more stable policy update. Initially, we calculate the probability ratio between the new and old policies, denoted as $r_t\left(\theta \right)$, with the following Formula (5):

$$r_t\left(\theta \right)=\frac{{\pi }_{\theta }\left(a_t|s_t\right)}{{\pi }_{{\theta }_{old}}\left(a_t|s_t\right)},$$

(5)

where ${\pi }_{\theta }\left(a_t|s_t\right)$ represents the probability of selecting action $a_t$ by the current policy in state $s_t$, while ${\pi }_{{\theta }_{old}}\left(a_t|s_t\right)$ represents the probability of selecting action $a_t$ by the original policy before the update in state $s_t$. Based on $r_t\left(\theta \right)$, PPO introduces a new objective function, denoted as $L^{CLIP}\left(\theta \right)$:

$$L^{CLIP}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[min\left(r_t\left(\theta \right){\widehat{A}}_t,clip\left(r_t\left(\theta \right),1-ϵ,1+ϵ\right){\widehat{A}}_t\right)\right],$$

(6)

where $ϵ$ is a hyperparameter. The second term in $min$, $clip\left(r_t\left(\theta \right),1-ϵ,1+ϵ\right){\widehat{A}}_t$ modifies the surrogate objective by clipping the probability ratio, eliminating incentives outside the range of $[1-ϵ,1+ϵ]$ for $r_t\left(\theta \right)$. Finally, the minimum value between the pre-clip and the post-clip is taken. This method is easier to implement and more effective compared to the approach proposed by TRPO [38].

In Algorithm 1, we provide the pseudocode for the specific steps of designing a Kalman filter using the PPO reinforcement learning algorithm. We also provide additional details in Algorithm 1 regarding the implementation and deployment of the ANN forward model in our proposed method.

3.4. Reinforcement Learning Approach for Optimizing Hairpin Filters

In this study, we utilized the PPO reinforcement learning framework within a continuous action space to execute the task. Herein, we delineate the reinforcement learning configuration employed for the hairpin filter design task, ensuring the reinforcement learning methodology is suited to the task at hand. The reinforcement learning environment is defined by states, actions, and rewards, with a specific focus on the reward definition. A well-structured reward can steer the agent towards learning more effective strategies and directly influence the speed of convergence and the final performance of the algorithm. The amalgamation of reinforcement learning and artificial neural network techniques markedly diminishes the time spent in the iterative search for the optimal policy. Nonetheless, it remains vital to accurately assess the agent’s behavior within the environment. Constructing a logical reward function that aligns with the practical design requirements is a fundamental aspect of reinforcement learning.

The ANN model is deployed as a substitute for an EM simulator, such as the Ansys High-Frequency Structure Simulator (HFSS), as discussed in the preceding section. The ANN model provides a more rapid response to the agent’s actions than solvers, thereby facilitating more efficient learning.

State: The state is represented by concatenating two types of feature vectors: (1) a vector comprising geometric design parameters of the hairpin filter, namely, [W, $Lt$, $L_1$, $L_2$, $d_1$, $d_2$, $tw$, $scpwg$], which are normalized based on the upper and lower limits of each parameter’s design space; and (2) the S-parameters (S11 and S21) of the corresponding designed filter at specific frequency points within the simulation frequency range. In this instance, we design within a simulation frequency band from 60 GHz to 125 GHz with a 1 GHz interval, resulting in a total of 66 frequency points for both S11 and S21, amounting to a total of 132 frequency points.

Algorithm 1 Automatic design of hairpin filters based on PPO.

1: Initializing actor policy network parameters ${\pi }_{\theta }$ and ${\pi }_{{\theta }_{old}}$; critic value network parameters $V\left(w\right)$.   2: Initialize the replay buffer D.   3: Randomly initializing the design of the hairpin filter.   4: for eposide = 1 to M do   5:    Calling the forward network of ANN to predict the S-parameters of the current design of the hairpin filter.   6:    Initializing the state $s_1$.   7:    for step = 1 to T do   8:      Select action $a_t$ at according to the current policy ${\pi }_{\theta }$.   9:      Adjusting the design of the hairpin filter to perform the corresponding action. 10:      Calculating the S-parameters of the newly designed hairpin filter based on the ANN model. 11:      Generating the new state $s_{t+1}$, and computing the reward $r_t$. 12:      Store the transition ($s_t$, $a_t$, $r_t$, $s_{t+1}$) in D. 13:      if data volume of D ≥ minimum batch size then 14:         Estimate advantages ${\widehat{A}}_t={\sum }_{t^{\prime }>t}^{}{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}-V(s_t;w)$. 15:         for k = 1 to K do 16:           ${\pi }_{{\theta }_{old}}$←${\pi }_{\theta }$ 17:           Calculate $L\left(\theta \right)$ based on Equation (6). 18:           Update the parameters of the policy network ${\pi }_{\theta }$:$\theta \leftarrow \theta +\alpha \nabla \theta L\left(\theta \right)$. 19:           Update the critic network by minimizing the loss $L\left(w\right)=\frac{1}{T}{\sum }_{t=1}^T{\left({\widehat{A}}_t\right)}^2$: $w\leftarrow w-\beta \nabla wL\left(w\right)$ 20:         end for 21:      end if 22:    end for 23: end for

Action: The agent’s actions govern its interaction with the environment. This framework employs a continuous action space defined as ${\left[-1,1\right]}^m$, where m corresponds to the number of geometric parameters for designing the hairpin filter. In this paper, m = 8. For instance, when $a_t^i=1$, it signifies an increase in the upper limit value of the ith geometric parameter size in the current design space of the hairpin filter.

Reward: The design of a hairpin filter is typically associated with electrical characteristics such as passband range, insertion loss, and return loss. Consequently, the reward function must reflect the quality of the relevant design criteria. Using return loss as an example, it gauges the extent to which the filter aligns with the signal transmission within the passband. Generally, designers aim for the return loss of the filter to be below a certain threshold within the passband and above a certain threshold within the stopband, as depicted in Figure 4:

The assessment of return loss (S11) can be achieved by assigning weighted scores to the performance at each frequency point in both the passband and the stopband. Specifically, within the passband, a straightforward approach involves calculating the score for each frequency point by determining the distance between a specific standard value l (for example, −15 dB) and the S11 at the corresponding frequency point. If the S11 surpasses the standard value at this point, the score is set to be negative; conversely, if the S11 is below the standard value, the score is positive. To avoid the tendency of the system to overly focus on individual frequency points with high scores while neglecting the overall performance across the entire frequency range, we can apply the function $g\left(x\right)=1-e^{-(1+x)}$ to scores greater than 0 to limit score inflation. Consequently, the score $F_p\left(f\right)$ for a single frequency point f within the passband can be represented by Equation (7):

$$F_p\left(f\right)=\left\{\begin{array}{cc}l-S_{11}\left(f\right)& if{S}_{11}\left(f\right)>l\\ g\left(l-S_{11}\left(f\right)\right)& if{S}_{11}\left(f\right)\le l\end{array}\right.$$

(7)

where $S11\left(f\right)$ denotes the dB value of S11 corresponding to the frequency point f, l represents the standardized metric within the passband, and $F_p\left(f\right)$ stands for the evaluation score for a single frequency point f.

The overall performance score of the entire passband, represented by $F_p$, can be derived by calculating the average score of all frequency points within the passband, as demonstrated in Equation (8):

$$F_p=\frac{1}{n}\sum _{i=0}^{n-1}{F}_p\left(f_i\right),$$

(8)

where n denotes the number of frequency points on the passband, i is the index of the frequency point, $F_p\left(f_i\right)$ represents the score estimation at the frequency point $f_i$, and $F_p$ signifies the overall score estimation after taking into account all the frequency points on the passband.

Similarly, we can calculate the performance score $F_s$ within the stopband. By combining these two scores with appropriate weights, we can compute the overall performance score $F_{S_{11}}$ for the return loss:

$$F_{S_{11}}={\beta }_1{F}_p+{\beta }_2{F}_s,$$

(9)

where ${\beta }_1$ and ${\beta }_2$ are the weighting coefficients, $F_p$ represents the composite scores on the passband, $F_s$ represents the composite scores on the stopband, and $F_{S_{11}}$ represents the composite scores on the S11 after combining both the stopband and passband scores.

If we want to consider the performance score of the insertion loss (S21) as well, a similar method can be applied and the detailed process is omitted here. Therefore, we can obtain the overall performance score of the filter as $F_{total}={\gamma }_1{F}_{S_{11}}+{\gamma }_2{F}_{S_{21}}$, where $F_{S_{21}}$ represents the score of the insertion loss, taking into account both the stopband and passband performance. Here, ${\gamma }_2$ and ${\gamma }_2$ are weighting coefficients.

Additionally, we introduce the concepts of short-term objective and ultimate objective to provide additional rewards $r_{extra}$:

$$r_{extra}=\left\{\begin{array}{cc}10& ifachieve{E}_g\\ 1& ifachieve{E}_s\\ 0& otherwise\end{array},\right.$$

(10)

where $E_s$ represents the short-term objective and $E_g$ represents the ultimate objective. Essentially, these short-term and ultimate objectives are the same as the design objectives mentioned earlier, but the indicators of the short-term objectives are easier to achieve than those of the ultimate objectives. If the current design meets or surpasses the short-term objective $E_s$, a reward of 1 is given to the system. If the ultimate objective $E_g$ is met or exceeded, the system receives a reward of 10. If neither objective is achieved, no additional reward is provided.

The ultimate reward function R is defined as follows:

$$R=\eta ·F_{total}+r_{extra},$$

(11)

where $\eta$ is a weighting factor, $F_{total}$ is the previous score for the overall performance of the filter, and $r_{extra}$ is the additional reward.

4. Results and Discussion

4.1. Training of the Proposed ANN

We processed the simulation data from the filters, yielding a total of 118,698 pairs of design-frequency-S parameter data. These pairs were randomly assorted, with 80% (94,958 pairs) designated as the training dataset and the remaining 20% (23,740 pairs) as the testing dataset. The artificial neural network (ANN) was trained over 200 epochs using a learning rate of $1\times {10}^{-3}$. Furthermore, we implemented a decay of the learning rate by a factor of 0.9 every five epochs. The ANN’s training outcomes are depicted in Figure 5a. To more effectively demonstrate the fluctuations in the loss function and accuracy during the later stages of training, we selected the range of 0 to 5 for the loss function and 96.5% to 100% for the accuracy. Observing the variations in the loss function and accuracy curves throughout the training process, it is evident that the neural network model converged to a low error level in a relatively brief period, and the trend progressively stabilized. The error converged to 0.4507, with an accuracy of 99.76% on the training set. On the testing set, the accuracy ultimately stabilized at 99.4%, with no significant overfitting detected.

We selected a test group to compare the prediction results of the ANN model with the simulation results of the EM. As shown in Figure 5b, it was found that the predictions of the ANN model highly conformed to the actual EM simulation results, accurately predicting the changing trends of S11 and S21. This suggests that the ANN model can replace the EM simulation software in predicting the electrical performance of the card filter in our experiments.

We further scrutinized the absolute error distribution of the ANN model. The absolute error distributions for S11 and S21 are depicted in Figure 5c,d, respectively. In both distributions (with the vertical axis employing a logarithmic scale), there are 19,186 data points for S11 with errors predominantly under 0.2, representing approximately 80.82% of the entire testing dataset. Within these, 95.0% of the data exhibit absolute errors less than 1, and the fraction of data pairs with absolute errors less than 3 reaches as high as 97.96%. For S21, around 75.88% of the data display errors less than 0.5, while 91.27% of the data possess prediction errors within 1. This suggests that the ANN model has effectively assimilated the nonlinear relationship between geometric dimensions and electrical behavior. As depicted in Figure 5b, the ANN model demonstrates excellent consistency with the actual simulation results, mirroring the frequency response characteristics of the designed filter. Consequently, we can infer that the established ANN model is aptly suited for predicting the behavior of the hairpin filter and can supersede the electromagnetic simulation kernel in predicting the electrical behavior of the hairpin filter.

4.2. The Experimental Results of Reinforcement Learning

In order to assess our proposed hairpin filter design optimization method, we executed the design within a frequency band of 60 GHz to 125 GHz. During the design process for the frequency range of 85 GHz to 100 GHz, we stipulated that the return loss S11 should be below −11 dB, and the insertion loss S21 should be above −4 dB. For the frequency ranges of 60 GHz to 79 GHz and 105 GHz to 125 GHz, our objectives were for S11 to exceed −1 dB and for S21 to be less than −25 dB. Each episode consisted of 100 optimization steps.

We implemented five training iterations on the reinforcement learning agent, tracking the trend of the average rewards the agent garnered throughout these design processes, as illustrated in Figure 6a. The shaded region represents the observed range of rewards (maximum and minimum values) during these five training iterations. In the actual 6000 episodes of training, we observed a continuous increase in reward, maintaining near a certain level. From the trend of reward changes, we can conclude that the agent gradually mastered the strategy of maximizing reward after a period of learning. This also indicates that the design results of the card filter are gradually approaching or reaching the design task indicators with the increase of completed episodes, optimizing the characteristics of the return loss and insertion loss of the design goal.

Figure 6b records the number of tuning steps used by the agent to reach the design goal for the first time in the training process. If the design goal is not reached in the current episode, the tuning steps are recorded as 100. It can be seen that from the 2000th episode, the agent begins to reach our design goal, and with the increase of episodes, it becomes more and more clear about its goal, completing our design goal with fewer and fewer operation steps.

We demonstrate the fluctuations in the S-parameter curves during the testing experiment. In Figure 7, we validate the design proposals generated by the PPO algorithm using HFSS simulation software in the process of optimizing the filter. It can be observed from the figure that the initial design of the hairpin filter did not meet our performance requirements. However, the frequency response curves showed a clear optimization trend during the algorithm design process, improving the frequency response characteristics within the specified frequency range. In the experiment, the design proposals provided by the algorithm ultimately met the performance requirements and achieved the predetermined design goals for S11 and S21 parameters. In most cases, the trained agent was able to complete the design within approximately 10 steps. Additionally, to verify the actual effectiveness of the ANN in the design process, we provide the ANN evaluation results for the final design proposal in Figure 7.

4.3. Discussions

The results demonstrate the effectiveness of the PPO algorithm in designing hairpin filters. The trained and learned PPO algorithm can modify the geometric dimensions of the filter to achieve the desired frequency response.

In the design of microstrip filters, the topology design of the filter structure is relatively mature, and most structures can be found in various handbooks and literature. However, engineers often face challenges in adjusting the size parameters of the filter. In the past, this process could only be manually completed by experienced engineers, which was expensive and time-consuming, lengthening the development cycle. In this work, by interacting with the design environment, the agent learns the filter design experience and replaces human engineers in executing this strategy. Additionally, we use a neural network model to replace the commonly used EM simulation by engineers, which speeds up the learning and design process, reducing computational costs.

Table 2. Comparison of optimization results for different methods.

	Design Target	Design Method	EM/ANN Model Evaluation Counts	CPU Operation Time
[39]	coupled-line filter	BO with local sampling	1077	5.32 h
[40]	SIW * filter	narrowband bandpass synthesis algorithm	101	44.75 h
[41]	waveguide filter	Homotopy Optimization + ANN	1500∼17,263	8.3 min∼12 min
[42]	Microstrip filter	MOEA/D-FDTD	≈2700	9.5 day
this work	Hairpin Filter	PPO + ANN	5∼15	in seconds

* substrate integrated waveguide.

summarizes the comparisons of different optimization targets, methods, and time costs in related fields in recent years. Among all the compared methods, our proposed method accomplishes the design task in the shortest CPU operation time.

However, it is important to note that the neural network model is trained using EM simulation data, which means that the accuracy of the ANN model cannot surpass that of the EM simulation. This may result in the agent missing out on certain reasonable designs or providing designs that do not meet the specifications, which is one limitation of this method. This could potentially be addressed by improving the model and increasing the training dataset. In the majority of cases, using the ANN model as a replacement for the EM simulation is still reasonable.

Furthermore, the discussions in this paper are limited to exploring fixed topology circuit structures. These circuit structures are reasonable but also restrict the exploration space of the agent. A potential research direction is to use artificial intelligence techniques to explore new topology structures, which may be irregular but potentially more reasonable.

5. Conclusions

In this paper, we propose a novel method for optimizing the geometric dimensions of filters using a new design approach. This method utilizes artificial neural network modeling techniques to replace EM simulation and employs the PPO reinforcement learning algorithm to learn design strategies through trial and error. We validate this method on a hairpin filter and demonstrate its ability to correctly adjust structural parameters to meet design specifications. Compared to other related research directions, our method shows significant advantages in terms of time efficiency. Finally, we believe that this method has the potential to be applied to the design of other microwave circuits.