Procedural- and Reinforcement-Learning-Based Automation Methods for Analog Integrated Circuit Sizing in the Electrical Design Space

Abstract

Analog integrated circuit sizing is notoriously difficult to automate due to its complexity and scale; thus, it continues to heavily rely on human expert knowledge. This work presents a machine learning-based design automation methodology comprising pre-defined building blocks such as current mirrors or differential pairs and pre-computed look-up tables for electrical characteristics of primitive devices. Modeling the behavior of primitive devices around the operating point with neural networks combines the speed of equation-based methods with the accuracy of simulation-based approaches and, thereby, brings quality of life improvements for analog circuit designers using the $g_{\mathrm{m}}/I_{\mathrm{d}}$ method. Extending this procedural automation method for human design experts, we present a fully autonomous sizing approach. Related work shows that the convergence properties of conventional optimization approaches improve significantly when acting in the electrical domain instead of the geometrical domain. We, therefore, formulate the circuit sizing task as a sequential decision-making problem in the alternative electrical design space. Our automation approach is based entirely on reinforcement learning, whereby abstract agents learn efficient design space navigation through interaction and without expert guidance. These agents’ learning behavior and performance are evaluated on circuits of varying complexity and different technologies, showing both the feasibility and portability of the work presented here.

1. Introduction

As a motivation and introduction, the current state of analog integrated circuit (IC) sizing automation, including the approaches relevant to this work, is outlined in the following subsections. Furthermore, comparisons are drawn between related work regarding machine learning (ML) and reinforcement learning (RL) in the field, further detailing the differences to the approach presented here.

1.1. Current State of Analog IC Sizing Automation

Despite comprehensive research regarding the design automation of analog Mixed-Signal ICs, in terms of topology synthesis [1], design optimization [2], or yield optimization [3], this domain has still not caught up with its digital counterpart [4]. As such, it remains a predominantly manual task relying heavily on human expert knowledge due to the complexity of the subject matter. While digital circuits are designed from a higher level, the gate level, due to their time- and value-discrete nature, analog circuit designers are still sizing individual transistors by hand, bottlenecking the rest of the design flow. Existing automation approaches are categorized as either optimization-based or knowledge-based.

The former employs optimization algorithms, such as Bayesian optimization (BO) [5] or evolutionary strategies (ESs) [6], for example, to find new solutions. Depending on the algorithm and complexity of the circuit, this may lead to long execution times, be sample inefficient, or return physically infeasible solutions [7].

The latter requires human experts to express their knowledge in an executable format, reproducing the results they have found previously [8]. Some overhead is associated with each new design, but value is accrued with each reuse. This is mainly based on the $g_{\mathrm{m}}/I_{\mathrm{d}}$ method [9] and the corresponding pre-computed look-up tables (LUTs), which combines the accuracy of simulation-based methods with the execution time of equation-based methods [10].

1.2. Procedural Analog IC Sizing

The main drawback is a lacking higher level of abstraction. Considering the continuous nature of analog signals, all physical effects and parasitics have to be considered during the design of analog integrated circuits. Hence, they are still designed on the transistor level, in contrast to a fixed set of gates in the digital domain, making it difficult for optimization-based automation approaches [7,11] to find solutions within such a high-dimensional design space in an adequate time.

Formalizing the expert knowledge necessary for the design of analog circuits lays the foundation for procedural automation approaches, which have been successfully employed in the layout domain in the past [12]. These procedures are able to generate layouts for commonly used analog circuits such as current mirrors or differential pairs. These frequently reoccurring structures have been deemed “building blocks” [13,14]. While it is possible to implement such generators for more complex, less frequently used circuits, the time investment is a considerable trade-off. Recently, similar approaches have made advances in the circuit design domain as well [15].

1.3. Machine Learning for EDA

Consequently, learning-based approaches have been emerging in recent years [16,17], attempting to address the yet-unresolved challenges surrounding analog IC design automation by means of ML and RL.

Both, neural networks (NNs) and deep learning (DL) have shown great potential in a wide range of tasks [18]. In the field of computer vision especially, convolutional neural networks (CNNs) enable the autonomous extraction of meaningful information from images, e.g., for object detection [19], semantic segmentation [20], or 2D/3D human pose estimation [21], to only name a few. In recent years, the interest in the application of DL for electronic design automation (EDA) has also increased [17,22], and methods for analog circuit sizing are an active research topic.

While it has been shown that the behavior of primitive devices can be modeled with NNs [23,24], this paper attempts to extend this approach to the building block [13] level. Considering the universal approximation theorem [25], NNs are able to map characteristics of such a building block to corresponding sizing parameters, e.g., $W/L$ ratios, given sufficient data.

Kahraman and Yildirim [26] showed that they can size a current mirror and a differential amplifier with the help of NNs, i.e., a multilayer perceptron (MLP) and a general regression NN. The networks learn a mapping from the performance parameters to the devices’ widths, while the lengths are kept constant.

Mendhurwar et al. [27] proposed a more general NN-based framework for circuit sizing. They obtained a large database of simulation data by performing parameter sweeps over primitive device models for different technologies and used these data for training NNs. Due to difficulties in fitting a single model to the large parameter spaces, Mendhurwar et al. used a binning algorithm to split the parameter space and train multiple sub-models on the resulting bins. Additional correction models were applied to further improve accuracy in specific failure cases.

A combination of an evolutionary algorithm and an NN was proposed by Islamoglu et al. [28]. When evaluating the performance of the individuals, the simulation results were not discarded, but used for training an NN that learns to predict the performances of individuals. The better this network is able to predict the performances, the less simulation runs are required, resulting in a reduction of the execution time by up to 64.80%.

1.4. Reinforcement Learning for EDA

Reinforcement learning is a framework for finding optimal behavior in sequential decision-making problems through interaction. The emergence of deep-learning- based function approximation and subsequently deep reinforcement learning (DRL) has led to tremendous successes in several challenging control problems, from board and video games [29,30] to autonomous driving [31] and robotics [32,33]. By virtue of its two simple provisions, (1) that the problem can be formulated as an Markov decision process (MDP) and (2) that there exists a measure of reward, which indicates the desirability of a given interaction, the framework can be applied to solve an even wider variety of tasks.

With AutoCkt, Settaluri et al. [34] applied DRL to the analog circuit sizing problem. Given a netlist, a test bench, and a target specification, AutoCkt can generate trajectories of actions, e.g., incrementing or decrementing transistor widths, thus satisfying the desired target specification. The actions the RL agent is allowed to take are restricted to specific intervals and step sizes, transforming a continuous action space into a discrete one. AutoCkt uses the algorithm proximal policy optimization (PPO) [35], which is a popular baseline because of its ability to handle large and continuous action spaces, as well as its relative robustness during learning. However, as an on-policy method, PPO discards sampled interactions after a single round of updates—a severe hindrance in a regime where sampling interactions are expensive, e.g., because of slow simulations. For reference, PPO can solve simple continuous control problems after a few tens to hundreds of thousands of interactions, usually sampled at thousands of frames per second, a sampling rate about two orders of magnitude higher than running the 25 $\mathrm{m}$$\mathrm{s}$ simulation used by Settaluri et al.

Wang et al. leveraged the graph structure of circuits and presented a graph convolutional network (GCN)-based RL circuit designer [36], which operates in continuous action spaces, such as the widths and lengths of transistors. Training is performed with the deep deterministic policy gradient (DDPG) [37] algorithm, a natural choice as DDPG is an off-policy actor–critic algorithm with the ability to sample interactions from a large memory buffer repeatedly, only discarding them after the buffer is overflowing. As a reward, the authors defined a figure of merit (FOM): the weighted sum of the normalized performance metrics, i.e., the distance between the target and actual performance—a technique known as reward shaping [38], which provides a dense reward signal, which, in some cases, can aid learning. GCN-RL is run for ${10}^4$ steps corresponding to a runtime of around 5 h and, thus, $1.8$ $\mathrm{s}$ per step. The time required per step is consistent with our approach, which is around 2 s per step.

Li et al. introduced a stochastic attention-based graph neural network (GNN) called circuit attention network (CAN) [39]. They followed the FOM definition of [36], but treated the normalized metrics as a random variable optimized for small variance. This leads to the preference of sizings with small performance variance induced by the layout.

Noting these prior works, we highlight the importance of sensible decisions concerning the definition of the analog sizing problem as an MDP and the choice of RL algorithms. Like Wang et al., we built our approach upon an off-policy algorithm, namely twin delayed deep deterministic policy gradient (TD3) [40], the successor to DDPG. TD3 attempts to alleviate several shortcomings of its predecessor, such as an overestimation bias in the value function leading to the agent getting stuck in local optima and, thus, not genuinely solving complex tasks with small solution spaces. Further, we employed hindsight experience replay (HER) [41], an augmented memory buffer designed to support learning in sparse reward regimes. The sizing task is only solved once all target specifications are met simultaneously, and successful episodes are rare in the agent’s initial exploration. HER effectively “moves the goalpost” when sampling episodes from the memory buffer; while the agent may not have reached the target specification, it has met a target specification. Thus, the agent learns to navigate the state (performance parameter) space effectively, making the most of a limited number of simulation interactions without the need for manual reward shaping.

Lastly, unlike previous works conducted purely in geometrical spaces, we propose to transfer the sizing problem into the electrical parameter space, significantly reducing the search space [42]. This paradigm shift outright prevents dead-end episodes due to infeasible configurations of geometrical parameters by only allowing primitive devices at sensible operating points, which in turn guarantees computable results from the simulation analyses.

This lines up with the preferred $g_{\mathrm{m}}/I_{\mathrm{d}}$ methodology [43] for manual design by human experts. Device geometries are merely a means to an end, while the electrical characteristics of individual devices are much more closely correlated with the overall circuit performance. It is generally difficult for human experts to intuitively relate transistor dimensions to circuit performance. Instead, they have a better sense of what the operating points of specific devices should be.

1.5. Structure

Section 2 introduces the concept of function mappings from the electrical design domain to geometrical sizing. Subsequently, the sampling of data and training are covered in Section 3. The trained models are then used in Section 4 to demonstrate the procedural design with an ML-powered $g_{\mathrm{m}}/I_{\mathrm{d}}$ methodology. This methodology for analog IC sizing automation was first reported in [44]. Section 5 describes how these function mappings are used in conjunction with RL agents, where they act as a performant interface to the $g_{\mathrm{m}}/I_{\mathrm{d}}$ methodology. Experimental results of this approach, which were first presented in [45], are recapped and discussed in Section 6.

2. Function Mappings for Circuit Sizing

First, let us consider a very simple design task, such as a voltage divider, made up only of resistors. Here, a designer would start by determining the required resistances, which are the values of the electrical ($\mathfrak{E}$) domain, independent of any particular technology. In a subsequent step, the designer would pick a specific resistor model from a given process design kit (PDK) and convert the resistances to corresponding geometrical ($\mathfrak{G}$) values according to the documentation. This voltage divider will behave the same in any technology, provided there is a way of converting the desired resistance ($\mathfrak{E}$) to the correct widths and lengths ($\mathfrak{G}$).

When sizing transistors, however, designers do not have this luxury of specifying desired electrical behavior and looking up a conversion into geometrical sizing parameters in the PDK manual. While the equations mapping terminal voltages and geometries to operating point parameters exist in the device models [46,47], they are not easily rearranged and only grow in complexity with each new version. As such, this would require function optimization with a simulator in the loop to find geometrical parameters that result in the desired electrical behavior. The search space, spanned by possible combinations of widths and lengths for multiple devices in a circuit, is considerable and difficult to constrain. It would be much more intuitive for a circuit designer, however, to constrain the parameters of the electrical domain, such as the speed ($f_{\mathrm{ug}}$) or the efficiency ($g_{\mathrm{m}}/i_{\mathrm{d}}$) of a device. In many cases it is possible to fix such an electric parameter to a specific value, greatly reducing the search space. A technology-dependent way of translating desired operating point parameters into widths and lengths would lift the design problem into the electrical domain. Here, the task can be viewed as entirely decoupled from the technology and geometries. Once a design point, consisting of the desired electrical behavior of the transistors, is found, it is the responsibility of such a translation function for a given technology to produce the correct widths and lengths, resulting in this specified electrical behavior. That way, the same operating point can be reproduced, provided the function exists for a PDK. If the electrical behavior of individual devices is reproduced, the performance of the overall circuit should be comparable as well.

Instead of analytically inverting the equations of the transistor model, in this approach, non-linear regression models are trained to learn and approximate the mappings shown in (1).

$$\begin{array}{c}{\rho }_{\mathrm{type},\mathrm{tech}}:\left[\begin{array}{c}\frac{g_{\mathrm{m}}}{I_{\mathrm{d}}}\\ f_{\mathrm{ug}}\\ V_{\mathrm{ds}}\\ V_{\mathrm{bs}}\end{array}\right]\mapsto \left[\begin{array}{c}\frac{I_{\mathrm{d}}}{W}\\ L\\ \frac{g_{\mathrm{ds}}}{W}\\ V_{\mathrm{gs}}\end{array}\right]\\ \phantom{(}\end{array}$$

(1)

This results in the translation function $\rho$ for a given PDK (tech) and device type, where this type is either NMOS or PMOS. As for the inputs to this model, the aforementioned speed and efficiency are considered expert knowledge because an experienced circuit designer has an intuition for choosing these values. Additionally, some prior knowledge in the form of the drain–source voltage ($V_{\mathrm{ds}}$) and bulk–source voltage ($V_{\mathrm{bs}}$) is used as part of the input. These latter parameters can be determined by the location of the device in the circuit.

Related work [13] shows that the design space can be reduced even further by breaking a circuit into building blocks, such as the current mirror or the differential pair, which are highlighted in Figure A1. While NNs can be trained to approximate a mapping from building block specific performances to individual sizing parameters for each device therein [26], this work focuses on an alternative approach. It is shown that sizing a single reference device and simply propagating the sizing parameters to related devices within a building block is sufficient [48]. This only requires training data for a single device and implicitly ensures that functionally related devices are matched. Hence, the sizing of a building block is reduced to a single NN evaluation, which will be elaborated on during the example in Section 4.

3. Data Sampling and Training

As mentioned previously, in Section 2, the foundation for this approach is LUTs containing the operating point parameters of a transistor [9]. NNs can be trained to learn a mapping of this data, approximating the simulation model. However, previous work has shown that it can be difficult for NN predictions to converge over the entire input space [27]. Instead of binning the input space and training a correction model, in this approach, the dataset is sampled [49] and transformed [50], changing the distribution of the outputs, as shown in Figure 1, yielding better convergence of a single model.

All mappings modeling primitive device behavior around the operating point presented in this work were trained using the same network architecture and training algorithm, which was found by iteratively extending the architectures of previous works [26,27]. The result is an NN with 4 inputs, 7 fully connected hidden layers, and 4 outputs, where rectified linear unit (ReLU) [51] is used as the activation for all hidden layers. These hidden layers consist of 128, 256, 512, 1024, 512, 256, and 128 neurons, respectively. The Adam optimizer [52], with a learning rate of $\alpha ={10}^{-3}$ and exponential decay rates for the moment estimates ${\beta }_1=0.9$ and ${\beta }_2=0.999$, is used to minimize the mean-squared error (MSE) between the LUT and predictions. Additionally, the mean absolute error (MAE) is calculated for the validation set while monitoring the training progress. Training on a dataset of $4\times {10}^6$ samples for 24 epochs with a batch size of 128 took ca. 35 $\min$ on an NVIDIA^® RTX ^™ 3090. Figure 2 shows how well the predictions agree with the LUT after training. The left shows the drain current density ($J_{\mathrm{d}}=I_{\mathrm{d}}/W$) over the efficiency, while the self-gain ($g_{\mathrm{m}}/g_{\mathrm{ds}}$) over the saturation voltage ($v_{\mathrm{dsat}}$) is depicted on the right.

The training data were obtained by characterizing primitive devices over a range of terminal voltages ($V_{\mathrm{ds}}$, $V_{\mathrm{ds}}$, $V_{\mathrm{ds}}$) and geometries (W, L), resulting in two ML models (NMOS and PMOS) per PDK.

4. Procedural Design Example

For demonstrating the viability of the methodology for human circuit designers, the symmetrical amplifier (SYM) shown in Figure A1a is sized to meet the specification given in

Table 1. Specification for procedural sizing examples.

Parameter	VDD	Vin,cm	Vout,cm	IB0	CL
Specification	3.30 V	1.65 V	1.65 V	3.00 μA	10.00 pF

. The strategy is expressed entirely in the electrical domain as a sequence of function evaluations for each building block in the circuit. When executing this procedure for a technology, the correspondingly trained models act as a drop-in replacement for $\rho$.

Initially, prior knowledge is considered by observing the specification, given in Table 1, and deciding on a biasing current $I_{\mathrm{d},\mathtt{MNCM}\mathtt{12}}=I_{\mathtt{B}\mathtt{1}}=2\times I_{\mathrm{B}0}$, as well as an output current $I_{\mathrm{d},\mathtt{MNCM}\mathtt{32}}=I_{\mathtt{B}\mathtt{2}}=4\times I_{\mathrm{B}0}$, resulting in the ratio $M=1:4$ of the PMOS current mirrors MPCM2. Usually, this is chosen to balance the power consumption and phase margin. Since this has to be analyzed separately by simulation, starting values $M_{\mathtt{cm}\mathtt{21}}=1$ and $M_{\mathtt{cm}\mathtt{22}}=4$ were selected. Thus, a sizing strategy is expressed as a sequence of function evaluations, where the inputs depend on the knowledge and intuition of an expert circuit designer and on prior knowledge of the location and connectivity of the device in the circuit. This procedure, which is reminiscent of the manual work flow with analytical equations, is illustrated in detail hereafter and captured in executable form. First, since the common mode output voltage $V_{\mathrm{out},\mathrm{cm}}$ is known, current mirror MNCM3 is considered with (2).

$${\rho }_{\mathrm{NMOS}}\left(\left[\begin{array}{c}{(g_{\mathrm{m}}/I_{\mathrm{d}})}_{\mathtt{cm}\mathtt{3}}\\ f_{\mathrm{ug},\mathtt{cm}\mathtt{3}}\\ 0.5·V_{\mathrm{DD}}\\ 0.0\end{array}\right]\right)\Rightarrow \left[\begin{array}{c}{J}_{\mathrm{d},\mathtt{cm}\mathtt{3}}\\ L_{\mathtt{cm}\mathtt{3}}\\ g_{\mathrm{ds},\mathtt{cm}\mathtt{3}}/W_{\mathtt{cm}\mathtt{3}}\\ V_{\mathrm{gs},\mathtt{cm}\mathtt{3}}\end{array}\right]$$

(2)

This defines the length $L_{\mathtt{cm}\mathtt{3}}$ and width $W_{\mathtt{cm}\mathtt{3}}=I_{\mathtt{B}\mathtt{2}}/J_{\mathrm{d},\mathtt{cm}\mathtt{32}}$ in terms of speed and efficiency and constitutes $M_{\mathtt{cm}\mathtt{31}}=M_{\mathtt{cm}\mathtt{32}}=2$ for a $1:1$ ratio. Next, the active load current mirrors MPCM21 and MPCM22 are considered with the single model evaluation given in (3), thereby sizing them identically in terms of the same $f_{\mathrm{ug}}$ and $g_{\mathrm{m}}/I_{\mathrm{d}}$.

$${\rho }_{\mathrm{PMOS}}\left(\left[\begin{array}{c}{(g_{\mathrm{m}}/I_{\mathrm{d}})}_{\mathtt{cm}\mathtt{2}}\\ f_{\mathrm{ug},\mathtt{cm}\mathtt{2}}\\ V_{\mathrm{DD}}-V_{\mathrm{out},\mathrm{cm}}\\ 0.0\end{array}\right]\right)\Rightarrow \left[\begin{array}{c}{J}_{\mathrm{d},\mathtt{cm}\mathtt{2}}\\ L_{\mathtt{cm}\mathtt{2}}\\ g_{\mathrm{ds},\mathtt{cm}\mathtt{2}}/W_{\mathtt{cm}\mathtt{2}}\\ V_{\mathrm{gs},\mathtt{cm}\mathtt{2}}\end{array}\right]$$

(3)

yielding the length $L_{\mathrm{cm}2}$ and the width $W_{\mathrm{cm}2}=(I_{\mathrm{B}1}/2)/J_{\mathrm{d},\mathtt{cm}\mathtt{22}}$, while the ratio $M_{\mathtt{cm}\mathtt{21}}:M_{\mathtt{cm}\mathtt{22}}$ was defined earlier. Let $V_{\mathrm{y}}=V_{\mathrm{DD}}-V_{\mathrm{gs},\mathtt{cm}\mathtt{2}}$, then the differential pair MND1 is defined in terms of speed and efficiency with (4).

$${\rho }_{\mathrm{NMOS}}\left(\left[\begin{array}{c}{(g_{\mathrm{m}}/I_{\mathrm{d}})}_{\mathtt{dp}\mathtt{1}}\\ f_{\mathrm{ug},\mathtt{dp}\mathtt{1}}\\ V_{\mathrm{y}}-V_{\mathtt{CM}}\\ -V_{\mathtt{CM}}\end{array}\right]\right)\Rightarrow \left[\begin{array}{c}{J}_{\mathrm{d},\mathtt{dp}\mathtt{1}}\\ L_{\mathtt{dp}\mathtt{1}}\\ g_{\mathrm{ds},\mathtt{dp}\mathtt{1}}/W_{\mathtt{dp}\mathtt{1}}\\ V_{\mathrm{gs},\mathtt{dp}\mathtt{1}}\end{array}\right]$$

(4)

where the width and length are obtained in the same way as previously described. Finally, (5) defines the sizing for MNCM1 in terms of electrical parameters.

$${\rho }_{\mathrm{NMOS}}\left(\left[\begin{array}{c}{(g_{\mathrm{m}}/I_{\mathrm{d}})}_{\mathtt{cm}\mathtt{1}}\\ f_{\mathrm{ug},\mathtt{cm}\mathtt{1}}\\ V_{\mathtt{CM}}\\ 0.0\end{array}\right]\right)\Rightarrow \left[\begin{array}{c}{J}_{\mathrm{d},\mathtt{cm}\mathtt{1}}\\ L_{\mathtt{cm}\mathtt{1}}\\ g_{\mathrm{ds},\mathtt{cm}\mathtt{1}}/W_{\mathtt{cm}\mathtt{1}}\\ V_{\mathrm{gs},\mathtt{cm}\mathtt{1}}\end{array}\right]$$

(5)

Consequently, a sizing procedure for this entire circuit, consisting of 10 devices, is wholly expressed by a sequence of 4 function evaluations and defines the strategy entirely in terms of electrical characteristics according to (6).

$$f_{\mathrm{SYM}}:\left[\begin{array}{c}{(g_{\mathrm{m}}/I_{\mathrm{d}})}_{\mathtt{dp}\mathtt{1}}\\ {(g_{\mathrm{m}}/I_{\mathrm{d}})}_{\mathtt{cm}\mathtt{1}}\\ {(g_{\mathrm{m}}/I_{\mathrm{d}})}_{\mathtt{cm}\mathtt{2}}\\ {(g_{\mathrm{m}}/I_{\mathrm{d}})}_{\mathtt{cm}\mathtt{3}}\\ f_{\mathrm{ug},\mathtt{dp}\mathtt{1}}\\ f_{ug,\mathtt{cm}\mathtt{1}}\\ f_{\mathrm{ug},\mathtt{cm}\mathtt{2}}\\ f_{\mathrm{ug},\mathtt{cm}\mathtt{3}}\end{array}\right]\mapsto \left[\begin{array}{c}{W}_{\mathtt{dp}\mathtt{1}}\\ W_{\mathtt{cm}\mathtt{1}}\\ W_{\mathtt{cm}\mathtt{2}}\\ W_{\mathtt{cm}\mathtt{3}}\\ L_{\mathtt{dp}\mathtt{1}}\\ L_{\mathtt{cm}\mathtt{1}}\\ L_{\mathtt{cm}\mathtt{2}}\\ L_{\mathtt{cm}\mathtt{3}}\\ M_{\mathtt{dp}\mathtt{11}}\\ ⋮\\ M_{\mathtt{cm}\mathtt{32}}\end{array}\right]$$

(6)

More generally, this achieves a mapping from the electrical domain onto the geometrical domain, $f_{\mathrm{SYM}}:\mathfrak{E}\to \mathfrak{G}$, for this particular circuit, where the transformation models $\rho$ act as a drop-in replacement for any technology. The geometrical sizing parameters obtained from (6) are used in conjunction with a simulator to extract circuit performance parameters. By either procedurally or interactively adjusting the electrical characteristics of the building blocks, an experienced circuit designer may approach any specification. However, for this example, a sequential-model-based optimization library [53] is used to find the target performances given in

Table 2. Target specification for procedural sizing examples.

Parameter	Unit	Target	Result
DC loop gain ($A_0$)	dB	>60.00	60.70
unity gain bandwidth (UGBW)	MHz	> 7.50	7.80
common mode rejection ratio (CMRR)	dB	>100.00	118.47
power supply rejection ratio (PSRR)	dB	>80.00	80.61
Slew Rate (SR)	V/μs	>4.00	4.52
phase margin (PM)	°	>80.00	80.03
statistical offset ($V_{\mathrm{off}}\left(1\sigma \right)$)	mV	<5.00	4.71

. For this, a scalar-valued objective function is defined according to (7).

$$o\left({\mathit{x}}_{\mathfrak{E}}\right)=c\circ s\circ f\Rightarrow l$$

(7)

where $f:{\mathit{x}}_{\mathfrak{E}}\mapsto {\mathit{x}}_{\mathfrak{G}}$ is a function converting the electrical characteristics of the building blocks into the geometric sizing parameters of individual devices, such as $f_{\mathrm{SYM}}$. $s:{\mathit{x}}_{\mathfrak{G}}\mapsto \mathit{p}$ denotes the interface to the simulator, which takes a vector of geometric sizing parameters ${\mathit{x}}_{\mathfrak{G}}$ and returns a vector with corresponding circuit performances $\mathit{p}$, such as the ones listed in Table 2. The cost function $c:\mathit{p}\mapsto l$ is a curried [54] version of $c^{\prime }(\mathit{t},\mathit{p})$, defined in (8), returning a scalar loss l, where $\mathit{t}$, a vector of performance targets of the same size n as $\mathit{p}$, is already applied, yielding c.

$$c^{\prime }(\mathit{t},\mathit{p})=\sum _{i=1}^n{e}^{t_i-p_i}$$

(8)

It is implied that the elements of $\mathit{t}$ and $\mathit{p}$ line up, and elements with the same index refer to the same performance parameter. With this, Bayesian optimization using Gaussian processes [5,53] and a function evaluation budget of 128 achieves the results shown in Table 2.

5. Reinforcement Learning Methodology

While the results achieved in the previous section could be improved by tuning the parameters of the optimizer [53], this type of approach is not pursued further in this work. Instead, we considered the formulation of the sizing problem as a function in terms of electrical characteristics, such as $f_{\mathrm{SYM}}$ in (6) for the symmetrical amplifier shown in Figure A1a, as an interface to the $g_{\mathrm{m}}/I_{\mathrm{d}}$ method not only for human expert designers and optimization algorithms, but also DRL agents. Because circuit simulations and performance extraction are expensive, RL agents are trained to reach arbitrary goal states with as few simulation steps as possible. In other words, the agents are not trained to find any optimum, but rather, navigate the design space as efficiently as possible by building intuition through experience, similar to a human designer. This is achieved by training policies and Q-functions, which take as the input a state $s\in \mathcal{S}$, as well as a desired goal $g\in \mathcal{G}$ [55], where the state is defined as a set of performance parameters describing the behavior of a circuit, while the goal is a subset thereof, such that there exists a predicate $h_g:\mathcal{S}\mapsto \{0,1\}$. As such, it is every agent’s designation to achieve any state $\left\{\widehat{s}\right|\widehat{s}\in \mathcal{S},h_g\left(\widehat{s}\right)=1\}$. If the goal space is multi-dimensional, which is the case here, a mapping $m:\mathcal{S}\mapsto \mathcal{G}$ is also required. The HER buffer makes use of this, to generate successful trajectories by finding the goal $g^{\prime }$, which is satisfied by a trajectory’s final state [41].

5.1. Motivation

As the motivation, before going into deeper detail about the implementation, an attempt was made to reproduce the results reported by [34], with the method presented here, utilizing a continuous action space in the electrical domain in conjunction with the HER buffer. For this comparison, two agents were trained on the Miller operational amplifier, shown in Figure A1b, one with an electrical ($\mathfrak{E}$) action space and the other with a geometrical ($\mathfrak{G}$) one. Figure 3 shows the average number of steps it takes these agents to reach a given goal state, sampled at the beginning of the episode, where this goal state is defined by the same parameters $A_0$, UGBW, PM, and current consumption ($I_{\mathrm{DD}}$) [34]. These results indicate that training with a sparse reward signal and HER in addition to the continuous action space improves the navigational capabilities of an agent even in the $\mathfrak{G}$ domain. Additionally, it shows how much easier it is for an agent to find a satisfactory solution in the electrical design space. Here, it takes the agent $1\le t\le 3$ steps to reach a goal state, which is $9\times$ faster than in the results reported by Settaluri et al.

5.2. Overview

For this implementation, $\mathcal{G}$ is a 9-dimensional space, spanned by the unity gain bandwidth (UGBW), phase margin (PM), slew rate (SR), current consumption ($I_{\mathrm{DD}}$), DC loop gain ($A_0$), statistical offset ($V_{\mathrm{off}}\left(1\sigma \right)$), common mode rejection ratio (CMRR), power supply rejection ratio (PSRR), and estimated area ($A_e$), where $A_e$ is the only parameter in the geometrical domain $\mathfrak{G}$. The condition $g\subseteq s$ defined earlier is satisfied by the mapping m shown in (9). Therefore, any goal g is defined as a 9-dimensional coordinate in the state space, where additional parameters of the state such as the input and output voltage ranges or the output referred noise density at different frequencies are not considered for this implementation.

$$m:\left[\begin{array}{c}\mathrm{UGBW}\\ \mathrm{PM}\\ ⋮\\ V_{\mathrm{IL}}\\ V_{\mathrm{IH}}\\ V_{\mathrm{OL}}\\ V_{\mathrm{OH}}\\ V_{\mathrm{N},\mathrm{X}\mathrm{Hz}}\end{array}\right]\mapsto \left[\begin{array}{c}\mathrm{UGBW}\\ \mathrm{PM}\\ \mathrm{SR}\\ \mathrm{CMRR}\\ \mathrm{PSRR}\\ A_0\\ V_{\mathrm{off}}\left(1\sigma \right)\\ A_{\mathrm{e}}\\ I_{\mathrm{dd}}\end{array}\right]$$

(9)

State-of-the-art approaches, discussed in Section 1.4, do not make use of HER, but instead, guide the learning behavior of an agent by defining an FOM, which scores the quality of a proposed sizing [36,39], where such an FOM is the weighted sum of the distances between achieved and desired circuit performance parameters. Reward shaping [38] like this requires expert-level domain knowledge and potentially encodes biases, of the designer crafting the function, into the reward signal. Additionally, modeling trade-offs between performance metrics within a specific design context requires explicit human intervention by adjusting the aforementioned weights [36]. All of these challenges are overcome by leveraging HER and a sparse, binary reward signal, as given in (10) [41].

$$r_g(s_t,a_t)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\mathrm{specification}\mathrm{is}\mathrm{met}\hfill \\ -1,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(10)

Generally, the objective in terms of RL, shown in (12) [40], is finding the optimal policy ${\pi }_{\varphi }$, with parameters $\varphi$, that maximizes the expected return $J\left(\varphi \right)={\mathbb{E}}_{s_i\sim p_{\pi },a_i\sim \pi }\left[R_0\right]$ [40], where the return is the discounted sum of rewards, given in (11) [40].

$$R_t=\sum _{i=t}^T{\gamma }^{i-t}r(s_i,a_i)$$

(11)

$${\pi }_{\varphi }=\underset{\varphi }{argmax}J\left(\varphi \right)$$

(12)

where $\gamma$ is the discount factor and $p_{\pi }$ the state transition probability subject to policy $\pi$ [40], in particular, a policy able to efficiently navigate the analog IC sizing state space, spanned by the circuit performance parameters given in (9), by means of adjusting the operating point parameters of building blocks within a circuit.

Figure 4 illustrates the RL framework implemented for this approach. It is parallelizable, gym-compatible [56], and uses Cadence Spectre [57] for circuit performance evaluation, ensuring compatibility with real-world PDKs. Initially, a custom TD3 [40] agent was implemented in HaskTorch [58], while the functionality and reproducibility were subsequently verified with stable-baselines3 [59].

5.3. Environment

A function f expresses the sizing of a circuit in terms of electrical characteristics, such as the previously described $f_{\mathrm{SYM}}$ for the symmetrical amplifier, and is the core component of the environment. After each reset, the environment samples a new random goal $\mathit{g}\sim \mathcal{U}\left(\mathcal{G}\right)$ and a random initial state ${\mathit{s}}_{t=0}\sim \mathcal{U}\left(\mathcal{S}\right)$ for the episode. Given any state ${\mathit{s}}_t$ and goal $\mathit{g}$ from the environment, an action ${\mathit{a}}_t\sim {\pi }_{\varphi }$ is sampled from the agent’s current policy, where $\mathit{a}\widehat{=}{\mathit{x}}_{\mathfrak{E}}$, as described in Section 4. The environment’s step function takes an ${\mathit{a}}_t$ and transitions into the next state ${\mathit{s}}_{t+1}$ by transforming the action into geometrical sizing parameters $f\left({\mathit{a}}_t\right)\Rightarrow {\mathit{x}}_{\mathfrak{G}}$, entering this sizing into the netlist and simulating this new state. Additionally, the environment returns the reward $r_g(s_t,a_t)$ as defined in (10).

5.3.1. States

The state ${\mathit{s}}_t\in {\mathbb{R}}^n$ of a circuit at time step t is an n-dimensional vector composed of performance parameters, such as the ones given previously in Section 5. Since there is no guidance by a state’s FOM, concatenating the goal ${\mathit{s}}_t\left|\right|\mathit{g}$ is necessary when leveraging HER so the agent has a measure of distance. The individual components of $\mathit{s}$ are of very different magnitudes, wherefore they are normalized based on an estimated range $\mathit{s}=\widehat{\mathit{s}}+ϵ$, where $\widehat{\mathit{s}}\in {[-1,1]}^n$ and $ϵ\in {\mathbb{R}}^n$. This also puts $\mathit{s}$ in proportion with $\mathit{a}$, which are both passed to the critic. All three circuits presented here share the same state space $\mathcal{S}$. Furthermore, we highlight that all components of $\mathit{s}$, and $\mathit{g}\subseteq \mathit{s}$ correspondingly, with the exception of $A_e$, are parameters of the $\mathfrak{E}$ domain.

5.3.2. Actions

The action ${\mathit{a}}_t\in {[-1,1]}^m$ at time step t is an m-dimensional vector, consisting of a normalized efficiency ($g_{\mathrm{m}}/I_{\mathrm{d}}$) and speed ($f_{\mathrm{ug}}$) for every building block, as well as the branch currents $I_{\mathbb{B}}$ in the circuit. Essentially, it is a version of ${\mathit{x}}_{\mathfrak{E}}$, described in Section 4, normalized such that all devices are guaranteed to be in saturation. The reasoning behind this choice of operating point parameters is detailed in Section 2. The main motivation for using a continuous action space over a discrete one is the issue of dimensionality [37]. In case of a discrete space with 1000 options for each geometrical parameter of each building block in the circuit, we would end up with a design space of size ${10}^{30}$ for the symmetrical amplifier, shown in Figure A1a, which is by far the simplest example considered in this work.

For each of the operational amplifiers shown in Figure A1, the dimensions of the action space are different, corresponding to the number of building blocks and branch currents available in the circuit.

The geometrical action space used for comparison, in Section 5.1, comprises widths (Wxx#), lengths (Lxx#), and multipliers (Mxx#), as denoted in Figure A1, which is a normalized version of ${\mathit{x}}_{\mathfrak{G}}$ according to technology constraints.

5.3.3. Rewards

The reward $r_t\in \{-1,0\}$ at time step t, as introduced previously in (10), is a binary signal. If a state $h_g\left({\mathit{s}}_t\right)=1$ for $t<T$ is encountered, the agent has found a coordinate in the state space $\mathcal{S}$ that meets the specification $\mathit{g}$, yielding a reward of 0; otherwise, the reward is always $-1$ [41].

5.3.4. Goals

The goal $\mathit{g}\in {\mathbb{R}}^{n^{\prime }}$ is an $n^{\prime }$-dimensional vector, where $n^{\prime }\le n$, which is sampled at the beginning of an episode and remains identical for all $t\in \{0..T\}$. This is analogous to a specification a human designer would pursue during the manual sizing process. The components of this vector and its relation to the state are detailed in Section 5.2. Similarly, the dimensions of $\mathcal{G}$ remain identical across all circuit environments shown in this work. When using HER and replaying trajectories with augmented goals ${\mathit{g}}^{\prime }$, the reward is calculated given the predicate $h_{{\mathit{g}}^{\prime }}\left({\mathit{s}}_{t+i}\right)$ for any future state ${\mathit{s}}_{t+i}$ [41], which is a set of inequalities, as indicated by a specification, such as the one described in Section 6, checking whether the achieved performance is adequate.

5.4. Agents

Our custom DRL agent was trained using a combination of TD3 [40] and HER [41], the reasoning for which is detailed in Section 1.4. At this time, no hyperparameter search or optimization was performed; instead, the established values found in the literature [40,41] were used and are listed in

Table A2. Hyperparameters for TD3 + HER agents.

Parameter	Value
Optimizer	Adam [52]
Actor Learning Rate	$1·{10}^{-3}$
Critic Learning Rate	$1·{10}^{-3}$
Discount Factor ($\gamma$)	$0.99$
Number of Hidden Layers	2
Number of Hidden Units per Layer	256
Non-Linearity of Hidden Layers	ReLU [51]
Number of Update Steps (E)	40
Target Update Interval (d)	2
Random Exploration Interval	10
Test Interval	5
Parallel Environments (P)	32
Target Smoothing Coefficient ($\tau$)	$0.005$
Noise Clipping (c)	$0.2$
Exploration Policy	$\mathcal{N}(0,0.1)$
Number of Steps per Episode	30
Buffer Size	${10}^6$
Batch Size	128
Goal Sampling Strategy ($\mathbb{S}$)	future [41]
Number of Additional Goals (k)	4

. Algorithm A1 further details the overall implementation. Both policy and critic networks have 2 fully connected hidden layers, with 256 neurons each. All hidden layers use ReLU [51] as their activation function, while the policy’s output layer uses tanh [51].

Parallelizing off-policy RL methods is not common practice; however, it was performed here to cope with the limitations of the environments regarding simulation time. Running all necessary simulation analyses with Cadence Spectre takes about 2 $\mathrm{s}$ on average. Compared to commonly used RL environments [56], this is at least two orders of magnitude slower. This is overcome by parallelization, such that the agent gets to interact with $P=32$ environments simultaneously. Given a state ${\mathit{s}}_t$ and goal $\mathit{g}$ at time step t for each parallel environment, the agent samples an action ${\mathit{a}}_t$, as defined in (13).

$${\mathit{a}}_t^{\top }\sim \pi ({\mathit{s}}_t\Vert \mathit{g})=\left[{(g_{\mathrm{m}}/I_{\mathrm{d}})}_{\mathtt{MX}},\dots ,f_{\mathrm{ug},\mathtt{MX}},\dots ,I_{\mathtt{BX}},\dots \right]$$

(13)

where ${\mathit{a}}_t^{\top }$ is the transposed action in the $\mathfrak{E}$ domain, as described in Section 5.3. During training, some exploration noise $ϵ$ will be added [40] before feeding the actions to the corresponding parallel environments. First, they are denormalized to obtain $g_{\mathrm{m}}/I_{\mathrm{d}}\in \mathbb{R}$, $f_{\mathrm{ug}}\in \mathbb{R}$, and $I_{\mathrm{B}}\in \mathbb{R}$, after which, $\rho$ for the technology of the training environment is used to convert these electrical characteristics into geometrical sizing parameters for each building block, as described in Section 4.

Simulating the environment with these sizings returns the next state, ${\mathit{s}}_{t+1}$, yielding the transition tuple $({\mathit{s}}_t,{\mathit{a}}_t,r_t,d_t,{\mathit{s}}_{t+1})$, which is stored in a preliminary replay buffer ${\mathcal{R}}_e$ as is commonly found in the literature [40]. It is important to note that the goal $\mathit{g}$ sampled at the beginning of the episode only influences an agent’s choice of action, but has no effect on the dynamics of the environment at all [41]. Therefore, any collected trajectory can be replayed during training with an arbitrary goal ${\mathit{g}}^{\prime }$. This lends itself perfectly to the objective of this approach, efficient design space navigation, as it teaches the agents how to reach any coordinate in $\mathcal{S}$ as quickly as possible, instead of finding one optimal coordinate. Even though agents cannot learn how to reach $\mathit{g}$ from any trajectory $\tau =\{{\mathit{s}}_0,\dots ,{\mathit{s}}_T\}$ where $k_{\mathit{g}}\left({\mathit{s}}_t\right)=0\forall {\mathit{s}}_t\in \tau$, they can certainly learn how to reach any intermediate state ${\mathit{g}}^{\prime }=m\left({\mathit{s}}_{t^{\prime }}\right)$ where $0\le t^{\prime }\le T$. Thus, successful trajectories can be created arbitrarily by replaying each trajectory with k augmented goals, where we pretend the agent was meant to reach ${\mathit{s}}_{t^{\prime }}$. Therefore, the agent is able to efficiently navigate $\mathcal{S}$ and reach arbitrary $\mathcal{G}$ therein, even if those were never encountered during training. Following the strategy $\mathbb{S}$, the HER buffer $\mathcal{R}$ [41] is filled by sampling k additional goals G for each transition in ${\mathcal{R}}_e$. $E=40$ optimization steps were performed with mini-batches $\mathcal{B}\sim \mathcal{U}\left(\mathcal{R}\right)$ sampled uniformly from the HER buffer after experiencing $P=32$ parallel episodes. The critics $Q_{{\theta }_1,{\theta }_2}$ were updated towards the minimum target value of actions selected by the target policy ${\pi }_{{\varphi }^{\prime }}$ during each iteration according to (14) [40].

$$\begin{array}{cc}\hfill {\mathit{s}}^{\prime }& =\mathit{s}\Vert \mathit{g}\hfill \\ \hfill \mathit{ϵ}& \sim \mathrm{clip}\left(\mathcal{N}\right(0,\sigma ),-c,c)\hfill \\ \hfill {\mathit{a}}^{\prime }& ={\pi }_{{\varphi }^{\prime }}\left({\mathit{s}}^{\prime }\right)\hfill \\ \hfill y& =r+\gamma min(Q_{{\theta }_1^{\prime }}({\mathit{s}}^{\prime },{\mathit{a}}^{\prime }+\mathit{ϵ}),Q_{{\theta }_2^{\prime }}({\mathit{s}}^{\prime },{\mathit{a}}^{\prime }+\mathit{ϵ}))\hfill \end{array}$$

(14)

The parameters of the online policy $\varphi$ are updated every dth optimization step according to the deterministic policy gradient [60]. Both target critics $Q_{{\theta }_1^{\prime },{\theta }_2^{\prime }}$ and the target policy ${\pi }_{{\varphi }^{\prime }}$ are updated after each episode with a target smoothing coefficient $\tau$. All hyperparameters are listed in Table A2.

The results presented in Section 6 were also reproduced with stable-baselines3 [59], due to the environment being fully gym-compatible [56]. This TD3 reference implementation does not support parallel environments, however, wherefore longer runtimes are to be expected.

6. Experimental Results

For the evaluation of the proposed method, the three operational amplifiers, shown in Figure A1, were selected primarily to showcase the robustness regarding this class of circuits. All agents were trained in the same nine-dimensional goal space $\mathcal{G}$, defined in Section 5. During the initial training, the agents only get to interact with the environment via transformation models $\rho$ for a real-world 350 $\mathrm{n}$$\mathrm{m}$ PDK. On the left in Figure 5, the success rate of each agent is shown, where 100% corresponds to the agent reaching the desired $\mathit{g}$ with $t<T$ steps, averaged over $P=32$ parallel environments. The success rates for training in the $\mathfrak{G}$ domain are omitted in this plot, since the agents fail to achieve any specification over the course of $M=50$ episodes, suggesting that a longer training time and reward shaping might be necessary in this domain.

Section 5 introduces efficient design space navigation as the main focus for this approach. Instead of judging the circuit performance with an FOM [34,36,39], the number of simulations t, such that $h_g\left(s_t\right)=1$, is the defining metric used to assess the agents’ performance. The number of steps, averaged over $P=32$ parallel environments, are plotted versus the training episode on the right side of Figure 5. In addition to the average number of steps $\overline{t}$ and success rate after $M=50$ episodes,

Table 3. All agents are trained with transformation models $\rho$ for a 350 $\mathrm{n}$$\mathrm{m}$ technology and evaluated later with models for a 180 $\mathrm{n}$$\mathrm{m}$ technology. All step values $\overline{t}$ are averaged over $P=32$ parallel environments. Shown is the number of steps until the first successful trajectory, the number of steps when an agent first succeeds with less than 5 steps, as well as the success rate and the number of steps to succeed after $M=50$ training episodes.

Environment	350 nm (Trained)				180 nm (Evaluated)
	First $\mathit{r}=0$	First $\overline{\mathit{t}}<5$	Success	$\overline{\mathit{t}}$	Success	$\overline{\mathit{t}}$
SYM ($\mathfrak{E}$)	$333.91$	$336.94$	100.00%	$1.00$	82.29%	$1.27$
MIL ($\mathfrak{E}$)	$203.50$	$218.41$	97.92%	$1.10$	75.00%	$1.00$
FCA ($\mathfrak{E}$)	$338.22$	$392.53$	100.00%	$1.03$	100.00%	$1.02$

also shows the average number of steps it takes an agent to reach the first success and the average number of steps before the agent achieves success with $t<5$ steps. By observing

Table 4. Initial and final step of a successful episode, by an agent trained on folded cascode amplifier (FCA) in 350 nm and tested on 180 nm.

Parameter	Unit	g	350 nm		180 nm
			s0	st	s0	st
$A_0$	dB	≥75.0	$-27.23$	$77.32$	$-59.52$	$76.03$
UGBW	$\mathrm{M}$$\mathrm{Hz}$	≥2.5	N/A	$2.51$	N/A	$2.58$
SR	$\mathrm{V}$/$\mathrm{m}$$\mathrm{s}$	≥500	N/A	$515.87$	N/A	$565.32$
PM	${}^{\circ }$	≥80.0	0	$85.54$	0	$82.79$
CMRR	dB	≥120.0	$116.69$	$134.70$	$43.98$	$125.24$
PSRR	dB	≥100.0	$107.49$	$127.53$	$17.18$	$109.21$
$V_{\mathrm{off}}\left(1\sigma \right)$	$\mathrm{m}$$\mathrm{V}$	≤1.5	$0.032$	$1.06$	$1.39$	$0.85$
$I_{\mathrm{DD}}$	$\mathsf{\mu }$$\mathrm{A}$	≤25.0	$70.87$	$24.08$	$202.61$	$24.30$
$A_e$	$\mathsf{\mu }$$\mathrm{m}$2	≤6.0	$10.03$	$5.83$	$113.72$	$4.24$
t	–	<30	0	2	0	2

, the benefits of choosing a continuous action space become evident. Upon each environment reset, the agent is confronted with a broken state of the circuit, where some performance metrics cannot even be extracted, which are denoted as N/A in the table. Regardless of how broken this random initial state is or how far it is from the desired goal, the agent proposes a suitable solution almost immediately. This would not be possible in a discrete action space, where the agent is impaired by the step size, such that the efficiency of navigation depends highly on the starting point. Furthermore, Table 4 indicates that, regardless of initial state ${\mathit{s}}_0$ and goal $\mathit{g}$, no explicit expert guidance or reward shaping is necessary. Since the agent is not trained to maximize any FOM, there is never the need for expert intervention or reward shaping [36]; instead, the agent simply navigates to a coordinate in $\mathcal{S}$ where $h_{\mathit{g}}\left({\mathit{s}}_t\right)=1$ such that $t\ll T$, regardless of whether some performances are already met or not. These observations and results satisfy the goal, defined in Section 5, of this approach. After training for $M=50$ episodes, the agents prove to be very capable of design space navigation based on the transformed action space in the $\mathfrak{E}$ domain. Whether this approach can be successfully applied to other circuit classes remains to be examined in future research. However, if the action and observation spaces are equally well defined with similar dimensions, comparable results are to be expected.

Technology Migration

Table 3 shows, in addition to the previously discussed metrics, the findings regarding the success rate and average number of steps of agents evaluated with transformation models of a different technology without re-training. For this experiment, agents trained in a 350 $\mathrm{n}$$\mathrm{m}$ technology were employed on the same circuit, but with different transformation models $\rho$ for a 180 $\mathrm{n}$$\mathrm{m}$ technology. As expected, the results confirm the findings presented in Section 4, where procedures written for one technology can be seamlessly re-used simply by swapping the transformation models. Essentially, the agents are oblivious to the technology; they merely propose operating points that yield a certain performance. Regardless of the technology, if all devices are at the same operating point, the overall performance should be comparable as well. By shifting the blame to the transformation models, there is no need for transfer learning [36] to address technology migration. Similar to human experts, the agents have acquired an intuition regarding the electrical behavior of building blocks and their influence on the entire circuit performance. It is important to note when considering different technologies that boundary conditions, such as the supply voltage, can change and some circuit topologies might not be able to achieve the exact same performance specifications. Therefore, these experiments were conducted with identical boundary conditions.

For further illustration, the other example in Table 4 shows how an agent manages to find the same specification in the same amount of steps t, but in a different technology. By never exposing the agents to technology-dependent parameters, they become significantly more versatile.

7. Conclusions

Findings regarding both the knowledge-based and learning-based analog IC sizing methods are discussed and compared to related work in the following sections.

7.1. Knowledge-Based Circuit Sizing

The state-of-the-art, manual $g_{\mathrm{m}}/I_{\mathrm{d}}$ method for sizing analog ICs based on LUTs was used as the foundation for procedural sizing automation. NNs were trained with the LUT data, effectively modeling the behavior of primitive devices around the operating point, given desired electrical characteristics. Using these models, a sizing procedure for an operational amplifier was expressed in terms of electrical characteristics ($g_{\mathrm{m}}/I_{\mathrm{d}}$, $f_{\mathrm{ug}}$) of building blocks [13] in the circuit.

When employing this method, the primary consideration is the generation of training data and the training of the ML models. With an NVIDIA^® RTX ^™ 3090, training for one such model took approximately 35 $\min$. The data-generating process had to be performed twice per technology, once for NMOS and PMOS. As soon as the models are available, they can be used in conjunction with an already existing procedure, initially composed of different models, seamlessly porting the sizing strategy to another technology. The effort required in training the first models and programmatically expressing sizing strategies is far outweighed by the resulting re-usability. Furthermore, the sizing strategies of experienced circuit designers are captured in a technology-independent way, which is valuable for both academia and industry.

7.2. Learning-Based Circuit Sizing

The findings of the procedural approach were the basis for the following learning-based approach. If an expert circuit designer can use the $g_{\mathrm{m}}/I_{\mathrm{d}}$ method to express an iterative sizing strategy in the electrical domain, it stands to reason that an artificial designer can be trained to learn a similar policy. We formulated the sizing task as a sequential decision-making problem, which allowed the use of RL to find a near-optimal solution given weak supervision in the form of a reward signal. Prior work regarding RL for analog IC sizing exists; however, it does not consider an action space in the electrical domain, nor does it focus on efficient design space navigation, which makes a direct comparison difficult.

Unlike related work [34,36,39], our approach trained RL agents with an action space in the electrical domain and a sparse binary reward signal indicating whether the agent had reached the required specifications. We employed HER to increase sample efficiency without relaxing the original problem, which would introduce bias to the optimization procedure. During training, the agent was rewarded for reaching arbitrary states, i.e., sets of specifications after a series of actions. This reward schedule deprecated the need for a hand-crafted FOM, which would otherwise supplement a sparse reward signal in the initial training stages when the agent is unlikely to reach optimal sets of specifications. Subsequently, the agent learned to navigate the state space, allowing it to reach any desired state with ease once training had concluded.

Wang et al. reported training for up to ${10}^4$ simulation steps, while our approach considered $50\times 30\times 32=4.8\times {10}^4$ samples initially. Table 3 shows that the first successes were achieved after $\frac{1}{3}\sum \{333.91,203.50,338.22\}\times 32=9340.05$, simulation steps on average across all three environments, which is just before the 10th episode. Additionally, we observed that, shortly after this, the agents were able to solve the environment in less than five simulation steps. According to Figure 5, the number of samples could be decreased to $30\times 30\times 32=2.88\times {10}^4$ without any impact on performance.

Leveraging an electrical action space makes the agents oblivious to the device geometries. As such, the method is inherently technology-independent, and no transfer learning is required, unlike other approaches [34,36]. The electrical domain, in conjunction with HER, improves the design space navigation abilities of the agents presented here, making it $9\times$ faster than the ones reported by Settaluri et al.

This behavior demands further analysis since, initially, this was considered a sequential decision-making problem. After training, however, the performance resembled that of supervised learning approaches. Instead of step-by-step progress toward a goal, the agent generates a suitable solution within one or two steps. A naively created dataset for supervised learning in such a space would be comparatively large. For example, the simplest circuit considered here, SYM shown in Figure A1a, with 10 input parameters and 10 values each, would result in a dataset with ${10}^{10}$ points. With the RL algorithm presented here, it seems the $28.8\times {10}^4$ data points collected after 30 episodes are sufficient to train an efficacious artificial designer.

Wang et al. drew a comparison between a designer with 5 years of experience, for whom it takes 6 $\mathrm{h}$ to size an operational amplifier. Although, most likely only a fraction of a human designer’s training is spent on one specific topology, it is still much more than the $4.5$ $\mathrm{h}$ training time of the artificial designers presented here. Additionally, the execution time of $t\le 3$ simulations of a trained agent is at least three orders of magnitude faster than this manual sizing process reported in [36]. Furthermore, because there is no reward shaping, no prior domain knowledge is biasing or influencing the learning behavior of the agents; they re-discover circuit sizing in the $\mathfrak{E}$ domain by themselves.

7.3. Summary

Overall, using an electrical design and action space proved very effective for human- and particularly artificial circuit designers in terms of both learning behavior and the portability of trained agents between different technology nodes. The procedural approach presented in Section 4 provides an interface for RL to the $g_{\mathrm{m}}/I_{\mathrm{d}}$ method, such that agents can learn the art of analog circuit sizing without being tied to a specific technology. Agents trained in this alternative action space gain an understanding of the electrical behavior of the building blocks [13] and do not require any re-training or explicit guidance by human domain experts when used with different technologies. Additionally, the results show that the learned policies were optimized in terms of efficient design space navigation.