In the script you provided, the trainable parameters are typically found in the layers

and modules that involve learnable weights and biases, which are used in neural
network operations such as linear transformations and convolutions. These
parameters are automatically learned through the training process using
backpropagation and are updated during the optimization step.

Here's a breakdown of the key components in the script that have trainable
parameters:

1. Linear Layers (nn.Linear): Each linear layer in the model has a weight
matrix and a bias vector, both of which are trainable. These are used in
the Generator, MultiHeadedAttention, and PositionwiseFeedForward modules
among others.
2. Embeddings (nn.Embedding): The embedding layers, used in
the Embeddings module, have trainable lookup tables that map input tokens
to continuous vectors. These vectors are updated during training to better
capture semantic relationships between tokens.
3. Layer Normalization (LayerNorm): The layer normalization component
includes trainable parameters for scaling (a_2) and shifting (b_2) the
normalized data. This helps in stabilizing the learning process.
4. Dropout Layers (nn.Dropout): While dropout layers themselves do not have
trainable parameters, they are crucial in regulating the training process by
randomly zeroing some of the elements of the input tensor during training,
which helps prevent overfitting.
Each of these components contributes to the overall capacity of the model to learn
from data. During training, an optimizer like SGD or Adam adjusts these parameters
to minimize a loss function, which measures the discrepancy between the model's
predictions and the actual data.

When initializing these parameters (e.g., in the make_model function), they are typically
set using specific schemes like Xavier/Glorot initialization, as seen in the script. This
approach is chosen to help in maintaining a stable variance across layers, which is
crucial for effective training of deep networks.
Overall, the script is structured to provide a complex model involving several layers
and components, each contributing to the model's ability to learn effectively from
large amounts of data, particularly in tasks involving sequence-to-sequence models
like machine translation or text generation.