What is weight?

Beginners
1

Hi all,

I have read a lot of HF pages, but I’m still don’t understand, what the weights meaning. Is this a factor for parameters in a formula? And what is a “parameter”? Is this like parameter in a Python function?

Thanks for your answers.

2

I think you probably mean model weights… :

A simple way to picture it might be a huge recipe.

A model has an enormous number of numerical “amounts” inside it. During training, those amounts are adjusted little by little depending on how well the model is doing. Those adjustable numbers are its parameters, and many of them are weights.

So when you see something like “download/load the model weights” on Hugging Face, it basically means getting those learned numerical settings instead of starting with an untrained model.

A little more technically

For a very simple mathematical example:

y = wx + b
  • x is the input.
  • w is a weight.
  • b is a bias.
  • If w and b are learned during training, both are parameters of the model.

This is essentially the same pattern as a linear layer. In a real neural network, weights are normally stored as much larger tensors, rather than as single numbers.

For example, in PyTorch:

import torch.nn as nn

layer = nn.Linear(3, 2)

print(layer.weight)
print(layer.bias)

PyTorch represents trainable model parameters using torch.nn.Parameter, and you can inspect them with named_parameters():

for name, parameter in layer.named_parameters():
    print(name, parameter.shape)

which gives something like:

weight torch.Size([2, 3])
bias   torch.Size([2])

During training, automatic differentiation / backpropagation tells the optimizer how those parameters should be changed.

A model parameter is therefore different from a Python function parameter:

def greet(name, loud=False):
    ...

Here name and loud are Python function parameters: values supplied when calling a function. They are not values learned by training.

One terminology wrinkle: strictly speaking, not every model parameter is a weight — a bias can be a parameter too. But in everyday ML/Hugging Face usage, “model weights” is often used more loosely for the model’s learned numerical state as a whole.

Hugging Face makes a similar distinction between an architecture and a checkpoint: the architecture is roughly the structure of the model, while a checkpoint provides learned weights for that architecture. from_pretrained() loads those pretrained weights.

Those weights are commonly stored in files such as model.safetensors, which contain many named tensors.