abesh@chakraborty ~ $ cat Why did LLM architectures shift to RMSNorm?
Why did LLM architectures shift to RMSNorm?
While I was reading the RMSNorm paper, I realized that the question they asked was genuinely interesting.
We are all familiar with LayerNorm, and we know that it has stood the test of time. Therefore, there is not much to write about its ability to successfully regularize the model. It normalizes the input by subtracting the mean and dividing by the standard deviation. It is a simple algorithm.
However, the RMSNorm paper asks a more interesting question. The authors begin by noting that the LayerNorm provides two theoretical benefits: re-centering invariance and re-scaling invariance. Moreover, they state that previous research has focused primarily on re-centering invariance. The authors hypothesize,
“In this paper, we hypothesize that the re-scaling invariance is the reason for success of LayerNorm, rather than re-centering invariance.”
The interesting thing about this hypothesis is that the authors decided to remove re-centering from the LayerNorm and keep only the re-scaling. This led them to discover RMSNorm.
In the standard Layernorm, we must first calculate the mean of the input data:
and subtract it from each element of the input. In contrast, the RMSNorm calculates the RMS of the input:
and normalizes the input by this value.
I think it is worth clarifying why this method is called RMS or root mean square.
The "mean" in root mean square is not the mean that LayerNorm subtracts. It is the average of the squared inputs:
and the root is the square root we then take of it. LayerNorm's mean, Eq. (1), never appears. Nor is the RMS a standard deviation: the standard deviation measures spread around the mean, while the RMS measures magnitude around zero. The two only coincide when the mean happens to be zero.
Therefore, the term mean in the RMSNorm can be confusing. Despite the name, RMSNorm computes no mean statistic at all. It only rescales the input by its magnitude.
It is important to note that the paper states this clearly,
“Intuitively, RMSNorm simplifies LayerNorm by totally removing the mean statistic in Eq. (3) [Eq. (1) here] at the cost of sacrificing the invariance that mean normalization affords.”
The paper does a good job of distinguishing between the two different concepts of mean.
The authors’ main hypothesis was not that there was an alternative way to calculate the mean, but rather that mean normalization was unnecessary.
The Interesting Thing About This Paper
The interesting thing about this paper is that it illustrates well how the authors’ questioned existing assumptions. They were not looking to improve upon LayerNorm. They asked themselves: what if mean normalization was not necessary? And they set out to test their theory. They developed a simple algorithm, RMSNorm, and found that it performed comparably to LayerNorm while also improving running time. The authors’ experiments confirmed their hypothesis.
For me, this is the kind of research that makes machine learning exciting. We are constantly trying to find ways to improve our models. However, sometimes the most interesting findings come from questioning long-held beliefs.