Positional encoding

From Quire, the open reference·Deep learning technique·10 min read

In deep learning, the positional encoding is a component of sequence models, especially Transformers. Input data such as text, images, or audio is converted to a sequence of numerical representations called tokens, and the position of each token is encoded as a vector. This combination of token embedding vectors and positional encoding vectors is a context window.^[1]

Positional encoding is used because the standard attention mechanism used in Transformers by default treats all tokens symmetrically over all positions (i.e. permutation-symmetric), thus effectively treating the input as a bag of tokens, rather than a sequence of tokens.

Typology#edit

Absolute vs relative positional encoding:

In an absolute positional encoding scheme, the position of a token within a context is encoded. That is, it matters whether a token is at position 1 or position 100. This was proposed in 2017, just before Transformer was published.^[2]
In a relative positional encoding scheme, the relative position of two tokens within a context is encoded. That is, it matters not whether a token is at position 1 or position 100, and it only matters whether one token is in front of another token by 1 or 100. This property is also called translational invariance. This was proposed in 2018, just after Transformer was published.^[3]

Learned vs designed positional encoding:

In a learned positional encoding scheme, the parameters of positional encoding are learned like any other parameter in a model, usually by backpropagation.
In a designed positional encoding scheme, the parameters are not learned, and picked by the researchers. That is, they are hyperparameters.

For example, BERT used two levels of positional embeddings, one for the per-token position, and one for per-segment position. Both are absolute and learned.^[4]^[5] It was reported that learned absolute positional encoding generally perform better than relative positional encoding when there are special tokens that always occur in a specific location, such as the [CLS] token of BERT-like architectures. However, with learned absolute positional encoding, a model cannot process contexts longer than what it is trained for.^[6]

Other possibly desirable properties of a positional encoding scheme include:^[6]

Symmetry: In a symmetric positional encoding scheme, the positional encoding of a token ${\displaystyle n}$ tokens before is the same as that of a token ${\displaystyle n}$ tokens after. This property only makes sense for non-causal Transformers, since in a causal Transformer, a token cannot pay any attention to its subsequent tokens.
Monotonicity: In a monotonic positional encoding scheme, the positional encoding increases attention weighing on nearby tokens compared to far-away tokens.

Sinusoidal#edit

The original transformer paper (Attention Is All You Need) uses a sinusoidal absolute positional encoding. Each token is converted to an token embedding vector ${\displaystyle x_{t$ , plus a positional encoding vector ${\displaystyle p(t)}$ .^[7]

{{mathimg|f279a30bc8eabc788f3fe81c9cfb674e72e858db|width=2.156ex|height=2.009ex|va=-0.671ex|alt={\displaystyle x_{t}} depends only on the identity of the token, not on ${\displaystyle t}$ . That is, the token "the" would have the same ${\displaystyle x_{t$ , no matter if it appears in the first or the last position. ${\displaystyle p(t)}$ depends only on the position of the token, not on the identity of the token. That is, the ${\displaystyle t}$ -th token always has the same ${\displaystyle p(t)}$ , no matter if it is "the" or "a".

Both the encoder and the decoder takes ${\displaystyle x_{t}+p(t)}$ as input. This is shown in the picture.

Both ${\displaystyle v_{t$ and ${\displaystyle p(t)}$ has the same number of dimensions, since they must be added. Both of them have even dimensions, since that is what the positional encoding vector requires.

Let the dimension be ${\displaystyle 2d}$ , then the position encoding vector is defined as ${\displaystyle p(t)=(\cos(\omega _{1}t),\sin(\omega _{1}t),\cos(\omega _{2}t),\sin(\omega _{2}t),\dots ,\cos(\omega _{d}t),\sin(\omega _{d}t)}$

where ${\displaystyle \omega _{k}=b^{-{\frac {k-1}{d$ .

Here, ${\displaystyle b}$ is a free parameter that should be significantly larger than the biggest ${\displaystyle k}$ that would be input into the positional encoding function. The original paper uses ${\displaystyle d=50,b=10000}$ .

More succinctly, ${\displaystyle p(t)=(e^{i\omega _{k}t})_{k\in 1:d},\quad ,\omega _{k}=b^{-{\frac {k-1}{d$ In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference.

Radix interpretation#edit

The positional encoding has been interpreted as implementing a base- ${\displaystyle b^{1/k$ positional number system for representing real numbers:^[8]

Position	Frequency	Factor	Tokens per cycle ("wavelength")	Tokens per cycle at ${\displaystyle d=50,b=10000}$
Lowest	1	${\displaystyle e^{i\omega _{1}t}=e^{it$	${\displaystyle 2\pi }$	6.3
Next lowest	${\displaystyle b^{-1/d$	${\displaystyle e^{i\omega _{2}t}=e^{ib^{-1/d}t$	${\displaystyle 2\pi b^{1/d$	7.5
Highest	${\displaystyle b^{-(d-1)/d$	${\displaystyle e^{i\omega _{d}t}=e^{ib^{-(d-1)/d}t$	${\displaystyle 2\pi b^{(d-1)/d$	52261

The effect of base- ${\displaystyle b}$ positional number system means the attention mechanism is to apportion its attention by coordinate. For low frequency (long wavelength) coordinates, RoPE implicitly states that these coordinates encode information that varies slowly over a sequence, and vice versa for high frequency (low wavelength) coordinates. Concretely, the first 2 coordinates are treated as encoding features that vary quickly every token, while the last 2 coordinates are treated as encoding features that vary slowyl over 10000s of tokens.

Symmetry#edit

This positional encoding method was designed based on previous work on convolutional neural network language model.^[2] In the author's words, "we hypothesized it would allow the model to easily learn to attend by relative position."

A relative position shift is just a linear transformation away: ${\displaystyle p(t+\Delta t)=\mathrm {diag} (p(\Delta t))p(t)}$ where ${\displaystyle \Delta t\in \mathbb {R} }$ is the distance one wishes to shift. More generally, any convolution of relative positions is also just a linear transformation away: ${\displaystyle \sum _{j}c_{j}p(t+\Delta t_{j})=\left(\sum _{j}c_{j}\,\mathrm {diag} (p(\Delta t_{j}))\right)p(t),\quad \forall c_{j}\in \mathbb {R} }$ for any constants ${\displaystyle c_{j$ . This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors.

ALiBi#edit

In the original Transformer, positions are encoded by a vector ${\displaystyle p(t)}$ that is added to the token embedding vector ${\displaystyle x_{t$ , and the Transformer only sees their sum ${\displaystyle x_{t}+p(t)}$ . ALiBi (Attention with Linear Biases) is a relative positional encoding scheme that partially decouples the position encoding from the token embedding, by injecting positional encoding directly into each attention mechanism block.^[9]

The attention mechanism block in the original Transformer is ${\displaystyle {\begin{aligned}{\text{Attention(Q,K,V)={\text{softmax\left({\frac {QK^{\mathrm {T} {\sqrt {d_{K+M\right)V\end{aligned$ where ${\displaystyle Q,K,V}$ stand for the matrices for the query, key, and value vectors, ${\displaystyle d_{K$ stands for the dimension of the key, and ${\displaystyle M}$ stands for the mask matrix. Explicitly, ${\displaystyle Q={\begin{bmatrix}q_{1}\\q_{2}\\\vdots \\q_{\ell _{\text{query\end{bmatrix,K={\begin{bmatrix}k_{1}\\k_{2}\\\vdots \\k_{\ell _{\text{key-value\end{bmatrix,V=&#$

The ALiBi attention mechanism modifies it to ${\displaystyle {\begin{aligned}{\text{Attention(Q,K,V)={\text{softmax\left({\frac {QK^{\mathrm {T} {\sqrt {d_{K+M+sB\right)V\end{aligned$ Here, ${\displaystyle s\in \mathbb {R} }$ is a learnable scalar, and ${\displaystyle B}$ is the linear bias matrix defined by ${\displaystyle B={\begin{bmatrix}0&-1&-2&-3&\cdots \\-1&0&-1&-2&\cdots \\-2&-1&0&-1&\cdots \\-3&-2&-1&0&\cdots \\\vdots &\vdots &\vdots &\vdots &\ddots \\\end{bmatrix$ in other words, ${\displaystyle B_{i,j}=|j-i|}$ . The idea being that the linear bias matrix is a softened mask. Just as ${\displaystyle 0}$ represent full attention paid, and ${\displaystyle -\infty }$ represents no attention paid, the linear bias matrix linearly decreases the attention logit. Note that in the causally masked Transformer, any attention paid to the future is blanked out by the causal mask matrix ${\displaystyle M_{\text{causal={\begin{bmatrix}0&-\infty &-\infty &\dots &-\infty \\0&0&-\infty &\dots &-\infty \\0&0&0&\dots &-\infty \\\vdots &\vdots &\vdots &\ddots &\vdots \\0&0&0&\dots &0\end{bmatrix$ so the upper triangular part of ${\displaystyle B}$ does not matter.

ALiBi allows pretraining on short context windows, then fine-tuning on longer context windows. Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the "bottom" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).

More generally, is a more general version:^[10] ${\displaystyle {\begin{aligned}{\text{Attention(Q,K,V)={\text{softmax\left({\frac {QK^{\mathrm {T} {\sqrt {d_{K+M+B\right)V\end{aligned$ where ${\displaystyle B}$ is a Toeplitz matrix. ${\displaystyle B={\begin{bmatrix}b_{0}&b_{-1}&b_{-2}&\cdots &\cdots &b_{-(n-1)}\\b_{1}&b_{0}&b_{-1}&\ddots &&\vdots \\b_{2}&b_{1}&\ddots &\ddots &\ddots &\vdots \\\vdots &\ddots &\ddots$ For example, Google T5 used a learned Relative Position Representation, where the diagonals in the positional bias matrix ${\displaystyle B}$ are grouped into bands to decrease the number of learnable parameters of ${\displaystyle B}$ .^[11]

Relatedly, Relative Position Representations was an early attempt by some of the original authors of the Transformer to make a decoupled relative positional encoding scheme.^[3] It has not been used much in practice since 2018. In this scheme, learned positional encoding vectors are added to the key and value vectors in an attention mechanism block.

RoPE#edit

RoPE (Rotary Positional Encoding) is another scheme for relative positional encoding. It works by rotating the key and value vectors input into an attention mechanism block.^[12] It was proposed in 2021 in a Chinese blog, and then verified by Eleuther AI, before its subsequent spread of popularity.^[13]^[14]

Consider a feature vector with even dimensions: ${\displaystyle x=(x_{1},x_{2},\dots ,x_{2d-1},x_{2d})\in \mathbb {R} ^{2d$ . Like in the original sinuisoidal positional encoding scheme, it can be written as a complex vector ${\displaystyle v=(x_{1}+ix_{2},\dots ,x_{2d-1}+ix_{2d})\in \mathbb {C} ^{d$ . Then, to encode it as positioned at ${\displaystyle t}$ , the ${\displaystyle t}$ -th feature vector is represented by rotating its entries: ${\displaystyle \operatorname {RoPE} (((x_{1}+ix_{2}),\dots ,(x_{2d-1}+ix_{2d})),t):=(e^{i\omega _{1}t}(x_{1}+ix_{2}),\dots ,e^{i\omega _{d}t}(x_{2d-1}+ix_{2d}))&$ where the frequencies ${\displaystyle \omega _{1},\dots ,\omega _{d}\in \mathbb {R} }$ are usually designed by hand. A standard design is ${\displaystyle \omega _{k}=b^{-{\frac {k-1}{d$ where ${\displaystyle b=10000}$ , the same base as the original Transformer, and common since the original RoPE paper and Llama.^[12]^[15]

In an attention mechanism, RoPE is implemented as ${\displaystyle {\begin{aligned}{\text{RoPEAttention(Q,K,V)={\text{softmax\left({\frac {\operatorname {RoPE} (Q)\operatorname {RoPE} (K)^{\mathrm {T} {\sqrt {d_{K+M\right)V\end{aligned$ where ${\displaystyle \operatorname {RoPE} (Q)={\begin{bmatrix}\operatorname {RoPE} (q_{1},1)\\\operatorname {RoPE} (q_{2},2)\\\vdots \\\operatorname {RoPE} (q_{\ell _{\text{query,\ell _{\text{query$ and similarly for the RoPE key matrix ${\displaystyle \operatorname {RoPE} (K)}$ .

Like the original positional encoding, it is translation-invariant and behaves as a base- ${\displaystyle b}$ positional number system. Specifically, translation symmetry means that ${\displaystyle \operatorname {RoPE} (x,t)\cdot \operatorname {RoPE} (x',t')=\operatorname {RoPE} (x,t+\Delta t)\cdot \operatorname {RoPE} (x',t'+\Delta t)}$ for any ${\displaystyle x,x'\in \mathbb {C} ^{n$ and any ${\displaystyle t,t',\Delta t\in \mathbb {R} }$ .

ReRoPE#edit

ReRoPE and Leaky ReRoPE:^[16]

Variants#edit

Extrapolatable Position Embedding (xPos) adds decay terms ${\displaystyle \zeta _{1},\dots ,\zeta _{d}\in [0,1]}$ to the rotation of RoPE, designed to damp out the high-frequency oscillations at long distances.^[17] ${\displaystyle \operatorname {xPos} (((x_{1}+ix_{2}),\dots ,(x_{2d-1}+ix_{2d})),t,\omega ,\zeta ):=(\zeta _{1}^{t}e^{i\omega _{1}t}(x_{1}+ix_{2}),\dots ,\zeta _{d}^{t$ though the decay terms are inverted for the key vectors: ${\displaystyle {\begin{aligned}{\text{xPosAttention(Q,K,V)={\text{softmax\left({\frac {\operatorname {xPos} (Q,\omega ,\zeta )\operatorname {xPos} (K,\omega ,\zeta ^{-1})^{\mathrm {T} {\sqrt &#123$ It is also limited to causal attention masking, since otherwise the attention scores will diverge.

RoPE-2D applies RoPE to tokens that are arranged on a plane. They are used in vision transformers. Similarly, Transformers that process videos may use RoPE-3D. In equations,

RoPE with Relative distances (RoPER), or value-output RoPE (VO-RoPE), applies RoPE to value vectors.^[18]^[19] The idea is to rotate the value vectors using RoPE. ${\displaystyle {\begin{aligned}{\text{RoPERAttention(Q,K,V)=\operatorname {RoPE} ^{-1}\left({\text{softmax\left({\frac {QK^{\mathrm {T} {\sqrt {d_{K+M\right)\operatorname {RoPE} (V)\righ$ The idea is as follows: the output matrix ${\displaystyle O}$ is an attention-weighted sum of the value vectors: ${\displaystyle O={\text{softmax\left({\frac {QK^{\mathrm {T} {\sqrt {d_{K+M\right)V}$ so relative positional encoding can be implemented by rotating each value vector absolutely, then undoing it in the output vectors, thus resulting in another relative positional encoding. ${\displaystyle \operatorname {RoPE} ^{-1}(O)={\begin{bmatrix}\operatorname {RoPE} (o_{1},-1)\\\operatorname {RoPE} (o_{2},-2)\\\vdots \\\operatorname {RoPE} (o_{\ell _{\text{output,-\ell _&#1$ note that ${\displaystyle \ell _{\text{query=\ell _{\text{output$ .

NoPE#edit

In NoPE (No Positional Encoding), there is no positional encoding. The idea is that a decoder's causal masking lets each token ${\displaystyle t}$ to see only tokens ${\displaystyle 1:t}$ , which is an implicit kind of positional encoding.^[20]^[21]^[22] Relatedly, pure convolutional networks are capable of implicit absolute positional encoding, by exploiting the zero-padding on the edge.^[23]

NoPE can represent both absolute and relative positional encodings. Empirically, when trained with gradient descent, a NoPE transformer would place a transformer using T5-style positional encoding (ALiBi, learned) produces similar distributions of attention weights for the same input sequences, measured using Jensen–Shannon divergence.^[24]

NoPE is rarely used for all attention modules of a Transformer, since it would degrade performance significantly. However, NoPE may be used for a fraction of attention modules. This was found in practice to be a worthwhile trade off of performance for computing cost. For example, Llama 4 alternates layers of NoPE and RoPE, with RoPE in 3 layers and NoPE in the 4th. They call it iRoPE ("interleaved RoPE").^[25]

Context extension#edit

Since the per-token cost of inference scales with context length, training cost is decreased by using shorter contexts during training. However, this would make the resulting model bad at using longer contexts. Consequently, a common way to train a model is to "train short, test long", i.e. training a model on short contexts, and use it on long contexts.^[26]

Context extension was a concern since the original Transformer paper. In it, the authors reported trying both learned positional encoding and desigend positional encoding. They found both to result in similar performance, and decided to use the designed positional encoding, since it "may allow the model to extrapolate to sequence lengths longer than the ones encountered during training".^[7]

A slightly more sophisticated method is train according to a curriculum: Begin by training on a short context length, then train on a longer context length. This can be repeated. For example, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K.

Context lengths can even be mixed during training. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than purely training on long contexts.^[27]

In general, context extension using RoPE consists of training a model on one context length ${\displaystyle L}$ with one set of RoPE angles ${\displaystyle \omega _{1},\dots ,\omega _{d$ , then its RoPE angles are modified to ${\displaystyle \omega _{1}',\dots ,\omega _{d}'}$ , and it is trained further on context length ${\displaystyle L'=sL}$ for some ${\displaystyle s>1}$ .

Some architectures natively support context extension. For example, convolutional seq2seq,^[2] and sliding window attention,^[28] are both insensitive to context length.

YaRN#edit

YaRN (Yet another RoPE extensioN) is a common way to extend context, used across many models, including DeepSeek, Qwen, Kimi, and GPT-OSS.^[29] It was built in steps.

In Positional Interpolation (PI),^[30] frequencies are stretched equally: ${\displaystyle \omega _{k}'=\omega _{k}/s}$ . This was found to degrade performance beyond ${\displaystyle s=8}$ , as the model loses the high-frequency resolution that tells adjacent tokens apart.

In Adjusted Base Frequency (ABF)^[31] (other names include "NTK-aware", "NTK-RoPE", "RoPE-ABF"), the base is increased: ${\displaystyle \omega _{k}'=(b')^{-{\frac {k-1}{d$ , where ${\displaystyle b'=bs^{\frac {d}{d-1$ is designed so that ${\displaystyle \omega _{1}'=\omega _{1},\quad \omega _{d}'=\omega _{d}/s}$ Equivalently, ${\displaystyle \omega _{k}'=\omega _{k}/s^{\frac {k-1}{d-1$ . The effect is that the highest frequency is not stretched, the lowest frequency is stretched by ${\displaystyle s}$ , and those in-between are interpolated.

In "NTK-by-parts", high frequencies are not stretched, low frequencies are stretched by ${\displaystyle s}$ , and those in-between are interpolated: ${\displaystyle \omega _{k}'=\gamma _{k}\omega _{k}+(1-\gamma _{k})\omega _{k}/s}$ where

${\displaystyle \gamma _{k}=1}$ if the feature rotates by more than ${\displaystyle \beta }$ times over the length of a context: ${\displaystyle \omega _{k}L>2\pi \beta }$ . Equivalently, the wavelength is short enough: ${\displaystyle \lambda _{k}<L/\beta }$ .
${\displaystyle \gamma _{k}=0}$ if the feature rotates by less than ${\displaystyle \alpha }$ times over the length of a context: ${\displaystyle \omega _{k}L<2\pi \alpha }$ . Equivalently, the wavelength is long enough: ${\displaystyle \lambda _{k}>L/\alpha }$ .
${\displaystyle \gamma _{k$ is linearly interpolated otherwise: ${\displaystyle \gamma _{k}={\frac {L/\lambda _{k}-\alpha }{\beta -\alpha$ .

YaRN uses NTK-by-parts with frequencies scales with ${\displaystyle \alpha =1,\beta =32}$ , meaning that wavelengths shorter than ${\displaystyle L/32}$ are not stretched, wavelengths longer than ${\displaystyle L}$ are fully stretched by ${\displaystyle s}$ , and those in-between are stretched by an in-between value.

YaRN alse scales up the logits in the attention mechanism. The idea is that stretching frequencies blurs the corresponding features, and scaling up the logits unblurs them. The authors experimented with context-extending models from the Llama and Llama 2 series, and found empirically that^[29] ${\displaystyle {\begin{aligned}{\text{Attention(Q,K,V)={\text{softmax\left(\underbrace {(1+0.1\ln s)^{2 _{\text{unblurring{\frac {QK^{\mathrm {T} {\sqrt {d+M\right)V\end{aligned$ worked well.

Dynamic scaling#edit

Because extending the context length involves a degradation of performance, during deployment, a model is used using its original context length ${\displaystyle L}$ , and when the context exceeds ${\displaystyle L}$ , switch to the extended context length ${\displaystyle L'}$ . This was found to cause a sudden drop of performance at the switching point. Consequently, dynamic scaling ("dynamic NTK") was used to allow graceful degradation.^[29]

The algorithm:

When context length ${\displaystyle \ell }$ is lower than the short context length ${\displaystyle L}$ , use the positional encoding of ${\displaystyle L}$ .
When context length ${\displaystyle \ell }$ is longer than the short context length ${\displaystyle L}$ , use the positional encoding obtained by extending the context by a factor of ${\displaystyle \ell /L}$ .

Dynamic scaling can be used even with no extra training on extended context ${\displaystyle L'}$ .^[29] It was first proposed in a post in the subreddit r/LocalLLaMA.^[32]

References#edit

^ Dufter, Philipp; Schmitt, Martin; Schütze, Hinrich (2022-09-01). "Position Information in Transformers: An Overview". Computational Linguistics. 48 (3): 733–763. doi:10.1162/coli_a_00445. ISSN 0891-2017.
^ a b c Gehring, Jonas; Auli, Michael; Grangier, David; Yarats, Denis; Dauphin, Yann N. (2017-07-17). "Convolutional Sequence to Sequence Learning". Proceedings of the 34th International Conference on Machine Learning. PMLR: 1243–1252. ISSN 2640-3498.
^ a b Shaw, Peter; Uszkoreit, Jakob; Vaswani, Ashish (2018). "Self-Attention with Relative Position Representations". arXiv:1803.02155 [cs.CL].
^ Google Research. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". GitHub. modeling.py, line 496. Commit eedf5716ce1268e56f0a50264a88cafad334ac61. {{cite web}}: |author= has generic name (help)
^ Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina (11 October 2018). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". arXiv:1810.04805v2 [cs.CL].
^ a b Wang, Benyou; Shang, Lifeng; Lioma, Christina; Jiang, Xin; Yang, Hao; Liu, Qun; Simonsen, Jakob Grue (2021). "On Position Embeddings in BERT". International Conference on Learning Representations.
^ a b Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (2017). "Attention is All You Need" (PDF). 31st Conference on Neural Information Processing Systems (NIPS 2017), 4-9 December 2017, Long Beach, CA, USA. Vol. 30. Curran Associates, Inc. arXiv:1706.03762. ISBN 978-1-5108-6096-4. Archived from the original (PDF) on 2024-02-21. Retrieved 2023-10-31.
^ 苏, 剑林 (2023-07-06). "Transformer升级之路：10、RoPE是一种β进制编码" [The Road to Transformer Upgrades: 10. RoPE Is a Base-β Encoding]. 科学空间 (in Chinese).
^ Press, Ofir; Smith, Noah A.; Lewis, Mike (2021-08-01). "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation". arXiv:2108.12409 [cs.CL].
^ Ke, Guolin; He, Di; Liu, Tie-Yan (2021-03-15). "Rethinking Positional Encoding in Language Pre-training". arXiv:2006.15595 [cs.CL].
^ Raffel, Colin; Shazeer, Noam; Roberts, Adam; Lee, Katherine; Narang, Sharan; Matena, Michael; Zhou, Yanqi; Li, Wei; Liu, Peter J. (2020). "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer". Journal of Machine Learning Research. 21 (140): 1–67. arXiv:1910.10683. ISSN 1533-7928.
^ a b Su, Jianlin; Ahmed, Murtadha; Lu, Yu; Pan, Shengfeng; Bo, Wen; Liu, Yunfeng (2024-02-01). "RoFormer: Enhanced transformer with Rotary Position Embedding". Neurocomputing. 568: 127063. doi:10.1016/j.neucom.2023.127063. ISSN 0925-2312. Archived from the original on 2026-01-15.
^ 苏, 剑林 (2021-02-03). "让研究人员绞尽脑汁的Transformer位置编码" [Transformer Position Encoding That Has Researchers Racking Their Brains]. 科学空间 (in Chinese).
^ Biderman, Stella; Black, Sid; Foster, Charles; Gao, Leo; Hallahan, Eric; He, Horace; Wang, Ben; Wang, Phil (2021). "Rotary Embeddings: A Relative Revolution". EleutherAI.
^ Touvron, Hugo; Lavril, Thibaut; Izacard, Gautier; Martinet, Xavier; Lachaux, Marie-Anne; Lacroix, Timothée; Rozière, Baptiste; Goyal, Naman; Hambro, Eric; Azhar, Faisal; Rodriguez, Aurelien; Joulin, Armand; Grave, Edouard; Lample, Guillaume (2023). "LLaMA: Open and Efficient Foundation Language Models". arXiv:2302.13971 [cs.CL].
^ Jin, Hongye; Han, Xiaotian; Yang, Jingfeng; Jiang, Zhimeng; Liu, Zirui; Chang, Chia-Yuan; Chen, Huiyuan; Hu, Xia (2024-07-11). "LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning". arXiv:2401.01325 [cs.CL].
^ Sun, Yutao; Dong, Li; Patra, Barun; Ma, Shuming; Huang, Shaohan; Benhaim, Alon; Chaudhary, Vishrav; Song, Xia; Wei, Furu (2022-12-20). "A Length-Extrapolatable Transformer". arXiv:2212.10554 [cs.CL].
^ 苏, 剑林 (2025-04-18). "Transformer升级之路：19、第二类旋转位置编码" [The Road to Transformer Upgrades: 19. The Second Type of Rotary Position Encoding]. 科学空间 (in Chinese).
^ "Rotary Positional Embeddings with Relative distances (RoPER)". labml.ai research blog. 2022. Retrieved 2026-10-26.
^ Haviv, Adi; Ram, Ori; Press, Ofir; Izsak, Peter; Levy, Omer (December 2022). Goldberg, Yoav; Kozareva, Zornitsa; Zhang, Yue (eds.). "Transformer Language Models without Positional Encodings Still Learn Positional Information". Findings of the Association for Computational Linguistics: EMNLP 2022. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics: 1382–1390. doi:10.18653/v1/2022.findings-emnlp.99.
^ Wang, Jie; Ji, Tao; Wu, Yuanbin; Yan, Hang; Gui, Tao; Zhang, Qi; Huang, Xuanjing; Wang, Xiaoling (August 2024). Ku, Lun-Wei; Martins, Andre; Srikumar, Vivek (eds.). "Length Generalization of Causal Transformers without Position Encoding". Findings of the Association for Computational Linguistics: ACL 2024. Bangkok, Thailand: Association for Computational Linguistics: 14024–14040. doi:10.18653/v1/2024.findings-acl.834.
^ Zuo, Chunsheng; Guerzhoy, Pavel; Guerzhoy, Michael (2025-01). Rambow, Owen; Wanner, Leo; Apidianaki, Marianna; Al-Khalifa, Hend; Eugenio, Barbara Di; Schockaert, Steven (eds.). "Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings". Proceedings of the 31st International Conference on Computational Linguistics. Abu Dhabi, UAE: Association for Computational Linguistics: 9418–9430. {{cite journal}}: Check date values in: |date= (help)
^ "dblp: How Much Position Information Do Convolutional Neural Networks Encode?". dblp.org. Retrieved 2026-10-27.
^ Kazemnejad, Amirhossein; Padhi, Inkit; Natesan Ramamurthy, Karthikeyan; Das, Payel; Reddy, Siva (2023). "The Impact of Positional Encoding on Length Generalization in Transformers". Advances in Neural Information Processing Systems. 36. Curran Associates, Inc.: 24892–24928. doi:10.52202/075280-1082.
^ "The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation". ai.meta.com. Archived from the original on 2026-10-21. Retrieved 2026-10-26.
^ Press, Ofir; Smith, Noah A.; Lewis, Mike (2022-04-22). "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation". arXiv:2108.12409 [cs.CL].
^ Gao, Tianyu; Wettig, Alexander; Yen, Howard; Chen, Danqi (July 2025). Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina; Pilehvar, Mohammad Taher (eds.). "How to Train Long-Context Language Models (Effectively)". Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics: 7376–7399. doi:10.18653/v1/2025.acl-long.366. ISBN 979-8-89176-251-0.
^ Child, Rewon; Gray, Scott; Radford, Alec; Sutskever, Ilya (2019-04-23). "Generating Long Sequences with Sparse Transformers". arXiv:1904.10509 [cs.LG].
^ a b c d Peng, Bowen; Quesnelle, Jeffrey; Fan, Honglu; Shippole, Enrico (2024-05-31). "YaRN: Efficient Context Window Extension of Large Language Models". International Conference on Learning Representations. 2024: 31932–31951.
^ Chen, Shouyuan; Wong, Sherman; Chen, Liangjian; Tian, Yuandong (2023-06-28). "Extending Context Window of Large Language Models via Positional Interpolation". arXiv:2306.15595 [cs.CL].
^ Xiong, Wenhan; Liu, Jingyu; Molybog, Igor; Zhang, Hejia; Bhargava, Prajjwal; Hou, Rui; Martin, Louis; Rungta, Rashi; Sankararaman, Karthik Abinav; Oguz, Barlas; Khabsa, Madian; Fang, Han; Mehdad, Yashar; Narang, Sharan; Malik, Kshitiz (June 2024). Duh, Kevin; Gomez, Helena; Bethard, Steven (eds.). "Effective Long-Context Scaling of Foundation Models". Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Mexico City, Mexico: Association for Computational Linguistics: 4643–4663. doi:10.18653/v1/2024.naacl-long.260.
^ emozilla (2023-06-30). "Dynamically Scaled RoPE further increases performance of long context LLaMA with zero fine-tuning". Reddit. {{cite web}}: More than one of |work= and |website= specified (help)

Categories Artificial intelligence Deep learning

This article is mirrored from “Positional encoding” on English Wikipedia, revision 1382349472, written by Wikipedia's contributors and licensed CC BY-SA 4.0. Editing it here creates a Quire copy with its own history.

Synced with Wikipedia revision 1382349472 · just now