Attention (machine learning)

In <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a>, attention is a method that determines the importance of each component in a sequence relative to the other components in that sequence. In <a href="/facts/Natural_language_processing/1hjMKsSN">natural language processing</a>, importance is represented by <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC9007265/">"soft"</a> weights assigned to each word in a sentence. More generally, attention encodes vectors called <a href="/facts/Lexical_token/T1JYWpIf">token</a> <a href="/facts/Word_embedding/7uRcBPqo">embeddings</a> across a fixed-width <a href="/facts/Context_window/WnogWVJY">sequence</a> that can range from tens to millions of tokens in size.
Unlike "hard" weights, which are computed during the backwards training pass, "soft" weights exist only in the forward pass and therefore change with every step of the input. Earlier designs implemented the attention mechanism in a serial <a href="/facts/Recurrent_neural_network/bx7hBVB1">recurrent neural network</a> (RNN) language translation system, but a more recent design, namely the <a href="/facts/Transformer_(machine_learning_model)/cDbjx6a8">transformer</a>, removed the slower sequential RNN and relied more heavily on the faster parallel attention scheme.
Inspired by ideas about <a href="/facts/Attention/pleJPaeQ">attention in humans</a>, the attention mechanism was developed to address the weaknesses of leveraging information from the <a href="/facts/Hidden_layer/uKzUskIX">hidden layers</a> of recurrent neural networks. Recurrent neural networks favor more recent information contained in words at the end of a sentence, while information earlier in the sentence tends to be <a href="/facts/Vanishing_gradient_problem/3hteovPr">attenuated</a>. Attention allows a token equal access to any part of a sentence directly, rather than only through the previous state.

Attention (machine learning) open-in-new

Attention (machine learning)