HighlightCurated by Aramai EditorialarXiv

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Analyzes encoder-decoder neural machine translation models, showing performance drops with longer sentences and more unknown words.

Neural machine translation is a then-new approach to statistical machine translation built purely from neural networks, using an encoder that maps a variable-length sentence to a fixed-length representation and a decoder that generates the translation. The paper analyzes two models: an RNN Encoder-Decoder and a newly proposed gated recursive convolutional neural network. Quality is good for short sentences without unknown words but degrades rapidly as sentence length and unknown words grow; the gated model learns grammatical structure automatically.

Based on: On the Properties of Neural Machine Translation: Encoder–Decoder Approaches · SSST@EMNLP

HighlightCurated by Aramai EditorialarXiv

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

ALBERT introduces two parameter-reduction techniques and an inter-sentence coherence loss to scale BERT pretraining with less memory and faster training.

Scaling up model size in language representation pretraining tends to improve downstream performance but eventually runs into GPU/TPU memory limits and longer training times. ALBERT proposes two parameter-reduction techniques that cut memory use and speed up BERT training, allowing it to scale far better than the original. It also adds a self-supervised loss modeling inter-sentence coherence, which helps tasks with multi-sentence inputs. The best model sets new state-of-the-art results on GLUE, RACE, and SQuAD while using fewer parameters than BERT-large.

Based on: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations · International Conference on Learning Representations

HighlightCurated by Aramai EditorialarXiv

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Empirically compares gated recurrent units (LSTM, GRU) against traditional tanh units in RNNs on polyphonic music and speech signal modeling tasks.

This paper compares different types of recurrent units in recurrent neural networks, focusing on sophisticated units with gating mechanisms such as the long short-term memory (LSTM) unit and the recently proposed gated recurrent unit (GRU). The units are evaluated on polyphonic music modeling and speech signal modeling tasks. Experiments show that the advanced gated units outperform traditional recurrent units such as tanh units, and that the GRU is comparable to the LSTM.

Based on: Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling · arXiv.org

HighlightCurated by Aramai EditorialarXiv

LoRA: Low-Rank Adaptation of Large Language Models

Proposes LoRA, which freezes pre-trained weights and injects trainable low-rank decomposition matrices into Transformer layers for efficient adaptation.

Full fine-tuning of large pre-trained language models becomes impractical at scale — deploying independent fine-tuned instances of GPT-3 175B is prohibitively expensive. LoRA freezes pre-trained weights and injects trainable rank decomposition matrices into each Transformer layer, cutting trainable parameters for downstream tasks by 10,000x and GPU memory by 3x versus fine-tuning GPT-3 with Adam. LoRA matches or beats fine-tuning quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, with higher training throughput and, unlike adapters, no added inference latency.

Based on: LoRA: Low-Rank Adaptation of Large Language Models · International Conference on Learning Representations

HighlightCurated by Aramai EditorialarXiv

LLaMA: Open and Efficient Foundation Language Models

Introduces LLaMA, foundation language models (7B-65B) trained solely on publicly available data, with LLaMA-13B outperforming GPT-3 on most benchmarks.

LLaMA is a collection of foundation language models ranging from 7B to 65B parameters, trained on trillions of tokens. The work shows that state-of-the-art models can be trained using publicly available datasets exclusively, without proprietary or inaccessible data. LLaMA-13B outperforms the 175B GPT-3 on most benchmarks, LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B, and all models are released to the research community.

Based on: LLaMA: Open and Efficient Foundation Language Models · arXiv.org

HighlightCurated by Aramai EditorialarXiv

Sequence to Sequence Learning with Neural Networks

Presents an end-to-end sequence-to-sequence learning approach using multilayered LSTMs to encode inputs to a fixed vector and decode target sequences.

Deep neural networks perform well on difficult tasks given large labeled training sets but cannot map sequences to sequences. This paper presents a general end-to-end sequence learning approach that uses a multilayered LSTM to encode the input sequence into a fixed-dimensional vector and a second deep LSTM to decode the target sequence. On WMT-14 English-to-French translation it reaches 34.8 BLEU versus 33.3 for a phrase-based SMT system, and 36.5 when reranking that system's 1000 hypotheses; reversing source word order markedly improved performance.

Based on: Sequence to Sequence Learning with Neural Networks · Neural Information Processing Systems

HighlightCurated by Aramai EditorialarXiv

Training language models to follow instructions with human feedback

Introduces InstructGPT: GPT-3 fine-tuned on demonstrations and human feedback rankings via RL to align language models with user intent.

Larger language models are not inherently better at following user intent and can produce untruthful, toxic, or unhelpful outputs. The authors align models by fine-tuning GPT-3 on labeler-written demonstrations, then further fine-tuning with reinforcement learning from human feedback using rankings of model outputs, producing InstructGPT. In human evaluations, outputs from the 1.3B-parameter InstructGPT are preferred over the 175B GPT-3 despite 100x fewer parameters, with improved truthfulness, less toxic generation, and minimal regressions on public NLP datasets.

Based on: Training language models to follow instructions with human feedback · Neural Information Processing Systems

HighlightCurated by Aramai EditorialarXiv

Evaluating Large Language Models Trained on Code

Introduces Codex, a GPT model fine-tuned on GitHub code, and HumanEval, a benchmark for functional correctness of programs synthesized from docstrings.

The paper introduces Codex, a GPT language model fine-tuned on public GitHub code, and studies its Python code-writing abilities; a distinct production version powers GitHub Copilot. On HumanEval, a newly released benchmark measuring functional correctness of programs synthesized from docstrings, Codex solves 28.8% of problems versus 0% for GPT-3 and 11.4% for GPT-J, and repeated sampling solves 70.2% with 100 samples per problem. The authors also examine limitations, such as long chains of operations and variable binding, and discuss safety, security, and economic impacts.

Based on: Evaluating Large Language Models Trained on Code · arXiv.org

HighlightCurated by Aramai EditorialarXiv

Language Models are Few-Shot Learners

Trains the 175-billion-parameter GPT-3 and shows strong few-shot task performance from text prompts alone, without fine-tuning.

Prior NLP gains came from pretraining plus fine-tuning needing many labeled examples, unlike humans who learn from a few examples. This paper trains GPT-3, a 175-billion-parameter autoregressive language model 10x larger than prior non-sparse models, evaluated via pure in-context few-shot prompting with no gradient updates. GPT-3 performs strongly on translation, QA, cloze, and reasoning tasks, rivaling some fine-tuned methods, though it struggles on some datasets and can write news articles hard to distinguish from human-written ones.

Based on: Language Models are Few-Shot Learners · Neural Information Processing Systems

HighlightCurated by Aramai EditorialarXiv

Neural Machine Translation by Jointly Learning to Align and Translate

Introduces a soft-attention mechanism letting an encoder-decoder network jointly align and translate without fixed-length bottleneck.

Neural machine translation models typically use an encoder-decoder architecture where an encoder compresses a source sentence into a fixed-length vector from which a decoder generates the translation. The authors conjecture this fixed-length vector is a bottleneck, and propose extending the model to automatically (soft-)search source-sentence parts relevant to each target word, without explicit segmentation. This matches existing state-of-the-art phrase-based system performance on English-to-French, and the learned soft alignments agree well with intuition.

Based on: Neural Machine Translation by Jointly Learning to Align and Translate · International Conference on Learning Representations

HighlightCurated by Aramai EditorialarXiv

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A replication study showing BERT was undertrained, and that a tuned pretraining recipe matches or beats later models.

Language model pretraining yields strong gains but careful comparison across approaches is difficult, since training is expensive, done on private datasets of varying sizes, and sensitive to hyperparameter choices. This paper presents a replication study of BERT pretraining that measures the impact of key hyperparameters and training data size, finding BERT was significantly undertrained. A better-tuned BERT can match or exceed every model published after it, achieving state-of-the-art results on GLUE, RACE, and SQuAD; the authors release their models and code.

Based on: RoBERTa: A Robustly Optimized BERT Pretraining Approach · arXiv.org

HighlightCurated by Aramai EditorialAnnual Meeting of the Association for Computational Linguistics

Bleu: a Method for Automatic Evaluation of Machine Translation

Proposes BLEU, a quick, inexpensive, language-independent automatic method for evaluating machine translation quality.

Human evaluation of machine translation is thorough but slow, costly, and its labor cannot be reused. The authors propose an automatic evaluation method that is fast, inexpensive, and language-independent, correlating highly with human judgments while adding little marginal cost per run. It is presented as an automated substitute for skilled human judges, useful whenever quick or frequent evaluation is needed.

Based on: Bleu: a Method for Automatic Evaluation of Machine Translation · Annual Meeting of the Association for Computational Linguistics