research
∙
05/15/2023
Symbol tuning improves in-context learning in language models
We present symbol tuning - finetuning language models on in-context inpu...
share
research
∙
04/18/2023
UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining
Pretrained multilingual large language models have typically used heuris...
research
∙
03/17/2023
CoLT5: Faster Long-Range Transformers with Conditional Computation
Many natural language processing tasks benefit from long inputs, but pro...
research
∙
03/07/2023
Larger language models do in-context learning differently
We study how in-context learning (ICL) in language models is affected by...
research
∙
01/31/2023
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
We study the design decisions of publicly available instruction tuning m...
research
∙
12/19/2022
DSI++: Updating Transformer Memory with New Documents
Differentiable Search Indices (DSIs) encode a corpus of documents in the...
research
∙
12/16/2022
Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification
With the devastating outbreak of COVID-19, vaccines are one of the cruci...
research
∙
12/09/2022
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Training large, deep neural networks to convergence can be prohibitively...
research
∙
11/03/2022
Inverse scaling can become U-shaped
Although scaling language models improves performance on a range of task...
research
∙
10/20/2022
Transcending Scaling Laws with 0.1
Scaling language models improves performance but comes with significant ...
research
∙
10/06/2022
Language Models are Multilingual Chain-of-Thought Reasoners
We evaluate the reasoning abilities of large language models in multilin...
research
∙
10/04/2022
Recitation-Augmented Language Models
We propose a new paradigm to help Large Language Models (LLMs) generate ...
research
∙
07/21/2022
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
There have been a lot of interest in the scaling properties of Transform...
research
∙
04/25/2022
ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference
State-of-the-art neural models typically encode document-query pairs usi...
research
∙
04/05/2022
PaLM: Scaling Language Modeling with Pathways
Large language models have been shown to achieve remarkable performance ...
research
∙
02/22/2022
A New Generation of Perspective API: Efficient Multilingual Character-level Transformers
On the world wide web, toxic content detectors are a crucial line of def...
research
∙
11/25/2021
PolyViT: Co-training Vision Transformers on Images, Videos and Audio
Can we train a single transformer model capable of processing multiple m...
research
∙
10/25/2021
The Efficiency Misnomer
Model efficiency is a critical aspect of developing and deploying machin...
research
∙
10/18/2021
SCENIC: A JAX Library for Computer Vision Research and Beyond
Scenic is an open-source JAX library with a focus on Transformer-based m...
research
∙
10/16/2021
Sharpness-Aware Minimization Improves Language Model Generalization
The allure of superhuman-level capabilities has led to considerable inte...
research
∙
10/16/2021
Improving Compositional Generalization with Self-Training for Data-to-Text Generation
Data-to-text generation focuses on generating fluent natural language re...
research
∙
09/30/2021
Born Again Neural Rankers
We introduce Born Again neural Rankers (BAR) in the Learning to Rank (LT...
research
∙
09/22/2021
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
There remain many open questions pertaining to the scaling behaviour of ...
research
∙
07/14/2021
The Benchmark Lottery
The world of empirical machine learning (ML) strongly relies on benchmar...
research
∙
06/29/2021
SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
Self-supervised contrastive representation learning has proved incredibl...
research
∙
06/23/2021
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
State-of-the-art models in natural language processing rely on separate ...
research
∙
05/12/2021
How Reliable are Model Diagnostics?
In the pursuit of a deeper understanding of a model's behaviour, there i...
research
∙
05/07/2021
Are Pre-trained Convolutions Better than Pre-trained Transformers?
In the era of pre-trained language models, Transformers are the de facto...
research
∙
05/05/2021
Rethinking Search: Making Experts out of Dilettantes
When experiencing an information need, users want to engage with an expe...
research
∙
03/01/2021
OmniNet: Omnidirectional Representations from Transformers
This paper proposes Omnidirectional Representations from Transformers (O...
research
∙
02/23/2021
Do Transformer Modifications Transfer Across Implementations and Applications?
The research community has proposed copious modifications to the Transfo...
research
∙
02/17/2021
Switch Spaces: Learning Product Spaces with Sparse Gating
Learning embedding spaces of suitable geometry is critical for represent...
research
∙
02/17/2021
Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters
Recent works have demonstrated reasonable success of representation lear...
research
∙
02/09/2021
Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection
Detecting out-of-distribution (OOD) examples is critical in many applica...
research
∙
12/01/2020
StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
There are two major classes of natural language grammars – the dependenc...
research
∙
11/08/2020
Long Range Arena: A Benchmark for Efficient Transformers
Transformers do not scale very well to long sequence lengths largely bec...
research
∙
10/19/2020
Surprise: Result List Truncation via Extreme Value Theory
Work in information retrieval has largely been centered around ranking a...
research
∙
10/06/2020
Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder
This paper demonstrates a fatal vulnerability in natural language infere...
research
∙
09/14/2020
Efficient Transformers: A Survey
Transformer model architectures have garnered immense interest lately du...
research
∙
08/17/2020
Generative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study
Large generative language models such as GPT-2 are well-known for their ...
research
∙
07/12/2020
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
Achieving state-of-the-art performance on natural language understanding...
research
∙
05/02/2020
Synthesizer: Rethinking Self-Attention in Transformer Models
The dot product self-attention is known to be central and indispensable ...
research
∙
04/26/2020
Choppy: Cut Transformer For Ranked List Truncation
Work in information retrieval has traditionally focused on ranking and r...
research
∙
04/13/2020
Reverse Engineering Configurations of Neural Text Generation Models
This paper seeks to develop a deeper understanding of the fundamental pr...
research
∙
02/26/2020
Sparse Sinkhorn Attention
We propose Sparse Sinkhorn Attention, a new efficient and sparse method ...
research
∙
01/20/2020
Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching
Transformer has been successfully applied to many natural language proce...
research
∙
12/21/2019
Jacobian Adversarially Regularized Networks for Robustness
Adversarial examples are crafted with imperceptible perturbations with t...
research
∙
12/11/2019
What it Thinks is Important is Important: Robustness Transfers through Input Gradients
Adversarial perturbations are imperceptible changes to input pixels that...
research
∙
08/27/2019
Interactive Machine Comprehension with Information Seeking Agents
Existing machine reading comprehension (MRC) models do not scale effecti...
research
∙
06/11/2019
