Vaden Masrani

In the era of costly pre-training of large language models, ensuring the intellectual property rights of model owners, and insuring that said models are responsibly deployed, is becoming increasingly important. To this end, we propose model watermarking via passthrough layers, which are added to existing pre-trained networks and trained using a self-supervised loss such that the model produces high-entropy output when prompted with a unique private key, and acts normally otherwise. Unlike existing model watermarking methods, our method is fully task-agnostic, and can be applied to both classification and sequence-to-sequence tasks without requiring advanced access to downstream fine-tuning datasets. We evaluate the proposed passthrough layers on a wide range of downstream tasks, and show experimentally our watermarking method achieves a near-perfect watermark extraction accuracy and false-positive rate in most cases without damaging original model performance. Additionally, we show our method is robust to both downstream fine-tuning, fine-pruning, and layer removal attacks, and can be trained in a fraction of the time required to train the original model. Code is available in the paper.

Variational inference (VI) is a popular method used within statistics and machine learning to approximate intractable probability distributions via optimization. Central to VI is the Evidence Lower Bound (ELBO), a variational objective function which lower bounds the log marginal likelihood, and can be used to jointly perform maximum likelihood parameter estimation and approximate posterior inference using stochastic gradient ascent. The core contribution of this thesis is the Thermodynamic Variational Objective (TVO), a novel variational objective derived from a key connection we make between variational inference and thermodynamic integration. The TVO both tightens and generalizes the ubiquitous ELBO, and empirically leads to improvements in model and inference network learning in both discrete and continuous deep generative models. Using a novel exponential family interpretation of the geometric mixture curve underlying the TVO, we characterize the divergence bound gap left by the TVO as a sum of KL divergences between adjacent distributions, with the forward and reverse KL’s corresponding to the lower and upper-bound TVO variants. To enable the TVO to be used in gradient- based optimization algorithms, we provide two computationally efficient score-function and doubly-reparameterized based gradient estimators, as well as two adaptive “schedulers” which choose the discretization locations of a one- dimensional Riemann integral approximation, a key hyperparameter in the TVO. Additionally, we show that the objective functions used in Variational Inference, Variational AutoEncoders, Wake-Sleep, Inference Compilation, and Rényi Divergence Variational Inference are all special cases of the TVO. Finally, we evaluate the TVO in two real-world settings - a stochastic control flow models with discrete latent variables, and multi-agent trajectory prediction with continuous latent variables built on top of a differentiable driving simulator - and find the TVO improves upon baseline objectives in both cases.

We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames. We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length. We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA self-driving car simulator.