Embarrassingly Parallel VI

Distributed, communication-free variational inference in nonconjugate models.

We develop a parallel variational inference (VI) procedure for use in data-distributed settings, where each machine only has access to a subset of data and runs VI independently, without communicating with other machines. This type of ``embarrassingly parallel'' procedure has recently been developed for MCMC inference algorithms; however, in many cases it is not possible to directly extend this procedure to VI methods without requiring restrictive conditions on the form of the model. Furthermore, most existing (nonparallel) VI methods are restricted to use on conditionally conjugate models, which limits their applicability. To combat these issues, we make use of the recently proposed nonparametric VI to facilitate an embarrassingly parallel VI procedure that can be used in nonconjugate models. We derive our embarrassingly parallel VI algorithm, analyze our method theoretically, and demonstrate our method empirically on a few nonconjugate models.

Read the Paper

Embarrassingly Parallel Variational Inference in Nonconjugate Models [pdf]. 
Willie Neiswanger,  Chong Wang,  Eric Xing. 

Get the Code

The source code can be downloaded here [git; zip]. Please see the README for installation instructions.