SciPost Physics

SciPost Phys. 14, 032 (2023) · published 14 March 2023

• doi: 10.21468/SciPostPhys.14.3.032
• pdf
• BiBTeX
• RIS
• Submissions/Reports
• 

Abstract

Restricted Boltzmann Machines are simple and powerful generative models that can encode any complex dataset. Despite all their advantages, in practice the trainings are often unstable and it is difficult to assess their quality because the dynamics are affected by extremely slow time dependencies. This situation becomes critical when dealing with low-dimensional clustered datasets, where the time required to sample ergodically the trained models becomes computationally prohibitive. In this work, we show that this divergence of Monte Carlo mixing times is related to a phenomenon of phase coexistence, similar to that which occurs in physics near a first-order phase transition. We show that sampling the equilibrium distribution using the Markov chain Monte Carlo method can be dramatically accelerated when using biased sampling techniques, in particular the Tethered Monte Carlo (TMC) method. This sampling technique efficiently solves the problem of evaluating the quality of a given trained model and generating new samples in a reasonable amount of time. Moreover, we show that this sampling technique can also be used to improve the computation of the log-likelihood gradient during training, leading to dramatic improvements in training RBMs with artificial clustered datasets. On real low-dimensional datasets, this new training method fits RBM models with significantly faster relaxation dynamics than those obtained with standard PCD recipes. We also show that TMC sampling can be used to recover the free-energy profile of the RBM. This proves to be extremely useful to compute the probability distribution of a given model and to improve the generation of new decorrelated samples in slow PCD-trained models. The main limitations of this method are, first, the restriction to effective low-dimensional datasets and, second, the fact that the Tethered MC method breaks the possibility of performing parallel alternative Monte Carlo updates, which limits the size of the systems we can consider in practice.

• Krasnoproshin et al., Random Search in Neural Networks Training
  Pattern Recognit. Image Anal. 34, 309 (2024) [Crossref]
• Fariselli et al., Deep convolutional and conditional neural networks for large-scale genomic data generation
  PLoS Comput Biol 19, e1011584 (2023) [Crossref]
• Decelle et al., Inferring effective couplings with restricted Boltzmann machines
  SciPost Phys. 16, 095 (2024) [Crossref]
• Garí et al., Learning restricted Boltzmann machines with pattern induced weights
  Neurocomputing 610, 128469 128469 (2024) [Crossref]
• Valle, The Capabilities of Boltzmann Machines to Detect and Reconstruct Ising System’s Configurations from a Given Temperature
  Entropy 25, 1649 (2023) [Crossref]
• Decelle et al., Unsupervised hierarchical clustering using the learning dynamics of restricted Boltzmann machines
  Phys. Rev. E 108, 014110 (2023) [Crossref]