Reproduce “Loss as the Inconsistency of a Probabilistic Dependency Graph: Choose Your Model, Not Your Loss Function” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Oliver E Richardson

A framework demonstrates that diverse machine learning loss functions, including cross-entropy, mean squared error, and ELBO, emerge naturally as quantifications of "inconsistency" within carefully constructed Probabilistic Dependency Graphs (PDGs). This approach advocates for model-driven objective function selection, rather than heuristic choices, by mapping explicit probabilistic assumptions to their corresponding optimization targets.

Key insights

  • Loss functions should be viewed as direct consequences of a carefully specified probabilistic model, particularly as a measure of the inherent inconsistency within that model.
  • The framework unifies diverse machine learning objectives, including cross-entropy, mean squared error, KL divergence, and variational inference objectives (ELBO), under the single concept of PDG inconsistency.
  • The strength of regularization terms in objective functions directly corresponds to the confidence assigned to prior beliefs within the PDG, providing a principled justification for their role.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.