After grumbling for years in our substacks about uses and abuses of probability in machine learning, @beenwrekt.bsky.social and I decided it's finally time to put down some concrete ideas in a manifesto with a DOI. Here it is, give us your best feedback. (1/4)
Separating Geometry from Probability in the Analysis of Generalization
The goal of machine learning is to find models that minimize prediction error on data that has not yet been seen. Its operational paradigm assumes access to a dataset $S$ and articulates a scheme for ...
arxiv.org
![Full LaTeX source: https://pastebin.com/mA6KjUJs
\begin{proposition}[Triangle-like inequality for KL divergence]\label{prop:kl-triangle}
Let $P$, $R$, and $Q$ be probability distributions with $P$ being absolutely continuous with respect to $R$ and $R$ being absolutely conotinuous with respect to $Q$.
Let $\kappa \in (1,\infty)$.
Then
\[
\dr{\text{KL}}{P}{Q} \le \frac{\kappa}{\kappa-1} \dr{\text{KL}}{P}{R} + \dr{\kappa}{R}{Q},
\]
where $\dr{\text{KL}}{P}{Q} := \ex{X \gets P}{\log(P(X)/Q(X)}$ denotes the KL divergence and\\$\dr{\kappa}{R}{Q} = \frac{1}{\kappa-1} \log \ex{X \gets R}{(R(X)/Q(X))^{\kappa-1}}$ denotes the R\'enyi divergence of order $\kappa$.
\end{proposition}](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:thfnaqgktdwqbm42radc322i/bafkreibz3gfiqvyvbld7jgfnb6kzac2j6qrxirofpwrxiv5mpz6j7r6md4)
