In the Philosophy of Information, the connection between Information and Bayesian probability is a well-established topic (e.g. 1, 2). This freely available book offers the opportunity to explore hands-on the relationship between Bayesian methods and probabilistic programming. This is not the typical textbook for Philosophers but worth exploring assuming prior skills in programming.
If Bayesian inference is the destination, then mathematical analysis is a particular path to towards it. On the other hand, computing power is cheap enough that we can afford to take an alternate route via probabilistic programming. The latter path is much more useful, as it denies the necessity of mathematical intervention at each step, that is, we remove often- intractable mathematical analysis as a prerequisite to Bayesian inference. Simply put, this latter computational path proceeds via small intermediate jumps from beginning to end, where as the first path proceeds by enormous leaps, often landing far away from our target. Furthermore, without a strong mathematical background, the analysis required by the first path cannot even take place.