Graduate
  1. Real Analysis  1.1. Set Theory and Logic  1.1.1. Sets and Operations  1.1.2. Relations and Functions  1.1.3. Logic and Quantifiers  1.1.4. Cardinality of Sets  1.2. Metric Spaces  1.2.1. Open and Closed Sets  1.2.2. Convergence of Sequences  1.2.3. Continuous Functions in Metric Spaces  1.2.4. Compactness and Completeness in Metric Spaces in Real Analysis  1.3. Sequence and Series  1.3.1. Convergence of Sequences  1.3.2. Infinite Series  1.3.3. Uniform Convergence  1.3.4. Power Series  1.4. Differentiation  1.4.1. Mean Value Theorems  1.4.2. Taylor Series  1.4.3. Functions of Several Variables  1.4.4. Partial Derivatives  1.5. Integration  1.5.1. Riemann and Lebesgue Integration  1.5.2. Improper Integrals  1.5.3. Measure Theory  1.5.4. Fubini’s Theorem  1.6. Functional Analysis  1.6.1. Banach Spaces  1.6.2. Hilbert Spaces  1.6.3. Operators on Spaces  1.6.4. Spectral Theory
  2. Abstract Algebra  2.1. Groups  2.1.1. Definitions and Examples in Groups in Abstract Algebra  2.1.2. Group Homomorphisms  2.1.3. Normal Subgroups  2.1.4. Sylow Theorems  2.2. Rings and Fields  2.2.1. Ideals and Factor Rings  2.2.2. Field Extensions  2.2.3. Galois Theory  2.2.4. Polynomial Rings  2.3. Modules  2.3.1. Module Homomorphisms  2.3.2. Free Modules  2.3.3. Tensor Products  2.3.4. Exact Sequences  2.4. Linear Algebra  2.4.1. Vector Spaces  2.4.2. Eigenvalues and Eigenvectors  2.4.3. Inner Product Spaces  2.4.4. Diagonalization
  3. Topology  3.1. General Topology  3.1.1. Open and Closed Sets  3.1.2. Compactness and Connectedness  3.1.3. Continuous Maps  3.1.4. Separation Axioms  3.2. Algebraic Topology  3.2.1. Fundamental Groups  3.2.2. Homotopy and Homology  3.2.3. Simplicial Complexes  3.2.4. Exact Sequences in Homology  3.3. Differential Topology  3.3.1. Manifolds  3.3.2. Smooth Functions  3.3.3. Tangent and Cotangent Spaces  3.3.4. Vector Bundles
  4. Differential Equations  4.1. Ordinary Differential Equations  4.1.1. First-Order Equations  4.1.2. Second-Order Linear Equations  4.1.3. Systems of Differential Equations  4.1.4. Stability Analysis  4.2. Partial Differential Equations  4.2.1. Classification of PDEs  4.2.2. Fourier Series  4.2.3. Green's Functions  4.2.4. Method of Characteristics  4.3. Dynamical Systems  4.3.1. Stability Theory  4.3.2. Limit Cycles  4.3.3. Chaos Theory  4.3.4. Bifurcation Theory
  5. Probability and Statistics  5.1. Probability Theory  5.1.1. Random Variables  5.1.2. Probability Distributions  5.1.3. Law of Large Numbers  5.1.4. Central Limit Theorem  5.2. Statistical Inference  5.2.1. Hypothesis Testing  5.2.2. Confidence Intervals  5.2.3. Bayesian Methods  5.2.4. Maximum Likelihood Estimation  5.3. Stochastic Processes  5.3.1. Markov Chains  5.3.2. Brownian Motion  5.3.3. Poisson Processes  5.3.4. Martingales
  6. Numerical Analysis  6.1. Numerical Solutions of Equations  6.1.1. Root Finding  6.1.2. Iterative Methods  6.1.3. Convergence Analysis  6.1.4. Error Bounds in Numerical Solutions of Equations  6.2. Numerical Linear Algebra  6.2.1. Matrix Factorizations  6.2.2. Eigenvalue Computations  6.2.3. Sparse Matrices  6.2.4. Preconditioning Techniques  6.3. Numerical Integration and Differentiation  6.3.1. Quadrature Methods  6.3.2. Finite Differences  6.3.3. Error Analysis in Numerical Integration and Differentiation  6.3.4. Adaptive Methods
  7. Complex Analysis  7.1. Complex Numbers  7.1.1. Polar Form  7.1.2. Complex Conjugates  7.1.3. Exponential Form  7.1.4. Roots of Unity  7.2. Analytic Functions  7.2.1. Cauchy-Riemann Equations  7.2.2. Power Series  7.2.3. Conformal Mappings  7.2.4. Harmonic Functions  7.3. Integration in the Complex Plane  7.3.1. Cauchy’s Theorem  7.3.2. Residue Theorem  7.3.3. Contour Integration  7.3.4. Integral Transforms
  8. Mathematical Logic and Foundations  8.1. Propositional Logic  8.1.1. Truth Tables  8.1.2. Logical Equivalences  8.1.3. Proof Techniques in Propositional Logic  8.1.4. Resolution  8.2. Predicate Logic  8.2.1. Quantifiers in Predicate Logic  8.2.2. Formal Systems in Predicate Logic  8.2.3. Completeness and Soundness  8.2.4. Model Theory  8.3. Set Theory  8.3.1. Cardinality and Ordinality  8.3.2. Axiomatic Systems  8.3.3. Zermelo-Fraenkel Set Theory  8.3.4. Forcing
  9. Optimization  9.1. Linear Programming  9.1.1. Simplex Method  9.1.2. Duality in Linear Programming  9.1.3. Sensitivity Analysis  9.1.4. Interior Point Methods  9.2. Nonlinear Programming  9.2.1. Gradient Descent  9.2.2. Lagrange Multipliers  9.2.3. Convex Optimization  9.2.4. Quadratic Programming  9.3. Combinatorial Optimization  9.3.1. Graph Algorithms  9.3.2. Integer Programming  9.3.3. Network Flows  9.3.4. Approximation Algorithms
  10. Discrete Mathematics  10.1. Graph Theory  10.1.1. Graph Representations  10.1.2. Planarity and Coloring  10.1.3. Shortest Path Algorithms  10.1.4. Network Flows  10.2. Combinatorics  10.2.1. Permutations and Combinations  10.2.2. Generating Functions  10.2.3. Recurrence Relations  10.2.4. Inclusion-Exclusion Principle  10.3. Cryptography  10.3.1. Number Theory Applications in Cryptography  10.3.2. Symmetric and Asymmetric Cryptography  10.3.3. RSA and Elliptic Curve Cryptography  10.3.4. Post-Quantum Cryptography

Graduate → Probability and Statistics → Statistical Inference ↓


Bayesian Methods


Introduction

Bayesian methods play a fascinating role in statistical inference, providing a framework for reasoning about uncertainty. This approach is based on Bayes' theorem, named after the 18th-century statistician and theologian Reverend Thomas Bayes. Unlike frequentist statistics, which uses probability only to refer to long-term frequencies, Bayesian statistics allows probabilities to express one's level of belief or certainty about an event.

Bayes' theorem

The basis of Bayesian inference is Bayes' theorem, which is expressed mathematically as follows:

P(H|E) = (P(E|H) * P(H)) / P(E)

This formula can be broken down into its components:

  • P(H|E): Posterior probability. The probability of the hypothesis H based on the observed evidence E
  • P(E|H): Probability. The probability of observing evidence E given hypothesis H is true.
  • P(H): Prior probability. The initial degree of belief in the hypothesis H before observing E
  • P(E): Marginal likelihood. The overall probability of the evidence under all possible hypotheses.

Basic example: tossing a coin

Consider a simple example where we want to know if a coin is biased towards heads. We observe ten coin tosses, seven of which are heads. We want to find the probability of the coin being biased using a Bayesian framework.

Example

Let H be the hypothesis that the coin is inclined to heads, and E be the evidence that heads occur seven times out of ten tosses. Now, we need to specify:

  • P(H): Our prior belief about the coin being biased. Suppose we believe that every coin has a 50% chance of being biased. Thus, P(H) = 0.5.
  • P(E|H): the probability of observing seven heads under the hypothesis. If this is biased, assume P(E|H) = 0.9.
  • P(E): The marginal likelihood can be calculated by considering all hypotheses. For simplicity, let's say P(E) = 0.5.

Now apply Bayes' theorem:

P(H|E) = (0.9 * 0.5) / 0.5 = 0.9

Therefore, there is a strong possibility that the coin is biased.

Priors

The prior probability P(H) communicates the initial belief before observing the evidence. In Bayesian analysis, the choice of prior can greatly affect the final result, especially with small data sizes. Priors can be informative or non-informative.

Informative preferences

Informative prior knowledge consists of specific, previous knowledge about the parameter of interest. In the coin example, if previous experiments show that the coin falls heads with a 70% probability, this information will guide our prior selection.

Non-informative preferences

Non-informative or weak forecasts do not provide much specific information about the hypothesis, often reflecting a state of relative ignorance. Common alternatives include uniform distributions where all outcomes are equally likely.

Back

Once the evidence is taken into account via Bayes’ theorem, we obtain the posterior probability, code{P(H|E)}, which incorporates all of our information about the hypothesis — the prior and the data combined. The posterior probability is the most important aspect of Bayesian inference, as it represents how our understanding of a hypothesis is modified by new data.

Possibility

Probability is a core component of Bayesian calculations. It measures how likely observed data are under different hypotheses. Mathematically, the likelihood, code{P(E|H)}, assesses the compatibility between the data and the hypothesis.

Marginal likelihood

The marginal likelihood, code{P(E)}, ensures that the posterior probabilities add up to 1. It involves summing the probabilities across all hypotheses. In practice, calculating the marginal likelihood can be complicated, especially in models with many parameters.

Advanced example: disease testing

Suppose a medical test checks for a disease with the following characteristics:

  • The sensitivity of this test is 95%, meaning it correctly identifies 95% of patients suffering from the disease.
  • The specificity of this test is 90%, meaning it correctly identifies 90% of healthy patients.
  • 1% of the population has this disease.

Example

Let H be the event that the patient gets the disease, and E be the positive test result.

  • P(H) = 0.01 (prior probability of having the disease)
  • P(E|H) = 0.95 (probability of testing positive if sick)

To calculate the overall probability of a positive test, code{P(E)}, consider both true and false positive results:

P(E) = P(E|H) * P(H) + P(E|H') * P(H')
P(E) = 0.95 * 0.01 + 0.1 * 0.99 = 0.1045

Finally, use Bayes' theorem to find the posterior:

P(H|E) = (0.95 * 0.01) / 0.1045 ≈ 0.091

Despite a positive test, the probability of having the disease based on this result is only 9.1%.

Updating beliefs

Bayesian inference is an iterative process. As more evidence is collected, you continuously update your beliefs using Bayes' theorem. Each new piece of evidence acts as a probability that modifies your previous beliefs to form subsequent beliefs. Over time, this process refines our understanding and improves decision making.

Conjugate priors

In many cases, choosing a conjugate prior simplifies the calculation. A conjugate prior is one that, when used as a prior, yields a posterior distribution of the same family, thereby simplifying the analytical solution. For example, in binomial probabilities, the beta distribution as a prior will yield a beta posterior distribution - so the distribution type remains constant.

Application

Bayesian methods have extensive applications in a variety of fields. Some of the notable ones are:

  • Medicine: For diagnosing diseases, Bayesian methods balance prior information about the prevalence of a disease with diagnostic test evidence.
  • Finance: Bayesian models are used to predict stock prices, incorporating both historical data and expert forecasts.
  • Machine learning: Bayesian techniques power probability models for tasks such as classification, clustering, and regression.
  • Natural language processing: Bayesian inference extends models such as topic models to identify patterns in text data.

Challenges

While powerful, Bayesian methods also present challenges. Complex models often require significant computational resources. It can be difficult to calculate the posterior distribution analytically, requiring approximation techniques such as Markov Chain Monte Carlo (MCMC).

Conclusion

Bayesian methods provide a flexible, consistent framework for statistical inference. By combining prior beliefs with new evidence, Bayesian inference refines understanding in a logical, intuitive way. Despite the calculation challenges in a wide range of situations, its principles shine in many real-world applications, making it invaluable in the statistician's toolkit.


Graduate → 5.2.3


U
username
0%
completed in Graduate

← Prev (5.2.2)
Confidence Intervals
Next (5.2.4) →
Maximum Likelihood Estimation

Comments 



Bayesian Methods

https://www.buddymath.com/g/5.2.3?bmal=en