Graduate
  1. Real Analysis  1.1. Set Theory and Logic  1.1.1. Sets and Operations  1.1.2. Relations and Functions  1.1.3. Logic and Quantifiers  1.1.4. Cardinality of Sets  1.2. Metric Spaces  1.2.1. Open and Closed Sets  1.2.2. Convergence of Sequences  1.2.3. Continuous Functions in Metric Spaces  1.2.4. Compactness and Completeness in Metric Spaces in Real Analysis  1.3. Sequence and Series  1.3.1. Convergence of Sequences  1.3.2. Infinite Series  1.3.3. Uniform Convergence  1.3.4. Power Series  1.4. Differentiation  1.4.1. Mean Value Theorems  1.4.2. Taylor Series  1.4.3. Functions of Several Variables  1.4.4. Partial Derivatives  1.5. Integration  1.5.1. Riemann and Lebesgue Integration  1.5.2. Improper Integrals  1.5.3. Measure Theory  1.5.4. Fubini’s Theorem  1.6. Functional Analysis  1.6.1. Banach Spaces  1.6.2. Hilbert Spaces  1.6.3. Operators on Spaces  1.6.4. Spectral Theory
  2. Abstract Algebra  2.1. Groups  2.1.1. Definitions and Examples in Groups in Abstract Algebra  2.1.2. Group Homomorphisms  2.1.3. Normal Subgroups  2.1.4. Sylow Theorems  2.2. Rings and Fields  2.2.1. Ideals and Factor Rings  2.2.2. Field Extensions  2.2.3. Galois Theory  2.2.4. Polynomial Rings  2.3. Modules  2.3.1. Module Homomorphisms  2.3.2. Free Modules  2.3.3. Tensor Products  2.3.4. Exact Sequences  2.4. Linear Algebra  2.4.1. Vector Spaces  2.4.2. Eigenvalues and Eigenvectors  2.4.3. Inner Product Spaces  2.4.4. Diagonalization
  3. Topology  3.1. General Topology  3.1.1. Open and Closed Sets  3.1.2. Compactness and Connectedness  3.1.3. Continuous Maps  3.1.4. Separation Axioms  3.2. Algebraic Topology  3.2.1. Fundamental Groups  3.2.2. Homotopy and Homology  3.2.3. Simplicial Complexes  3.2.4. Exact Sequences in Homology  3.3. Differential Topology  3.3.1. Manifolds  3.3.2. Smooth Functions  3.3.3. Tangent and Cotangent Spaces  3.3.4. Vector Bundles
  4. Differential Equations  4.1. Ordinary Differential Equations  4.1.1. First-Order Equations  4.1.2. Second-Order Linear Equations  4.1.3. Systems of Differential Equations  4.1.4. Stability Analysis  4.2. Partial Differential Equations  4.2.1. Classification of PDEs  4.2.2. Fourier Series  4.2.3. Green's Functions  4.2.4. Method of Characteristics  4.3. Dynamical Systems  4.3.1. Stability Theory  4.3.2. Limit Cycles  4.3.3. Chaos Theory  4.3.4. Bifurcation Theory
  5. Probability and Statistics  5.1. Probability Theory  5.1.1. Random Variables  5.1.2. Probability Distributions  5.1.3. Law of Large Numbers  5.1.4. Central Limit Theorem  5.2. Statistical Inference  5.2.1. Hypothesis Testing  5.2.2. Confidence Intervals  5.2.3. Bayesian Methods  5.2.4. Maximum Likelihood Estimation  5.3. Stochastic Processes  5.3.1. Markov Chains  5.3.2. Brownian Motion  5.3.3. Poisson Processes  5.3.4. Martingales
  6. Numerical Analysis  6.1. Numerical Solutions of Equations  6.1.1. Root Finding  6.1.2. Iterative Methods  6.1.3. Convergence Analysis  6.1.4. Error Bounds in Numerical Solutions of Equations  6.2. Numerical Linear Algebra  6.2.1. Matrix Factorizations  6.2.2. Eigenvalue Computations  6.2.3. Sparse Matrices  6.2.4. Preconditioning Techniques  6.3. Numerical Integration and Differentiation  6.3.1. Quadrature Methods  6.3.2. Finite Differences  6.3.3. Error Analysis in Numerical Integration and Differentiation  6.3.4. Adaptive Methods
  7. Complex Analysis  7.1. Complex Numbers  7.1.1. Polar Form  7.1.2. Complex Conjugates  7.1.3. Exponential Form  7.1.4. Roots of Unity  7.2. Analytic Functions  7.2.1. Cauchy-Riemann Equations  7.2.2. Power Series  7.2.3. Conformal Mappings  7.2.4. Harmonic Functions  7.3. Integration in the Complex Plane  7.3.1. Cauchy’s Theorem  7.3.2. Residue Theorem  7.3.3. Contour Integration  7.3.4. Integral Transforms
  8. Mathematical Logic and Foundations  8.1. Propositional Logic  8.1.1. Truth Tables  8.1.2. Logical Equivalences  8.1.3. Proof Techniques in Propositional Logic  8.1.4. Resolution  8.2. Predicate Logic  8.2.1. Quantifiers in Predicate Logic  8.2.2. Formal Systems in Predicate Logic  8.2.3. Completeness and Soundness  8.2.4. Model Theory  8.3. Set Theory  8.3.1. Cardinality and Ordinality  8.3.2. Axiomatic Systems  8.3.3. Zermelo-Fraenkel Set Theory  8.3.4. Forcing
  9. Optimization  9.1. Linear Programming  9.1.1. Simplex Method  9.1.2. Duality in Linear Programming  9.1.3. Sensitivity Analysis  9.1.4. Interior Point Methods  9.2. Nonlinear Programming  9.2.1. Gradient Descent  9.2.2. Lagrange Multipliers  9.2.3. Convex Optimization  9.2.4. Quadratic Programming  9.3. Combinatorial Optimization  9.3.1. Graph Algorithms  9.3.2. Integer Programming  9.3.3. Network Flows  9.3.4. Approximation Algorithms
  10. Discrete Mathematics  10.1. Graph Theory  10.1.1. Graph Representations  10.1.2. Planarity and Coloring  10.1.3. Shortest Path Algorithms  10.1.4. Network Flows  10.2. Combinatorics  10.2.1. Permutations and Combinations  10.2.2. Generating Functions  10.2.3. Recurrence Relations  10.2.4. Inclusion-Exclusion Principle  10.3. Cryptography  10.3.1. Number Theory Applications in Cryptography  10.3.2. Symmetric and Asymmetric Cryptography  10.3.3. RSA and Elliptic Curve Cryptography  10.3.4. Post-Quantum Cryptography

Graduate → Probability and Statistics → Statistical Inference ↓


Maximum Likelihood Estimation


Maximum likelihood estimation (MLE) is a statistical technique used to estimate the parameters of a statistical model. This fundamental technique is widely applicable in various fields such as economics, finance, biology, machine learning, and others. In simple terms, MLE helps you find the parameter values for the model that make your observed data most probable. The main idea is to maximize the likelihood function, which is the probability of observing the data given a set of parameters.

Understanding the basic concepts

Before delving into the intricacies of maximum likelihood estimation, it is essential to understand some basic statistical and probability concepts.

Statistical models

A statistical model is a mathematical representation that describes random variables and their relationships. For example, suppose you have a set of data points. A statistical model might express these data points as normally distributed, with a mean (mu) and a standard deviation (sigma).

Parameters

Parameters are aspects of the model that can be adjusted to fit the data. In the normal distribution example above, the parameters are the mean ((mu)) and the standard deviation ((sigma)).

Possibility

The likelihood of a set of parameter values is defined as the probability of observing the data given those parameters. It is not a probability itself, but rather a function of the parameters. In more formal terms, if (theta) represents the parameters of the distribution and (X) represents the observed data, then the likelihood is (L(theta | X)).

The procedure for maximum likelihood estimation

Let us analyse the steps involved in maximum likelihood estimation:

Step 1: Choose a statistical model

The first step is to choose an appropriate statistical model based on the nature of your data. For example, if you are dealing with height measurement data, modeling it as a normal distribution may be appropriate.

Step 2: Define the likelihood function

The next step is to define the likelihood function for the model you choose. For example, assume your data is normally distributed; the likelihood function for the data sample (X = x_1, x_2, ldots, x_n) and the parameters (mu) (mean) and (sigma^2) (variance) is:

L(mu, sigma^2 | X) = prod_{i=1}^{n} frac{1}{sqrt{2pisigma^2}} expleft(-frac{(x_i - mu)^2}{2sigma^2}right)

Step 3: Maximize the likelihood function

To find the values of the parameters that maximize this likelihood function, we usually work with the log-likelihood function because it is often easier to handle:

ell(mu, sigma^2 | X) = sum_{i=1}^{n} left( -frac{1}{2} log(2pisigma^2) - frac{(x_i - mu)^2}{2sigma^2} right)

You then take the derivative of the log-likelihood function with respect to the parameters and set them to zero to solve for the parameters.

Step 4: Solve for the parameters

For the normal distribution:

frac{partial}{partial mu}ell(mu, sigma^2 | X) = 0 quad Rightarrow quad hat{mu} = frac{1}{n}sum_{i=1}^{n} x_i
frac{partial}{partial sigma^2}ell(mu, sigma^2 | X) = 0 quad Rightarrow quad hat{sigma}^2 = frac{1}{n}sum_{i=1}^{n} (x_i - hat{mu})^2

The solutions, (hat{mu}) and (hat{sigma}^2), are maximum likelihood estimates for the mean and variance.

Visual example

Let’s imagine how the likelihood function works for a simple dataset and a model with a single parameter, (p), which represents the probability of success in a binomial distribution. We’ll take data from a series of independent coin-flips resulting in 4 heads out of 10 flips.

The parameter pPossibilityP

This example shows the likelihood function for the binomial distribution. The red dot indicates the parameter value (p) that maximizes the likelihood function, which in this case is about 0.4 or 40% probability of getting heads.

Properties of maximum likelihood estimators

Maximum likelihood estimators have several notable properties that make them particularly useful in statistical inference:

Stability

As the sample size increases, an estimator is consistent if it converges to the probability of the true parameter value. MLEs usually have this property under standard conditions, meaning that they become more accurate as you collect more data.

Capacity

In the context of unbiased estimators, efficiency refers to the fact that the MLE achieves the smallest possible variance among all unbiased estimators of the parameter. This variance is known as the Cramer-Rao lower bound.

Normal state

Under certain regularity conditions, the distribution of the MLE tends towards a normal distribution as the sample size increases. This is particularly useful for constructing confidence intervals.

Examples of MLE in different models

Example 1: Estimating the parameter of the exponential distribution

Consider an exponential distribution with parameter (lambda). If you have a dataset (X = x_1, x_2, ..., x_n), then the likelihood function is given by:

L(lambda | X) = prod_{i=1}^{n} lambda exp(-lambda x_i)

Taking the logarithm:

ell(lambda | X) = n log(lambda) - lambda sum_{i=1}^{n} x_i

Setting the derivative to zero gives:

frac{partial}{partial lambda}ell(lambda | X) = frac{n}{lambda} - sum_{i=1}^{n} x_i = 0
hat{lambda} = frac{n}{sum_{i=1}^{n} x_i}

Example 2: Estimating parameters in a linear regression model

In a simple linear regression model of the form (y = beta_0 + beta_1 x + epsilon), where (epsilon sim N(0, sigma^2)), the likelihood function is:

L(beta_0, beta_1, sigma^2 | y, x) = prod_{i=1}^{n} frac{1}{sqrt{2pisigma^2}} expleft(-frac{(y_i - beta_0 - beta_1 x_i)^2}{2sigma^2}right)

Maximizing this likelihood involves finding estimates for (beta_0), (beta_1), and (sigma^2). However, this quickly turns into solving the usual equations or using matrix calculus to obtain:

hat{beta} = (X^TX)^{-1} X^T y

for a vector of coefficients in matrix form (beta).

Advantages and disadvantages of M.L.E.

Understanding the advantages and disadvantages of MLE can help you decide whether it is an appropriate method for parameter estimation.

Benefit

  • Flexibility: MLE can be applied to many different distributions and scenarios. The fundamental concept of maximizing the likelihood of observed data aligns well with a variety of situations.
  • Asymptotic properties: As discussed above, MLEs have some desirable asymptotic properties, such as consistency, efficiency, and normality, which make them statistically robust for large samples.
  • Explainability: This method produces a straightforward result – the estimated parameters make the observed data 'most likely', given the assumptions of the model.

Loss

  • Complexity: For complex models, the likelihood function can be complicated, and maximizing it may require sophisticated numerical methods. This can be computationally intensive.
  • Sensitivity to model assumptions: MLEs are highly dependent on the accuracy of the model. Misspecifying the model can lead to biased parameter estimates.
  • Drawbacks of finite sample: In small sample sizes, MLE may not exhibit its asymptotic properties such as efficiency, resulting in less reliable estimates.

Conclusion

Maximum likelihood estimation stands as a cornerstone technique in statistical inference, providing a structured and powerful approach to parameter estimation in a wide variety of statistical models. However, it requires careful consideration of the choice of model, as well as a readiness to tackle computational challenges in complex models. Despite its limitations, its flexible application and asymptotic properties secure its continued relevance and widespread use in both theoretical and applied statistics.


Graduate → 5.2.4


U
username
0%
completed in Graduate

← Prev (5.2.3)
Bayesian Methods
Next (5.3) →
Stochastic Processes

Comments 



Maximum Likelihood Estimation

https://www.buddymath.com/g/5.2.4?bmal=en