Review on Probability and Statistics

Please check https://online.stat.psu.edu/statprogram/reviews for a full list of review in algebra and calculus.

Probability

  • Definitions: sample space, events, probability rules(addition, multiplication)

  • Conditional probability: \(P(A|B) = \frac{P(A\cap B)}{P(B)}\).

  • Independent: If \(A\) and \(B\) are independent, then \(P(A\cap B) = P(A)P(B)\).

  • i.i.d. rv: Independent and identically distributed random variables. For example: \(X_1, ..,X_n\) are i.i.d. \(\sim N(0,1)\).

  • Bayes’ rule: \(P(A|B)= \frac{P(B|A)P(A)}{P(B)}\)

  • \(P(A|B) + P(A^C|B) = 1\)

  • Discrete and continuous random variables: know the pmf or pdf, expectation \(E(X)\) and variance \(V(X)\).

    Distribution \(f(x)\) \(E(X)\) \(Var(X)\)
    Uniform(\(a,b\)) \(\frac{1}{b-a}\) , \(a<x<b\) \(\frac{a+b}{2}\)
    Bernoulli(\(p\)) \(p^x (1 - p)^{1 - x}, \quad x \in {0,1}\) \(p\) \(p(1-p)\)
    Binomial (\(n,p\)) \(\binom{n}{x} p^x (1 - p)^{n - x}, \quad x = 0,1,\dots,n\) \(np\) \(np(1-p\) )
    Poisson (\(\lambda\)) \(\dfrac{e^{-\lambda} \lambda^x}{x!}, \quad x = 0,1,2,\dots\) \(\lambda\) \(\lambda\)
    Normal(\(\mu,\sigma^2\)) \(\dfrac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\dfrac{(x - \mu)^2}{2\sigma^2}\right), -\infty < x<\infty\) \(\mu\) \(\sigma^2\)
    Exponential (\(\lambda\)) \(\lambda e^{-\lambda x}, x>0\) \(\frac{1}{\lambda}\) \(\frac{1}{\lambda^2}\)
    Gamma \((\alpha, \beta)\) \(\dfrac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha - 1} e^{-\beta x},x>0\) \(\alpha\beta\) \(\alpha\beta^2\)
    Beta \((\alpha, \beta)\) \(\dfrac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} x^{\alpha - 1}(1 - x)^{\beta - 1}, 0 < x < 1\) \(\dfrac{\alpha}{\alpha + \beta}\) \(\dfrac{\alpha\beta}{(\alpha + \beta)^2(\alpha + \beta + 1)}\)

  • Law of Large Numbers (averages stabilize)

  • Central Limit Theorem (sample mean \(\approx\) Normal)

Statistics

  • Sampling distribution (t distribution, t test)

  • Point estimation: sample mean, sample variance, maximum likelihood idea

  • Hypothesis testing

    • Null hypothesis \(H_0\) and Alternative hypothesis \(H_a\)

    • Type I error \(\alpha\) and type II error \(\beta\)

    • p-value and test statistics

    • Power of a test \((1-\beta)\)

  • Confidence intervals for mean : interpretation and construction

  • Likelihood function

Simulation Study

Instead of relying on real-world experiments, you will generate data from a known probability model, analyze it, and repeat this process many times to see patterns. You know the truth in simulation, so you can test whether statistical model perform as promised, or analyze the results where math is hard but simulation is easy.

  1. Model: Pick a distribution you want to study (random sample from normal distribution, Bernoulli distribution, etc)

  2. Generate data: use random number generator to create synthetic samples

  3. Apply: run the statistical procedure you are interested in (e.g. confidence interval, central limit theorem)

  4. Repeat: do this thousands of times to see how the method behaves

  5. Summarize results: estimate coverage, bias, standard error, average sample size, etc.

Tips:

  1. Always set the random seed to make the results reproducible. You will have the same numbers, plots, and conclusion each run.
  2. Start small, then scale up. Test your code on 1 run, then 10 runs, then 1000 and more.
  3. Use vectorization if possible, instead of the for loop.
  4. Track key quantities at each run, and store results in a vector/array/matrix.
  5. Always label and comment your codes.
  6. Always report the estimation with the standard error.