Bayesian Econometrics Introduction

Introduction to Bayesian Econometrics¶

This course will cover the application of Bayesian statistical methods for econometric inference. Broadly speaking we will

1. Briefly discuss sampling methods for classical statistics
2. Introduce Bayes Rule and Provide an Application
3. Examine the use of Monte Carlo Markov Chains
   • Link to Bayes Rule
   • Metropolis-Hastings and other Samplers
   • Chain "convergence" and diagnostics
4. Application: OLS, Time-Series Econometrics, Heirarchical Models

The frequentist paradigm, arguably the dominant statistical paradigm in the social sciences (and what you all have studied), relies on the following notions:

• $\beta$ is not random but neither is it known. It is a fixed quantity.
• To uncover information about $\beta$, we observe part of some process (e.g. $\mathbf{y=x\beta+\epsilon}$).
• For statistical inference, we rely on repeated trials of $\mathbf{y}$ and $\mathbf{x}$, even if this repetition rarely (if ever) occurs in the social science context.
• $\mathbf{y}$ and $\mathbf{x}$ are considered random
• The model typically attempts to uncover information about $\mathbf{\beta}$ by examining the likelihood function

where $\mathbf{b}$ are our estimates of $\beta$

The bayesian paradigm tackles the issue of estimating $\beta$ by

• Treating $\beta$ as random and unknown
• Treating $\mathbf{y}$ and $\mathbf{x}$ as fixed and non-random (at least once they are recorded in your dataset)
• Uncovers information about $\mathbf{\beta}$ by examining the posterior likelihood

where $\mathbf{b}$ are our estimates of $\beta$

In quite a lot of instances, these two approaches give you the same estimate for $\beta$. Until recently, Bayesian Statistical modeling wasn't used because calculating the posterior likelihood was computationally challenging, but recent advances in the theory and construction of Monte Carlo Markov Chains and computational ability has really opened the door for Bayesian analysis for problems that might not be estimated using the frequentist paradigm (ie. Maximim Likelihood). There is an ongoing holy war in the two statistical camps, during the semester I will attempt to highlite the pros and cons of each paradigm without taking a position on which one is better. My philosophy is that if it gets the job done, use it while being aware of limitations and advantages.

Repeated trials in frequentist statistics¶

A good jumping off point for this course is to understand the use of sampling techniques in a classical statistical paradigm.

• Underlying all statistical inference that you have learned in statistics and econometrics is the idea of repeated trials. Bootstrapping highlites this really well.
• We will begin with an exploration of bootstrapping and see the implementation steps.

Load Tobias and Koop data for time==4¶

These standard errors aren't robust (although we could ask for that). As we saw in Cross-Section, we can non-parametrically bootstrap confidence intervals of our parameters. While bootstrapping is implemented in many ols routines, we implement them manually to see how it works:

Steps for bootstrapping:

1. Loop $R$ times ($R$ = # of boostrapped replicates)
2. For each $r\in R$, sample $N$ times with replacement from rows in $\mathbf{Y}$ and $\mathbf{X}$, noting that $N$ is equal to the number of observations in the original dataset $\mathbf{Y}$ and $\mathbf{X}$. Denote each of these samples as $\mathbf{X}_r$ and $\mathbf{Y}_r$.
3. Estimate an OLS model and recover estimate for replicate $r$: $\hat{\beta}_r = (\mathbf{X}_r'\mathbf{X}_r)^{-1}\mathbf{X}_r'\mathbf{Y}_r$
4. Store results
5. Calculate the standard deviation (our estimate for the standard errors) of each parameter estimate, or construct non-parametric confidence intervals (preferred)

Note, for our purposes we don't need to see the R regression results. We only need to store them and move on to the next replicate. Let's investigate further the res object we created above.

How to Recover OLS Estimates from Statsmodels¶

Sampling with replacement¶

Fundamental to the idea of bootstrapping is to sample with replacement to preserve the shape of the underlying distribution of our parameters. We can do this quite easily using numpy tools.

To do this, suppose we want to sample with replacement from 5 rows of data indexed by [0,1,2,3,4].

So we can sample with replacement from row numbers (index) and then use that to select the rows we need for that replicate. For each replicate, run the model or calculate the statistic we are interested in, store the results, and then repeat for all subsequent replicates.

Bootstrap for our OLS Model¶

The following code uses the ideas above to perform the following steps for each replicate $r$ of $R$ total replicates:

1. Sample with replacement from rows of our dataset
2. Calculate the statistic of interest (our OLS parameter estimates)
3. Store results

From the above results, we see that the 95% confidence interval for education is in the range [0.0668,0.10517] (approximately, as these numbers will change each time you run the code above). It is tempting to interpret the confidence intervals above as There is a 95% chance that $\beta$ is in the range [0.0668,0.10517]. However, recall that in the frequentist paradigm $\beta$ is fixed and non-random. So it is either in the range [0.0668,0.10517] or it isn't. What is random is the range [0.0668,0.10517]. So a better way to verbalize the confidence interval is that if you repeated your regression analysis many times, 95% of your calculated confidence interval would contain the true parameter $\beta$.

Let's plot our bootstrap replicates, by parameter. Note that the values of each of these replicates are completely independent (or should be). Replicate 3 doesn't in any way inform us about replicate 4. For each parameter, we show the estimated parameter (solid black line) and two parameter estimates:

1. The standard 95% confidence interval (dashed red line):

2. The 95% confidence interval calculated based off the 2.5 and 97.5 percentiles from our boostrap database (dashed blue line).

Focusing on educ¶

For all parameters:¶

Visualizing Covariances $\beta$:¶