Splines!

Splines are a popular way to fit data that allow for a good fit when the data has some sort of global curvature (see the graph below for an example). A cubic spline is a popular example. It allows for a cubic polynomial to be fit within some neighborhood between so-called “knots.” So for the graph below you might have a knot at age 0 and another knot at age 50, yet another knot at age 100, and so on. Within each segment of the data defined by these knots you fit a cubic polynomial. You then require that at the knot locations where the different polynomials meet they must join together in a smooth way (this is done by enforcing the first and second derivatives be equal at the knot).

Below a spline with 20 evenly spaced knot locations is shown for some dummy data. I’ve also graphed a 95% confidence interval of the spline predictions in dotted red.

cubic spline

Here’s the code I used to produce it.

# Load libraries
library(reshape)
library(mgcv)
library(ggplot2)
library(data.table)
 
# Load data
setwd("Masked")
data <- read.csv(file = "Masked", header = TRUE)
attach(data)
 
# Create 20 evenly spaced knots
splitDistance <- (max(age) - min(age))/19
knots <- seq(min(age), max(age), by = splitDistance)
 
# Fit a penalized cubic regression spline with 20 evenly spaced 
# knots using cross validation
model <- gam(strontium.ratio ~ s(age, k=20, fx=F, bs="cr", m=2), 
data=data, family=gaussian, sp = .45, knots = list(age = knots))
# Predict points to plot data 
plotPoints <- seq(min(age), max(age), by = .05)
yHat <- predict.gam(model, list(age = plotPoints), se.fit = TRUE)
 
# Create 95% confidence interval
upper95 <- yHat[[1]] + 1.96*yHat[[2]]
lower95 <- yHat[[1]] - 1.96*yHat[[2]]
 
# Prepare data for plotting
spline <- as.data.frame(cbind(plotPoints,yHat[[1]]))
setnames(spline,"V2","value")
CI <- as.data.frame(cbind(plotPoints,upper95,lower95))
CI <- melt(CI, id = "plotPoints", 
variable_name = "Confidence.Interval")
 
# Plot data
ggplot() +
geom_point(data = data, aes(x = age, y = strontium.ratio)) +
geom_line(data=spline, aes(x = plotPoints, y = value), 
colour = '#3399ff') +
geom_line(data=CI, aes(x = plotPoints, y = value, 
group = Confidence.Interval),linetype="dotted",colour = '#ff00ff') +
ggtitle("Cubic Spline") + 
xlab("Age") + 
ylab("Strontium Ratio")

Created by Pretty R at inside-R.org

To show the effect of knot placement we can generate 20 random knot locations for multiple models and compare them. Since these are so-called natural splines linearity is enforced after the last knot. Here, Model 3 had it’s first knot generated quite far on the righthand side of the Age axis and so it’s linear to the left of that point.

10 Splines

Again, here’s the code.

# Create 10 models with random knot placement
set.seed(35)
models <- as.data.frame(lapply(1:10,
                        FUN = function(i) {
                        knots <- sort(runif(20, min(age), 
                        max(age)))
                        model <- gam(strontium.ratio ~ 
                        s(age, k=20, fx=F, bs="cr", m=2), 
                        data=data, family=gaussian, 
                        method="GCV.Cp", 
                        knots = list(age = knots))
                        return(predict(model,
                        list(age = plotPoints)))
                        }))
 
colnames(models) <- lapply(1:10,
                    FUN = function(i) {
                    return(paste("Model",i))
                    })
 
# Plot splines
models <- cbind(plotPoints, models)
models <- melt(models, id = "plotPoints", variable_name = "Splines")
 
ggplot() +
geom_point(data = data, aes(x = age, y = strontium.ratio)) +
geom_line(data = models, aes(x = plotPoints, y = value, 
group = Splines, colour = Splines)) +
ggtitle("A Plot of 10 Splines") + 
xlab("Age") + 
ylab("Strontium Ratio")

Created by Pretty R at inside-R.org

LaTeX in WordPress

As part of a recent post on ridge and lasso regression techniques in R I wanted to present two mathamatical formulas. I checked for WordPress plugins that used LaTeX, but I learned you have to migrate your blog from WordPress.com to WordPress.org and I didn’t want to have to deal with that. Turns out I don’t have to! WordPress itself supports many of the features of LaTeX. You simple write “$latex” before your math equation, use standard LaTeX syntax, and then close with another “$”. Just like in LaTeX itself!! Below are two example equations I presented in my post:

Ridge Regression: \mathbf{\hat{\beta}^{ridge} = \displaystyle arg\min_{\beta}\sum^{n}_{i=1}(y_i-(\beta_0 + \beta^Tx_i))^2 + \lambda\|\beta\|_{2}^{2}}

Lasso Regression: \mathbf{\hat{\beta}^{lasso} = \displaystyle arg\min_{\beta}\sum^{n}_{i=1}(y_i-(\beta_0 + \beta^Tx_i))^2 + \lambda\|\beta\|_{1}}

Lasso and Ridge Regression in R

Lasso and ridge regression are two alternatives – or should I say complements – to ordinary least squares (OLS). They both start with the standard OLS form and add a penalty for model complexity. The only difference between the two methods is the form of the penality term. Ridge regression uses the \mathbf{\mathit{l}_{2}}-norm while lasso regression uses the \mathbf{\mathit{l}_{1}}-norm. Specifically the forms are shown below.

Ridge Regression: \mathbf{\hat{\beta}^{ridge} = \displaystyle arg\min_{\beta}\sum^{n}_{i=1}(y_i-(\beta_0 + \beta^Tx_i))^2 + \lambda\|\beta\|_{2}^{2}}

Lasso Regression: \mathbf{\hat{\beta}^{lasso} = \displaystyle arg\min_{\beta}\sum^{n}_{i=1}(y_i-(\beta_0 + \beta^Tx_i))^2 + \lambda\|\beta\|_{1}}

Ridge regression has a closed form solution that is quite similar to that of OLS: \mathbf{\hat{\beta}^{ridge} = (X^TX + \lambda I)^{-1}X^Ty} The only difference is the \mathbf{\lambda I} term. I read (or heard) somewhere that ridge regression was originally designed to deal with invertability issues in OLS. By adding the small \mathbf{\lambda I} term we get the inner product to move away from possible singular matrices.

In ridge regreession when \mathbf{\lambda} is close to zero the ridge solution approachese the least squares solution. As \mathbf{\lambda} increases the solutions are forced toward zero. You can see this behavior graphed below for a sample dataset.

CoefvLambda

The code to produce this graph is quite simple.

# Load libraries
library(MASS)
library(reshape)
library(psych)
library(ggplot2)
 
# Get data
data read.csv(file = "Masked")
# Prepare data
X data[,-10]
X scale(X, center = TRUE, scale = TRUE)
response data[,10]
response scale(reponse, center = TRUE, scale = FALSE)
 
# Get ridge solution for every lambda
betas matrix(rep(NA), nrow = 7, ncol = 100)
rownames(betas) colnames(X)
lambda 1:100
betas lapply(1:length(lambda),
               FUN = function(i) {  
               return(solve(t(X)%*%X + diag(x = lambda[i], 
               ncol(X)) )%*%t(X)%*%response)  
               })
 
# Prepare data for plot
betas data.frame(t(betas))
betas cbind(lambda, betas)
betas (betas, id = "lambda", variable_name = "coviariate")
 
# Plot
ggplot(betas, aes(lambda, value)) + 
geom_line(aes(colour = coviariate)) + 
ggtitle("A Plot of Coefficients vs. Lambda for Shellfish Data") + 
xlab("Lambda") + 
ylab("Coefficient Value") + 
theme(legend.position = "none")

Lasso on the other hand does not have a closed form solution so numerical techniques are required. The benefit of lasso, however, is that coefficients are driven to exactly zero. In this way lasso acts as a sort of model selection process. The reason ridge regression solutions do not hit exactly zero, but lasso solutions do is because of the differing nature of the geometry of the \mathbf{\mathit{l}_{1}}-norm and \mathbf{\mathit{l}_{2}}-norm. In two dimensions the difference can be viewed in the picture below. Lasso regression constraints make a diamond aligned with the \mathbf{\beta_{2}} axis (in this case). The contours inscribed by the solutions (the red circles) can easily intersect the diamond at it’s tip and force \mathbf{\beta_{1}} to zero. This is much more unlikely in the case on the right for ridge regression where the \mathbf{\mathit{l}_{2}}-norm constraint inscribes a circle (in fact there is zero probability that \mathbf{\beta_{1}} will be exactly zero).

Screen Shot 2014-04-19 at 11.19.00 PM

For lasso regression, then, solutions end up looking like the graph below. As \mathbf{\lambda} increases each coefficient is eventually driven to zero.

betavloglambda

All of this begs the question, “what lambda values is ideal?” To determine this cross validation (CV) is often used. For each \mathbf{\lambda} it is possible to use CV to obtain a good estimate of the model error. You then repeat this process for all \mathbf{\lambda} within some range. The plot ends up looking like the graph below where the green line is the result of a general cross validation process and the red line uses leave-one-out cross validation. It’s interesting to note that while CV sounds like an iterative process (and in concept it is) there are fairly simple closed-form solutions for computing the CV error. I was very surprised when I first learned that! You can see below that our model should set lambda to around 4 or 5 to minimize the error (the precise value can be found by examining stored output).

CV Scores

The code to produce this graph is shown below.

# Get data
data read.csv(file = "Masked")
# Prepare data
X data[,-10]
X scale(X, center = TRUE, scale = TRUE)
response data[,10]
response scale(response, center = TRUE, scale = FALSE)
 
lambdaSeq 1:100
LOOCVscore rep(0, length(lambdaSeq))
GCVscore rep(0, length(lambdaSeq))
 
# Get leave-one-out CV score
LOOCVscore lapply(lambdaSeq,
                     FUN = function(i) {
                     lambda [i]
                     hat = X %*% (solve(t(X) %*% X + 
                     lambda*diag(ncol(X))) %*% t(X))
                     prediction = hat %*% response
                     Lii = diag(hat)
                     return(mean(((response - prediction)/
                     (1 - Lii))^2)) 
                     })
 
# Get GCV score
GCVscore lapply(lambdaSeq,
                   FUN = function(i) {
                   lambda [i]
                   hat = X %*% (solve(t(X) %*% X + 
                   lambda*diag(ncol(X))) %*% t(X))
                   prediction = hat %*% response
                   Lii = tr(hat)/nrow(X)
                   return(mean(((response - prediction)/
                   (1 - Lii))^2)) 
                   })
 
CV as.data.frame(cbind(lambdaSeq, LOOCVscore, GCVscore))
CV (CV, id = "lambdaSeq", variable_name = "CVscores")
ggplot(CV, aes(lambdaSeq, value)) + 
geom_line(aes(colour = CVscores)) +
ggtitle("A Plot of Cross Valadiation Scores vs. Lambda") + 
xlab("Lambda") + 
ylab("Cross Validation Score")

Created by Pretty R at inside-R.org

Another common statistic often presented when doing lasso regression is the shrinkage factor, which is just the ratio of the sum of the absolute value of the coefficients for the lasso solution divided by that same measure for the OLS solution. Remember small \mathbf{\lambda} forces the lasso toward the OLS solution, which implies a shrinkage factor near one should also correspond with the lasso having a solution close to OLS. You can see that patten holds below. (Code to produce this graph is shown below).

Lassobetas shrinkage

As I mentioned there is no closed-form solution that can determine the coefficients in the case of lasso regression. There are a number of methods to search through and find these betas, however. Luckily, minimizing the lasso solution results in a convex optimization problem so given enough iterations we can converge very close to the global solution. The method presented below is my implementation of the shooting algorithm. The shooting algorithm is stochastic: it picks a random beta, removes it from the model, calculates the effect of this deletion, and assigns the left out beta a value proportional to this effect, which is based on a formula I won’t present here. We were meant to determine our own criteria for stopping the loop (you could theoretically repeat this process forever). I choose a method that makes sure all betas have been updated and then calculates the MSE of the current model. If it’s within some tolerance – playing around the number suggested .001 was pretty good – the loop stops and we move on to the next lambda value. Otherwise we must continue iterating until all betas have been updated once more and the tolerance check repeats.

We were told to plot our iterations as a function of the shrinkage factor. The performance using my convergence criteria was much better than with the 1,000 iterations we were told to start with as a test case. The median number of iterations for my solution to converge quite close to the 1,000-iteration method was 103, meaning my algorithm improved performance by 10 times (at least measured by iterations).

iterationsvsConv

# Read in data
data read.csv(file = "Masked")
 
# Perpare data
X as.matrix(data[,-10])
X scale(X, center = TRUE, scale = TRUE)
response data[,10]
response scale(response, center = TRUE, scale = FALSE)
 
# Initialize variables
set.seed(35)
lambdaSeq seq(from = 1, to = 501, by = 10)
betaVectorConv matrix(NA, ncol = 7, nrow = length(lambdaSeq))
iterationsVector rep(NA, length(lambdaSeq))
aj 2*colSums(X^2)[1]
 
# Get lasso solution using shooting algorithm for each lambda
for(i in 1:length(lambdaSeq)) {
lambda [i]
betas solve(t(X) %*% X) %*% t(X) %*% response
betaUpdate rep(NA,7)
exit = FALSE
iterations  0
MSECurrent 0
MSEPast 0
 
  while(exit == FALSE) {
    j sample(1:7,1)
    betaUpdate[j] 1 
    iterations 1 
    xMinJ [,-j]
    betaMinJ [-j]
    xj [,j]
    cj 2*(t(xj) %*% (response - xMinJ %*% betaMinJ))
 
    if(cj < -lambda)      { betas[j] (cj + lambda)/aj } 
    else if(cj > lambda)  { betas[j] (cj - lambda)/aj } 
    else                  { betas[j] 0               }
 
    if(sum(is.na(betaUpdate)) == 0) {
    yHat sum((response - yHat)^2)
      if(abs(MSECurrent - MSEPast) < .001) {
      iterationsVector[i] TRUE
      }
      MSEPast rep(NA,7)
    }   
  }
  betaVectorConv[i,] }
 
#Calculate shrinkage
colnames(betaVector) colnames(X)
betaLS solve(t(X) %*% X) %*% t(X) %*% response
betaLS sum(abs(betaLS))
betaLasso abs(betaVectorConv)
betaLasso apply(betaLasso,1,sum)
betaShrink # Prepare shrinkage data for plot
betaShrinkPlot as.data.frame(cbind(betaShrink,betaVector))
betaShrinkPlot (betaShrinkPlot, id ="betaShrink",
variable_name = "betas")
 
# Plot shrinkage
ggplot(betaShrinkPlot, aes(betaShrink, value)) + 
geom_line(aes(colour = betas)) +
ggtitle("Plot of Beta Values vs. Shrinkage Factor with Conv. Crit.") + 
xlab("Shrinkage Factor") + 
ylab("Beta Value")
 
# Prepare and plot iterations
betaIterPlot as.data.frame(cbind(betaShrink, iterationsVector))
ggplot(betaIterPlot, aes(betaShrink, iterationsVector)) + 
geom_point(aes(colour = iterationsVector)) +
ggtitle("Plot of Iterations vs. Shrinkage Factor with Conv. Crit.") + 
xlab("Shrinkage Factor") + 
ylab("Iterations")

Created by Pretty R at inside-R.org

LaTeX in Stata

When doing my second HW for correlated data I came across some Stata packages that export LaTeX code directly. They are pretty similar to xTable in R. The first is the latabstat command, which is great at producing summary statistics. It outputs a LaTeX file that you can cut and paste into whatever program you are using (I use TexMakeker).  Sometimes a little additional formatting is needed, but overall it’s a much better alternative to creating LaTeX tables from scratch. Here are a few examples.

Screen Shot 2014-04-19 at 9.48.28 PM

Screen Shot 2014-04-19 at 9.48.43 PM

The second is eststo function, which is part of a separate package you have to install. It’s great for getting regression output together in a table. You can call the eststo function multiple times and then use the esttab function, which will collate all results that were prefixed with eststo. See the code below for an example use. Here are a few result tables. Again, a little modification is needed, but they look pretty good straight out of Stata.

Screen Shot 2014-04-19 at 9.48.52 PM

Screen Shot 2014-04-19 at 9.49.15 PM

Screen Shot 2014-04-19 at 9.49.24 PM


//  Load data 
C:\Users\jjmacky\Documents\Stat 540 HW 2
insheet using seizurewide.csv, clear

//  Inspect data 
br
codebook
inspect

/************ Problem 1 ************
//  Part i. 
//  Examine data by treatment status 
//  Normalize data so all variables are 
//  in terms of counts per week 
replace bl = bl/8
replace y1 = y1/2
replace y2 = y2/2
replace y3 = y3/2
replace y4 = y4/2

//  Simple table output 
tabstat y1 y2 y3 y4 bl age, by(trt) stat(n mean sd) long format ///
nototal save
latabstat y1 y2 y3 y4 bl age, by(trt) stat(mean sd) long format  ///
nototal


//  Part ii. 
//  Examine data by age 
//  Create age groups 
gen age_group = 4 if age replace age_group = 3 if age < 40
replace age_group = 2 if age < 30
replace age_group = 1 if age < 20
label define age_label 1 "Teen" 2 "Twenties" 3 "Thirties" 4 "Forties"
label values age_group age_label
//  Produce table 
tabstat bl y1 y2 y3 y4, by(age_group) stat(n mean sd) long format
latabstat bl y1 y2 y3 y4, by(age_group) stat(mean sd) long format
//  Produce graph
scatter y time, by(age_group) ytitle("Seizures per Week") xtitle("Time")

//  Part iii. 
//  Produce spaghetti plots 
//  Run this after normalizing data by seizure rate 
reshape long y, i(id) j(time)
drop if trt == 1
spagplot y time, id(id) ytitle("Seizures per Week") xtitle("Time")
//  Run this seperatly 
reshape long y, i(id) j(time)
drop if trt == 0
spagplot y time, id(id) ytitle("Seizures per Week") xtitle("Time")
//  Another possible plot 
label define trt_label 0 "Control" 1 "Treatment"
label values trt trt_label
meansdplot y time, by(trt) inner(1) ytitle("Seizures Per Week")  /// 
xtitle("Time")

//  Part B, C, D 
//  Run regressions 
//  Create variables 
gen yplus1 = y + 1
gen log_yplus1 = log(yplus1)
gen log_seizures = log(y)
gen log_age = log(age)
gen log_bl = log(bl)
//  Remove observations with no seizures (we are told to do this) 
by id (time), sort: drop if y == 0
//  Run clustered regression 
eststo: quietly regress log_seizures trt log_age log_bl, cluster(id)
//  Compare to no clustering 
eststo: quietly regress log_seizures trt log_age log_bl, robust
//  Now repeat with yplus1 as the response 
eststo: quietly regress log_yplus1 trt log_age log_bl, cluster(id)
eststo: quietly regress log_yplus1 trt log_age log_bl, robust
//  Produce Latex Result 
esttab using reg1.tex, replace


//  Part E 
//  Test effects on different times 
//  Run without reshaping 
//  Transform variables 
gen log_y1 = log(y1)
gen log_y2 = log(y2)
gen log_y3 = log(y3)
gen log_y4 = log(y4)
gen log_age = log(age)
gen log_bl = log(bl)
//  Run regressions 
eststo: quietly regress log_y1 trt log_age log_bl, cluster(id)
eststo: quietly regress log_y2 trt log_age log_bl, cluster(id)
eststo: quietly regress log_y3 trt log_age log_bl, cluster(id)
eststo: quietly regress log_y4 trt log_age log_bl, cluster(id)
//  Produce Latex Result 
esttab using reg2.tex, replace

/************ Question 2 ************/
insheet using dentalwide.csv, clear
egen numsubject = seq()
drop subject
//replace subject = numsubject 
reshape long distance, i(numsubject) j(agefrom8)
replace agefrom8 = agefrom8 - 8
gen male = cond(sex == "Male", 1, 0)

xtset numsubject agefrom8
eststo: quietly xtgee distance c.agefrom8##male, corr(independent) robust
eststo: quietly xtgee distance c.agefrom8##male, corr(exch) robust
eststo: quietly xtgee distance c.agefrom8##male, corr(ar1) robust
eststo: quietly xtgee distance c.agefrom8##male, corr(unstr) robust
esttab using gee.tex, se nostar replace

Formatted By Econometrics by Simulation

Microsoft Access

I’m learning Microsoft Access as part of a class from UW’s Information School. A lot of people are down on Microsoft, but in my opinion they usually do a good job with Office products. Access is pretty easy to learn and use and it seems powerful. It’s actually a great way to organize information even if you aren’t going to use much of the database functionality. I put some of the World Bank LSMS data I’ve been analyzing for my thesis and it’s a great way to visualize the linkages. The World Bank data has over 30 files with various one-to-one, many-to-one, and one-to-many relationships and Access seems like it would be a good platform to get a handle on the data. Although you can’ t do hardcore data analysis with Access, I think for simple data exploration using reports, queries, and forms could be quite useful.

Below are a couple of screenshots of some of the project assignments we’ve had that simulate a community college administration office.

Relations

Relationships 2

Markov Chain Monte Carlo Model Composition

One important element of prediction and classification using regression methods is model selection. A first principle of most models is parsimony, we want to add as few predictors (explanatory variables) as is necessary to do the best we can at predicting our feature set (outcome). Often not every predictor is necessary so there is some combination that produces a balance between parsimony and accuracy.

There are simple tests to help find a model with this balance.  One such test is the Akaike Information Criterion (AIC) named after the Japanese statistician Hirotugu Akaike (who was quite avuncular if his Wikipedia profile picture is any indication of his temperament). In math speak, the AIC maximizes the likelihood function by using convex optimization and has an added penalty as more predictors are added to the model.

There are simple ways to select a model. One easy way is to try every possible combination of predictors and see which one has the best AIC. At some point adding additional predictors will fail to produce much improvement in predicting the feature set and we know we can stop. This is pretty easy if you have five or ten predictor, but the number of models to test grows exponentially.

For example, a recent dataset we were given for homework in a graduate statistics course I’m taking, included 60 predictors. Each predictor can be either added to the model or left out resulting in \mathbf{2^{60}} (\mathbf{10^{18}}) possible models (including the null model with only the intercept). To give a sense of how large that is, it’s beginning to approach the number of starts in the observable universe (\mathbf{10^{24}}). Even modern computers would take too long to test every possible model.

And some models have even more variables. Many more. High-dimensional models with thousands of predictors are not uncommon. One popular example in machine learning uses MRI images where each pixel (called a voxel in this setting because it represents a 3-D slice of the brain) represents a predictor. Here there are 20,000 possible predictors: each voxel of the image can be in or out of the model, representing weather that part of the brain is important in predicting some neurologic outcome. (In practice particular – as common sense would imply – there is some dependency in choosing voxels, which makes the task slightly different).

One alternative to testing every possible combination of predictors is using a Markov Chain Monte Carlo Model Composition(MC3). It’s just a fancy way of saying that we choose a model at random and then move randomly (but intelligently) toward models that have a better AIC score.

We were asked to implement such an algorithm recently for my class. It was a pretty fun project that took some creative thinking to make efficient. The MC3 algorithm is pretty simple. We pick a random model with a random number of predictors. Then we look at all of the valid models that leave one of these predictors out as well as the valid models that add one predictor. This is called the “neighborhood” of the model. We choose one of these new models in the neighborhood at random and compute a modified version of the AIC. If it’s better than our current model’s AIC we move to that model and repeat the process. Otherwise we randomly select another candidate model, compute it’s AIC, compare scores to the current model and repeat until we move to a new model.

We were given several starter functions by our professor. All three are pretty trivial to implement oneself aside from isValidLogisticRCDD(), which implements a model validity test from a statistics paper using deep theoretical concepts far beyond mathematical knowledge.

My results are shown below. We were meant to run the algorithm for 10 instances of 25 iterations each. I store all of the relevant information in a list so when the algorithm is done running I get a nice little output. I just realize the “Iteration [number]” at the top of each column should really say “Instance”. Oops!! Just went back and fixed that in my code.

Screen Shot 2014-04-18 at 7.22.21 PM

The R code is shown below. It looks a little funky because we are transitioning to writing in C and our professor wants us to start getting into the C mindset. I’m also trying to write more loops using the apply() function instead of the traditional for loop. There are all kinds of debates online about under what cases apply() is faster than a for loop and if it’s even true in general anymore. At any rate it’s good to familiarize myself with different techniques and look at problems in different ways. I wrote several helper functions to find a models neighborhood and test the valid members in it. The MC3 function appears near the bottom along with a main() function.

#######################################################
# James McCammon 
# Description:
# MC3 uses several custom helper functions to determine
# the best AIC of a model using a markov chain process.
#######################################################
 
 
#######################################################
# Given Functions
#######################################################
getLogisticAIC <- function(response,explanatory,data)
isValidLogistic <- function(response,explanatory,data)
isValidLogisticRCDD <- function(response,explanatory,data)
 
 
 
##########################################################
# Custom Functions
##########################################################
 
############### checkModelValidity Function ###############
# Checks if a particular model is valid by first doing
# preliminary empirical test and then moving on to a more
# robust theoretical test
checkModelValidity <- function(response,explanatory,data) {
  prelimCheck <- isValidLogistic(response,explanatory,data)
  if(!prelimCheck) { return(FALSE) }
  robustCheck <- isValidLogisticRCDD(response,explanatory,data)
  if(!robustCheck) { return(FALSE) }
  return(TRUE) 
}
 
############### findNeighboursBelow Function ###############
# Given an arbitrary vector this function finds the set of
# all vectors generated by leaving one element out.
findNeighboursBelow  <- function(elements) {
  if(is.null(elements)) { return(NULL) }
  length = length(elements);
  neighbours = vector("list",length);
  neighbours = lapply(1:length,
                      FUN = function(i) {
                      return(elements[-i])
                      })
  return(neighbours);
}
 
############### findNeighboursAbove Function ###############
# Given an arbitrary vector this function finds the set of
# all vectors generated by adding an element not already in
# the set given an upper bound of possible elements to add.
findNeighboursAbove <- function(elements, last) {
  fullModel = seq(1,last);
  notInModel = setdiff(fullModel,elements);
  length = length(notInModel);
  neighbours = vector("list",length);
  neighbours = lapply(1:length,
                      FUN = function(i) {
                      return(union(elements, notInModel[i]))
                      })
  return(neighbours);
}
 
############### findValidNeighboursAbove Function ###############
# This function finds all of the valid neighbouring models produced
# by using the findNeighboursAbove function.
findValidNeighboursAbove <- function(response, explanatory, last, data) {
  neighbours = findNeighboursAbove(explanatory,last);
  length = length(neighbours);
  keep = rep(NA, length);
  keep = lapply(1:length,
                FUN = function(i) {
                return(checkModelValidity(response, 
                neighbors[[i]], data))
                })
  validNeighboursAbove = neighbours[as.logical(keep)];
  return(validNeighboursAbove);
}
 
##########################################################
# MC3 Function
##########################################################
 
MC3 <- function(response, explanatory, iterations, instances, 
lastPredictor) {
  require(rcdd);
  finalResults = NULL
 
  for(k in 1:instances) {
    # Make sure you start with a valid model
    initialModelValid = FALSE;
    while(!initialModelValid) {
      modelSize = sample(explanatory,1);
      initialModel = sample(explanatory, modelSize);
      initialModelValid = checkModelValidity(response, initialModel,      data);
    }
 
    # Get measures for initial model
    currentModel = initialModel;
    currentAIC = getLogisticAIC(response,currentModel,data);
 
    # Check to see if we should start at the null model
    nullModelAIC = getLogisticAIC(response = response, explanatory =    NULL, data = data)
    if(nullModelAIC < currentAIC) {
      currentModel = NULL;
      currentAIC = nullModelAIC;
      initialModel = NULL;
    }
 
    # Store initial AIC for later output.
    initialModel = currentModel;
    initialAIC = currentAIC;
 
    # Get valid neighbors of current model
    currValidNeighAbove = findValidNeighboursAbove(response, 
    currentModel, lastPredictor, data);
    currValidNeighBelow = findNeighboursBelow(currentModel);
    currAllNeigh = c(currValidNeighAbove, currValidNeighBelow);
    # Compute score
    currentScore = -currentAIC - log(length(currAllNeigh));
 
    # Iterate for the number of times specified.
    for(i in 1:iterations) {   
      # Compute measures for the candidate model
      candidateIndex = sample(1:length(currAllNeigh),1);
      candidateModel = currAllNeigh[[candidateIndex]];
      candValidNeighAbove = findValidNeighboursAbove(response, 
      candidateModel, lastPredictor, data);
      candValidNeighBelow = findNeighboursBelow(candidateModel);
      candAllNeigh = c(candValidNeighAbove, candValidNeighBelow);
 
      candidateAIC = getLogisticAIC(response, candidateModel, data);
      candidateScore = -candidateAIC - log(length(candAllNeigh));
 
      # If necessary move to the candidate model.
      if(candidateScore > currentScore) {
        currentModel = candidateModel;
        currAllNeigh = candAllNeigh;
        currentAIC = candidateAIC;
        currentScore = candidateScore;
      }
      # Print results as we iterate.
      print(paste('AIC on iteration', i ,'of instance', k , 'is', 
      round(currentAIC,2)));
    }
 
    # Produce outputs
    instanceResults = (list(
                       "Instance"            = k,
                       "Starting Model Size" = modelSize,
                       "Initial Model"       = sort(initialModel),
                       "Initial AIC"         = initialAIC,
                       "Final Model"         = sort(currentModel),
                       "Final AIC"           = currentAIC));
    finalResults = cbind(finalResults, instanceResults);
    colnames(finalResults)[k] = paste("Instance",k)
  }
  return(finalResults);
}
 
##########################################################
# Main Function
##########################################################
 
main <- function(data, iterations, instances) {
  response = ncol(data);
  lastPredictor = ncol(data)-1;
  explanatory = 1:lastPredictor;
  results = MC3(response, explanatory, iterations, instances, 
  lastPredictor);
  return(results);
}
 
##########################################################
# Run Main
##########################################################
setwd("~/Desktop/Classes/Stat 534/Assignment 3")
data = as.matrix(read.table(file = "534binarydata.txt", header = 
FALSE));
main(data, iterations = 25, instances = 10);

Created by Pretty R at inside-R.org

We were meant to run 10 instances of the algorithm for 25 iterations each. My best run was instance 8. I print out the AIC at every iteration to help keep track of things and give me that nice reassurance that R isn’t exploding at the sight of my code, which is always a fear when nothing is output for more than 5 minutes.

[1] "AIC on iteration 1 of instance 8 is 123.21"
[1] "AIC on iteration 2 of instance 8 is 104.17"
[1] "AIC on iteration 3 of instance 8 is 104.17"
[1] "AIC on iteration 4 of instance 8 is 97.31"
[1] "AIC on iteration 5 of instance 8 is 88.52"
[1] "AIC on iteration 6 of instance 8 is 88.25"
[1] "AIC on iteration 7 of instance 8 is 88.25"
[1] "AIC on iteration 8 of instance 8 is 86.64"
[1] "AIC on iteration 9 of instance 8 is 86.64"
[1] "AIC on iteration 10 of instance 8 is 83.67"
[1] "AIC on iteration 11 of instance 8 is 83.67"
[1] "AIC on iteration 12 of instance 8 is 83.67"
[1] "AIC on iteration 13 of instance 8 is 81.77"
[1] "AIC on iteration 14 of instance 8 is 81.77"
[1] "AIC on iteration 15 of instance 8 is 56.96"
[1] "AIC on iteration 16 of instance 8 is 56.96"
[1] "AIC on iteration 17 of instance 8 is 54.45"
[1] "AIC on iteration 18 of instance 8 is 54.45"
[1] "AIC on iteration 19 of instance 8 is 52.69"
[1] "AIC on iteration 20 of instance 8 is 52.69"
[1] "AIC on iteration 21 of instance 8 is 52.69"
[1] "AIC on iteration 22 of instance 8 is 50.71"
[1] "AIC on iteration 23 of instance 8 is 50.71"
[1] "AIC on iteration 24 of instance 8 is 50.71"
[1] "AIC on iteration 25 of instance 8 is 50.71"

Created by Pretty R at inside-R.org

I Finally Learned LaTeX!

I finally got around to learning LaTeX last night, which is a document preparation system for math and statistics papers. The best way I know how to describe it is like HTML, but for documents. It’s much easier to learn and use than HTML, but does have a particular syntax to make text bold, add section headers, and so on. Here’s a screenshot to give a feel of the interface (I use TexMaker). The code on the left produces the PDF on the right.

Screen Shot 2014-04-09 at 12.53.52 AM

LaTeX documents have a very distinctive look. You may recognize the format if you’ve taken a math class in college because many professors write their assignments using it.

I was even able to find that the listings package does syntax highlighting. Unfortunately, Stata is not supported. I played around with telling LaTeX that I was using Python, Java, etc. until the highlighting rules for that language matched somewhat with the way Stata should be highlighted. Java was the best match I found. There is also a package called minted, but it requires a little more work for installation and I didn’t have the time to mess around with it. I’m not even sure if it supports Stata, but it does mention support for over 150 languages. Here is the final product, which looks pretty good I think.

Screen Shot 2014-04-09 at 1.00.04 AM

Screen Shot 2014-04-09 at 1.00.16 AM

Screen Shot 2014-04-09 at 1.00.32 AM

Screen Shot 2014-04-09 at 1.00.42 AM

The Firm as Economic Integrator

Postscript

I once got paid $18 an hour to edit graduate papers for a year. I had some rules. One rule was that the quality of a paper was usually inversely related to the number of times the word “modalities” was used. If you’re using “modalities” you’re likely trying too hard. There’s a better way to say what you need to say. Too many grad students mimic Foucault. They don’t understand that Foucault was a great thinker, but not a great writer, at least if writing is meant to be read. Unfortunately, most graduate students seem to write to show other graduate students and their professors how smart they are. If they were actually smart they would take the time to learn how to write well.

If I use my own rules to judge the quality of this paper it doesn’t do so well. In addition to being poorly written, it doesn’t give due care to the economic sociology literature despite attempting to be an economic sociology paper. Use it as an example of how not to write.

Abstract

I argue that the use of the phrase “the market” when defining and describing economic integration is both a rhetorical and logical mistake. That a more useful and accurate mode of thought is to envision the firm as the institution that generates exchange and acts as the epicenter of a broader, society-wide economic integration. I claim that this new theory allows for a better description of the real world. I conclude with a brief discussion of some applications.

Economic Integration

Traditional economic analysis has focused on “the market” as the name to describe the coming together and exchanging of goods by people throughout history, thereby subsuming economic integration within material exchange, what Mark Granovetter called “undersocialized” analysis (Granovetter 1985)[1]. Karl Polanyi reinterpreted the market process as what he termed, “instituted economic integration,” and attempted to extricate different types of integration across time and space that had become confounded by traditional analysis in the social sciences (Polanyi 1957). In doing so, Polanyi hoped to give a more nuanced view of the specific type of integration involved in the modern market, making analysis contingent on historical particularities.

Perhaps equally important as Polanyi’s primary argument that distinct modes of economic exchange have existed across time and space is the implication that economic exchange itself is a type of “integration.” It is a binding force that both carries with it a particular bundle of social relations and is itself embedded within an ongoing and evolving social context (for more on embeddedness see Granovetter 1985).

Whereas Polanyi’s project was an attempt at moving away from the market to a more macohistorical view that allowed for “the market” to become referable in a context outside of itself, I suggest moving to a more microeconomic-sociological context where the firm is viewed as the main actor in analysis of economic integration.

“The Market” – A Rhetorical (and Logical) Mistake

One key problem in traditional economic analysis is a rhetorical one: the variegated meaning of the term “market.” It wants to mean so many things. On one hand it is a higgle-haggle style bazaar; on another it is a once-a-week pop-up style gathering where people sell trinkets or flowers or banjos made of cigar boxes; on yet another it is the corner grocer (as in “I’m going to the market to buy some milk”); and on still another it is the “marketplace” of ideas. We could continue: labor markets, stock markets, commodity markets, currency markets. What all of these types of “markets” have in common is that they haven’t change much over time (aside from technology perhaps). Further, they all invoke images in which half of the parties have something to sell and the other half have the money to buy, and each is in a sort of dance looking to find their perfect partner and make an exchange.

It is no surprise then given these uses of the term “market,” that the phrase “Market Economy” or “Free Market” conjures visions of people moving freely about their daily business, at some times buying certain goods while neglecting others (thereby creating “demand crowds”) and at other times working in some business or another for production (thereby creating “supply crowds”). The problem is that what constitutes the economic component of one’s daily life has changed greatly over time, and so the term presuppose a certain generality that is at odds with particular historical episodes. This is not simply true across “modern” and “pre-modern” societies, but within modernity itself. One example is what Alfred Chandler chronicled in his book The Visible Hand (Chandler 1977).

Chandler outlines roughly two periods. The first, 1790 up until the 1840s, was a period of increased specialization—a movement from general merchants to specialized enterprises. “In the 1790s…[the merchant] carried out all the basic commercial functions. He was an exporter, wholesaler, importer, retailer, shipowner, banker, and insurer. By the 1840’s, however, such tasks were being carried out by different types of specialized enterprises (Chandler 1977 pg. 15).” In the second period, 1850 to the post-WWII era, Chandler recounts the rise of the managerial firm, starting with the railroads. “A small number of large, managerially administered enterprises replaced a large number of the small personally run transportation, shipping, and mercantile firms that had previously carried goods from one transshipment point to another (Chandler 1977 pg. 122).” This required “a type of cooperation between business enterprises [that] was an entirely new phenomenon (Chandler 1977 pg. 123).”

So by the late 1860’s there had emerged a type of managerial firm with new organizational structures, interfirm connections, and social relations. Chandler’s project is to show the significant ways in which these firms changed over time. Indeed, the types of firm structures, interfirm connections, and social relations that were present in 1860, or even in 1960, cannot be assumed in whole to describe the firms of 2011 because they simply do not. This profound evolution, however, is obfuscated by deference to “the market” as the main driver of economic integration. In each epoch, then and now, market mechanisms were at work determining prices and allocating resources (with varying degrees of government intervention). But this processes is quite different than the one of economic integration—the means by which one satisfies material wants within a social context because the social context itself has changed. So too has the variety and type of goods available for material want drastically increased. These two subjects—satisfaction of material wants and the social context in which it takes place—are the two components that make up economic integration.

Aggregation of economic integration to the market level can at most tell us that social relations are transmitted during exchange, but tells us very little about the nature of the specific relations involved or how they might have changed over time. It can tell us that reciprocity and redistribution involved different social relations than the modern system, but leaves to the imagination a richer description. Shifting the focus of economic integration from the aggregated marketplace to the firm level can solve this problem.

The Firm as Economic Integrator

Focusing on the firm as the economic integrator has two logical underpinnings. In the first place, firms are the engine of the market, fueling both the supply of, and demand for, the panoply of exchangeable goods and services. Whereas in past societies, as Polanyi demonstrated, forms of redistribution or reciprocity might have been the means to satisfy material wants, in a modern society individuals integrate via firms—they both purchase goods from firms and simultaneously act as producers for firms to earn income.

In the second, the parallel rise of capitalism and the managerial firm, has created a new component of identity that was previously nonexistent (or at least extant in a very different form). And so firms more clearly identify and delineate the social relations involved in economic exchange than does “the market.” In short, people don’t identify much with the statement, “I am an actor in the broader market economy.” However, they might very much identify with the statement, “I’ve been an engineer at The Boeing Company for twenty-five years.”

Along with the rise of capitalism (both pre- and post-industrial) there has been a parallel increase in the division of labor, in employment specialization, and, since 1850, in the proliferation of the managerial firm. One effect has been a perpetual increase in the choice of goods available to consumers (so great that it led psychologist Barry Schwartz to coin the term “The Paradox of Choice”(Schwartz 2004)). So while in 1790, as Chandler notes, families self-produced much of their own goods themselves, as the choice and complexity of goods has increased it is all but impossible to satisfy material wants alone, or even in a small group. Thus there has been a simultaneous increase in material wants and a decrease in the ability for self-production. This is not to say that the material well-being of consumers has suffered as a result—quite the opposite. It is simply to point out that this process has resulted in the rise of the firm as the central node of economic integration.

The Firm as the Engine

We can see the importance of the firm in satisfying material wants by briefly examining the broader context of the economy. First, let us turn our attention to the National Income Identity:

Y = C + I + G + (X-IM)

That is, the demand for aggregate products and services, and indeed any particular product or service, is given by the sum of the demand from consumers (C), from firms (here ‘I’ stands for firm Investment), and from government (G). The notation (X-IM) simply denotes the net sum of exports minus imports. Exports are generated at the firm level as supply to those outside a particular geo-political region, and imports are the demand for goods and services, themselves provided by outside firms. So ‘X-IM’ and ‘I’ clearly emanate from the firm, but so does demand from consumers (C) since their wages also originate at the firm level. Government may seem exempt from our calculation, but anything more than a cursory glance will reveal that: (a) much government spending redounds upon firms, and (b) government accrues tax revenue from wages, again paid by firms. This is not to say government has no part to play in the economy, indeed they are likely the institution with the most important role.

This is simply to say that firms are the dominant player in economic integration (a more complete theory will need to further examine the roll of government in economic integration, but this is a project for a later date).

Integration from the supply side is even more evident. If supply does not come from firms, from where does it originate? It is true that a very small number of individuals create products originating from their household and attempt to sell them at some sort of common marketplace shop. But the key is to recognize that these exchanges that are economically integrated outside of the firm-based market are but a patina of dust on the rich and gargantuan tapestry of exchange that takes place with the firm at its center. To imply that the amorphous “market” is the key economic integrating force is to overlook the firm as the very institutional social structure that acts as the engine of the market, providing both goods and services and the means to purchase them.

The Firm and Identity

The interconnectedness of firms and social identity is perhaps best demonstrated when discussing individuals who are economically disintegrated. Those with reduced economic integration define themselves, and are defined by others, by their exclusion from the firm. For example, the “unemployed”—the population that is looking to find a job. Where? With a firm. Discouraged workers—the group that has given up hope. Of what? Of finding a job with a firm. Retired persons—what have they retired from? From working for a firm. How do they live? From pensions, Social Security, 401Ks, or savings, all of which were accumulated during work with a firm. Informal workers—what do they do? The same sort of work that could be done by a firm, but usually for less money and often for friends or family.

We have touched thus far mostly on pecuniary matters, but it is important to remember that each of these transactions carry with them a set of social relations. This is the entire purpose of focusing on integration rather than simply exchange. The focus thus far has simply been to reconceive economic integration away from “the market” and to point out that while it may be a useful rhetorical device for discussing the mechanism of resource allocation, it is a silly way to discuss economic integration. We now turn our focus to the matter of social relations by embarking on two applications of our new theory.

Applications

Having severed our old definition and created a new one we must now discuss its implications.

Using this new theory we can make one immediate addendum to traditional economics and the so-called Theory of Value. Since the Marginal Revolution, traditional economics has theorized that value is determined when prices are transmitted back through the chain of production. Every purchase at the margin accrues with every other purchase, aggregately creating a demand for a good or service and imputing down through the chain of production the value of all resources required to render the good (in competition with all other final goods that require each of the same inputs). In this way the supply and demand for final goods also determines the value of both intermediate goods (including labor) and natural resources.

By analyzing economic integration at the firm level we see that an additional process is at work. Since social relations are also interconnected within the firm structure it is clear that along with price is a transmission of a dyad of social relations. First, an imputation of the value of both the resources and the social relations that acted together towards production. Since certain social structures within firms are more effective at producing particular goods than others, both act together towards production and so both are rewarded at purchase. This is one mechanism, but not the only, whereby social relations become embedded. Second, as Mark Granovetter referenced (see Granovetter 1985), empirical studies have shown that supplier-buyer relationships are often very social. These relationships, then, are also affected by imputation since interfirm contact is, at least to some extant, reliant on factors such as sales, new order shipments, and price changes. Consequently there is both an intrafirm and interfirm component of the social relation transmission when our new framework is applied to the traditional Theory of Value.

Let’s look more closely at the supply side social relations we just discussed, that is, the relations within firms. As economic integration has grown to be primarily instituted in the firm, and as firms have grown, it has engendered new types of social structures and redistributed power relations.

With the growth of larger firms dominated by managers power and obedience have evolved. The relations within firms are complex and cannot receive a full treatment here. However, it is clear that sometimes economic considerations dominate behavior within firms (as New Institutional Economics might predict) while at others they are subservient to social relations (as Economic Sociology argues). One obvious change chronicled by Chandler and illuminated by the latter field has been the increase in the types of organizational roles within firms. This process has redistributed power, but not in the way one might think.

In Why People Obey professors Gary Hamilton and Nicole Biggart use empirical data from organizational behavioral studies to argue convincingly that it is the organizational structures themselves that have power (Hamilton and Biggart 1985). Neither managers nor their superiors can act belligerently, not simply because of supervision and discipline from above, but because power and legitimacy themselves stem from gaining the respect of others within the firm. All employees, then, must role-play according to the requirements of their position. It is the structure and nature of these games where power and discipline originate.

But with the rise of the managerial firm there has also been a rise of rationality so that the roles to be played quite frequently impose a discipline of bureaucracy and ordinality: there must always be a process and steps must be taken in seriatim; all problems have solutions that can and must be solved, in one way or another. So there is an environment of achievement, of surpassing difficulties, of extraordinary reason. There are deep social norms that both reproduce those present in the larger society and are created emergently by the processes of the firm itself, projecting back upon society the logics of its inner culture.

The important factor is that these structures of power and obedience are markedly different than those that were present during the 1860s when the first managerial firms emerged. There were various types of firm arrangements in place as industrial capitalism began to develop and so too was there a method to satisfy material wants and an accompanying social structure—an economic integration. Importantly, though, these firm arrangements and their resulting forms of economic integration have been superseded since the rise of capitalism with the managerial firm and its new type of economic integration. But even these managerial firms have continued to change, along with society, itself and so the story continues. To say that “the market” was at work throughout this historical episode tells us nothing about the evolution of these social relations or the vastly different ways individuals have come to satisfy their material wants. A different model must be used, one centered on the firm itself.

Yet another application of our theory is to the drastic change in the way consumers purchase goods. Indeed, 2011 was a record setting year for online sales (BusinessInsider). As technology continues to usher in new methods of interfacing with firms, material wants will continue to be satisfied outside of traditional social settings. Standard economic analysis would examine this phenomenon only superficially, but a firm based theory of economic integration would allow an exploration of both the economic and sociological implications including the way firm structure, interfirm connections, and social relations might adapt to this change (again I leave this project for a later date).

Conclusion

We started by noting that economic integration is a particular view of exchange that is different than that normally analyzed by traditional economics. Some problems with this view were discussed, perhaps stemming from the mixed meaning of the word “market”, resulting in a poor framework if one wishes to discuss economic integration (as oppose to other types of economic analyses where the market level might be appropriate).

A new framework was proposed that uses “the firm” loosely defined as the main source of economic integration. We put to the side the interesting and larger question of how changes in firm structure that accompanied the rise of capitalism have influenced different types of economic integration over time. Instead, we focused only on the structure of managerial firms after the change has been “completed” (the process, of course, is ongoing).

Finally, two brief applications of the new theory were discussed. The first combined economic integration and the neo-classical Theory of Value, and showed that both value (in the monetary sense) and social relations were imputed with the purchase of goods. The second example demonstrated one area, power relations, where a change in economic integration has resulted in a change in power relations.

Herein has been a cursory treatment of this topic. The theory most certainly needs refined and must be subjected to a broader set of historical episodes and processes and must be examined to see if it holds up to more rigorous empirical analysis.

Endnotes

[1] As Granovetter demonstrates, even fields such as New Institutional Economics use a sort of robotic view of human behavior rooted in assumptions of behavior that stem from years of the economics discipline focusing on “the market.”

Bibliography

Anon. US Online Sales Reached A Record $6 Billion Last Week – Business Insider. http://articles.businessinsider.com/2011-12-05/news/30476321_1_comscore-chairman-gian-fulgoni-online-sales.

Chandler, Alfred. 1977. The visible hand: the managerial revolution in American business. Cambridge  Mass.: Belknap Press.

Granovetter, Mark. 1985. “Economic Action and Social Structure: The Problem of Embeddedness.” American Journal of Sociology 91 (3) (November 1): 481-510.

Hamilton, Gary G., and Nicole Woolsey Biggart. 1985. “Why People Obey: Theoretical Observations on Power and Obedience in Complex Organizations.” Sociological Perspectives 28 (1) (January 1): 3-28.

Polanyi, Karl. 1957. Trade and market in the early empires  economies in history and theory,. Glencoe  Ill.: Free Press.

Schwartz, Barry. 2004. The paradox of choice: why more is less. 1st ed. New York: Ecco.

 

The Future of the Greek Economy

Postscript

I wrote this paper as part of an intentional finance class I took from UW’s school of public affairs. It’s not the best thing I’ve ever written, but there’s some decent analysis in it. I wrote it before I learned Adobe Illustrator so there are some pretty ridiculous looking graphics. As I remember, we were suppose to put ourselves in the place of a policy advisor. The heading for mine was “Report for Mr. Yiannis Stournaras, Greek Minister of Finance.”

Executive Summary and Problem Statement

On Thursday, 2 August 2012 Mario Draghi, head of the European Central Bank (ECB), held his latest press conference. “The euro is irreversible,” he said, “It stays…It is pointless to bet against the euro.”[1] However, Mr. Draghi also made it clear that the continued flow of funds to troubled countries would come with requirements for economic reform.[2] While he did not announce immediate ECB action, Draghi did state that in upcoming weeks the ECB would begin the purchase of short-term Greek bonds in a continued effort to help the Greek economy.

However, the austerity measures that have been required by the ECB, and by its informal umbrella institution, the Troika, have been unpopular with the Greek people. In addition to austerity, two economic alternatives have been suggested in recent months – internal devaluation and an exit from the Eurozone – each claiming to provide the best chance of resolving the now five-year long Greek recession. Is continued Troika assistance really the best alternative?

This report will outline four possible policy options for the Greek government moving forward. Included is one package of reforms that are recommended independent of the other three policy options:

  • Maintaining the status quo and accepting Troika assistance
  • Leaving the Eurozone and returning to the Greek drachma
  • Defaulting on the current debt obligations

None of these solutions is perfect and all of them will require sacrifice and, likely, economic hardship in the short term. They also have varying degrees of political and social feasibility. After a detailed analysis it seems clear that the least harmful path is indeed Troika assistance including some form of austerity reforms. However, as will be suggested in the concluding recommendation, these reforms could possibly be restructured by negotiating with Troika officials. As unpopular as austerity has been with the Greek populace, other policy options would likely turn out to be equally unpopular and have the additional effect of hurting political relations between Greece and the rest of Europe.

The stakeholder positions of the various groups involved in the Greek debt crisis are shown in the stakeholder map below. Note two things. First, the interests of the stakeholders are far from aligned. Second, these are the interests as they stand today. Hedge funds, for instance, having already locked in their investment portfolio with Greek debt, care about full repayment and little else. Meanwhile, the interest of the Troika and various European countries are more complex.

This is not to say that these are the interest as they should be. For instance, Greece should probably shift some portion of its interest away from maintaining Greek entitlement payments and toward more governmental and business reforms. The discussion in this report will make this point clear.

Untitled

History

Untitled

The current crisis in Greece is a confluence of factors including steadily increasing public debt, stagnant government revenue, large public sector obligations, a history of government corruption and low transparency, a poor business environment, all combined with a European-wide slowdown in output.

Greece enjoyed fairly strong growth from 2001 through 2007, but has been very hard hit in the years since the Euro crisis started. Problems worsened throughout 2009 and 2010 with the release of a series of budget deficit revisions, most notably a change in the 2009 figure, which began at 3.7% of GDP[3] and eventually topped out at 15.4%.[4]

This only increased suspicions of Greek corruption in a country that already ranked among the lowest countries in all of continental Europe in the Corruption Perceptions Index. Indeed, Greece’s ranking steadily decreased between 2009 and 2011.[5]

Untitled

Unfortunately, Greece could ill afford such budget deficits, which added to an already mounting public debt that peaked at 167% of GDP in 2010. Government revenue, meanwhile, remained stagnate, hovering around 40% of GDP since 2001.[6]

Rather than decline, however, government expenditures have increased since 2008 from 45% to 50% of GDP.[7] This is a result of a shrinkage in GDP from the Greek recession, combined with a failure to adequately cut expenditures accordingly. A significant portion of government expenditure has been dedicated toward public employment and support programs. For instance, one report found that before the required ECB reforms, “On average, Greeks retired at 58 years old and received 96% of pre-retirement income.”[8]

As the Greek crisis worsened three financial institutions came together in what has become known as the Troika. These are the European Council, the ECB, and the International Monetary Fund (IMF). The Troika release two separate bailout funds: €110 billion in May of 2010 and a second €130 billion package in February of 2012.[9] All told, Greece debt stands at €356 billion.[10]

This brief history demonstrates the enormity of the problem facing Greece today. The following sections will discuss four separate policy options with the first being a set of necessary general reforms.

General Reforms

The current Troika loans are contingent on political and economic reforms that can be separated into two types. One type is a contentious set of so-called “internal devaluation” reforms. The other set contains a sensible set of government and business reforms that are long overdue. These should be undertaken regardless of the broader economic measures that will be discussed in the next part of this report. The Greek economy cannot plunge into a five-year recession and experience two elections in 2012 alone, only to keep things as they have been.

Tax reform

Tax payment is notoriously lax in Greece. Only about 25% of the €40 billion owed in taxes are collected annually.[11] The government can ill afford the loss of so much potential revenue in the middle of the worst debt crisis the country has ever seen. One option is to moderately reduce taxes, but drastically increase collection from 25% closer to the averages seen in most developed economies, thereby increasing government revenues while simultaneously giving concessions to tax payers.

Improved transparency

Greek political transparency must improve for its economic situation to improve. Greek government corruption has cost the country dearly, especially the fiscal budget deficits that were repeatedly covered up, leading to artificially high investment in Greece and worsening the credit situation. Implementation of consistent long-term measures to improve transparency and fiscal reporting will help restart foreign direct investment in Greece, but at levels that are commensurate with its true financial condition.

Business reform

Greece ranked 100 on the 2012 World Bank Doing Business Index. This is an extremely low ranking for a country of Greece’s stature and represents the significant challenges to starting and efficiently running a business in the country. The good news is that many other countries have made quick and substantial progress in reforming their own business environment. Such reform is necessary if Greece is to spur domestic and international investment, and would likewise improve the fortunes of the Greek people – 99.9% of businesses in Greece are small and medium-sized enterprises (SMEs).[12]

The government must also continue the fight against widespread abuse by business cartels. These cartels have caused prices in Greece to continue to rise even as the economy has shrunk by 13%.[13] This hurts consumers and limits the flexibility of government policy.

Benefits cuts

Public wages rose 50% between 1999 and 2007[14], an extraordinary rate. Cutting wages is a different discussion, but stopping the disproportionate increase in public sector wages is a necessary step to help stem the chronic deficit problem.

Keeping with the Status Quo

The first option is to maintain the current plan of receiving Troika bailouts and move forward with economic reforms. Again, one set of Troika requirements discussed above is necessary regardless of whether the rest of the bailout package reforms are accepted.

The second set of bailout reforms, geared toward creating a so-called “internal devaluation,” are more politically and socially contentious. The difficulty of these reforms has already come to light. After the June passage of an austerity package expected to save €28 billion, two days of violent riots ensued during which 300 protesters and police were injured.[15] Yet as of 4 August 2012 only 100 of the required 300 reforms had been completed.[16] Some of these are not set to take effect until 2013 or ‘14 and further Troika funding is unavailable until all 300 reforms are in place. Given the already widespread public discontent of the limited reform measures that have been enacted, when the full set are completely implemented public discontent is likely to grow, possibly resulting in further violence.

An internal devaluation uses internal structural changes to a country’s economic system in an effort to reduce production costs, and through this channel prices for goods. This improves competitiveness in world markets and increases exports, thereby spurring economic growth. As growth increases and public debts are reduced, the structural changes can be slowly reversed.

There are two possible methods to impose an internal devaluation. The first is a mandated decrease in wages, while the second uses adjustments to payroll and value added taxes (VAT).

In first the method, wages are reduced, inducing a recession, but lowering production costs and increasing exports, eventually reversing the recession and restarting economic growth. This strategy is not unprecedented as a solution to the worldwide financial crisis. Estonia embarked on such a path throughout the past few years. In Estonia, internal devaluation led to 2010 GDP growth of 2.3%; by 2011 growth stood at an impressive 7.5%![17] Additionally, public debt in Estonia is now far below any other Euro country. However, it is important to remember that the political and social situation in Estonia is far different than that in Greece.

Growth did come in Estonia, but it was preceded by several years of economic hardship as its leaders refused to borrow and instead fought to keep government spending low. Initially, GDP dropped by over 14% in 2009 and unemployment hit 16%.[18] Estimated GDP loss for Greece is 15.8% if the current internal devaluation measures continue, but could be much worse.[19] Unemployment in Greece is already 22% and could easily slip even higher if events in Estonia are a precedent. Nevertheless, it is this first method that the Troika has so far favored.

The second method of devaluation is to encourage exports more “stealthily” by increasing the VAT and decreasing payroll taxes.[20]  This second method keeps wages at their current levels while also keeping operating revenue roughly the same (since income taxes are unchanged). At the same time the lower payroll tax decreases production costs significantly – in Greece the employer contributes 28% of wages toward such taxes.[21] Additionally, a higher VAT discourages consumption of imports leading to a positive balance of trade. The hope is that in the long run payroll taxes can rebound or other forms of public support can be found to recoup the necessary short-term loss to the social security fund, eventually refilling the fund before permanent cuts to entitlements are required. This second method seems to be more politically and social feasible, but would require a renegotiation of the bailout reforms as they currently stand.

Troika action has by no means been all bad, however. Greek banks have been heavily reliant on the ECB for liquidity during the crisis and even during the pre-crisis era 2.5% of Greek GDP came from EU transfer payments.[22] Additionally, the recent ECB announcement of the purchase of short-term Greek debt seems reasonable given the extraordinary increase in Greek two-year bond yields. Combined with the €240 billion in bailout funds and consistently strong rhetoric of Euro survival, it seems clear that overall the Troika is committed to seeing Greece, and indeed the Eurozone, succeed. Likewise, austerity is not an unreasonable reform considering the state of Greek public debt, revenue, and expenditure. Austerity of some kind will be required, although the degree and form it takes may be able be negotiated with the Troika.

Untitled

Leaving the Eurozone

Leaving the Eurozone would involve a return to the drachma and subsequent currency devaluation. Greek exports would increase and this would improve economic prospects in the medium and long run. Again, in the short term a recession is likely since domestic wealth and savings are decreased by the devaluation at the same time that imports would become more expensive. As a result, aggregate demand would likely fall in the immediate aftermath of the devaluation. This is somewhat offset, however, by an increase in exports and tourism in the short run, and in the long run it is suggested that these two factors would together pull the country out of the recession and restore long-term economic growth.

Additionally, a move to the drachma would mean an increase in the debt overhang – the amount of debt that Greece owes to foreign creditors. This increase is due to the fact that most Greek debt is denominated in foreign currencies and so cannot be devalued along with the domestic currency. Additionally, this foreign denomination means the debt cannot be “inflated away” with loose monetary policy after a return to the drachma as is true when debt is domestically held or otherwise denominated in the domestic currency.

An analogy to Iceland is instructive as to the possible consequences of increased debt overhang. In 2008 Iceland began a devaluation of its currency, the Krona. As the chart below demonstrates, the external debt shot up to roughly three times its previous levels.

Untitled

Greece would have to either negotiate a payment of its debts in drachma – at an unfavorable drachma to Euro exchange rate – or purchase Euro itself on the open market in order to repay its obligations. Obviously, this would be a great burden on the Greek economy. One option that could be used in conjunction with an exit from the Eurozone is a debt default, which will be discussed in the next section. If Greece chooses not to default, it would have to service its debt at regular intervals, which would entail a substantial regular payment from Greece to its creditors.

This increase in debt must be weighed against the simultaneous increase in Greek tourism and exports. After its devaluation, for instance, Iceland was considered a “bargain” as a travel destination.[23] A full quarter of Greek GDP comes from tourism[24] and a significant increase in tourism could indeed be great news for growth. This is especially true if a return to the drachma is viewed by the Greek people as a superior policy, and therefore leads to more social and political stability and an end to rioting and protests, which otherwise would most certainly discourage tourism.

On the whole, however, the merits of a devaluation may be less significant than has been suggested in other reports. Imports account for 30% of Greek GDP and are mostly inelastic – goods that customers are reluctant to cut back on even if they increase in price – such as oil, food, and pharmaceuticals.[25] Exports, meanwhile, only account for 25% of GDP; and some major export industries, such as shipping, are dollar denominated and therefore immune from a drachma devaluation and its otherwise lowering of prices.[26] Together, this may mean that the relatively more expensive imports caused by a devaluation outweigh the benefits of cheaper exports, perhaps making the economic situation worse, not better.

Lastly, the logistics of a Eurozone exit are challenging, but certainly not impossible. First, Greece would have to take a so-called “banking holiday,” temporarily freezing the banking sector to stop capital from leaving the country in anticipation of a drachma reinstatement. Next, over a period of several days the government would set an exchange rate and redenominate the banking sector into drachma. After banks reopened, Greek citizens could go to any branch and exchange their Euro for drachma, which the government would have to print and distribute to banks. Producers would also have to update all of their prices in terms of drachma.

Loan Default

A third option is a loan default. Under this plan, Greece would decide not to pay back any of its debts, or, alternatively, pay back carefully selected loans – perhaps those held domestically, for instance – thereby reducing public debt substantially. This plan is not an option if continued Troika funding is desired and in practice it would likely be combined with a Eurozone exit and drachma devaluation.

A more ideal option would be to structure a large-scale debt write down, but such a debt restructuring already took place in March of 2012 when private bond holders of Greek debt voluntarily agreed to take a 53% loss totaling €105 billion.[27] It is unlikely, then, that there is remaining low-hanging fruit, as it were, when it comes to renegotiating debt. Pay back or complete default seem to be the only two possible scenarios at the moment.

The obvious benefit of this approach is the drastic reduction of the large Greek debt. Combined with a currency devaluation, it would avoid the debt overhang problem and provide an opportunity for economic recovery. Such a strategy would drastically increase the cost of borrowing, however, and would shut Greece out of international lending markets for several years at a minimum.

The reason a Eurozone exit is inevitable under a default is simple. Suppose for a moment that Greece did not exit the Euro. Since revenues are not yet covering government expenditures, and because borrowing would no longer be an option, a default would mandate immediate measures to balance the budget. This would entail either raising taxes, laying off public sector workers, cutting welfare benefits, or some combination of the three.[28] Greek banks would also loose Eurozone loan support, resulting in a failure to recapitalize and insolvency, quickly leading to a large number of bank failures.[29] Together these factors would induce a deep recession and would likely result in widespread riots and violence.[30] However, combined with a return to the drachma and devaluation, these events could be avoided or dampened.[31] However, this plan is still subject to the economic weaknesses of a return to the drachma that were outlined in the previous section.

There are other concerns in defaulting as well. A default would likely hurt political relationships with other European countries. A substantial amount of Greek debt is held either by other European governments or by private banks in European countries. Together US$44.3 billion is held in France, US$13.3 billion in Germany, US$10.5 billion in the UK, and combined Italy, Switzerland, and Span hold approximately US$5 billion.[32] Defaulting on those creditors would undoubtedly cause negative shocks throughout their national economies, and the move would be seen as fiscally irresponsible, perhaps even leading to a breakdown of normalized political relations with other members of the EU.

Lastly, a default might also mean the seizure of Greek assets abroad as foreign governments, private banks, and, especially, hedge funds – which also hold several billion in Greek debt – attempt to recoup their losses.[33] Some hedge funds, known colloquially as “vulture funds,” make seizing assets an explicit strategy.[34] And seizures are quite common throughout history having occurred in other sovereign defaults involving Brazil, Peru, Liberia, Zambia, and the Democratic Republic of the Congo, to name just a few.[35]

Recommendation

Ultimately, remaining with the status quo appears to be the most politically feasible policy available and offers the best chance at economic success. The New Democracy Party, which was elected 17 June 2012, ran on a pro-bailout platform and garnered the highest percentage of votes from the Greek electorate.[36] Investors worldwide reacted positively to this news[37] and it is important to continue to demonstrate political follow-through and stability going forward by keeping with the policy platform on which the party was elected.

Thus far, riots, sometimes violent, have not been uncommon. But the Troika is likely sensitive to the political stability of Greece and does not desire nationwide Greek violence any more than the incumbent Greek government. If discontent among Greeks grows to the point that the radical Syriza party – which has a contentious relationship with the rest of Europe[38] – is likely to win election, or, worse, a coup is forecast, the Troika would likely be willing to postpone or reduce austerity measures. In the end, of course, some austerity is necessary for the Greek economy to recover. Likewise patronage, business cartels, and government corruption must be rolled back. All of these measures will temporarily hurt some segment of the Greek populace, but in exchange long-term economic health is likely to be stronger. It is up to Greek politicians to sell this idea to the people.

The Troika and individual European nations, most notably, Germany, do not always agree on the best European-wide economic policy. However, in the final analysis they are both committed to Greek success. A Greek default or Eurozone exit would ripple throughout Europe, causing investor panic in Spain, Italy, and Portugal, eventually spreading to France and Germany.[39] For this reason, Greece has some leverage over the Troika and can perhaps use this power to negotiate a reduced set of austerity requirements. Specifically, an internal devaluation using the VAT and payroll tax method is politically and socially preferable to the Troika’s current wage reduction plan. One further suggestion, then, is to take this change to the Troika using Greek social resistance and possible riots as leverage.

Whatever happens, Greece has a tough line to walk between the desires of the Greek people and the reforms proposed by the Troika. This line, though, is still preferable to Greece turning its back on the rest of Europe as would be necessary in a return to the drachma or currency devaluation. In this particular case, what is best for the Eurozone is what is also best for the Greek government and people.

Charts and Graphs

All charts and graphs besides the Stakeholder map, which I created, are sourced from Wikipedia.

Endnotes

[1] Jones, Johnson, and Watkins, “Draghi Kills Hope of Instant Action.”

[2] “Draghi’s Bold Move in Euro Chess Game.”

[3] Catao, Fostel, and Ranciere, “Fiscal Discoveries, Stops and Defaults.”

[4] Jolly, “2009 Greek Deficit Revised Higher.”

[5] “Transparency International – Country Profiles.”

[6] Eurostat, “Government Revenue, Expenditure and Main Aggregates.”

[7] Ibid.

[8] Roscini, Schlefer, and Dimitriou, “The Greek Crisis: Tragedy or Opportunity?”.

[9] The Associated Press, “Key Dates in Greece’s Debt Crisis.”

[10] “Q&A.”

[11] “The Greek Economy.”

[12] Hyz, “Small and Medium Enterprises (SMEs) in Greece – Barriers in Access to Banking Services. An Empirical Investigation.”

[13] “The Greek Economy.”

[14] “Q&A.”

[15] The Associated Press, “Key Dates in Greece’s Debt Crisis.”

[16] “The Greek Economy.”

[17] Greeley, “Krugmenistan Vs. Estonia.”

[18] Ibid.

[19] Weisbrot and Montecino, “More Pain, No Gain for Greece.”

[20] Editors, “Greece’s Least Bad Option Looks to Be Internal Devaluation.”

[21] Wikipedia contributors, “Taxation in Greece.”

[22] “Greece.”

[23] “With Devaluation of the Krona, Iceland Is Now a Hot Spot.”

[24] “Greece.”

[25] Ibid.

[26] Ibid.

[27] Wikipedia contributors, “Greek Government-debt Crisis.”

[28] Giles, Birkett, and Jones, “Consequences of a Greek Eurozone Exit.”

[29] Ibid.

[30] Ibid.

[31] Ibid.

[32] “Q&A.”

[33] “Greece.”

[34] Ibid.

[35] Ibid.

[36] Brown, “Greek Elections.”

[37] Ibid.

[38] Ibid.

[39] Giles, Birkett, and Jones, “Consequences of a Greek Eurozone Exit.”

Bibliography

Brown, Abram. “Greek Elections: Investors, Take A Moment To Cheer Pro-Bailout Party’s Victory – Forbes.” Forbes, June 17, 2012. http://www.forbes.com/sites/abrambrown/2012/06/17/greek-elections-investors-take-a-moment-to-cheer-pro-bailout-partys-victory/.

Catao, L., A. Fostel, and R. Ranciere. “Fiscal Discoveries, Stops and Defaults.” (2011). http://www.parisschoolofeconomics.eu/IMG/pdf/Catao-Fostel-Ranciere-oct2011.pdf.

“Draghi’s Bold Move in Euro Chess Game.” Financial Times, August 2, 2012. http://www.ft.com/intl/cms/s/39ff72c0-dcab-11e1-a304-00144feab49a,Authorised=false.html?_i_location=http%3A%2F%2Fwww.ft.com%2Fcms%2Fs%2F0%2F39ff72c0-dcab-11e1-a304-00144feab49a.html&_i_referer=#axzz22LATAiaq.

Editors, the. “Greece’s Least Bad Option Looks to Be Internal Devaluation: View.” Bloomberg, n.d. http://www.bloomberg.com/news/2012-01-04/greece-s-least-bad-recovery-option-looks-to-be-internal-devaluation-view.html.

Eurostat. “Government Revenue, Expenditure and Main Aggregates”, n.d. http://appsso.eurostat.ec.europa.eu/nui/show.do?query=BOOKMARK_DS-054156_QID_-76724309_UID_-3F171EB0&layout=TIME,C,X,0;GEO,L,Y,0;UNIT,L,Z,0;SECTOR,L,Z,1;INDIC_NA,L,Z,2;INDICATORS,C,Z,3;&zSelection=DS-054156UNIT,PC_GDP;DS-054156INDICATORS,OBS_FLAG;DS-054156INDIC_NA,B9;DS-054156SECTOR,S13;&rankName1=SECTOR_1_2_-1_2&rankName2=INDIC-NA_1_2_-1_2&rankName3=INDICATORS_1_2_-1_2&rankName4=UNIT_1_2_-1_2&rankName5=TIME_1_0_0_0&rankName6=GEO_1_2_0_1&pprRK=FIRST&pprSO=PROTOCOL&ppcRK=FIRST&ppcSO=ASC&sortC=ASC_-1_FIRST&rStp=&cStp=&rDCh=&cDCh=&rDM=true&cDM=true&footnes=false&empty=false&wai=false&time_mode=ROLLING&lang=EN&cfo=%23%23%23%2C%23%23%23.%23%23%23.

Giles, Chris, Russell Birkett, and Cleve Jones. “Consequences of a Greek Eurozone Exit.” Financial Times, May 21, 2012. http://www.ft.com/intl/cms/s/2/0a35504a-0615-11e1-a079-00144feabdc0.html#axzz22zmX3gKi.

“Greece: Better To Stay Put In Euro?” Seeking Alpha, n.d. http://seekingalpha.com/article/646871-greece-better-to-stay-put-in-euro.

“Greece: Here Come the Vulture Funds.” The Guardian, May 17, 2012. http://www.guardian.co.uk/commentisfree/2012/may/17/greece-vulture-funds.

Greeley, Brendan. “Krugmenistan Vs. Estonia.” BusinessWeek: Global_economics, July 20, 2012. http://www.businessweek.com/articles/2012-07-19/krugmenistan-vs-dot-estonia#p1.

Hyz, Alina. “Small and Medium Enterprises (SMEs) in Greece – Barriers in Access to Banking Services. An Empirical Investigation.” International Journal of Business and Social Science 2, no. 2 (February 2011).

Jolly, David. “2009 Greek Deficit Revised Higher.” The New York Times, November 15, 2010, sec. Business Day / Global Business. http://www.nytimes.com/2010/11/16/business/global/16deficit.html.

Jones, Claire, Miles Johnson, and Mary Watkins. “Draghi Kills Hope of Instant Action.” Financial Times, August 2, 2012. http://www.ft.com/cms/s/c07cf4d2-dc86-11e1-bbdc-00144feab49a,Authorised=false.html?_i_location=http%3A%2F%2Fwww.ft.com%2Fcms%2Fs%2F0%2Fc07cf4d2-dc86-11e1-bbdc-00144feab49a.html&_i_referer=#axzz22bccWj8w.

“Q&A: Greek Debt Crisis.” BBC, June 18, 2012, sec. Business. http://www.bbc.co.uk/news/business-13798000.

Roscini, Dante, Jonathan Schlefer, and Konstantinos Dimitriou. “The Greek Crisis: Tragedy or Opportunity?” Harvard Business School, September 16, 2011.

The Associated Press. “Key Dates in Greece’s Debt Crisis.” BusinessWeek: Undefined, June 15, 2012. http://www.businessweek.com/ap/2012-06-15/key-dates-in-greeces-debt-crisis.

“The Greek Economy: Promises, Promises.” The Economist, August 4, 2012. http://www.economist.com/node/21559974?zid=307&ah=5e80419d1bc9821ebe173f4f0f060a07.

“Transparency International – Country Profiles”, n.d. http://www.transparency.org/country.

Weisbrot, M., and J. A. Montecino. “More Pain, No Gain for Greece: Is the Euro Worth the Costs of Pro-Cyclical Fiscal” (2012). http://gesd.free.fr/paingain.pdf.

Wikipedia contributors. “Greek Government-debt Crisis.” Wikipedia, the Free Encyclopedia. Wikimedia Foundation, Inc., August 10, 2012. http://en.wikipedia.org/w/index.php?title=Greek_government-debt_crisis&oldid=506652815.

———. “Taxation in Greece.” Wikipedia, the Free Encyclopedia. Wikimedia Foundation, Inc., August 3, 2012. http://en.wikipedia.org/w/index.php?title=Taxation_in_Greece&oldid=494104235.

“With Devaluation of the Krona, Iceland Is Now a Hot Spot.” U-T San Diego, n.d. http://www.utsandiego.com/news/2009/jan/04/1t04advicem18931-no-headline/.

 

Whose Reality Counts?: Fifteen Years Later

Postscript

This one goes out to all the feminists out there. I wrote this as part of a development management class, thus the frequent digressions into development economics. I don’t necessarily agree with everything I’ve written here – Ed Glaser is one of my favorite economists, for instance; he was integral in revitalizing the field of urban economics. That said, it’s an important exercise to take a point of view when writing, especially if it’s not your own. Far too many people vacillate or pull punches and it makes their writing weak. Be bold people.

Whose Reality Counts?: Fifteen Years Later

“There is but one social science,” economist George Stigler is reported to have once said, “and we are its practitioners.” Though Stigler meant his statement as a slight at the qualitative nature of economics’ “softer” alternatives, the “we” in his pithy remark can also double to mean “we men,” a distinction that has been borne out by a long history of female exclusion from the social sciences in general and from economics in particular.

The three themes embedded in Stigler’s remark – (1) the promotion of economics as the pinnacle of social science research employing (2) rigorous measurement and mathematical analysis thereby unconsciously resulting in (3) women having lower status within the profession – though not first identified by Robert Chambers, were perhaps best articulated by him. These ideas were included as part of his bigger project on professionalization within the social sciences and the need for participatory practices in development, which were collectively formalized in his late ‘90s work Whose Reality Counts?

It would be pleasant to believe that in the fifteen years since Chambers’ book was first released the social sciences have “learned their lesson,” lead by an infusion of humility and inclusion within the economics profession. Unfortunately, Chambers’ critiques can be levied just as easily today as they were when he initially wrote his book.

Economists as Kings and Scientists

The economics profession is still very much living in the shadow cast by Paul Samulson who first formalized its study in the 1940s by applying advanced mathematical models to what was previously a largely qualitative field – there is no math in Keynes’ General Theory, for instance. All of that has changed. Led by economists, mathematics has now spread to virtually every social science; it has leaked into such non-mathematical fields as history, sometimes going by the name “cliometrics,” several sub-branches of modern philosophy, and, according to Chambers, even cultural anthropology. Only the post-positivists have remained immune from the influence of mathematics, but then that’s how they make a living.

As Chambers argues forcefully in his book, economists are at the “head of the class” according to traditional status hierarchies within the social sciences. This fact is easy enough to observe. The President has a Council of Economic Advisors; there is no Council of Sociologists. When hundreds of billions of dollars in government bailout funds are at stake it is the economists, not the historians or philosophers or feminists, that are brought to the table for discussion. Demographers are not appointed to the position of Treasury Secretary. The Federal Reserve controls the money supply and sets interest rates, power that ultimately radiates to affect the lives of billions of people worldwide; yet, an ethnomusicologist has never been considered for the job. Only those in security studies enjoy equal power in the U.S. government, with their appointments as heads of the various U.S. intelligence agency posts, presidential advisors, and, often, the powerful position of Secretary of State. Similarly prominent positions reserved for economists exist in most Western-style governments worldwide.

That economists have come to hold positions that deal with economic matters seems logical enough; however, the hubris they display through the use of formal mathematics and sophisticated statistical techniques is anything but logical. The term “social science” was meant to denote a type of analytical rigor directed toward the social world, not the conversion of “people to things,” as Chambers himself chides, or the notion that complex social systems can be studied and controlled in the same manner that a laboratory physicist can careful control and monitor her experimental setup. If there is an analogy between the social and physical sciences, economists are more like geologists – they know something, but are unable to precisely predict the location and veracity of the next earthquake, tidal wave, or volcanic eruption, and do not posses sufficient technology to easily and swiftly deal with such catastrophes when they do occur.

Of course, economists do not phrase their doctrine of mathematical rigor this way, but rather as a sanguine belief that mathematics, applied properly, can tease out casual relationships with precision as well as offer prescriptive policies. For example, economist Ed Glaeser, in his recent op-ed “The Role of Economics in an Imperfect World,” phrased things this way:

“Hubris has been part of the human condition, with or without math, long before the Black-Scholes asset-pricing formula. Mathematical models create discipline. They ensure that we specify our assumption and that our conclusions then follow from our assumptions. Statistics then provide us with indispensable tests of our theories.”[1]

But if this were true we would expect – or at least hope – that testing of various theories would generate consensus within the profession, anchoring a public policy question with normative prescriptions so that that the next question could be addressed and so on until a socially efficient outcome was reached. But surveys show that economists are still widely divided on most important public policy issues.[2]

And what is true about economic consensus in general is true for development economics in particular. Late last year, for instance, Twitter exploded after Daron Acemoglu and James Robinson brazenly responded to Jeffery Sachs’ Foreign Affairs review of their popular book Why Nations Fail?[3] Sachs also famously disagrees (on nearly everything) with NYU economist William Easterly, but so too does Easterly disagree with the various “poverty trap” theories put forth by Paul Collier in his best-selling The Bottom Billion.[4] If data has done anything to resolve these controversies it doesn’t show.

As Chambers puts it, “Figures so selected are then accepted, repeated, cumulatively misquoted, and used, consciously or unconsciously, to reinforce predisposition and prejudice.”[5] Economist Tyler Cowen calls this phenomenon “mood affiliation,”[6] the notion that one first unconsciously chooses a “mood” and then selects or interprets results to comport with this pre-selected world view. In the universe that Chambers and Cowen envision, data does little to resolve fundamental disputes over theory as Glaeser suggests.

In fact, such disagreement even extends to the data-heavy Randomized Controlled Trial (RTC) craze. For instance, in a 2008 blog post – again involving Jeffery Sachs – Harvard development economist Dani Rodrik cited an RTC to support Sachs’ view that insecticide-treated bed nets should be given away for free rather than sold at a nominal price.[7] However, Mead Over from the Center for Global Development quickly responded by arguing that the RTC did not at all support Sachs.[8]

Intellectual disagreement, of course, is to be encouraged. Indeed, it can be a powerful force in creating new knowledge and moving research programs forward. However, when paired with the hubris of those such as Stigler or Glaeser it can be dangerous. This is doubly true when economists are hoisted into powerful government positions and then expected by both government officials and the general public to carefully predict and steer the economy as a physicist would a laser beam. The latter two parties would do well to realize the limitations of what is possible in matters of economics, but so too should economists chisel away their patina of scientism and refrain from promoting themselves as surgeons who simply opted to operate on matters of public policy instead of on human patients.

Holy Measurement Error Batman

Part of the idolatry of the economics profession is the belief that “data” can erase all sins. To the extent that reliable data can serve as evidence to support an argument or claim there is no harm done. But the fabrication of unreliable data where data simply doesn’t exist can indeed be harmful. In Chambers’ book he references Gerry Gill’s paper Ok, The Data’s Lousy, But It’s All We’ve Got, summarizing Gill’s piece with his own colorful observation:

“At worst, they [economists and consultants] grub around and grab what numbers they can, feed them into their computers, and print-out not just numbers but more and more elegant graphs, bar-charts, pie diagrams and three-dimensional wonders of graphic myth with which to adorn their reports and to justify their plans and proposals.”[9]

In The Black Swan Nassim Taleb offers an elegant parallel for much of the data hunger in economics.[10] In his analogy you have just boarded a plane in Atlanta that is destined for New York City. Suddenly a flight attendant’s voice can be heard over the intercom, “The pilot has misplaced the map for the NYC airport,” he says, “but don’t worry he’s going to use the map for Chicago’s airport.” But Chicago is not New York; the two airports are in different parts of their respective cities; the runways are facing different directions and require different approach patterns; JFK is positioned next to Jamaica bay while O’Hare is fifteen kilometers inland from Lake Michigan. The dissimilarities are nearly endless. If placed in such a scenario Taleb argues any rational person would immediately disembark the plane. In economics, however, researchers proudly pat themselves on the back, remarking the good fortune that they found data to apply to their cause.

Perhaps the single best example of this phenomenon in all of economics comes from the recent developments in African GDP revisions. Economists have long known that African nations had severe obstacles in collecting the sort of data needed to construct accurate GDP figures: poor infrastructure, widespread corruption and institutional failures, low-skilled government workers, technology shortages, and so on. Yet, GDP figures were constructed anyway and indiscriminately imported into the databases of prominent international organizations such as the OECD, IMF, and World Bank. Economists have used the data for thousands, perhaps tens of thousands, of regressions to determine both the causes of poverty and to assess the multitude of macro-level poverty alleviation interventions that have been tried over the decades. And now it turns out that unreliable GDP data may invalidate a large portion of that work.

How bad is it? Alwyn Young from Britain’s London School of Economics reconstructed GDP for 29 Sub-Saharan countries using household-level income gathered in DHS health surveys. He estimates that for the past two decades the region has been growing between 3.4 and 3.7 percent per year, or roughly four times the figures reported in international data sources.[11] Compounded over time the difference between his findings and official statistics are enormous.

To give some sense of the magnitudes, in 2010 Ghana’s government revised its GDP figure upwards by 60%, meaning that roughly $13 billion worth of economic activity had been overlooked.[12] This is all the more alarming because “[o]ver the past thirty years Ghana has been one of the most scrutinized, measured, studied, pick-over economies in Africa.”[13] If the pending investigation in Nigeria yields a similar GDP revision it would be the equivalent of “40 economies roughly the size of Malawi’s” hiding in the African country.[14] Similar news is certain to appear Africa-wide as more governments undertake revision investigations at the behest of international aid organizations and researchers.

With such enormous misinformation about GDP figures – much of it dating back decades – what is really “known” about development economics becomes quite murky. This should serve as a tale of caution to economists, and indeed all social scientists, about the fallacy that some data is better than none at all.

The Second Sex

One key concept for Chambers is what he terms “Normal Professional Status” – a status hierarchy within and between practicing professions composed of five primary factors: education and training, competence and specialization, gender, influence and wealth, and location. Regarding gender Chambers writes, “The high-status professionals are mainly men, while those of lower status are mainly women, or have a higher proportion of women.”[15] Cambers continues by specifying several examples of women being positioned at the bottom of the academic ladder. Again, little has changed in the time since he wrote. Female exclusion is still prevalent both in economics and in the social sciences more broadly.

While Chambers wrote in 1997 that no woman had every won the Nobel Prize in Economics, as of 2013 only Elinor Ostrom has achieved that honor, though even she was technically a political scientist. When the Chronicle of Higher Education created a visual representation of academic papers authored by women over the four centuries between 1665 and 2010 (using data from the University of Washington’s Eigenfactor Project), it found that only 9.7% of economics papers were female-authored. Since 1991 the percentage has increased, but is still paltry at just under fourteen percent.[16]

Again, what is true in general is also true for development in particular. It was not until 2011 when the International Monetary Fund (IMF) was first headed by a woman, Christine Lagarde; meanwhile, The World Bank has never had a female president. There has been only one female USAID administrator, Henrietta Fore, who served two years from 2007 until 2009. Yet, since its founding in 1946 USAID has disbursed a third of a trillion dollars in development loans and grants[17], the World Bank nearly $450 billion since it began[18], while the IMF currently has over a trillion dollars in pledged or committed resources.[19]

In this regard, the economics profession is not alone. The pattern of female underrepresentation is replicated throughout the social sciences. When women are represented they tend to be in those specialties associated with what feminists scholars have identified as feminine values such as community, connection, sharing, nature, life, and interdependence[20]: household decision making and subjective well-being in economics, gender and family studies in sociology, women’s studies and concept of self in anthropology, women candidates in U.S. election studies, feminist history in history, early childhood and infant learning in cognitive science, pregnancy outcomes in occupational health, hospital queuing in operations research, society and fertility in demography, minority students in education, and, of course, Lie algebras in mathematics (meanwhile a measly 1.5% of papers on Riemannian manifolds have been authored by women).[21]

But these feminine values are not appreciated in a system that treats “truth” and “measurement” eponymously. Instead, women are pushed either to the edges of their fields or must choose lower-status academic professions that place a greater value on concepts of qualitative research and intuition. This is not to say that only women express or appreciate these feminine values, but as Chambers points out, men who are empathetic to such ideas are driven to positions of lower status. Nor is it to say that all women are more interested in “community” than in “measurement.” However, it does seem true that these feminine values come more easily to women than to men and as such women are more often required to make career decisions that are at odds with personal sensibilities.

Conclusion

The problems identified by Chambers fifteen years ago are very much alive today. Economists still enjoy high status positions due, in part, to the way they are perceived by the general public and the important government positions they fill. They in turn sustain their status by using advanced mathematics to lend an air of science and control to the profession, in practice, however, rarely moving toward consensus. The emphasis on measurement and data depreciates feminine values and thus drives women to the fringes instead of incorporating their viewpoint as an important and necessary alternative. Here’s hoping that fifteen years from now Chambers’ concerns will be a distant memory.

Endnotes

[1] Glaeser, “The Role of Economics in an Imperfect World.”

[2] Klein, “An Amazing Consensus Among Economists: Not.”

[3] Murphy, “Thanksgiving Eve Twitter Debate: Sachs Vs Everyone.”

[4] Easterly, Easterly on Growth, Poverty, and Aid.

[5] Chambers, Whose Reality Counts?.

[6] Cowen, “The Fallacy of Mood Affiliation.”

[7] Rodrik, “Jeff Sachs Vindicated.”

[8] Over, “Sachs Not Vindicated.”

[9] Chambers, Whose Reality Counts?.

[10] Taleb, Taleb on Black Swans, Fragility, and Mistakes.

[11] Blattman, “Africans Are Richer (and Getting Richer) Than You Think.”

[12] Jerven, “Lies, Damn Lies and GDP.”

[13] Moss, “» Ghana Says, Hey, Guess What?”.

[14] Jerven, “Lies, Damn Lies and GDP.”

[15] Chambers, Whose Reality Counts?.

[16] The Chronicle of Higher Education, “Women as Academic Authors, 1665-2010.”

[17] Services, “Standard Program Report.”

[18] The World Bank, “Total Disbursements by Country.”

[19] International Monetary Fund, “Factsheet — The IMF at a Glance.”

[20] Tong, Feminist thought.

[21] The Chronicle of Higher Education, “Women as Academic Authors, 1665-2010.”

Bibliography

Blattman, Chris. “Africans Are Richer (and Getting Richer) Than You Think.” Chris Blattman, October 30, 2012. http://chrisblattman.com/2012/10/30/africans-are-richer-and-getting-richer-than-you-think/.

Chambers, Robert. Whose Reality Counts?: Putting the First Last. First Edition. Intermediate Technology Publications, 1999.

Cowen, Tyler. “The Fallacy of Mood Affiliation,” March 31, 2011. http://marginalrevolution.com/marginalrevolution/2011/03/the-fallacy-of-mood-affiliation.html.

Easterly, William. Easterly on Growth, Poverty, and Aid, February 11, 2008. http://www.econtalk.org/archives/2008/02/easterly_on_gro.html.

Glaeser, Edward. “The Role of Economics in an Imperfect World.” Economix Blog. Accessed February 13, 2013. http://economix.blogs.nytimes.com/2011/05/10/the-role-of-economics-in-an-imperfect-world/.

International Monetary Fund. “Factsheet — The IMF at a Glance.” Accessed February 12, 2013. http://www.imf.org/external/np/exr/facts/glance.htm.

Jerven, Morten. “Lies, Damn Lies and GDP.” The Guardian, November 20, 2012. http://www.guardian.co.uk/business/2012/nov/20/economics-ghana.

Klein, Dan. “An Amazing Consensus Among Economists: Not.” EconLog, February 6, 2012. http://econlog.econlib.org/archives/2013/02/an_amazing_cons.html.

Moss, Todd. “Ghana Says, Hey, Guess What? We’re Not Poor Anymore! – Center for Global Development: Views from the Center.” Center for Global Development, November 5, 2010. http://blogs.cgdev.org/globaldevelopment/2010/11/ghana-says-hey-guess-what-we%e2%80%99re-not-poor-anymore.php.

Over, Mead. “Sachs Not Vindicated.” Center for Global Development, January 18, 2008. http://blogs.cgdev.org/globalhealth/2008/01/sachs-not-vindicated.php.

Rodrik, Dani. “Jeff Sachs Vindicated,” January 15, 2008. http://rodrik.typepad.com/dani_rodriks_weblog/2008/01/jeff-sachs-vind.html.

Services, USAID Economic Analysis and Data. “Standard Program Report,” February 8, 2011. http://gbk.eads.usaidallnet.gov/query/do?_program=/eads/gbk/programReport&submit=submit&output=2&unit=N&program=GBK419902.

Taleb, Nassim. Taleb on Black Swans, Fragility, and Mistakes. Podcast, May 3, 2010. http://www.econtalk.org/archives/2010/05/taleb_on_black_1.html.

The Chronicle of Higher Education. “Women as Academic Authors, 1665-2010,” 2012. http://chronicle.com/article/Woman-as-Academic-Authors/135192/.

The World Bank. “Total Disbursements by Country.” World Bank Finances. Accessed February 12, 2013. https://finances.worldbank.org/browse/select_dataset?nofederate=true&suppressed_facets%5B%5D=domain.

Tong, Rosemarie. Feminist thought : a more comprehensive introduction. Boulder, Colo.: Westview Press, 2009.