Week 12.3 percentile plots

One thing I have trouble doing with the leaderboard is quickly scanning how many reps you need to rank in a certain percentile.  This is especially true after Thursday, where droves of scores make scrolling through the leaderboard the devil.

Here - I make it easy! Thanks to Andrew, I've managed to scrape the leaderboard and plot cumulative distributions for week 3.   Want to know how many reps you need for the 50th percentile?  Just go to the y-axis, go to 0.5, and run your finger across until it hits the pink (Ind. Women) or Blue (Ind. Men) lines.   Drop down, and then there's the target number of reps.  Going from reps to percentile is also pretty easy, just reverse the process.

Caveat, this is still pretty early, so the curves could change a bit, especially at the extremes (ie near 0 or near 1).  Ballpark though, I bet it'll work for most people.

Oh, the little bump on the pink line near low percentiles?  That's apparently a few folks that have trouble doing a toe-to-bar.  I wondered about that myself.

For those interested in the code, I've enclosed the matlab file I used to scrape the pages and organize it how I like it.  It's definitely rough around the edges, so if somebody would like to use it and make it better, by all means!

WOD 12.1 - Early plots and analysis

Thanks to miked and killers411, newcomers to the crossfit analysis scene, we have some early plots and discussion points!  Thanks guys!

Miked has plotted the real distributions and fitted distributions of the WOD 12.1, for all athletes, including masters.  He's also very nicely calculated the mean and std of each of these groups and included them in the legend.  Notes:  the masters distributions are hard to see because the y-axis is number of athletes, and of course the number of athletes in the non-masters groups dwarf that of the masters.

Also, he's plotted performance against height and weight, and done some modeling of workout performance and workload.  Outstanding stuff and it'll be fun to see some more.

Here's the link from miked.

Killers411, has done a nice number comparison of each region, which suggests the open has grown 2.5 times since last year!  The fight for regionals will certainly be tougher with all these newcomers.

...and the link from killers411.

Group thought: Questions to ask during the 2012 open....

Jeff is doing a great job providing a complete dataset this year (see last post for a link), which will allow many people to play with the data and come up with interesting analysis.  It's a ton of hard work and we're very fortunate to have him donating his time to this effort.

That said, what are some of the burning questions people want to ask of the dataset?  Hopefully by getting the questions out, we can have folks discussing the data and questions together.

Here are some of the things I'd like to see during the games.

1)  I'm fond of the height/weight plot for each workout, as I think it gives a quick informative picture of what each WOD.  By week 3 or week 4, is it possible to predict what the last wod should be?  One of my standing predictions is that wall ball will be a part of the open.  I think it's one of the few exercises that helps taller athletes.  Erg likely falls into that category too, but I have trouble thinking that would be required.

2)  At the end of the games, will it be possible to answer how much a year's worth of training helps?  This is a bit tricky, because we'll need someway of understanding if the overall performance of open participants has changed over the years.  Has crossfit's emergence and popularity shifted the performance curve down? 

Feel free to post your own questions, about a specific WOD or of the games overall.

The Crossfit Open 2011 dataset, for download

At some point I thought I would write something up more formally about the 2011 Open, but that moment has certainly passed.  Many folks have asked for the data from the 2011 Open, and here it is!

The funny thing is, I have some suspicions that the 2011 Open dataset might be better than the 2012 dataset for the height/weight analyses.  Why?  Part of it is the registration process.  In 2011, when athletes registered, they were asked immediately for their height and weight.  In 2012, these questions have been eliminated, although folks can enter this information freely in their profile.  As a result though, I expect that volunteered information for height/weight will fall off dramatically.  Who knows though, with how big the Open is getting maybe it won't matter?

Click here for a .csv file of the 2011 Open dataset, courtesy of work done by Greg Perkins.  The dataset includes all athletes, including athletes that did not finish all six workouts.  The column headers should be:

- athlete ID,nameURL,age,sex&division,height,weight, overall-points,overall-rank, score1,rank1, score2, rank2, score3,rank3, score4,rank4, score5,rank5, score6,rank6  

Click here for some helpful matlab scripts, including one that breaks the overall dataset into separate structures for each competition category.

And here for some information about the .csv file and descriptions of the matlab scripts.

Calling all fellow xfit nerds - help!

So the open is just around the corner... and so is my PhD thesis defense!  I defend April 12th, and this next month or two is going to be jam packed for me.  I'm trying to assemble one more manuscript before I finish and I have yet to start writing - yikes.

I'm almost certain that I won't be able to maintain the blog as well as I did last year - and let's be clear, Greg Perkins was a HUGE help in helping me acquire the data in last couple of weeks of 2011.  Without him I probably would have spent many more hours just trying to mine the data.

So at this point, I'm trying to start a discussion among people who would be willing and capable of helping out this year.  I can host the 2012 blog, or somebody else can if they prefer and want to make it look all fancy (instead of the terrible book background I've been using).  I can share the matlab scripts I used to generate the plots last year, which can be easily adapted to other languages.  

I'm just brain storming now, but we need people who can do the following:

1)  Understand how HQ is posting the workout scores on the website, and figure out a way to dump these scores along with all the other athlete information (name, region, age, height, weight) etc.  Greg, are you still around these days?  Do you think your nid trick will work like last year?

2)  Take data dumps in relatively real time and play around with the data, pose a few interesting questions and make some rough plots.  Ideally, this person would also be writing something to go along with their plots.  I can facilitate here a bit and help the crowd.

3)  Folks that can take the plots and clean them up a bit for presentation purposes.

I think that's all for now.  Please comment if you think you have the skill and desire to help!

Need help making it to Regionals? Try moving to Central East…

Welcome back everybody!  It seems like some folks are stumbling upon the blog once again now that the 2012 Open is around the corner.   After getting a few encouraging emails I felt renewed (reenergized?)to post again.

Where were we?   Ah yes, the Central East region boast some serious crossfitters, including last year’s world champion, Rich Froning, and the 2011 winner of the Open, Dan Bailey.  On the women’s side, Central East had the pleasure of hosting the beautiful and talented Julie Foucher.  Why would I recommend a move to that region?

Well if you’re striving for regionals, you might look at those who qualified last in each region, to figure out which one is the ‘easiest’.  Last year, and I believe this year as well, the top 60 men and women in each region move on to regional section of the competition.  The difference then, between qualifying and not qualifying is beating that 60^th person.

Plotted below is a chart showing the overall 2011 Open placement (listed in parathesis) of the 60^th person in each region.   I’ve left Asia, Africa, and Latin America off the chart, where Crossfit is still expanding and the number of athletes is small relative to other regions. 

In the United States, the Central East’s 60^th person finished behind all other US regions, for both women and men.   The region to region comparisons can be striking.  For example, Southern California’s 60^th male finished 508^th in the open, while Central East’s finished at 1042*.   Another way of stating it – Central East’s 60^th man would have placed 101st in Southern California.  On the women’s side, Central East’s 60^th placer would have come in at 113^th in the Mid-Atlantic region, which… cough cough… borders the Central East region! 

Comparison of the last place qualifiers (ie 60th man/woman) for regionals*.  The numbers represent the overall 2011 Open ranking of these athletes.   Asia, Africa, and Latin America not shown.

The suggestion to move for my US readers is, of course, made in jest.  Whether the competition last year predicts the competition this year is uncertain, and I hesitate to make any bold predictions.   It’ll be fun to find out this year though!

*Note:  My definition of 60th placer may not be the 'true' last regional qualifier because I'm using the overall worldwide Open results, rather than breaking down the athletes into regions (which is how HQ actually does it), and then back calculating their overall Open results.  The reason I've done this is simple: laziness.  I've done enough checking though to say with strong confidence that the plot overall is quite accurate, just not perfectly accurate.

Was the CrossFit Open fair?

If you cruise across many CrossFit webpages, it’s pretty easy to find people grumbling over the Open workouts.  Some decried at the number of body weight exercises – the double unders of 11.1,  the box jumps of 11.2, and the burpees at 11.4, while others scowled (including myself) at the squat clean and jerk weight of 11.3.  The fact is, designing the Open is probably more difficult than most people are willing to give credit for.  Sure, the act of thinking of different workouts isn’t especially hard, we’ve all done it, but there are so many factors in the Open that it’s a challenge to satisfy them all.  I’m speculating a bit here, but I assume HQ had a few broad goals in mind for the Open:  (1) the workouts should separate the fit from the unfit (2) the workouts should be relatively accessible to everybody (3) there should be equal opportunity to perform well no matter your body type.   Designing the workouts to satisfy all three requirements is no easy task and requires careful thought and planning. 

As many who have read the blog, I’ve been commenting on some aspects of the third point in recent weeks.  How did the overall performance depend on the biometrics such as weight and height?  Was the Open fair in this regard?  After looking at all the data, I would argue that the answer is more ‘yes’ than ‘no’.   I’m not saying the programming was perfect, but overall the workouts seemed pretty balanced.   

Let’s look at data from the main division for men (I’ll do ladies soon enough) for competitors that completed all six workouts.   First, I take each competitor’s overall rank score and convert it into a number between zero and 100, where Dan Bailey represents 100 and some 17 year old named Thomas Thompson represents zero.  As some may recall, I called this scaled score by a different name - ‘overall percentile’.  I have stopped doing that here because it’s not quite right.  Thanks to the weird rule about ties, the overall rank scores have a skew towards better scores, so the scaled score that actually represents the 50^th percentile is around 55. Also for reference, a scaled score of 30 represents the 20^th percentile.  I may change the scaled score metric to percentile, but roll with me for now. 

Here I’ve presented the average scaled score (color) grouped by their height (y-axis) or weight (x-axis). Let’s take an example to explain further.   If you took all the male athletes from about 175 to 180 pounds that were 6’2 to 6’3 (here labeled with a pink dot to aid the eye), you would find that the average overall scaled score was around 45-50.   

Overall open performance broken down by weight and height.  Between BMI 22.5 and BMI 30.5, performance is relatively similar -  in contrast to the individual workouts shown below.  

Let’s backup though, to define what I’m thinking about when I evaluate ‘fairness’.  In a perfect egalitarian world, we would expect to see uniform color across this plot, ie, the average performance for every given height and weight is similar.  Deviations from uniform color have increasing degrees of unequality.  Could we imagine an Open where we have uniform color?  I could definitely imagine it, but practically speaking, it may be difficult, especially keeping in mind the broad goals of the Open discussed above.

Let’s discuss the plot.  Broadly, the plot has a skewed look that roughly travels along similar body ‘thicknesses’.  To illustrate this, I have plotted iso-BMI lines plotted on the charts, denoting height and weights where BMIs are the same.  In between the two extreme BMI lines on each chart is something reassuring to see – that the performance of the average athlete is relatively similar (reds and yellows), and there is little weight bias.

For now, set aside the topic of the ideal crossfit ‘thickness’, as many have expressed interest in (I’ll address that in another post).

What’s the point here?  The overall plot shows that for a given height, there exists an optimal weight range where performance is similar for other optimal height/weight combinations.   This is huge. As we’ll see in the breakdowns of the individual workouts below, this definitely did not have to be the case.   In the overall plot, as you travel along the 26.5 iso-BMI curve, performance is relatively constant as you slide from left to right.  For many workouts, I’m thinking especially of 11.2 and 11.3, performance across a BMI curve is not equal as you shift, indicating a weight bias even when controlling for body thickness. 

Thus, either by accident or good design, HQ managed to design a program where many body thicknesses performed on par with one another.   I think they should be applauded for that.

Be clear that inequalities do exist.  Performance rapidly falls off at BMIs below 22.5 and above 30.5.  Does this data imply that individuals in these low performing regions should be packing some weight gain 2000, or consider some good ol’ fen-fen?  If doing better at CrossFit is your primary goal – sure.  Perhaps though you play sports where being agile (low thickness) might be helpful or other sports (say football), where being thicker is better.  It’s up to the person for sure.   The Open data can only speak about what’s optimal for the Open, and predictions about other activities may not hold.

Below I have included the same height/weight plots as above, for each individual workout.  I think it’s fun to take a look at them, but I won’t go completely crazy discussing them.

Individual workouts, broken down by height and weight.  They sure are pretty, no?

Couple of major points or questions:

0) On the color scale for each plot, 240(40.6) represents the raw score and the percentile score (following the Open method of breaking ties).

1) We often think of the exercises involved in order to ask which workouts are similar to others.  Do these plots graphically agree with our intuition?  For me, I guess I’m relatively surprised at how completely flipped 11.3 and 11.6 are, given the similarities of the thruster/squat-clean movements.  The central tendencies are almost completely rotated.

2)  11.3 is wild to look at.  Iso-BMI lines have equal performance until you hit a BMI of around 30.5, where more weight is better.

3)  11.4 is one of the few workouts that shows a strong height effect.  Perhaps wallballs? 

4)  11.1 is intrinsically the fairest workout, maybe because being bad at double-unders affects people of all body shapes.

6)  11.2. and 11.6 look moderately redundant, at least in terms of their graphics.