Wednesday, April 13, 2011

Week 3 Dropout Statistics

Thanks to everybody who's commented so far on the blog!  There are some open questions to be decided for sure, in particular the height relationship on performance.  Rest assured it's being looked and will be presented in the future.

Erg.  I wanted to be able to cite specific numbers regarding the dropouts from week 2 to week 3.  I have learned, however, that HQ isn't showing a perfect leaderboard each week, which probably doesn't surprise a few folks given the comments I've seen on some of the pages.  For example, when I gathered the week 3 data for the men, there were a few hundred names that were not present in the week 2 data set.  I presume these folks actually had scores for week 2, but for whatever reason were not posted on the leaderboard at the end of week 2.  That alone would be fine, but in general, I think it means I can't totally trust that week 3's leaderboard contains all the continuing athletes.  Thus, I think the histograms below might be off by a few percent.

Overall the dropout percentage was similar across M/F boundaries - 23% (Men), and 24% (Women).  I estimate that the true number is +/- a few percent at most.

The first plot below shows, for each 5 pound female weight class, what percentage of athletes dropped in week 3.  Remember, this plot does not represent all athletes, just athletes that bothered to fill out their weight information.  While the absolute numbers might be slightly off, my prediction from earlier seems to have held.  Almost 50% of athletes under 110 pounds did not complete week 3.  Cuts were felt all around though, and even the lowest drop percentage was still around 15%.

From a performance standpoint, how did these athletes do?  This next plot shows the scatter of week 2 versus week 1 scores.  Blue dots represent athletes who finished week 3, and red dots are athletes who dropped (wk3 score = 1, or nothing).  A quick inspection reveals many of the drops occuring in the lower scores (lower left), but a surprising number are in the middle.

Week 3 heavily pruned female athletes that were light (left), and who received lower scores in weeks 1 and 2 (right).  Note: I am not  confident that the red dots in the highest performing areas (upper right) of the scatter chart are real dropouts.  They might be absent from the leaderboard, as discussed above.

The same plots for the men are less dramatic.  Interestingly, while the dropout percentage has a strong trend downward, the drops by wk1 and wk2 performance seem more scattered.  I can only conclude that some drops might result from people just not having the time to do the workout as directed, rather than some limitation in performance.

Sadly, I couldn't muster one rep for week3's WOD.   Our workout group had just started power cleans a month ago, and I had recently managed to clean my weight (~140).  In a fairly dumb move, I attempted a squat clean at 165 and not only managed to fail miserably, but also managed to sprain my wrist. 


Dropouts followed a similar weight trend as females (left), but surprisingly their performances seemed to be fairly uniform (right), except maybe in the very elite categories.  Ignore the big bar at 275, it's from low number stats.

15 comments:

  1. In addition to performance vs height, if you're feeling ambitious you could look at age (non-masters). I found that I can see age when I hold mouse over name on leaderboard, but its not listed on profiles. Regardless, fascinating stuff. I enjoy reading your posts.

    ReplyDelete
  2. I second the interest in age vs. performance. I'm pushing up towards masters (but not there yet) and would be very interested to see how I compare to those nearest my age. Also, don't feel too bad - I couldn't get one rep either! Tried to submit a score of zero (hey, I did several attempts!), but they wouldn't take it, so I'm dropped too. Oh well - will still hit the open wods for fun.

    Hope you keep doing these analyses - I'm an xfit nerd as well and this is seriously interesting stuff.

    ReplyDelete
  3. Thanks again for posting these and sorry to hear you're out of the comp.

    Does anyone have an opinion as to whether or not they should strip the rankings in previous weeks of those who drop out? If they don't it would seem that the order of the workouts will affect the final ranking.

    ReplyDelete
  4. Thanks again for the wonderful analysis!!

    Even though a bunch of people won't be able to post a result for Workout 3, and therefore won't appear on the overall leaderboard again, I think they will be still able to submit scores for later workouts and see their ranks *in that workout*. (A friend didn't submit workout #1, but did submit #2, and I was able to find the score he did post even though he didn't have any overall rank.)

    It sounds like you are scraping the data from the Games site from a traversal of the leaderboard, which as you noted isn't always so reliable (data appearing later, calculations having issues at any give time, etc.). If so, I might be able to make it a little more convenient...

    Feeling inspired by your example, I put together a little program last night to extract all info available for every athlete (whether or not they appear in a leaderboard) -- I'm planning to make it periodically update a little data collection of what's known about everyone in the competition, and you're welcome to the code and/or data if you think it might be useful.

    There's one sad wrinkle, though: while the approach I took is great for getting a complete list of participants and any/all scores that have been submitted by them, there's one important piece of data that I simply can't get for everyone (or even most everyone): their sex. The only way I can see for determining an athlete's sex (aside from guessing based on name or whatever) is implicit, signaled only by their showing up in a male leaderboard vs. a female leaderboard, at *some* time. But I'm too late to get that data for gobs of people now that we've passed Week 3 with its huge number of DNFs. :^(

    I would be so grateful if you could share the association between userid and sex (or even just name and sex) that you got from the leaderboards back in weeks one and two. If that might be possible, please drop me a note at my lowercase firstname AT ecosmos DOT com -- thanks!

    ReplyDelete
  5. Amit, I think they must recalculate the rankings after each round based on who is left in the competition. I was 6000+ in the first workout, but now my ranking for WOD 1 is 4000+, so my improvement is consistent with the estimated proportion of DNFs (and assuming they're not just missing 2000 scores for week 3).

    ReplyDelete
  6. Greg, I would love to get my hands on the data. How did you scrape it?

    ReplyDelete
  7. I don't know if you've noticed, but those who are posting via video submission have not been on the leaderboard for a couple of weeks now. All of those would probably show as dropouts, making the dropout rate seem higher than reality. This is an issue that hopefully HQ is working on, although I haven't heard.

    ReplyDelete
  8. Hi, Dimitriy. The way I approached gathering data is primarily through the head-to-head comparison page (for name, region, age, scores and current sore-ranks) and secondarily through the individual's account page (for weight and height, if they gave it, because it doesn't appear in a nice form on the other page). I discovered the value of the head-to-head page when I was frustrated by not being able to look at a user's page to see what they did for any given workout: Once you go to the trouble of pulling up an individual in that head-to-head page, you can see every workout they have posted a score for, and you can see the rankings that may exist for those performances relative to others at that point in time, and so on.

    The final piece was noticing how the page has the dynamic lookup boxes that end up filling in name and noting their "nid" -- but that's all just for a nice user experience: clicking 'submit' takes you to a URL that includes the nids, and that's how the lookup actually happens. For example, here's a head to head comparison of me and Jonathan here, no interaction required: http://games.crossfit.com/compare/8853/235703

    One doesn't need to supply the second nid, making the content more regular for scraping -- and the page doesn't barf if the supplied nid doesn't have an account for whatever reason. So gathering the data on all accounts just requires iterating over nid values from from 0 up, probing for actual accounts and gathering data until 26-thousand-and-whatever have been found. Later, refreshing the data is much faster because you only have to probe the nids that correspond with real accounts. (And this can be done any time -- it was only critical to gather data in a certain window of time when the leaderboards were full of names after weeks one and two closed, to get as much implicit data about sex as could be gathered.)

    ReplyDelete
  9. Greg, thanks a lot for the explanation. I'll see if I can make that work over the weekend.

    ReplyDelete
  10. So you probably have a leader board that is accurate?

    ReplyDelete
  11. Great stuff - keep it up! I too would love to see performnce/age, if you have more "spare" time. Thanks -from another XFit nerd.

    ReplyDelete
  12. Sorry about the wrist. That sucks even more than being out of the comp!

    ReplyDelete
  13. Hi, love your blog! It inspired me to do a little informal weight-performance analysis of my own for my masters women 55-59 bracket.

    I looked at OHS in WOD 4, which was clearly the biggest challenge so far for our group for lifting components. Our weight for the OHS was 75#. Only 3 women completed all OHS; none of them did a muscle-up; the modal number of OHS for our age bracket was 0. Note that this is *after* the WOD 3 pruning had already taken place, so women who couldn’t get a clean fell off the boards. So we know that ALL of these women can clean 75#. But OHS at 75# is another matter.

    DATA: Weight and number OHS for WOD 4
    NOTE: My data (N=30) was incomplete, for two reasons. First, some of the women didn’t give their weight (although we have better compliance than the younger open women – 66% instead of 47%--guess we are less sensitive about this!). Second, I was working from my printout from the Scoreboard during the brief period (a matter of hours) between when they got the ranks properly sorted for WOD 4 and then blanked out the board again in preparation for WOD 5. I only have the first page, which has the top 48 women. However, there were only 55 of us left at the end of WOD 4 and I don’t think any of the 7 women for whom data is missing at the bottom got any OHS. So including whatever portion of them also gave their weight wouldn’t likely affect my results much.

    RESULTS:
    Av weight for women with 0 OHS: 129.5 [n=18]
    Av weight for women with 1-9 OHS: 137.2 [n=6
    Av weight for women with 10-30 OHS: 147.5 [n=6]

    Correlation of weight/OHS number for those with *any* OHS: .54

    Using weight class as a predictor, of the 10 women in the 140 and above weight class, 5 got some OHS.

    Of the 10 women in the under 130 weight class, only 1 got some OHS.

    Next week I’ll do a better job of capturing the data for all the women in my bracket and try some more elaborate analyses across multiple WODs.

    ReplyDelete
  14. Nice job Harrow!

    I'm glad you shared your results and congratulations on competing. The workouts look pretty brutal in that age group for sure. I think it would be nice to promote more participation in that age group - hopefully HQ agrees with this point in the future. Best wishes for the rest of the WODs!

    ReplyDelete
  15. J Young, like your info. How do I contact you?

    ReplyDelete