I completed Daniël Lakens’ excellent MOOC on Coursera last year (2016). In Week 7, the assignment required pre-registering a hypothesis to a small research question of sorts, based on any data that one can get from IMDB. As this was a “peer-graded” assignment, it now seems to be locked, so I can’t see the exact details I submitted (and the comment I intended to post here).

Funny how it sometimes is. The one assignment on open science. Though in some way that does emphasise the benefits of open deposits of data. My little project can still be found on OSF.

Anyway, here was my main hypothesis: “Directors should improve in “skill” as they gain more experience over time. As such, their later films should score better ratings than their earlier films. This result is what I would expect to find when looking at the works of many different directors (higher ratings for later films). Competing theories may specify that directors already gained their most important skills in roles prior to their first time in the hot seat (in film school, or as producers or actors), that there are too many random factors involved (quality of scripts, studio support, etc.), or that many directors first make a name for themselves with a big hit that gets them more opportunities, but that they then have difficulties living up to this initial hit.”

One point of pre-registering a hypothesis is to be able to point at the time-stamped registration as proof that your hypothesis wasn’t formulated post-hoc, especially if the hypothesis was more unusual/the findings seem surprising to others. Had I followed this logic, this could have turned into a particularly satisfying exercise. Instead, I discarded the initial hypothesis that gave me the idea (bolded above) and lamely presented it as an alternative explanation instead. Yes, I talked myself into the prediction that my initial hunch would be wrong, and that the average director would instead improve over time. Naturally, I found a significant effect in the other direction instead. This is not to say that the effect is true, but in the context of this exercise it would have been a pretty sweet triumph.

Now to the main purpose of this brief reflection: If the assignment sounds cool, do check the whole course out! If it doesn’t sound cool, I promise that the rest is! It taught me some new things, solidified my knowledge in some other areas, and even provided me with some teaching materials for a research methods course that I myself held soon after that. Plus, I was in the very first “cohort”, so the course can only have gotten better since then! (I was even involved in “beta testing”, so you’re very welcome for all the inconsequential bits that were ironed out thanks to me.)