parable of the parachute
on statistical evidence, bureaucracy, and epistemology, and cute!
we put too much weight on statistical evidence these days. we also put way too little weight on other types of evidence, in particular evidence that easily verifies causality which happens not to be statistical. and also we put too much weight on the absence of statistical evidence.
of course, the randomized control trial is among epistemically strongest pieces of evidence, but there’s still so much that can go wrong… it is the RCTs that are carried out competently, with enough attention to enough detail that is the gold standard. there are a lot of devils in a lot of details.
imagine a Science Team and trying to find a way for people to jump from high places without hurting themselves. they consider all kinds of stuff, like putting them inside suits with a lot of cushion, or using giant rubber bands, or …
Leo says something out of left field as he tends to (he’s weird, autistic even): what if we use a sail?
nonono, hear me out for a minute, really. we have the person hold the sail by two corners …
what a moron. everyone knows that sails work only to propel ships and they need wind to work.
it’s even worse than that! chimes someone else. boats ‘make their own wind’, so to speak, as anyone who knows about ships can tell you. you absolutely need the boat for the sail to work. moving along.
nonono, really…hear me out i won’t call it a sail anymore, let’s call it hm uh a large piece of cloth. have you noticed that when leaves or pieces of clothing fall, they don’t fall very fast?
this idiot thinks that sails and leaves are the same …heavens! can we move on to something useful, please?
but really, have you noticed that?
i guess i have, but so what? once you put a person inside the clothes the person-plus-clothes assembly falls down just as fast — but i guess we’ve never verified that. we’ve only checked that clothed people fall fast. we should see if removing the clothes helps! i propose we do an RCT. we are careful scientists! and thus, the Science Team has a new research agenda and eagerly go pursue it.
while conducting the first RCT, a debate spanning multiple weeks ensues about whether their sample is representative, and possible confounders and god-knows-what about various statistical methods they deploying to deal with various preliminaries. they decide to redo the study a few times. the studies seem to all find a tiny effect that suggests that clothed people fall more slowly, and they spend another few weeks debating the methodology and whether the meta study is correct in concluding that the studies replicate. they also get in trouble for not clearing their experiments with the IRB.
meanwhile, Leo went and tried this: he grabbed a table cloth and jumped from the table while holding it from two diagonally-opposite corners.
to his disappointment, it didn’t seem to have much effect, but he noticed that what he had in mind was a “negative sail” in “full tension” and that he was having a very hard time at arriving at that configuration while jumping from the table.
he had some other insights, too, though. he didn’t need to be doing the jumping himself, he could use something of comparable size and weight to a person and attach it to the cloth. that way he can just experiment safely, vary the height with ease, and so on…
but also, that posed problems: Leo isn’t exactly a viable for a Strongman competition — he probably can’t move a rock that weighs half as a person. he’s very sad because he has a pretty good hunch that somewhere in the family of ideas he’s exploring is a way to make it all work, and it’d all be much easier if he were working with the whole Science Team on this, but they are very busy debating which one of them understands the meaning of the worlds correlation and control now. this is all so sad.
fortunately, he runs into rz. it is hard for rz to tell whether Leo is onto something or not, but that is reason enough for encouragement.
so, rz reminds him that obvious & true things only seem that way in retrospect. prospectively, every new idea feels either doomed, stupid, or irrelevant at the beginning. consider working with someone who is more physically capable and maybe focus only on being able to run the tests with ease. rz thinks that’s the sort of thing pg would say.
this has a moderate reinvigorating effect on Leo, and he heeds the advice. a few weeks go by, during which Leo finds someone to share the load with and decides that a good approach might be to first worry about making slower objects fall sufficiently slowly. he figures if we can make an egg land safely, we can probably scale things up to make a person land safely.
fun fact, dear reader, IIRC attempting safe egg (or maybe a watermelon) landings following drops from a 4-story atrium used to be a tradition / right of passage for physics freshmen where i went to school. it’s possible that i’m misremembering the tradition, but there’s also that for one of our labs, my buddy and i designed and tested a hockey-puck parachute. it is a lot harder than it seems. we said that in turn about the floor, the puck (sorry!), and the process!
meanwhile, the Science Team has split into factions according to how they view probability and which sort of summary statistics are best. and they might as well be making progress there because they are trying to run a 1000-subject RCT where the subjects jump off of a 5 foot platform either in baggy clothes or spandex, but the IRB clearance has taken 6 weeks so far.
Leo and Igor go off and prototype the “negative sail” for an egg. to make sure they aren’t cheating by padding the egg they use a rubber-band contraption to hold the egg and they tie to the sail with string. Leo struggles with the question about how big to make the piece of cloth and what shape and kinda gets sad.
rz still can’t tell whether Leo is onto something, but keeps encouraging him, this is huge if true, afterall, and suggests that he can certainly afford a few dozen cracked eggs — ahem, splattered. just try something random: ask the gypsy woman, open a geometry book at page 37, whatever. to iterate on the size, go in factors of two, try a random value, then double it, then halve it, then proceed in the direction that looks more promising. and actually, for the shape, maybe just use rectangles until there’s reason not to. rz thinks that’s the sort of thing pg would say.
Leo heeds the advice again, and him and Igor go to work. and indeed they are able to make an egg land safely consistently from table-height. they get excited and start trying higher and higher heights, expecting that their contraption would fail at some point. but they are now dropping the egg from the tallest building in town, and it seems that the egg always lands at the same slow velocity. this really defies Leo’s understanding of the world. heh that’s funny… he marks it as interesting in his notes.
the Science Team is still stuck trying to understand the impact that clothes have on fall safety via studies, and they have designed all kinds of studies to test for the properties of various types of clothes. the IRB keeps slowing them down, but they got the hang of it, so now they know how to go from proposal to experiment in a few weeks. some of the factions of the Science Team are very angry because they are satisfied that baggy clothes are the safest, but other factions keep insisting on controlling for various other properties of the clothing.
Leo tries to go to a meeting of the Science Team to let them know about his success with the egg and proposing investigation into how to use the negative sail for bigger objects. he says that he doesn’t see why he needs an RCT to know that the negative sail works in the case of the egg, which upsets everybody. the Science Team only does Serious Science, and that requires an RCT. in their anger, they also ask him to define the words correlation and control. and while he provides adequate definitions of both, they aren’t the word-for-word sanctioned ones, and all the factions unite in pronouncing him an ignorant. the meeting adjourns and they all go celebrate that even though they don’t have a workable parachute, yet, at least they’ve ousted one ignorant. Leo explains what wtf means to Igor.
Leo and Igor test on larger and heavier objects with great success. they start having an intuition for how big the sail needs to be for any one object. finally, Igor proposes that since he can lift 200lbs (with effort), despite the arduousness of the carrying, they could test the design on a make-do human-shaped dummy made of sandbags with adequate weight.
boy did those experiments suxxx to set up, but they are successful!
so, Leo announces to the Science Team that he’ll be conducting a public demonstration of his negative sail by jumping off of a building. the Science Team snickers about how Leo does not know how to think about risk. Nero calls them all imbeciles and blocks them on twitter, and is among the first to show up to the public demonstration, so several of them also show up to try to say something back to him.
Leo jumps and lands slowly, and the demonstration is an enormous success. the Science Team members that were present immediately understand how this whole contraption works and declare the finding obvious. its just a parachute, duh? of course it slows down whomever is wearing it. why did he even need to make such a big demonstration to prove something obvious? he’s weird, autistic even, it seems.
prizes and tenure are awarded at the subsequent meeting of the Science Team for all the papers written detailing how parachutes work, and for refining the design until it is feasible to mass produce it and use it in all relevant settings. nobody did an RCT to verify that parachutes work.
Leo and Igor are happy.
Leo remembers the “heh that’s funny” about the terminal velocity …
so yeah… there are other ways to know things besides running studies and having controls. most of everyone’s knowledge about the world is based on evidence that has nothing to do with statistics or studies. but it was fun to write the parable above. :D
also, fun fact, this is a lot like how the airplane was first developed! if you haven’t read about the Wright Brothers, consider doing so. it is interesting, more than you think.
EDIT (thanks, Sam!): somehow i didn’t know about Jack Parsons until i was today years old. the story of the rocket is also something like the above. interesting, more than you think!