It's kinda like in swimming terms, who's faster, phelps or leclos. But you put each of them in pools in the ocean subject to currents and waves, you let the salinity and temperature vary considerably without measuring it, you change the height of the starting blocks, the pool is "roughly" 50m long, but you aren't quite sure because each wall is free floating.

How many trials do you need to run to get a meaningful number?

clinical trials are controlled and randomized, and typically have very large sample sizes for phase 2 trials.

Anybody who knows statistics knows that with enough sample sizes, you can tease apart even the smallest differences (standard error, sample size is in the denominator) and something this straightforward lends itself extremely well to repeat outdoor testing.

It's obvious that you don't know anything about clinical trials. The effect sizes that are detected in medical trials are typically large (50%+). Small effect sizes (2-5%) are basically undetectable in anything but the very largest trials, which involve tens thousands of people. Even these are very hard to separate from confounders. The gains between top bicycles are small, between 1-5%.

You're also saying that every major bike company in the world is wrong.

Think about that.

Yes, and we should do the same for bike trials. You will absolutely need far, far smaller sample sizes, as the testing is much more similar and fixed, and the samples aren't from as diverse a range as the general human population. You're dealing with one rider, one route, and similar weather conditions (wait for the days, do it it SoCal if you must.) MUCH easier to test than squaring off two bike frames against each other and asking whether there is a meaningful, measurable speed difference outdoors.

The 50%+ required drug effect you throw out there is patently false, that's for sure. Not sure where you get that one.

No, it wouldn't.

Do you seriously think testing how fast a bike frame outdoors is more complex than testing medications across a range of human patients in the public? Yet that's what exactly medical drug and clinical trials do - across expansive populaces and across many different disease types. It doesn't always work, but it is effective enough that it is a critical point in medical testing to acknowledge that even if you can prove a 'statistically significant result p<.0005', the small magnitude of the finding is clinically irrelevant.

To someone like me, all these claims of 'wow, outdoor bike testing to tease out real world-outdoor frame differences' are just handwaving in the face of the very possible, if not likely reality, that there is essentially no measurable difference in speed between these frames once you're in an outdoor, noncontrolled racing environment.

Gear that has a measurable difference, like aerobars, don't suffer from such problems, precisely because the magnitude is greater.

If the magnitude of the difference is so small that it's undetectable in race-like conditions, and requires a $100k wind tunnel to detect it, the most likely explanation is that the aero effect is so little that it is meaningless in the face of real world variables.

In the tunnel I've used historically, I'd suggest doing this set of 12 runs:
beta =[0 10];
drop = [0 -2 -4];
reach = [0 8];

...for each of the frames. this ought to give you the statistical power to detect a CxA difference of ~0.0025 with a p value of around 0.05 (about 80% of the time). more importantly, though, it has a false positive rate of 0%, and a reversal rate of about 0.1 %. This set of runs would take about 90-100 minutes in a tunnel and on top of that, you'll get real good heuristics for reach and drop (and insight into any interactions).

If you want to field test this in a velodrome, I'd suggest doing this matrix ~6 times depending on how good the repeats look (you can't do the beta runs in the velodrome, so you'll need to double up the runs to compensate) in order to give equivalent stat power. Oh dear, that sounds like a piece of work! Based on the field testing I've done, that test would take a whole heckuvalot longer than 2 hours.

Many RCTs will start with the assumption of a medium Cohen's d of .5...that's where he got it...

"If the magnitude of the difference is so small that it's undetectable in race-like conditions, and requires a $100k wind tunnel to detect it, the most likely explanation is that the aero effect is so little that it is meaningless in the face of real world variables."

You keep leaving out THE critical difference between bike performance and clinical trials. A tiny difference in blood pressure does not make a clinical difference, that is true. On the other hand, if a 1 mm Hg difference in pressure gave one patient a 1 million dollar "health" prize while the next got got nothing, all of a sudden it matters a great deal.

It is immaterial if the advantage of a big frame gives 3 minutes to a rider, but that difference is obscured by the time variations of more important variables on race day. He still got those 3 minutes.


If he wins by three minutes or less, that means everything.


Your analogy with clinical trials only holds water if your are talking about MOP racers who do not care about their time or placing. Then those few minutes do not really matter. But a lot of people are chasing podiums, or PRs and care a lot about small time differences.

Bike radar doesn't need to...I already did it :-)

http://bikeblather.blogspot.com/...00&max-results=5

I don't think I would call ~2.5s/km of difference "potentially insignificant". You just lost that virtual bet you made ;-)

Great link

It is immaterial if the advantage of a big frame gives 3 minutes to a rider, but that difference is obscured by the time variations of more important variables on race day. He still got those 3 minutes.

If you were in the market for a new aero bike (your budget was $7,500) and Wind Tunnel testing was done in a "side by side test" at A2 by an independent entity and the difference was 100 grams of drag: 2 watts.

Is that a difference that you think you could translate to the real world (absent of any additional data)?

Would you purchase the faster bike (according to the Wind Tunnel testing) if it cost $7,500 vs the other at $4,500?

or

Would that 2 watts be to close to the margin of error to justify the additional cost?