Wanting to upgrade my P3 to either a BMC TM01 or a Cervelo P5, but noticed that neither company has released “white papers” on their bikes. Of course both marketing machines will spout that theirs is the fastest bike in the world…AND I don’t purport to be an expert on the drag coefficient numbers etc, but a comparison would be nice to see…
I wonder if after the recent splurge in “white” papers released, in the last round of super bike releases, and the often lack of objective testing done, companies no longer wish to release these in house tests as they don’t want the negative connotation of a biased testing procedure?
I like to see them, but I can see a companies reluctance. First you are going to have to assign someone with a lot of knowledge of the testing to answer questions on various sites or you’ll be accused of hiding something. 2nd bias can come in two forms. One is unintentional, if you test using a specific protocal and design the bike to your test it should ace it. The other is intentional, things like leaving off a competitors bike that tested well.
Either way a white paper is at least a little grey.
Any given white paper is often more an exercise in marketing than science due to the variables involved. What you can get out of all these competing whitepapers is a general sense of which bikes consistently test well (ex. the P4 does well in most tests, the Ordu doesn’t). Amongst that group of fast bikes, the fastest one for YOU is the one you fit best on (i.e. you can get get you position dialed in without using a ton aero-killing headset spacers).
Take five athletes and test them on the P5, Shiv, SC, DA, and Ilicito and I guarantee that one bike will not test fastest for all five. So you can use the approach above, do what Crowie did (buy each bike and some tunnel time and find out which is best) if you have mad $$$, or go the full-on marketing route and just buy the bike ridden by whomever had the fastest bike split at Kona last year.
" What you can get out of all these competing whitepapers is a general sense of which bikes consistently test well"
Thanks for the replies.
I think this sums up the main reason why I like seeing the test results. Reading rappstars article in Lava “the opacity of the white paper”, it’s evident that the number of variables in the companies’ test protocols leaves far too many margins of error to be able to say I trust one over the other, but as you rightly pointed out. Consistently “good” results across various tests has to count for something!
I noted on the QR thread that we are reaching a point where it is becoming exponentially harder to develop the “fastest” bike, so the “value” of releasing whitepapers is diminishing. The bikes out there today, especially the “superbikes” are so freaking good that it makes it hard for the smaller and mid-level companies to compete.
Add into that the point Mark Cote has made, the recent post by SD and the points raised by Jordan in his Lava article and things start getting real cloudy, real fast. We all look at the test results to see which bike is “fastest” (see the Venge vs. S5 thread). If you aren’t going to “win” the shoot-out, why release data? And if you do “win” the shootout, people will throw the BS flag on your protocols, the competitive set, the position of the dummy (or if one was used), etc.
One of the key issues I have is that, coming from a medical background, proof of a product comes from repeatability in results using standard testing protocols. These results are then scrutinised by the wider medical field and correspondence entered into between the scrutinisers and the original authors. Within reputable medical journals there are also clauses used by the journals that require publication of data despite negative findings.
Now extrapolating that to this scenario, I can appreciate that Cervelo undertake some very stringent testing (as explained by your link - thank you). However (and please excuse my cynicism), your testing is still essentially a matter of n=1. Admittedly, that 1 is very transparent and for all intensive purposes very accountable, but without repeatability from other companies (using the same tunnel, same set ups, same testing conditions etc) there will always be some skepticism with the results. If every other major company tested the same way, against a standard protocol, the “n” would equal at least 10 (Trek, Cannondale, Specialized, QR, Felt, BMC, Cervelo, Blue, Scott, Giant, Orbea etc).
Speaking as a member of the general public, Cervelo and yourself should be applauded for your openness to reveal your testing procedure(s), but as your goal is to sell the maximum number of bikes to the public, which has a direct relationship to how aero/fast that bike is perceived, this leads to the possibility of bias in your result (or perhaps bias in the results you wish to release).
Thanks for your thoughts. I understand where you’re coming from.
Inside Cervelo we’ve discussed authoring a draft of some kind of standard wind tunnel testing protocol document and inviting a inter-company discussion aimed toward industry-wide acceptance, but so far the internal debate has stalled such a project. As you can no doubt understand, we feel that revealing some of our hard-earned (over 16 years!) wind tunnel know-how could potentially hurt Cervelo’s competitive advantage by teaching our competitors, but on the other hand such a standard could potentially help add credibility to wind tunnel test results in general, just as you suggest, by adding consistency, repeatability, independent verification, etc.
What do you (and others) see as potential pros and cons for the various parties that might be interested in such a standard?
In the absence of an industry standard, one approach would be to make your own standard more transparent by not conducting the testing yourselves. So you would document how the testing is to be carried out, and it would be carried out by someone independent, with Cervelo staff simply observing to check that the documented test process is correctly followed. Then when you publish the results, you publish them along with the documentation of the test process that was given to the people who carried out the testing.
This would, of course, still have the issue of giving away knowledge. But it would at least allow someone else in theory to replicate your results if they follow the published test process, which would need to include a CAD file to allow someone to make their own identical DZ dummy, for example. It would be a unilateral standard, that others can adopt if they want, but if others don’t adopt it, customers can see every last detail of what you’ve done and judge for themselves if the test was a fair one.
As an example of the sort of difficulty the customer has at the moment, there is a substantial disparity between Cervelo’s P4 test and the Tour magazine test, when looking at the relative performance of the P4 and SC9 at zero yaw, with the gap around 2.5x larger in Cervelo’s test. The only differences I can see from the published data are the wheels used and the fact that the dummies were different. Perhaps I would still be as puzzled by the disparity if even more details were published, but I would like to at least be able to have a go at spotting possible sources for the discrepancy, with exact details of frame size, stack, reach, how it was achieved in terms of Ventus stem angle options and SC stem options vs aero spacers under the pads etc.
I still see such a suggestion as one-sided, and not in Cervelo’s favor. I think you can understand why we would be reluctant to go to all that work to simply make (even more) public exactly the things we’ve spent so much time, effort and yes, money to discover on our own, for our own benefit, over such a long time. I’m having a hard time seeing where Cervelo could gain benefit enough to make it clearly worth doing. Help me argue inside Cervelo *for *such a plan.
As for the TOUR magazine testing, I know those guys and they’re nice guys, smart people, I mean really smart, but their wind tunnel testing is just beginning, roughly where Cervelo was over ten years ago. They know this; just look at the error bars on their data and compare how many bike ranking places they cover.
FYI here is a partial list of things they could change to improve their testing. I sincerely hope they make these changes (and more!) because the breadth of the customers they reach and the authority they have (IMHO not justified in this arena, yet) is unparalleled.
These reasons and more are why I don’t (yet) give much weight to their wind tunnel test results.
I’m not familiar with any of the TOUR folks, but x2 on everything else. My comments in another thread related to the effects of poorly executed/presented work aren’t limited to what comes from OEs.
It may be that I’m unusual, but I am heavily influenced by the quality and transparency of the information a company releases to build a case for their products. I absolutely would not have bought a Speed Concept to replace my Transition if Trek hadn’t produced the white paper that they did. And although I have a P5 on order, the shop would be ordering it for stock anyway, and are okay with me not going through with it if the wind tunnel data isn’t satisfactory when it appears. So basically, Damon, your case to Cervelo should be that Cervelo has to make a strong case to its customers.
I feel that a huge problem with the P4 was that the data just didn’t appear soon enough. If the data that was released in March 2011 had been released earlier, it is entirely possible I would have bought a P4 instead of a Speed Concept in late 2010. I like the look of the BMC TM01, but without any wind tunnel data from the company, it would be such a huge gamble to buy one, that I’m just not willing to go for it. Even if I were willing to gamble the cash, I’m not willing to gamble the time it would take me to establish how fast it is relative to my current bike, I feel that is a job the manufacturer should be doing for their customers, it’s just not feasible or efficient for every customer to individually buy all the bikes that look like they might be fast and test them.
But of course what would be even more efficient would be to have an industry standard protocol, and every time a manufacturer releases a new product, it is simply tested by a 3rd party company to that standard, and the results published. The whole industry could fund this 3rd party company to conduct these tests and publish the results. It would then simply be about each company trying to make a bike that performs the best, analogous to how cars are all tested for fuel economy according to a standard (at least they are here in the UK, I don’t know if you have the same thing in the US). Even though the test is standardised, not all companies are equally good at making cars that perform well at it. So I don’t see that the best companies have lost anything through the process of standardising car fuel consumption tests, it has given them chance to shine. The companies who should fear such testing are those who don’t have the expertise to create a product that will perform well in the test, but then they can market their products in other ways such as aesthetics.
So the ultimate question is, if there were an industry standard test, do you believe that Cervelo bikes would win? Are you confident enough in your expertise, that you believe you can still in future years stay ahead of the competition and keep producing the fastest bikes? If the answer to those questions is yes, I would say it is in your interests for it to happen. The alternative is to have independent testing by the likes of Tour magazine, which you may be less happy with, but people are going to keep on looking to 3rd parties to conduct such tests if they aren’t happy with what the manufacturers are producing. People have become more knowledgeable about these matters through forums such as these, and I don’t think the genie can simply be put back in the bottle, it’s out there for good now, the desire for this information isn’t going to go away.
Lots of interesting debate in this thread, and I agree with a lot of what you said here Steve. I too dissect manufacturer white papers pretty thoroughly, and am not satisfied with the lack of data on the BMC.
However, it’s interesting you use the fuel economy test as an example of a 3rd party test (I know you weren’t saying it was the ideal test), because it illustrates some of the problems with instituting this kind of thing for the bike industry. The danger is that you get manufacturers designing for the test, not the task. Ever tried to actually hit those combined fuel economy numbers? Unless you have a diesel it’s almost impossible. Similarly, the EU-regulated “drive by” noise test is utterly useless. Manufacturers of noisy cars didn’t make their cars less noisy, they just put valves in the exhaust that limit noise when on the partial throttle used when conducting the test. All the latest Ferrari cars pass that test, but they sound like hell’s bells when on power.
Even though I took issue with QR in their “we design bikes for low yaw” thread, I’m actually glad they’re doing that, because it gives the customer choice. Do we really want a test that weights 0-15 degrees yaw in a particular way and therefore effectively force all bikes to be judged according to that methodology? And Damon sounds pretty convincing when talking about how a lot of LSWT expertise actually resides within the manufacturers making the bikes. I’m not sure how much I would trust an independent test (ie not necessarily more than white papers) unless they were very good and very experienced. Where do we find these people, and how do we get manufacturers to fund them? There’s always going to be one winner and x losers, meaning it’s probably not worth it for the majority of manufacturers. Simon Smart and the Scott team did a great job with the Plasma 3 and Foil, but I don’t know how many other people you could tempt out of F1 or similar to get into our pokey little industry.
However, it’s interesting you use the fuel economy test as an example of a 3rd party test (I know you weren’t saying it was the ideal test), because it illustrates some of the problems with instituting this kind of thing for the bike industry. The danger is that you get manufacturers designing for the test, not the task. Ever tried to actually hit those combined fuel economy numbers?
Nonetheless, even if you can’t hit the absolute numbers, I do find that generally a car that has better figures in the test gives better fuel economy on the road.
Even though I took issue with QR in their “we design bikes for low yaw” thread, I’m actually glad they’re doing that, because it gives the customer choice. Do we really want a test that weights 0-15 degrees yaw in a particular way and therefore effectively force all bikes to be judged according to that methodology?
No, definitely not, I’d always want to have the raw drag vs yaw data available.
And Damon sounds pretty convincing when talking about how a lot of LSWT expertise actually resides within the manufacturers making the bikes. I’m not sure how much I would trust an independent test (ie not necessarily more than white papers) unless they were very good and very experienced. Where do we find these people, and how do we get manufacturers to fund them? There’s always going to be one winner and x losers, meaning it’s probably not worth it for the majority of manufacturers. Simon Smart and the Scott team did a great job with the Plasma 3 and Foil, but I don’t know how many other people you could tempt out of F1 or similar to get into our pokey little industry.
I just think it has to be cheaper to have it done once than to have all the companies doing their own comparisons. Someone like Simon Smart might be a good person for people in the industry to approach to take on the job, but clearly he’d need to do it instead of designing products, to avoid a conflict of interest. Even if a company knows its products might not be as aero as the best, they may still see aero data as compelling for them if they feel they provide very good performance at a lower price point. Or even showing that their performance is acceptably close to the best might be enough if they have other merits, e.g. some people will happily ride a bike that is 2W slower if they prefer how it looks. So I don’t think it is all negative for everyone except the most aero.
**Nonetheless, even if you can’t hit the absolute numbers, I do find that generally a car that has better figures in the test gives better fuel economy on the road. **
Generally yes, but their have been some clunkers, such as the Honda hybrid thats part of a class action suit.
No, definitely not, I’d always want to have the raw drag vs yaw data available.
The big issue is with one standard test, all companies will start desinging to do well on that test. If you are comany A and think the test is great fine, but if company B thinks the test favors certain yaws more than it should its a problem. Do they make what they believe is faster, and have to try to explain that the test everyone else uses is bad or do they cave and make a slower but better testing bike?
I just think it has to be cheaper to have it done once than to have all the companies doing their own comparisons.
Very few, if any companies do this. Take QR, they got a lot of critcism for mincomplete tests and their response was we don’t test to show we’re faster we test to help design the bike (paraphrased).
In answer to your original question … yes, apparently, white papers are now black-listed. Cervelo is not releasing the data on the p5 tunnel testing. Not good news.
