Jloewe wrote:
....I don’t mean to say that lab results are useless but more that they’re exaggerated. Even if the data is 100% spot on that result is not going to carry all the way to the real world. If I find a way to make a car 50% more efficient on a closed track taking it out into the Wild you’re never going to get close to those results. It’s the same with everything we use on the course. If you’re getting 15W in the wind tunnel you’re getting <15W in the wild. I studied a lot of science in college. One of the drawbacks of a controlled environment is that we don’t live in a controlled environment therefore our real world results will always be deminished. Whether it’s the amount of theoretical watts gained by a new piece of gear or the training effect of a new technique. Once you start including real life scenarios the effect is always deminished. The real world where we face wind, and hills, and traffic. Where the road isn’t always even. That’s what I mean. Any lab results are inflated by the controlled environment. Especially rolling resistance and aero properties. Particularly because of velocity.
Prime example... look at Garmin battery or car MPG. Most people, unless they go specifically out of their way don’t get anywhere close to what was advertised because of too many factors.
Okay, let's use this example.
Since Garmin watches neither store the same amount of usable energy, nor use the same amount of power in all configurations and conditions, any single figure can only be representative and based on a bunch of assumptions. That's not a weakness in controlled testing, it's simply the world not being as simple as you may like. There's a difference.
The same goes for car fuel economy figures. In both cases you can assume the manufacturer selects test cases for advertisements that provide favourable figures and cast the product in a positive light. You are therefore likely to see worse figures in your own use. That's selective data use by manufacturers, not a weakness in controlled testing.
I really think you're misinterpreting incorrect application of the data from controlled testing, as a flaw in test methods. This is inaccurate. In fact what you're observing is a flaw in application.
It is certainly not possible to determine the veracity or applicability of controlled test data for variables of this scale via uncontrolled field testing, especially when you don't have a means of measuring the more critical parameters.