Actually, I fall into that category. (Oly 5k about 19:xx and bike spilt about 1:03xx. Swim = terrible!)
I learned to swim 2 years ago. Zero swim background. Spent an entire year on technique and nothing but technique, and only started working hard recently.
I'm sure I have some technique problems, but I've been going to a good masters program of late, and the good coach there (locally respected) agrees that at this point, I'll only make minor gains with technique. Definitely nothing like 20-30sec/100 off.
It all comes down to background. I've been running for over 15 years, and ran 8 marathons, two under sub-3. Add my genetic big legs, and it's no surprise that I can run/bike at a respectable clip.
But on swimming, the limiter is clearly my swim-related endurance and power. There simply aren't enough capillaries built up in my swim muscles to function efficiently, and I can feel it. Even though my cardio might be capable of a good bike/run, my arms/lats/back are weak, and force my heart to pump super hard just to keep up. It feels exactly as I do when I do a one-legged bike drill - HR will still spike extremely high since your heart is trying to pump more blood through your entire body to supply a fast-fatiguing area.
It took me years to go from a 25min 5k to a sub-18, and a huge amount of work. (I'm not one of the gifted ones who did it with <35mpw). There's no reason to believe that it won't take similar work to get similarly fast in swimming, years, possibly.
If you doubt this is true, consider the fact that at least two guys in our local tri club who started triathlon several years ago (earlier than me) nearly qualified for the Olympics in swimming in their prime but aren't even close to FOP on the bike/run despite focusing on bike/run now. (Granted, they are almost definitely not training as hard as they did on their swim days, but still, point remains.) Swim fitness and run fitness do not necessarily overlap.