Swimming on a 1:20 base interval* has NOTHING to do with your parents. I recall the first time I did 100s on 1:20. I was 9. Most of the kids in the group were 10-12, but there was even a 6 or 7 year old in there. There are tons of tiny little 4 four-foot-six-inch children at school right now who will swim on a 1:20 base interval this afternoon. It has nothing to do with genetics. Frequency, intensity, duration. Measure and quantify each amount each week. Steadily increase overall training stress. The thing the 4’6’’ inch kids have going for them is they swim more than you or anyone in your Master’s group does. That is why they are faster. Don’t compare yourself to what other triathletes do at a Masters workout. That will only lead you to mediocre results. Watch what the 10 year olds do; they swim every day, they swim hard, they swim for over an hour, THEY GET FASTER. At the same time you will kill them on the bike and on a run; that doesn’t really make sense does it?
Genetics might start to play a role at around 5:00 for a 500, but pretty much any boy/man can get to 5:00 for a 500 with a little hard work at the pool on a very regular basis. I think 5:00 is actually pretty conservative.
*Disclaimer: I am assuming you are talking about swimming freestyle. If you are talking about kicking without fins, without a board, then 1:20 is a tough interval. If you are talking about breaststroke, then 1:20 is a fairly tough interval.
LOL!!! Perhaps the funniest thing ever written on ST!
Yes, it’s “Frequency, intensity, duration.” as you said, PLUS TECHNIQUE (which CAN be limited by genetics… you might not be able to physically do the right technique OR be unable to “get it”). Those 6-7 year olds have been swimming for what, 3 years? 2 years? Here comes the n=1 part. I’m 3 years into it, and I don’t think I could swim a 25y in the :15 it would take to crack a 1:00 pace. And I’d bet I’ve put in more work than your usual 6-7 year old.
So I’d take the near opposite of your theory. Only a very very small fraction of people could ever get to 5:00 for a 500y. So why can so many people do it? Because they are the small minority that made it through the filter. The masses got weeded out. They ended up playing other sports, or swimming in the dumpy rec center as became “strange denizens of the pool”!!! 
I don’t always agree with you, but I am taking your side on this one.
I’m currently working with a swim coach and I’m pretty happy to be swimming consistent sets of 4x3x100 (he’s a TI coach so I am working a lot on technique). My times are 1:33-35 and I’m doing them on 20 secs rest. To go from that to 1:00 100s for 500 yds will be impossible for me. I say impossible because I could not swim enough to get to that point and still do any running or biking (with our without PowerCranks
Just as I can categorically state that I’ll never crack a 3:00 marathon, I can categorically state my swim could NEVER get that good…a combination of my current form & fitness coupled with my POTENTIAL…just isn’t going to happen. Anyone who says that 5:00 500 is conservative…is either kidding themselves or is simply just so immersed in being that good they don’t know what it’s like to be a “normal” age grouper without a lot of athletic talent.
I train around John Reback (sister to Laura Bennet) and see him at the track and on the road frequently. It’s depressing! He is not out there any more than I am, he doesn’t work any harder than I do…he is just naturally way more athletically gifted than I am or ever will be! Of couse, he trains too, but still, he could go for a year and not train and STILL beat me if I had been training 15 hours a week. That’s just the roll of the genetic dice. Yeah, training intensity, duration, frequency and focus play into it. But when push comes to shove, if I am operating at my VO2 max and so is he…he’s gonna win. And sorry boys and girls, VO2 Max is only trainable by about 10% (depending on which study you read). If everyone could become Lance Armstrong, or Chris McCormack, don’t you think more people would? The fact is, athletes of that caliber make up only a small fraction of a percent of the population…hence the term “elite”.