What they're saying is just not true. There is a power and fitness component to swimming, but growing up I swam with plenty of very fast kids who were otherwise not particularly strong. Technique is more than just how you move through the water, it's also how you apply force to the water.
There is a correlation between those two things you noticed. Once you start your pull, you need to pull hard, your hand should be accelerating all the way until it exits the water. You may try doing some sets with fins to let you feel this without tiring yourself out. Another place to work on it is by doing some fast lengths. Important to give yourself plenty of rest when working on swimming fast or it'll be counterproductive.
There is a correlation between those two things you noticed. Once you start your pull, you need to pull hard, your hand should be accelerating all the way until it exits the water. You may try doing some sets with fins to let you feel this without tiring yourself out. Another place to work on it is by doing some fast lengths. Important to give yourself plenty of rest when working on swimming fast or it'll be counterproductive.