This topic piqued my interest, so I did a little Googling and found this summary of a study about fat and swimmers:
http://www.pponline.co.uk/encyc/0346.htm
…
To see if feasting and fattening is really a good idea for swimmers, scientists at the University of Miami artificially increased body fat levels by 2 per cent or more in a group of 10 male and female swimmers who had been swimming competitively for at least three years. ‘Fatness was enhanced by fitting latex pads under a spandex triathlon suit in the swimmers’ adominal, hip, thigh, chest, back, and buttock areas. Microscopic balloons were added to the latex so that the pads had the same density as actual body fat. Male swimmers attached a total of 3.3 pounds of artificial fat, while females donned an extra four pounds. Each athlete swam a 50-yard freestyle race as fast as possible, with and without the pads.
While the latex pads did improve flotation, they also slowed the swimmers down considerably. The athletes could rip through their 50-yard sprints in about 26.6 seconds without the added ‘fat’ but required around 27.4 seconds with the additional fat on board. Thus, each additional pound of fat slowed 50-yard swim times by approximately .2 seconds.
Why did added 'fat' slow performance, even though it improved buoyancy? While supplemental fat can reduce friction drag, it can actually expand something called 'form drag,' which is determined by the dimensions of a swimmer's body. Specifically, as a swimmer fattens up in the abdomen, thigh, and buttock areas, swirling eddy currents form around these protruding areas and can slow swimming velocity appreciably.
A second kind of drag - ‘frontal surface resistance’ - can also make it harder for corpulent swimmers to get through water. Frontal surface resistance is a function of how much body you actually have. If you have a big body, you have more frontal resistance, because there’s more body to push against the water (‘more front for the water to confront’). Instead of slipping through the water like a torpedo, you constantly bump into it. As water crashes against your large surface, it slows you down. Thus, even though fat helps by getting you out of the water, the part of a fatty body which remains in the water impedes progress. Perhaps the ideal free-style swimmer would have a fat back and sides but a slim anterior, while a backstroke specialist would have a paunchy tummy and flat buttocks.
…
Also, the Miami study doesn’t necessarily mean that fat is always bad for swimmers. The swimmers in the Miami research were not bone thin; males possessed 11-per cent body fat while females checked in at 21 percent. It’s possible that thinner swimmers might have actually benefited from increased body fat. The idea would be that an ultra-slim swimmer might slip too far down in the water column, hiking frontal drag to performance-hampering levels. Also, given the prevalence of eating disorders in athletes, it’s unwise to put pressure on swimmers to lose large chunks of weight. The bottom line is that we don’t yet know the true impact of body fat on swimming performance - or which level of fat produces the fastest times.
There’s some bonus information in there about wetsuits and triathlon times (summary: wearing a wetsuit makes you 7% faster).
I’m sure this isn’t the last word on the subject, but it does suggest that a little more fat does mean a little more drag - never mind moving more mass through the water. And it jibes with my own anecdotal experience.