The 1.5 miles is an approximate straight-line distance from the boat to the Yacht Club.
The fastest swimmers will be able to swim closest to a straight line. Slower swimmers will swim a large arc that goes toward S. Francisco, first, then toward the Yacht Club. The current toward the Golden Gate Bridge will be ~4 mph, so all swimmers will be carried in that direction with the current. In essence, you're swimming across a river with a brisk current and landing at the opposite shore about 1 mile downstream. The swift current accounts for the relatively quick times for the distance. There is typically a moderate chop and swells in the middle of the bay that are 3-4 feet.
Larry Creswell
http://www.athletesheart.org, @athletesheart