On passable roads (with all other things being equal), when inflated to appropriate width-specific pressures and when mounted on matched width-specific rims, wider tires/wheels are pretty much always slower than otherwise identical but narrower tires/wheels.

So what you're feeling is real ...

---

Advanced Aero TopTube Storage for Road, Gravel, & Tri...ZeroSlip & Direct-mount, made in the USA.
DarkSpeedWorks.com.....Reviews.....Insta.....Facebook
--

Perhaps you missed this article: https://www.bicyclerollingresistance.com/...prix-5000-comparison
(Continental Grand Prix 5000 Comparison: 23, 25, 28, and 32 mm Compared)

Almost no rims are optimized for aero on 32mm tires, so that's going to hurt. If it a course with lots of acceleration and decelarations you also have extra weight at the rim. You should be able to get them down to the exact same Crr for the same brand tire in 23 or 25