Help me understand differences in bike sizing/measurement

My first bike was a 2006 Fuji Newest 2.0 (size LG) that I rode from 2009-2014. For 2016 I’ve been on a 2012 Cannondale CAAD 10 (size 54).

For the past few years I’ve constantly struggled with comfort on the bike. As I’ve thought more and more about it, I realized I used to be on a bigger bike with the Fuji compared with the CAAD and I’m wondering if the CAAD is a size too small for me.

I just looked up the geometry charts and I think I’m right…but I was hoping someone could help clarify where the big differences are. It seems the top tube length on the Fuji is 2.5cm longer. Would this make that much of a difference?

PS As a side note, I’m on a Med (54/53cm effective top tube length) Fuji D6 for my race bike and after a quick fit with Slowman at my local Road Show, he said it would be better if I was on the M/L (56) or even the L (58cm). This also has me thinking the CAAD is too small.

Here are the size charts:

http://i66.tinypic.com/b6zkth.gif

http://i64.tinypic.com/1h8si1.jpg