I would go with the XL.

Large with 40mm spacers and +8deg stem is a LOT of rise above the headtube.
XLarge with 40mm spacers and -8deg stem is still a fair amount of rise and you have plenty scope to reduce height just by removing spacers. If you ever needed more than that you could go to a stem with a larger angle and end up way below where you are on your existing setup. The frame is not limiting you in this regard.
Don't let the change of a stem be a deciding factor. It's a cheap enough component to change, if the supplier won't swap it for you FOC.

I'm very surprised a fitter suggested 40mm of spacers to be honest. It's not really good practice to use that many. I think many companies recommend a max of 30mm.
Another thing to consider is that in addition to reducing the distance from the head tube bearings to the handlebars (the big moment arm on the L size would concern me) the longer head tube on the XL will provide a better supported steerer purely by having more distance between the bearings.

Of course there's also the other factor, to some it's the most important: Loads of spacers and a rising stem will look rubbish.

Thanks for the input.

The 40mm of spacers is really 25mm of spacers plus the headset top cap which is 15mm. My initial thought was to go for the XL since since 40mm is spacers sounded like a lot, but when I went into the shop they had a large built up with 40mm and an upturned stem and it didn't look bad.

When the fitter was adjusting the height of the front end I really couldn't tell much of a difference, so I think I have some flexibility there, but changes in reach were much more noticeable. The difference in reach between the 2 frames is 19mm. Getting the XL doesn't leave me much room to play with if I want to decrease the reach in the future, especially since I would have already shortened the stem down to 100mm.

What's the reach on the bars like?
If the fit is decent, it's unlikely you'll need a big reach adjustment later but if there's scope to, say, change saddle position it could happen. You can drop to a shorter stem. An 80mm or 90mm stem would be okay. You can also change to bars with less reach if the original bars are not already quite compact.

I recently was in the same boat as you as I can fit on most 56cm and 54cm road bikes. I would generally opt for the 56cm, but recently bought myself a 54cm Trek Madone. I have enjoyed being able to tinker with the length and getting lower by flipping the stem and slowly removing spacers while using a 120mm stem. I don't think you could go wrong with either size, but dropping to a lower size has been a fun experience for me so far.

The bars on the fit bike had a 70mm reach. The stock handlebars on the Defy come in 70 and 75mm but Giant's website doesn't specify if the larger bikes come with the 75s or not. Wouldn't shortening the stem down to 80-90mm start to affect handling?