Using a riser block isn't going to make it harder or easier it simply changes your bike angle from neutral/level to simulating inclines which allows you to change the way your muscle groups are used/engaged, more quads for example in climbing (seated).

Does the, at most, 3" between then lowest and highest setting on the riser block really make a difference?

It can, depending on how hard you are working. You can always increase the incline by putting something under the riser, phone book, wood block etc.