I recently picked up a new set up wheels which are wider then my previous ones. These new ones have an external width of 24 mm. Im running FSA Gossamer brakes and only have maybe 1 mm between pads and the new wheels. I've tried to set the brakes to their max width by loosening the nut on the side but this only give me that 1 mm on either side.

My question is, is it ok to remove the concave washer that is used to adjust the brake pad or should I just sand down the brake pads?