The only small issue with an alloy brake track is having to swap pads as well as adjusting brakes as most of the wheels I use on my TT bike are carbon. It's not a deal breaker, just mildly annoying.

Like you said the other option is to get a set of Jets which would make wheel changes super easy but that's a bit more than I'd like to spend at the moment, especially considering I'm quite fond of my PD24s.