Do you have a rear rack? You will need one for any type of pannier setup.
Racks for fatbikes are kinda big, cumbersome and expensive due to the width needed for tire clearance. If you do go this route, Arkel dry lite panniers are really nice. I have a set and they work well.
A better choice might be a frame bag of some sort, try Relevate designs for ideas. Kinda pricey but very good quality.
Backpacks are a hassle. Hot, heavy, no fun on the neck and shoulders. Let the bike carry the weight!

I bike 4 km to my pool and back with a small, lightweight daypack on my back. You don't need much. Just something big enough to hold your gear. Twenty minutes is no distance to ride with your swimwear, unless you have a ton of it. I carry, a towel, paddles, pull buoy/flotation shorts, sometimes fins. I don't change out of my wet shorts after the swim as I bike home to shower and change.

Don't overthink it. It's only 20 minutes. It's not a major expedition.

Something like this - Foldable Pack

---

Trust me I’m a doctor!
Well, I have a PhD :-)

I don't have a rack *yet* but I'm looking for those as well. I'm thinking of getting this bag: https://www.bikeradar.com/...ckpack-review-51023/

Forget the rack. Put the backpack where it belongs, on your back.

My commute is 35 mins by bike and I still use a backpack.
Panniers are meh.