Wouldn't recommend taking out the 12 tooth cog at all. I calculated the gear ratios for my bike and the jump from a 53-11 gear ratio to a 53-12 gear ratio is over 10 gear inches which is absolutely massive already, it's most likely the biggest jump in gear ratios in your drive train.
If you need that gap filled, you can buy the individual 12 tooth 10 speed cog. I threw the link down below.

https://www.jensonusa.com/...eYb0PmUaAkXqEALw_wcB

My thinking is that I’m only gonna be using 11 or 12 while headed downhill and they would just be a way to add a little speed to gravity’s work. So the jump wouldn’t be as impactful as it is further back in the cassette (where it would admittedly be a smaller jump).

The two smallest cogs in a Shimano cassette are usually specific to those places, so you can't just swap those out or around. You can probably find an MTB cassette that starts with 11-13 that you can use those cogs from.

Could you pull the 16, 17 or 18 out and use the 11,12 from one cassette on the other?