Personally, I would be leery of roller skiing anytime soon after back surgery. Perhaps you are looking months ahead and I am unnecessarily worried. I didn't grow up on nordic skis and it sounds like you didn't either. I am not very agile; you likely have a better starting point. Over the years, I have worked my way into wave 2 Birkie, may or may not ultimately get into wave 1. And still I am a cautious roller skier. Also, there is a good chance that you are more brave than me.
I echo the sentiment about skill over fitness. Yes, partly because finesse beats brute effort across snow. Just like swimming. But also for simple safety. If you try hard, you will fall hard.
My approach to off season ski training is to use roller skiing for technique only. Swim or use ski erg for upper body fitness. Run or bike, or use stair machine, for lower body fitness.
I count on falling every time. Usually I fall when I am really trying to prolong glide on one ski, catch a pebble, go down. These are low speed tumbles and I have never gotten more hurt than a little scrape.
You didn't ask for this advice, but here it is anyway. (These tips came from a helpful middle aged non competitive saint of a woman who educated me nicely for 30 minutes one day. It was a pivotal moment for me.)
find a small safe area of flat smooth pavement, like the corner of a quiet parking lot
--balance on one boot, no skis, reach down and touch the ground, 10x, repeat both sides
--stand on one boot without ski, put second boot into ski, return to put first boot into ski, if you sit on your butt then you are not ready
--stand on two skis, do 20x little up and down hops
--stand on two skis, do 20x little hops rotating from 11 o clock to 1 o clock, then do 9 to 3, then do 180s, then try a 360
--stand on two skis, get feet a little wider than shoulder width, turn tips in about 3-5 degrees, do 10 quick stutter steps in place, this is your basic stop technique
--now, skate about 5-10 strides slowly, and do the stutter stop, repeat many times gradually getting more speed
--now, skate about 10 strides perpendicular toward the grassy edge, transition to jog when the wheels bog down in the grass
--skate in a large figure 8 for 5 minutes, then make it smaller and tighter
--skate about 50 meters moving left and right around cones of an imaginary slalom
--learn to skate backward
--learn to do froggies
find a couple of miles of good pavement and flat terrain, enjoy yourself
winter is too short (for most of us), you can easily have a positive impact on your skiing with off-snow off-season activities