Wheel at the end of its life?

My current set of wheels (Bontrager Race Lites) are 4 years old. The front has been bullet proof. I’m starting to have some issues with the rear. It won’t stay true. There does not appear to be any damage or dents in the rim. I haven’t crashed or hit any major holes etc although I am a big guy and I do ride fairly hard (I did break 2 spokes last year when a stick got up into the wheel though). I true up the wheel and within a couple weeks its wobbling again.

I can keep fiddling with it but my question is, how do you know when a wheel is at the end of its life? I’d rather not have to find out when I’m 50 miles from home laying in a ditch so what are the signs of “old age” in a wheel?