Disclaimer: I considered myself a roadie, and I have plenty of good roads to ride around me. I do, however, have a nice gravel bike with very few miles on it. Once in a while, I enjoy the change but it never pulls me.
First of all, I really don't like the term 'gravel' as it really doesn't capture the kind of riding that I do at least. I prefer to refer to my 'gravel bike' as my mixed condition bike as that's what I use it for.
The kind of riding I like to do is explore, ride in areas that generally have low or no cars, and have terrain that I would consider to be more technically difficult - particularity when speed is introduced. So an ideal ride in my area that ticks those boxes is a long 4 to 5 hour ride that is remote, and includes pavement, double track through mtn bike trails, gravel roads, and unmaintained roads. I like to explore new areas that give me that sense of adventure. Another big draw to this type of riding is setting up my bike and tweaking it so it works best in these demanding conditions.
Edit: This type of riding is also more conducive to socializing with guys in my club.