I'm doing it right now. I did my first solo around the traffic pattern a few weeks ago at 15 hours, and currently have 23 hours total. I'm very much a noob at this shit... so take my advice accordingly...
Step 1 is to do a discovery flight. This is a 1/2 hour to 1 hour introductory flight lesson. Just do an internet search for local flight schools, pick up the phone and schedule it. I'd recommend doing this before doing any significant studying, getting a medical or committing any significant amount of money to anything. Unless you've already spent some time in small planes you don't really know how you will handle it. It's an entirely different feeling than riding in the back of an airliner and it's not for everyone. If you end up some combination of terrified or puking all over yourself it may end right here.
Step 2 is to get your medical. Go
here to find a doctor. For a PPL you need a class 3. This should be trivial for anyone on a triathlon forum, but just in case there is something preventing you from getting a medical you would rather know right now than after you've spent several thousand on dual instruction. You don't need the medical to fly with an instructor, but you do to act as pilot in command (i.e. fly solo). Just a note, there is currently a bit of a regulatory shakeup happening with the 3rd class medical and apparently the day is fast approaching when you will not need it. I haven't paid much attention to this since my medical is good for 4 more years, but reading
this it honestly looks easier to just go in and get the class 3.
At this point you have a few options depending on your location and budget. The most straightforward path is to go to an organized flight school. They will have a set curriculum, with ground instruction, flight instruction and plane rental all under one roof. This will almost always be the most expensive option. It will vary with your location and the quality of the planes, but typically you're looking at around $125/hr for a very basic 4-seat trainer (C172 or PA-28) plane and $60/hr for the instructor if you go this route.
The cheapest route, although it requires some capital up front and a bunch of legwork, is to buy a clapped out old 2-seater for <$20k and hire an independent instructor. The going rate for these guys around me (Twin Cities, MN) is about $40/hr. These planes don't depreciate much (they're all older than you at this point), so you're basically flying for the cost of gas, oil and insurance after you sell it. The problem here is that you're assuming some risk. If your engine takes a shit you could be selling the plane for scrap (a new engine costs more than scrapping the plane and buying another old plane). I personally wouldn't recommend this unless you already know a great deal about aviation and airplanes and have an in with a good mechanic, which is probably not the case since you're asking about aviation on a triathlon forum.
The middle ground is a flying club. This a very nebulous term that can mean just about anything, but the gist of it is that a bunch of people share access, and in many cases ownership equity to a plane (or multiple planes) and you typically pay some monthly fee for fixed costs like insurance and hangar, with an hourly fee for plane usage. Since these are typically set up as non-profits there is usually a significant discount vs. renting from a flight school/FBO. IMO this is the sweet spot for most people who aren't filthy rich (in which case you can just buy whatever plane you want and hire an instructor) or young kids looking for an airline career (in which case a formal flight school would get them out the door with their ratings and hours quicker, albeit at a price).
Two more pieces of advice. You will see some rental prices in Hobbes hours and some in tach hours. This a HUGE difference. The Hobbes meter runs at the same rate whenever the plane is running. It doesn't matter if you're sitting on the ground idling or at full power during climbout. The tach is calibrated to tick over an hour for an hour spent at typical cruise power. It will run much slower at idle. During training you spend a lot of time putzing around at way less than cruise power, so for the same time in your logbook you're paying ~30% less if you're renting by the tach. Most clubs charge tach time because it's more reflective of the actual wear and tear on the plane and it gives the members an incentive to not beat the hell out of the planes by flying around with the throttle firewalled all the time. Most FBO's and schools charge Hobbes because... reasons I guess.
The last thing is to not worry about how fancy the planes are. You can train in something brand new that has all the latest glass cockpit gizmos, but you will pay accordingly. All that crap is a distraction at best when you're still trying to get your stick and rudder skills up to snuff. It's actually easier to pass your PPL checkride in a minimally equipped plane. They can't test you on something that's not there. I've been told it's the other way around once you get into more advanced ratings, but for the PPL simpler is better.