Let's talk ST training log + Garmin/Suunto

we’re ready to contemplate the next step with our training log, which is to uptake the output of microelectronic devices. i’d like to go straight to devices that have gps units, like suunto X10, garmin 305, 310, 405, etc., and when we accept files into the log, it should be more than just HR, but also the GPS data (and perhaps weather data).

so, my questions are these:

  1. TRAINING LOG SOFTWARE: from a developers point of view, which company is going to be easier to work with? suunto has its “trek manager” and “track exporter” software, i don’t know anything about it. what sort of files does it import? if we enable our training log to accept thetse HR, GPS, Weather type data, is there an industry standard? or must we choose between garmin and suunto, or any other company we work with?

  2. what should we import? of course, HR, but i’d like to import from the gps to make a “route.” also, if we choose to import weather data, we can make fields for this in our training log.

  3. i know nothing about any of this, i’ve never owned one of these units. i assume there is client side software made by each company, you load it into your computer, and our training log interfaces with this client side software. correct? who’s further along? how do you get an ant+ transmission from your unit into your computer? this must be some additional piece of hardware for your computer, yes? and, who’s both mac + pc compatible?

and what other questions should i be asking before i just set my developers loose and start writing checks?

File Formats:
-Many types out there with no “standard” for a full range of data (gps, HR, cadence)
-I’m not an expert but I think Garmin’s .TCX file format is close to a standard in that it is XML based format with many data fields
-WKO+ is able to directly download from devices (bypassing manufacturer software and working/parsing the data in a raw format before you save it)
-I think you would have to develop a file conversion engine which can work with all the different file formats so your training log can work with a uniform data structure

Data to import:
-GPS data (I believe this is recorded in time stamped absolute spatial coordinates which can be overlaid on a map to produce a route)
-Speed/Distance (this is probably directly calculated from the GPS data post analysis)
-Interval/Lap data
-HR
-Altitude/barometer data
-Power

Brainstorm, cool things you could do:
-Fetch weather data from weather.com for the start and end of the workout using the location/time stamp
-Correct data files using accurate elevation profiles from USGS or other source (my garmin does a crappy job on its own of determine elevation gains/losses)
-Build a REALLY good open ended analysis engine so a user can examine all sorts of data parameters… e.g. Avg HR vs User selectable running pace range over the previous 3 months binned into weeks.

The way I see the interface working is: (consequently this is almost exactly what trainingpeaks does)
User attaches device to their computer
User downloads the workout data using the manufacturers or aftermarket software (Garmin Training Center or WKO+)
The file is saved in this program and its native format
User opens the ST training log and points to the appropriate workout file
ST Log uploads the file and converts/parses/analyzes the data

I would suggest that you take a look at Traininpeaks, they have a pretty neat setup.

I think the biggest hurdle you face is building a data analysis engine. Sure average pace and HR over a simple long run is nice, but for a track session it doesn’t tell the whole story. Also how can we effectively gauge fitness changes or workload intensity.

Goodluck!

how to gauge workload intensity, and fitness changes: as it now is, in the ST log, there are two methods of gauging intensity: you tell the training log what your intensity is; and you log in your HR. but there is no method for determining absolute intensity via logging your HR, because we have no input for ave HR x amplitude of HR during the curve of the workout.

when we start accepting file uploads, we’ll have that ability, plus the ability (ideally) of looking at addn metrics that are intensity-specific: power; speed; total elevation gained; grade of elevation; and then one can also add weather to the mix, if you consider heat and humidity accretive to intensity.

algorithms and schemes to parse the data and come up with an intensity gauge will come later (except for our “aerobic points” accumulator which we offer now).

my imminent goal is to crack the nut of data input. once i get the data in there, i can determine later what are the best ways to use it; or, the athlete and his coach can determine for themselves how to use it.

but it’s the social aspect of the training log that’s of most interest to me. here’s an example: we have a 90ish mile loop here in the L.A. area, it’s all in the mountains, about 11,000 vertical feet of climbing, and the compound here is right on the loop. i envision a virtual almost-century around this loop, on a particular day, maybe in a couple of months, where a bunch of slowtwitchers ride the loop, starting wherever on the loop is proximate to them. so, there will be a bunch of us riding, but we’re all starting/stopping at a different point, and going in both directions, depending on what suits each person. i think it might be fun for everyone to upload their GPS files, like a digital race report. what would we do with the data? i don’t know. among other things, perhaps it might be fun to see what the min and max distance and altitude gains are, so we can see whose GPSs are recording the data “correctly” and whose units are outliers.

in general, tho, our log is always going to trend toward furthering one notion: the virtual enclave. personally, i think what a lot of athletes are missing in their training are partners. for example, we had bjorn and jordan up here for a couple of weeks at the same time. i think they did maybe one swim together. that was it! now, me, i think that was an opportunity missed. but each had a coach, and each coach had workouts for these guys, and both coaches were either very disciplined; or just plain silly by not seeing the opportunity that each athlete had to use each other to take both to a higher place.

since most people are limited as to how they can change their “enclave paradigm” i’d like the training log to help them. if i go out and ride the angeles crest loop by myself, then it’s just me. but if i know i’m one of 50 people riding it that day – even if we’re all riding alone – we’ll all be out on the loop, and we’ll all compare our stories. i’m more likely to go out and ride the loop that day; and to ride it with vigor. our training log will always consider the virtual enclave its #1 mission. that’s why “challenges” were an early priority for us.

“routes” are important to us also, and i’d like to create a library of favorites, so that people can incorporate “hidden treasures” into their own list of routes. if we can get people to borrow routes, instead of everyone creating identical routes over and over, then we can generate a list of best performances on those routes. all, of course, is designed to aggregate people into a virtual enclave.

to that end, it’s always going to be a goal to generate a training log that’s as inclusive as possible. so, when we create a client side app that uptakes data from devices, it ought to be one that handshakes with as many devices as possible. hence the poll you see on side of the forum right now. i’d like to know how tough this is going to be; whether that app needs to absorb data from both suunto and garmin, for example, and how hard or easy that’s going to be to build.

Platforms

  Garmin Training center runs on both Windows and Mac, and imports / exports .tcx files, which is an xml-based file format.  Garmin has published the schema (roughly speaking, xml-speak for "file format") at http://www8.garmin.com/...enterDatabasev2.xsd, which makes it very easy for a developer to read and write .tcx files from an application. 

None of the others (Timex, Suunto, Polar) run on Macs. The Suunto Track Exporter will export to .kml (Google Earth) files; Timex will export (and maybe import) GPS data in NMEA format. Both of these formats can be readily converted into other common useful formats. I don’t know what Polar will import or export, let alone any info on file formats or how to talk to the device directly to stream data off it onto a network connection.

Data

If what you are interested in sharing are routes, then it seems to me you have a couple of decisions:

  1. what to take as allowed uploads
  2. what (if any) transforms to do on them (into a single common format, for example)
  3. what formats to hand out as downloads

(1) and (3) are likely to be very device-specific formats, but (2) need not be.

For strictly GPS data, the format with the widest support in the GPS industry is GPX - see http://www.topografix.com/gpx_resources.asp for more info on who supports it. There are tools for converting to and from GPX for pretty much every other format of interest (.tcx (Garmin), .kml (Google Earth), etc.). The schema is extensible so it’s possible to add additional fields (weather, etc.) if needed.

Tools

One such format conversion tool is Gpsbabel, which is an open-source tool that can deal with an enormous variety of gps data formats and sources - including reading directly from devices including Garmins. It does more than you’ll ever need. It does deal with non-gps fields like heart rate, and maybe some weather-related fields, but it’s current version knows nothing of fields like watts. GPSVisualizer is another open source tool useful in talking with devices and converting data formats if needed.

Garmin’s MotionBased website uses either a locally installed application (the MotionBased Agent) or a browser plugin to facilitate communications and transfer to / from locally connected devices. They also make their capabilities available to third parties - see http://developer.garmin.com/web-products/. Your developers might find one of the two tools above useful as a starting point in helping to go from a local device to your website if you go the route of talking with the device directly instead of doing just file uploads.

As far as getting data from an Ant+ device onto a laptop, that takes an Ant+ USB stick as the hardware piece, and some software infrastructure that knows how to speak the Ant+ protocol. The sticks come with an Ant+ device like a Garmin Forerunner 50 or 60, as that’s what enables the device to talk with the vendor software that already speaks Ant (e.g., Garmin’s Training Center). There may be licensing issues or fees involved for developing software that speaks Ant; I don’t know. The main ANT site is http://www.thisisant.com/

That’s all I can think of for now.

Chris

It’s pretty clear from this that a big goal is to help people connect via their past, current, and planned future activities. This is at least partly a search problem, so I took a quick look at what I could search for. What I found is that I can filter the global set of routes by country, city name, sport, and distance. This suggests a few additional capabilities:

  1. look for routes by tag
  2. look for routes within X miles / km of a given city / zip code
  3. look for people within X miles / km of a given city / zip code
  4. look for people who have done a given route, or are planning on a given route on a certain day

Do people also have tags, such that I could be found by them?

To borrow an idea from iTunes, a ‘Genius’ feature for routes might be interesting - if you liked a given route, then the system might suggest others based on some measures of similarity - location, hilly or flat, total elevation gain, proximity to major geographic features or landmarks, etc.

“… if we can get people to borrow routes, instead of everyone creating identical routes over and over, then we can generate a list of best performances on those routes. all, of course, is designed to aggregate people into a virtual enclave.”

Certainly if a person has borrowed a route, then when their performance data is uploaded they can specify the known route it is to be applied to. If they don’t specify the route but actually rode an already stored one, it seems to me to be desirable for the system to be able to still detect that they rode a known route, otherwise you wind up with the “identical routes over and over” problem anyway. It’s an interesting problem in computational geometry, and one that I think can be accurately and very quickly answered, but I’ll check with some coworkers that specialize in this sort of problem.

EDIT: It turns out that given an appropriately organized global database of rides, determining that

  1. a potentially new route has no existing matches in the database (where ‘match’ is defined as an arbitrarily high degree of geographical overlap)

or

  1. finding matching routes

can both be done very quickly, and that this is not significantly affected by differences in starting points, direction of travel, or device sampling / recording rates.

Hey all. I’ve done some developing against the Garmin TCX format, and thought I’d chime in with some findings.

Firstly, if you want to store the GPS track, then you need to figure out how the backend storage mechanism is going to work. For one, the GPS track is not GPX format, which is a shame. So assuming you want to support other devices with a single framework, you may want to convert the Garmin Track to GPX. Also, GPS data in XML format is HUGE. I strongly recommend that you utilize compression and store that in a blob field in your database, or even straight to the file system. (I don’t know the Slowtwitch log backend so I can’t say for sure which make sense. I’d guess it’s Perl/MySql).

Another option is storing tracks as decimal pairs of lat/lon straight in the database (with the timestamp). I haven’t studied the problem close enough to say whether or not that makes sense in the long run. That would probably take more space than the zipped GPX file, but you would at least gain better analytic ability.

The TCX format by the way is created by training center on demand, and isn’t actually stored on the Garmin device. Even “behind the scenes” Training Center uses it’s own binary format to store your data. It only creates the TCX when you ask for it. This makes uploading data automatically a bit trickier because as far as I know, Garmin does not have any public API to assist with this, so the user has to manually ask Training Center for the TCX, then upload that via web form.

The TCX itself stores 0…n activities, depending on how many items were in the selection on export. Each activity has 1…n laps, and the laps have 0…n trackpoints. The data Garmin displays on the actual device is stored at the lap level. The device doesn’t actually “go back” and look at the GPS data it stores. Neither does Training Center for that matter.

The issue with this is that if you opt to store just GPS data, and then calculate distance and time based off of that, you are going to get different numbers than what Training Center and the device say. Issues you will run into are gps signal, or routes that double back on itself quicker than the GPS can create a track point, and most importantly, Stops during the activity. With a careful algorithm you can get around the “stops during activity”, however the distance piece will be a real problem. I used the Levenstein formula to calculate distance between two GPS points, and it works pretty ok with a default setting for curvature of the earth, however the twistier the route, the more inaccurate your results will be come.

The best bet is to store the lap totals (distance, time, HR, and presumably power would be there for 705 user), and then convert and compress the GPS data for viewing via google maps only. If you don’t want to convert (from TCX->GPX or proprietary format), then you could still compress the track, and have a field which indicates what source the data is from which your front end software can handle.

I’ve written a bunch of code that does all the above, however it’s in .Net 3.5, so I’m afraid it may be little use to you.

Thanks for all the tips! The functionality for this is more or less done. (I’m the developer of the ST log btw). Some of the data is zipped and blobbed as you suggest. Storing lat/long/hr points is not feasible in an sql db, it takes too long to write (you end up with an awful lot of insert statements, and size is a problem too). Is it ok if I PM you in the future if I need any more tips/help? No time is set as to when this functionality will be in the ST log, there are some other things we need to take care of first, but it’s high on the list.

Btw, the log is Java + MySQL

Sure, always willing to lend a hand (or suggestions).