north-south feeling

I’m finally finished with the acquisition, parsing and the filtering of the data. The non-visual data that is. I’ve spend the last couple of days working on it. It’s pretty dull work but essential for displaying the data correctly. So now I’ve got 14 exquisite files which hold all the data (date and time, heart-rate, GPS coords and activities) for all 14 days. My Flash app will only have to loop through them. Bit by bit I’ll work on the photographs and I hope the whole work will be finished and online by next Wednesday.

An interesting book on working with data (for which I unfortunately don’t have enough time to work through) is Visualizing Data by Ben Fry, yes one of the developers of Processing. He’s a real genius in working with data, take a look at some of his projects.

Yesterday I discovered something very disturbing. The XML file with the GPS track was missing 40 tracks in the log for the fourth. Not at the beginning or end but just, somewhere…?! This file has thousands of nodes and is almost 1 Mb big. A node looks likes this:
<trkpt lat=”52.387110000″ lon=”4.909017000″>
 <ele>48,5</ele>
 <time>2009-01-05T07:46:35Z</time>
</trkpt>
That’s very easy for a computer to read but not for a human.

screen dump of compare_times output

I had to come up with a way to pinpoint the gaps. So I started to write a program called compare_times. I wanted a way to compare the times logged by the Suunto watch to the times stored by the GPS logger. Usually I don’t need the time node of a GPS track but now I could use it to see if the times weren’t matching at places. I output the times side by side to a file so I could compare them. And yes indeed, at some point they were not matching anymore.

Times in the GPS log that need my attention

So now I had to enhance my program so that it would point out to me exactly where things were going wrong. It’s impossible for a human to see for example two tracks missing at say 17:48:13 and 17:48:23 on a whole day. I decided that I would count the tracks in every minute. If there weren’t six tracks (logs) in one minute something was wrong. It sounds easy but it wasn’t all that easy to program. But I worked it out and ended up with a clean list of attention points so I knew where to look.
Mending the data was more work then writing the program. This was especially true for the log of the fifth. There 200 tracks were missing! Sometimes six minutes of data were just not there. Now I’ve got the day data of day 1 to 4. I hope the other days won’t be as bad.

I’m on the train going to Amsterdam for the second part of my project. I must say I wasn’t really looking forward to it. Not so much going to Amsterdam but doing all the measuring for a whole week. It’s pretty intense somehow, it cages you… I’ve also discovered that it’s a hell of a lot of work to get the data right. So most of the week I’ll be struggling with data. I’m still behind with my Breda data It’s all just going so slooooooooooooow. But I’ll try and be more organized this time. I’ll try to sort the data as soon as I’ve collected it and put it on my computer. This is especially handy for the GPS data. I’ll have to stick with the days for that one because the files are so long it takes ages to select say, half a day.
Yesterday as I was packing my things I had the bright idea to install the Suunto software on my laptop to see if it would work. Well… it didn’t. It was 7 pm and the most important piece of software wasn’t working. I tried everything. Even installing service pack 3. That resolved one problem but now I just get an ‘Abnormal program termination’ error. Meaning that the software just didn’t start. I wrecked my brain and finally decided to try and install the software on my old Sony laptop. It worked! But I do have to drag two laptops to Amsterdam. My bags were so heavy! But now I’m on the train, doing my measuring and I feel more relaxed. It will be fine this week.

Today I finished my program for combining data from different files. There will be three different files at the end of a day. A text file with lines and tab separated data containing all the data recorded by the Suunto watch. A gpx file that contains GPS data in xml format. And a text file with lines and tab separated data which holds time and activity data.

I placed myself for the daunting task of combining these different files and formats into one tab separated text file. As Flash can’t output a text file I chose Processing for the job. It appeared the capabilities to work with strings are rather limited compared to ActionScript or PHP. But they do have a very convenient XML element class. It was actually the first time I worked with an XML file in this way (that is extracting attributes from an XML node). The loadStrings function is very handy for loading external data and putting it into an array at the same time. In one big loop I cycle through all the files, add a concatenated line at the end of every loop. The result is a tidy text file with only the necessary data in a structured format.

A day well spend

Merging three files into one (day_data.txt)