Welcome to thatlinuxbox.com Thursday, November 21 2024 @ 10:46 AM UTC

Export entries from dailymile with my export tool

  • Saturday, February 21 2015 @ 01:43 PM UTC
  • Contributed by:
  • Views: 10,997
Linux, Open Source, and Tech Stuff

Share
  • Google Plus
  • Facebook
  • Twitter
  • Reddit
  • LinkedIn
  • Digg

I have been using dailymile as my fitness training log since 2010. I have now logged over 1500 entries and 6600 miles total. I don't want to lose that information. I would like to be able to do more advanced anlalytics on my personal fitness data than the dailymile web app provides.

The built-in dailymile.com export feature is abysmal. The file generated includes only a small number of fields (date, activity_type, distance, time, felt, elevation_gain).  Seriously, they include elevation gain but do not include the name of the workout or the description?

Because of these limitations, I contacted dailymile support. The response was something along the lines of "you could always write your own tool using the API." So I did.

Announcing v1.1 of my dailymile export tool. The current version is written in Python.  The software archive can be downloaded from:

https://github.com/danstoner/dailymile_export/releases

The github repo for the project is located:

https://github.com/danstoner/dailymile_export

Here is the basic usage info:

$ python dailymile_export_to_tsv.py -h
usage: dailymile_export_to_tsv.py [-h] [-d] [-g] username

Script to download entries from the dailymile API for a particular user into a
tab-delimited file.

positional arguments:
  username     The dailymile.com username of the account to export.

optional arguments:
  -h, --help   show this help message and exit
  -d, --debug  Enable debug level logging.
  -g, --gear   Retrieve gear data also.

 

Here is some sample output of the running script and the generated output file:

 

$ python dailymile_export_to_tsv.py danstoner
INFO:root:First API Request: https://api.dailymile.com/people/danstoner/entries.json?page=1
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): api.dailymile.com
INFO:root:Fetching: https://api.dailymile.com/people/danstoner/entries.json?page=2
INFO:root:Fetching: https://api.dailymile.com/people/danstoner/entries.json?page=3
INFO:root:Fetching: https://api.dailymile.com/people/danstoner/entries.json?page=4
...

$ ls -latr *.tsv
-rw-rw-r-- 1 dstoner dstoner     96 Feb 20 09:50 danstoner_dailymile_export.20150220095025.27.tsv

$ head -n4 danstoner_dailymile_export.20150220095025.27.tsv
id    url    timestamp    title    activity_type    felt    duration_seconds    distance    distance_units    description
1980554    http://www.dailymile.com/entries/1980554    2010-05-22T16:00:00Z    Hogtown Creek Greenway    Running    injured        1.5    miles    Brand new shoes and this is what I get.
1980574    http://www.dailymile.com/entries/1980574    2010-05-25T04:00:00Z    None    Running    alright        2.25    miles    Incorporated my first barefoot run.
1980599    http://www.dailymile.com/entries/1980599    2010-05-29T04:00:00Z    None    Running    alright        2.25    miles    CALVES!  1.5 mi of this was barefoot.

 

I have taken my export file and loaded it into MySQL and PostgreSQL. From there, I can query and filter the results.

For example, I can answer the question "How many days did I run 15 miles or longer during the last 3 months of my marathon training?"  Note that I can't just query for rows where the distance column is greater than 15 since some days I might have my running broken into 2 or more entries in my running log (warmup is one log entry, track workout is another). Here is my postgres query after loading the data with the pgimport_dailymile_export_tsv.sql helper script (included in the repo).

 

fitness=> SELECT entry_date::date, sum(distance) AS daily_miles FROM imported_entries WHERE distance IS NOT null GROUP BY 1 HAVING sum(distance) >= 15 AND now()::date - entry_date::date < 90 ORDER BY 1 DESC;
 entry_date | daily_miles 
------------+-------------
 2015-01-29 |     17.7900
 2015-01-22 |     20.3100
 2015-01-15 |     18.2400
 2015-01-11 |     24.0000
 2015-01-08 |     17.1700
 2015-01-04 |     15.0000
 2014-12-23 |     16.5600
 2014-12-19 |     17.4200
 2014-12-06 |     20.0000
(9 rows)

 

I have more enhancements planned for the export script. In particular I want to be able to retrieve my gear info (shoe mileage) which will be trickier since this information is not provided in the dailymile API but will probably need to be retrieved via web scraping.

I might also be providing the script in other languages.

If my dailymile export tool is useful to you, or you have suggestions or feedback, please leave a comment here or as a github issue.

Export entries from dailymile with my export tool | 0 comments | Create New Account

The following comments are owned by whomever posted them. This site is not responsible for what they say.


User Functions

Login

Connect

thatlinuxbox.com is the home of Dan Stoner's Personal Blog, Photos, and More (opinions, rants, techno-babble, and possibly a few useful tidbits of knowledge).

Questions or Comments about this site? Contact danstoner _ at _ gmail.com.

RSS Feed for this blog

Other places to find me on the web:

Twitter

LinkedIn

GitHub

Support This Site

If you like something that you find on this site, please consider making a purchase through one of the links below or sending me an item from my Amazon Wish List.


The Clymb


Awesome VPS hosting by Linode.com