Evan Savage

Fitbit: APIs, crossfilter, and d3.js

In this post, I present
fitbit-crossfilter, which uses
the Fitbit API, crossfilter and d3.js
to provide an interactive visualization for exploratory analysis.

The Inspiration #

It was early April 2012. By this point, I'd been through a stint of pen-and-paper
self-tracking for panic recovery.
I'd just received my Fitbit in the mail.

Earlier that year, I applied to the EECS PhD program at UC Berkeley with
this statement of purpose. I was fascinated by this idea that pervasive
gameplay really could make us all better
, that somewhere beyond the rat wheel
of gamification was hidden a Shangri-La of game-driven awesome.

That unfortunately didn't pan out, and I was left with the age-old question:

What do I do with this idea?

It was around this time that, in a moment of exquisite
digital serendipity, Meetup suggested I check out the
Bay Area Quantified Self Meetup Group.

Quantified Self? What's that?. As I explored the group page, I felt
a rush of clarity: this was exactly what I'd been doing! There's a whole
community of people turning their lives into games in the name of
self-betterment!

I bit the bullet and forked over hard cash to sign up for
QS Show&Tell #25 at the
California College of the Arts. It was everything I'd hoped for.
One presenter dissected 30 years of medical data and correlated it with
his marital status. Another showed off a cyclist threat detection system
cobbled together by mounting a webcam and sonar unit to his handlebars.
There was a rich vein of inquiry into awesome here. I was hooked.

Beau Gunderson of
Singly presented zeo-crossfilter.
That was the turning point. I saw what he had done and said

Hey, I can build that!

And so fitbit-crossfilter was born.

The Tools #

As mentioned, fitbit-crossfilter is a mashup between
the Fitbit API,
crossfilter,
and d3.js.
I'll go over each part with examples.

Fitbit API #

The Fitbit API uses OAuth for authentication. If you've never
confronted OAuth before, it can be confusing. To compound the confusion, every
API provider seems to do it slightly differently.
The
official Fitbit docs are opaque, the
OAuth specs are even more opaque, and
the unofficial apis.io listing is just wrong:

$ curl -X GET -u '<username>:<password>' http://api.fitbit.com/1/user/-/profile.json 2>/dev/null | jsonpp
{
"errors": [
{
"errorType": "oauth",
"fieldName": "n/a",
"message": "No Authorization header provided in the request. Each call to Fitbit API should be OAuth signed"
}
]
}

I turned to oauth2, a Python library that makes it easier to carry out
this handshake. First, we get a temporary access token:

# Fill in your app parameters here.
FITBIT_APP_KEY = '<app key>'
FITBIT_APP_SECRET = '<app secret>'

import oauth2
consumer = oauth2.Consumer(key=FITBIT_APP_KEY, secret=FITBIT_APP_SECRET)
client = oauth2.Client(consumer)
resp, content = client.request('http://api.fitbit.com/oauth/request_token, 'GET')
token = oauth2.Token.from_string(content)
# NOTE: the auth URL uses www.fitbit.com as the domain, NOT api.fitbit.com
auth_url = 'http://www.fitbit.com/oauth/authorize?oauth_token={0}'.format(token.key)
print auth_url

Now we need an OAuth verifier. This will be used to retrieve the real
access credentials. Visit auth_url in your browser,
log into Fitbit, and click Allow. You'll be redirected to the OAuth callback
specified in your app. Use the value of the oauth_verifier GET param on your
token from before to keep going:

token.set_verifier('<oauth_verifier>')
client = oauth2.Client(consumer, token)
resp, content = client.request('http://api.fitbit.com/oauth/access_token', 'POST')
access_token = oauth2.Token.from_string(content)

With this, we can now retrieve useful information:

request_url = 'http://api.fitbit.com/1/user/-/profile.json'
oauth_request = oauth2.Request.from_consumer_and_token(consumer, token=access_token, http_url=request_url)
# Despite what the docs say, you need to generate a plaintext signature.
oauth_request.sign_request(oauth2.SignatureMethod_PLAINTEXT(), consumer, access_token)
headers = oauth_request.to_header(realm='api.fitbit.com')

import httplib
connection = httplib.HTTPSConnection('api.fitbit.com')
connection.request('GET', request_url, headers=headers)
resp = connection.getresponse()

import json
data = json.loads(resp.read())

I encountered a few difficulties in figuring this out:

You can see the full implementation here, along with
an example of its use.

crossfilter #

Square's crossfilter is a JavaScript library for efficiently performing
multidimensional range queries. I've included an interactive example
below.

crossfilter uses two types of objects to represent a multidimensional dataset:

The totally-ordered part is essential, since that makes it possible to
perform range queries. A quick code snippet might help explain this further:

var L = [], N = 10, M = 2;
for (var i = 0; i < N; i++) {
L.push([i, Math.floor(M * (N - i - 1) / N)]);
}
var c = crossfilter(L);
var d0 = c.dimension(function(x) { return x[0]; });
var g0 = d0.group();
var d1 = c.dimension(function(x) { return x[1]; });
var g1 = d1.group();
d0.filterRange([3, 8]);

At this point, we can inspect the dimensions and groups to understand the
effect of filterRange():

> JSON.stringify(d1.top(Infinity))
'[[4,1],[3,1],[7,0],[6,0],[5,0]]'
> JSON.stringify(g1.all())
'[{"key":0,"value":3},{"key":1,"value":2}]'

Note that the range [3, 8] is actually interpreted as the semi-open interval
$ [3, 8) $. Note also that the elements of g1.all() are of the form
{key: k, value: v} where v is the number of elements x with
3 <= x[0] && x[0] < 8 && x[1] == k.

d3.js #

D3.js is a JavaScript library for manipulating documents based on data.

Using HTML, SVG, CSS, and JavaScript, you can build some pretty stunning
visualizations.
Again, check out the interactive example below. For more
examples, the D3 Gallery is
many kinds of awesome.

A Quick Demo #

A
B

If you're viewing this through an RSS reader, the above demo won't show
correctly. You can view it on my blog.

Insights From My Data #

You can see the live dashboard here. Some of the highlights:

Again, you can play around with the dashboard here
to find patterns in my Fitbit data.

How To Use fitbit-crossfilter #

I've placed my live fitbit-crossfilter dashboard into demo mode, but you can
fetch and view your data as follows.

First, you will need a Fitbit app with Partner API access; see
this page for more details on setting that up. Use the following
application settings:

Now copy settings.py.nopasswd to create your settings file:

$ cp settings.py.nopasswd settings.py

Edit the bottom of settings.py:

SYNC_ENABLED = True
DEFAULT_USER = None
FITBIT_CONSUMER_KEY = <your app key>
FITBIT_CONSUMER_SECRET = <your app secret>

Start the server, login, and sync your data:

$ python manage.py runserver 9001
# visit localhost:9001/login in the browser to do the OAuth handshake
# visit localhost:9001/sync-user-data in the browser to sync data

When the syncing completes, you'll be redirected to your dashboard.