Internet Speed Tests on a Raspberry Pi

image

A couple of weeks ago soynerdito soynerdito shared a bot he made to tweet internet connection speed results every now and then.

I then remembered I had a Raspberry Pi I wasn’t using at all. So I built a small app that runs speed tests every 3 hours. It’s based on speedtest-cli which is a Command line interface for testing internet bandwidth using speedtest.net.

The only difference with the one I built is that instead of tweeting the results, it posted all available data to an endpoint. I built a Flask app that can be used to store those results for different users as well as share them via a JSON API.

If you’re interested in running your own, check this Gist out. It contains a README file that will help you setup your box(doesn’t have to be a Raspberry Pi). It also includes a sample Flask app to store your own results. If you’re interested in using the one I’m hosting you could do so, just let me know and I’ll send you a URL to store results and one to share or consume. It’d be interesting to see different kinds of results at a larger scale, not just my own.

Here are the results graphed since I started running it. Interesting to see how so many spikes in my download speeds. Those are probably related to download in other computers or probably streaming Netflix. Upload speed as expected doesn’t really fluctuate that much.

Photo credit: John Talbot

The easiest way to add WebSockets to Django

image

TL;DR – I came up with a very simple solution to handle WebSockets in Django apps. You just install django-websocket-request, run a script and you have WebSockets for your Django app. The cool thing is that this solution makes Django believe its getting a (somewhat) normal HTTP request so you can reuse almost all of you application’s code. Plays nicely with Django REST Framework and with plain function-based views and class-based views. Check out the demo and it’s source code on GitHub.

The Details

We’re building Blimp 2 which means a lot of changes in our current application, infrastructure, and how we’d tackle old and new problems. One of the decisions we made regarding our application was how our frontend and backend would be organized and how they would interact.

Blimp’s backend is currently built on Django. It serves our frontend’s HTML, handles our private and public API, and all our business logic. That’s how most web apps are usually built, but Blimp is already a client-side JavaScript heavy application. Right now a normal request goes something like: You request a URL, there’s some backend work, queries to the database, caching, some more logic, render an HTML response, load third party CSS and JavaScript libraries, load our JavaScript application, some more logic, and finally render it.

After months of usage and growth, we’ve noticed a few key improvements we can build on. The new version will have a backend app that will only serve JSON, no HTML. Our frontend app, served from another location, will then consume data from that API. We want the frontend app to use WebSockets when possible, falling back to XHR.

Web frameworks like Django are built for the HTTP request/response life cycle, so everything from middleware, authentication, and views takes in an HTTP request, and at the end is an HTTP response. On the other hand, a WebSockets server implementation knows nothing about that kind of lifecycle and what it means.

Our main objectives were:

  1. Reuse the same serialization and deserialization logic between the HTTP and WebSockets API.
  2. Reuse all business logic for all available resources.

First ideas that came to mind were to abstract all of what we wanted to reuse by writing them as methods for our models. That way we would write what we needed once, sharing it between the two implementations. This seemed like the right answer at the moment, but after a couple of hours, it proved not to be. Since we are building on top of the awesome Django REST framework, we were going to have to subclass tons of their Generic Views and Mixins, which didn’t sound as bad at the moment, but still being very skeptical we decided to look for other possible solutions.

By then we knew that we wanted a single REST API available via two different transports, HTTP and WebSockets. The best case scenario for me was to avoid rewriting anything of what we have already working for the new HTTP API, just to have it work via WebSockets. Then it clicked. I remembered that Sails.js does something similar to what we wanted to achieve.

Sails supports transport agnostic routing, which allows your controllers/policies to automatically handle Socket.io / WebSocket messages. In the past, you’d have to maintain a separate code base to make that happen.

So in more formal terms, we wanted to support transport agnostic routing, allowing us to use everything in Django’s request/response lifecycle to automatically handle WebSocket messages.

Solution

image

WebSocketRequest is the surprisingly simple solution I came up with to solve this.

WebSocketRequest is a simple class that requires a JSON string containing the following keys: method, url, data, and token. The method key can be any HTTP method: GET, POST, PUT, DELETE, PATCH, HEAD, or OPTIONS. The url key is an absolute URL without the domain name. The data key is an optional dictionary in which the key-value pairs in used to create the method’s data payload. The token key is also optional, and used to recreate an HTTP Authorization header, Authorization: JWT YOUR_TOKEN_HERE. You might want to check out my last blog post to learn more about authentication with JSON Web Tokens and if you’re using Django REST framework, you might like to check out django-rest-framework-jwt.

WebSocketRequest works as following:

  1. Validates JSON string payload.
  2. Creates an instance of Django’s RequestFactory.
  3. Dynamically calls one of RequestFactory’s methods, which returns a WSGIRequest object
  4. Resolves the given URL
  5. Instantiates the resolved view passing along the request, any positional and keyword arguments from the resolved URL.

Yeah, RequestFactory, you read that right. You might be familiar with it if you’ve written tests in Django, but if you haven’t, the RequestFactory provides a way to generate a request instance that can be used as the first argument for any view. The only down side to this is that it does not support middleware, which might be a problem for some.

You can install WebSocketRequest via pip.

pip install django-websocket-request

As always source is available on GitHub.

Demo time!

I’ve setup a demo application running on Heroku. Source code is also available on GitHub.

http://dwr-example.herokuapp.com/

Note that Django is not running at all. Tornado is serving a static HTML file and is routing the websocket requests. django-websocket-request then does the magic.

Throttling is enabled and snippets are erased periodically. If you run into any errors feel free to bother me on Twitter about it.

I definitely would like to hear about possible problems of this solution. Where do you think that this fails? How about for a production environment? Where can it be improved?

Photo Credit: Alice Bartlett / Martin Börjesson

Ember.js + Django

Quick deck about how easy it was to switch an Ember.js app from using localStorage adapter to using an adapter for Django REST Framework.

To show the simplicity of it, I took the simple Markdown Editor I built a couple of days ago, Poe, and switched the localStorage adapter with one that works with Django REST Framework.

Presented at the prPIG meetup on September 26, 2013.

Python shell history

image image

If you’re not a fan of enhanced Python interpreters like IPython or BPython but still do heavy use of it you might have notice how annoying it could get whenever you have to quit it, started again, and remember what was the last things you ran. I do this all the time, usually for simple things like testing out built-in functions or quick hacks.

PYTHONSTARTUP is an environment variable which value is the name of a readable file and is executed before the first prompt is displayed when starting the Python shell.

The following script creates a history file in youra home directory(similar to ~/.bash_history), on exit appends your latest work to this file, limits the number of entries saved to 1000, and as a bonus enables autocomplete on tab. 

After creating the file, you need to set the PYTHONSTARTUP environment variable. Best way is to add the following to your ~/.bash_profile.

export PYTHONSTARTUP=$HOME/.python-startup.py

Bonus: If you use Django, you can take advantage of your history file as well by using Django Extensions’ shell_plus with the –use-pythonrc flag.

python manage.py shell_plus --use-pythonrc

Vagrant VM for StatsD + Graphite

image

StatsD is this little awesome network daemon built by Etsy used to aggregate stats like counters and timers to a pluggable backend service like Graphite. You can read more about how it works on Etsy’s Measure Anything, Measure Everything blog post.

Since I’ve been already playing around with Vagrant and Chef I decided to just put together a simple Vagrant VM for StatsD and Graphite with the help of @hectcastro’s graphite, statsd, and node.js cookbooks.

To get started

First, clone the repo from GitHub. You’ll also need to have Vagrant and the librarian gem installed.

$ librarian-chef install
$ vagrant up
$ vagrant ssh

Graphite

Graphite’s credentials default to username root and password root.

$ open http://localhost:8080

StatsD with Python

You can use a client for Python like statsd.

from statsd import StatsClient
statsd = StatsClient()
statsd.incr('foo')