Craig Russell

How to Use Outlook Web Access in Google Chrome on Linux

2015-01-27T00:00:00+00:00

If, like me, you work at an organisation that uses MS Exchange for their internal email you're probably familiar with the Outlook Web App (OWA), the web interface to your email account. Users of Chrome on Linux will probably be aware that OWA doesn't support that browser/device combination and are forced to use the crappy "light version" of Outlook.

However, it is possible to get the fully featured OWA on Chrome, with a few extensions.

First off install the User-Agent Switcher for Chrome. This tool will let you make Chrome lie to OWA, making it think your using Chrome on Windows, which is supported.

After installing, go to the Options for the extension.

Click Custom User Agents from the left-hand menu
Create a new User Agent config like this:
- New User-agent name: on Windows
- New User-Agent String: any Chrome UA string for "Windows"
- Group: Chrome
- Append? Append
- Indicator Flag: Win
Click Add

You should see your new option listed under Chrome. Next we'll make Chrome always use this User Agent string when we use OWA.

Click Permanent Spoof list from the left-hand menu
Create a new rule like this:
- Domain: Your OWA domain e.g. email.example.com
- User-Agent String: on Windows (the one we just created)
Click Add

Now if you go to your OWA, the login page shouldn't force you to use the light version any more.

But wait, we're not quite done yet...

Everything about your email experience in OWA should work fine, unless you try to add an attachment to an email. This is becaue OWA uses a deprecated JavaScript function, showModalDialog, which Chrome dropped support for last year.

This is kinda annoying, but easy to sort out.

Just install the showModalDialog shim extension and problem solved!

Google Analytics Reports in Google Sheets

2014-03-21T00:00:00+00:00

The new Add-ons in Google Sheets allow you to do all sorts of exciting things with spreadsheet data. My favourite (for now) is the Gogole Analytics Add-on, which makes it incredibly easy to pull analytics report data in to a spreadsheet using the Google Analytics API.

Add-ons are only available in the New Google Sheets, so if you're not yet using them hit "Try the new Google Sheets" at the bottom of a spreadsheet.

To install the Gogole Analytics Add-on hit, Add-ons > Get Add-ons... in the menu. Search for "Google Analytics" in the store, then install it. You'll probably have to authorise the app to access your drive account.

Once installed there'll be options for it on the Add-ons menu.

Click Add-ons > Google Analytics > Create New Report

This will open up a wizard (are they still called that?) to let you configure your first report. Give it a name e.g. "My Report". Choose the account, property and profile you want to use. And select the metrics and dimentions. If you're not sure try setting metrics to "Visits" and Dimensions to "Source/Medium". Then hit Create Report.

This will create a new worksheet containing the report configuration.

To run the report click Add-ons > Google Analytics > Run Reports

It'll take a moment or two, then give you a notification. If there are any problems with your report configuration, you'll see a message here.

The results of the report will be created in a new worksheet, titled after your report. Now you've got the data and can do all the usual spreadsheet magic.

You can configure multiple reports on the Report Configuration worksheet, they will all be run at the same time and each save their data in a seperate worksheet. Each time you run a report, it'll overwrite the previous data. Unless you rename a report, in which case it'll create a new worksheet under that name.

Building Reports

Getting the data you want in a report can take a bit of trial and error. I'd reccommend using the Google Analytics Query Explorer to develop your reports. This tool gives you a configurable interface to the Reporting API. Use it to try out different configurations, then once you're happy, copy the settings in to the spreadsheet.

Working with Dates

Use the built-in date functions in Google Sheets to create reports which run over relative time periods. This'll save you having to manually update the date field every time you run the report. Here's a few examples:

Today =TODAY()
One week ago: =DATE(YEAR(TODAY()),MONTH(TODAY()),DAY(TODAY())-7)
The first day of last month: =DATE(YEAR(TODAY()),MONTH(TODAY())-1,1)
The last day of last month: =DATE(YEAR(TODAY()),MONTH(TODAY()),0)
The first of January last year: =DATE(YEAR(TODAY())-1,1,1)

Formatting Values

Google Sheets has extensive formatting options, but sometimes you'll need to tweak the values returned by the Analytics API to get them to work:

Percentage values (e.g. % New Visits, Bounce Rate) need to be divided by 100 to format correctly
Duration values (e.g. Av. Time On Site) need to be divided by 86400 (seconds in a day) to format correctly.

Problems

There is a bug in the new Google Sheets, which prevents charts from appearing in PDF exports or when printing worksheets. This is a bit annoying if you want to distribute reports. Charts are included in exported XLSX documents, but the formatting can get a bit messed up sometimes.

Missing Blobs in Git

2014-01-11T00:00:00+00:00

I've just had to fix a missing blob in a repo. Here's how I did it.

you can see exactly what's missing with git fsck.

> git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (137/137), done.
missing blob 2d31815dd6357b5c7573a4eebc6792b168b4d6a4

We need to find which file the blob refers to.

> git ls-tree -r HEAD | g 2d31815dd6357b5c7573a4eebc6792b168b4d6a4
100644 blob 2d31815dd6357b5c7573a4eebc6792b168b4d6a4  lib/oauth2client/crypt.py

Now we can reinstate it.

> git hash-object -w lib/oauth2client/crypt.py
2d31815dd6357b5c7573a4eebc6792b168b4d6a4

So everything should be ok again.

> git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (137/137), done.

API Keys with Git Stash

2013-11-21T00:00:00+00:00

Working with API keys in git repositories can be a pain. You need the keys to test your project, but you don't want to publish them to the world. This gets even more confusing when you have multiple versions of the keys that you are working with. The stash command in git is a great tool for solving this problem.

The first step is to keep your API keys in a seperate file from the rest of your code. How you do this will be different depending what language, framework, envirenment &c. you use. For this demonstration I'll use python.

Here we have a simple program consisting of two files.

# config.py
API_KEY = "__YOUR_API_KEY__"

# app.py
import config
print config.API_KEY

We can commit these changes to our repo together. (NB: there may also be an app.pyc file, but we can easily ignore this)

$ git add config.py app.py
$ git commit -m "Basic application with config"

We can edit the config file to contain our private API key

# config.py
API_KEY = "abc123_development_key"

And we'll also make some changes to our app.

# app.py
import config
print "This is your API key:"
print config.API_KEY

Now we can commit chanes to the app file, leaving the config file change uncommitted.

$ git add app.py
$ git commit -m "Improved app"

So far, so simple. But what if we want to have multiple versions of our api keys? This is where git stash comes in handy. The stash command allows you to save uncommited changes to one side so that you can work without them, then reapply these changes later.

Lets stash our changed config file.

$ git stash save "Development API Key"

Now we save a different configuration.

# config.py
API_KEY = "abc123_production_key"

And stash this change too.

$ git stash save "Production API key"

We can list all our stashes.

$ git stash list
stash@{0}: On master: Production API Key
stash@{1}: On master: Development API Key

Now can can easily swap between different versions of the config file.

Applying production key from stash:

$ git stash apply stash@{0}

# config.py
API_KEY = "abc123_production_key"

Applying development key from stash:
(Note you must revert the previous change first)

$ git checkout config.py
$ git stash apply stash@{1}

# config.py
API_KEY = "abc123_development_key"

JavaScript - How to Iterate Arrays Correctly

2013-10-15T00:00:00+00:00

There are several different ways to iterate over Arrays in JavaScript. Each of these can behave differently when there are undefined values. Here, I demonstrate each and explain their behaviour.

For these examples, I'm using the JS console in Chrome Dev Tools. Other browsers may format responses differently.

First off, create an Array with an explicitly set "undefined" element.

> a = ['zero','one',undefined,'three']

Then add another element at a higher index. You must create it this way, you'll see why in a bit.

> a[5] = 'five'

When we inspect the Array, we can see the two undefined elements.

> a
  ["zero", "one", undefined, "three", undefined × 1, "five"]

Notice that the element at a[4] is shown as undefined × 1 not simply undefined. This subtle difference tells us that this is an implicit undefined value in the Array, meaning it doesn't actually exist in memory. Whereas the undefined value at a[2] was explicitly defined, so does exist in memory.

Now we can see how different iterators handle these different "undefined" values.

The first examines every index from 0 up to the largest index in the array. Because indices in Arrays are non-sequential, Array.length returns the highest index + 1. So this iterator examines the full range of potential indices, returning the undefined value it finds in the element at a[2] and (crucially, but subtly different) returning undefined when it finds no element at a[4].

> for(var i=0, len=a.length; i<len; i++){ console.log(a[i]) }
  zero
  one
  undefined
  three
  undefined
  five

The second examines each index incrementally from 0 upwards. Because the assignment of each array value to a variable is also the test to continue the for loop, the iterator stops when it encounters the first undefined value (Try !!(e=a[1]) and !!(e=a[2]) in your console).

> for(var i=0,e; e=a[i++];){ console.log(e) }
  zero
  one

The third iterates over indices only. So this only returns explicitly defined elements (including a[2]) and ignores the implicit undefined element.

> for(var i in a){ console.log(a[i]) }
  zero
  one
  undefined
  three
  five

At first glance it may seem weird that these iterators can behave so differently (especially to developers coming to JS from other languages). But when we understand the nature of non-sequential indices in Arrays, how variables can be created with undefined values and type-casting of undefined values to Boolean, the necessity of these different techniques becomes clear. And we realise that there is no "correct" way to iterate JavaScript Arrays.

A New Direction for Easy Open Data

2013-08-29T00:00:00+00:00

About 6 months a go I published Easy Open Data, a tool to make it easy to publish custom-formatted open data from Google Spreadsheets. The response I received was wonderful, and it has been gone on to be used by people to open up all sorts of interesting data.

Member pages now widely-updated. Twitter/Facebook links added thanks to http://t.co/FHkFTbyI8g and @gavinsblog (eg: http://t.co/Fp2cSlfRSx )
— kildarestreet.com (@KildareStreet) April 29, 2013

This project attracted the attention Michael Roberts who works with International Development agencies. He saw that a tool like this was ideal for cash poor, time poor, NGOs who are required to publish aid spend information in the IATI standard he helped to develop. Together we put forward a funding bid for a project to develop the tool, to promote and support it's use among the ID community. Unfortunately we didn't receive the funding we needed.

So, that's the past. Now I want to talk about the future.

Despite this disappointing set-back, we are both still very committed to developing the tool and promoting it's use.

I have decided to rewrite the code base from scratch, there are a few reasons for this. Some of the feedback I've had from users has identified some incredibly useful features that I hadn't considered in the initial version. To deliver these would require some fairly fundamental changes to the tool. Rather than chopping around the already messy code base, it's seemed a good idea to start over.

I also wanted to move the product to use Google App Engine. Given that the tool leans so heavily on Google Drive, I thought it reasonable to move closer to that eco-system. Easy Open Data is written in Ruby. As GAE only supports Java, Python and Go, this would have to change. To me Python was the most sensible choice, its a common language among open data enthusiasts, with a strong data mining heritage and used in high-profile products like CKAN.

We will be publishing the tool under an open source licence (TBD, but probably MIT). At present two releases are planned.

Version 0.1 will be the first open release of the code base and is planned for November. This version of the tool will be minimally functional, in a fit state for developers to begin experimenting with, but not end users.

Version 0.2 will be a more polished, featureful, release. The first that is fit for end users. This can be expected early next year.

You can see track the development of these releases and planned features on the issue tracker on GitHub. If you have any suggestions or thoughts, feel free to add them there.

The last big development is the name. I chose "Easy Open Data" without much forethought (I'm no marketeer). But I always felt that this name fails to explain that this is a tool for publishing open data. I also felt that a new name would help to differentiate this new incarnation of the tool from the previous one.

The new tool is called Open Data Press.

"Easy Open Data" will, from now on, only refer to the legacy version.

But what of this legacy? Over the past few months Easy Open Data has attracted a reasonably sized user-base. Some of whom have made impressive use of the tool. If you are among them, I'll be contacting you in the coming weeks. Asking for your input about the future of EOD and inviting you to join the new Open Data Press project.

Lastly, I'd like to thank everyone who's offered their support of this project so far. I'm thrilled by how the concept has resonated so well with so many people. Your feedback has been invaluable. I'm really excited for the future of this project, and I can't wait for you to see what we've got planned.

Unit Testing Third-Party REST APIs in Python

2013-08-06T00:00:00+00:00

I've wrestled with this problem for some time, and finally cracked it. Here's how to write unit tests for methods that use third-party REST APIs.

The principle is actually fairly basic, in your tests you have to "mock" the role of the third party in any requests. So for an oAuth2 authentication handshake, you have to play the role of the identity provider. This sounds simple enough, but figuring out how to implement it can be a real pain.

I've got a project that makes use of various Google APIs. Some of these are supported by the Google API Client python libraries but some aren't. In those cases I use httplib2 directly and parse the responses myself. To simplify the handling of authentication across both I use the oAuth2Client libraries also.

To write unit tests for my project, I've had to mock responses for all these different API requests.

Mocking requests basically means overriding the bit of code that sends a request out to a server. Instead of making a request to the third party, you return a dummy response, formatted to pass any verification in the client library.

The libraries I'm using use httplib2 behind the scenes, as do many (most?) python REST API libraries. Overriding the request method, I can return a dummy response to ensure that in a test environment, I control the responses.

This is my mock http client class. The request method returns an appropriate response for recognised URIs. For unrecognised URIs a message is logged, highlighting the URI that I need to mock a response for. It also returns an empty Response object, so I don't get distracted by NoneType errors.

from httplib2 import Response
import logging

class MockHttp(object):
    def request(self, uri, method="GET", body=None, headers=None, redirections=1, connection_type=None):

        if 'https://accounts.google.com/o/oauth2/token' in uri:
            headers = {'status': '200'}
            body = '{"access_token":"dummy_token"}'
            return Response(headers), body

        logging.error("No response for '%s'" % uri)
        return Response({}), ''

In actual projects I keep the response body content in seperate files as some of them can be quite large. I've left that part out for brevity.

Using it in a unit test is very straight forward. Simply replace the http client instance in your application with the mock client. In this example main is my WebApp2 application and google_api is my wrapper library for Google API calls.

import unittest
from tests.utils import MockHttp
import main
import google_api

class TestAuthentication(unittest.TestCase):

    def test_oauth2callback(self):
        # Replace http client in wrapper library with mock client
        google_api.httplib2.Http = MockHttp
        uri = '/auth/oauth2callback?code=dummy_code'
        response = main.app.get_response(uri)
        self.assertEqual(response.status_int, 200)

That's pretty much it.

Can't believe it too me so long to figure this out.

Semantic HTML is the Gateway Drug to Open Data

2013-08-04T00:00:00+00:00

James Padolsey says that semantic HTML is dying.

What happened to semantic HTML?

He's right.

For a while the semantic web was all anyone was talking about. You couldn't turn around without hearing about how important microformats were, or the rel attribute. Remember the fight over the HTML5 datetime attribute?

People used to care about this stuff.

Now, not so much.

Now people care about linked open data (which they should, by the way, I certainly do). But in the fuss, good old semantic HTML has been forgotten. It's easy to see why. Linked open data is sexy and exciting, everyone cares about linked open data. Hell, some of the most powerful Governments in the world care about linked open data!

In contrast, engouraging people to write better markup just isn't exciting enough.

But it is still important.

Very important.

Because semantic HTML is the gateway drug for open data.

Many organisations are still unable to contribute to the linked open data project. For them time, resources, understanding, will, exposure and all sorts of other barriers prevent them for 'making the leap' in to open data - the concept is too alien.

But semantic HTML. That's easier step to take.

Chris Gutteridge says that, in putting their information online, organisations are already doing open data, they're just doing it badly.

Semantic HTML is not doing open data well, but it is doing open data a little less badly.

And "HTML done a little less badly" × "a lot of online information" = "a much better web".

So, stay enthused about linked open data, but don't forget that good old HTML still has a part to play.

P.S. The SEO argument also helps too.

How to store Visitor ID in Google Analytics

2013-07-29T00:00:00+00:00

Google analytics doesn't store records for each individual visitor, but with a bit of JavaScript magic, it can be done.

Google Analytics uses several cookies to help monitor visitors on your site, among the data stored in these is a visitor identifier. This is a unique number that GA uses but doesn't store. It is possible to read this value out of the cookie and push it back up to GA in a custom variable, where it can be used in reports.

Note that the _setCustomVar parameter is wrapped in a function, so that it isn't called until after the GA code has run. This ensures that the cookie has already been set before trying to read it.

On it's own this is interesting information, but it's made all-the-more useful when tied together with your CRM or data warehouse.

We Need a Longer View of the Content Lifecycle

2013-07-04T00:00:00+00:00

Recently I've been trying to understand different content management strategies and the implications these have for the underlying technology. Most of what I've read fails to consider the long-term utility of content, considering only the newest to be useful. We need to develop a longer view of the life of web content. We should plan for content to live, not for years, but centuries.

There are no shortage of opinions about how you should be managing your web content. Most models of the content life-cycle focus on the generation and optimisation of web content. Typically this process focusses on the newest content, while older content is forgotten and allowed to sink in to obscurity. Unvalued and unloved, the effort put in to managing content is squandered in the first few weeks of it's life.

If your site has been around for long enough, a quick glance at your analytics will show you that somewhere out there, people do still value your old content. People are asking questions and finding the answers you provided years ago. Strangely many organisations seem blind to the value their old content has down the long-tail of SEO.

Many modern CMS' allow content to be shown at multiple locations in a site. A single content item can be combined with others and presented differently. When old content can be presented in new ways, it can attract new visitors and encourage old ones to explore new areas of the site. In this way old content can be re-purposed to serve new goals.

Clearly organisations should value their older content and redeploy it in support of their present objectives. But I want to take a longer view of the life of content.

Libraries and museums house books and texts that are centuries, even millennia, old. These are valuable records of people and knowledge that may otherwise be lost to history. And until a couple of decades ago most information was recorded in the same way, on paper, in books, journals and catalogues.

The ease that we can now publish online has seen an explosion in the amount of information we are making available to the world. However, the rate at which content is being created is likely only matched by the rate at which it is lost. Server moves, lost backups, incompatible file formats, budget cuts and wilful deletion are just a few reasons why organisations fail to preserve their old content.

Projects like the Wayback Machine and the UK Web Archive take snapshots of parts of the web. But we cannot blindly defer this responsibility to these organisations. It would be a real tragedy if future historians look back upon this time, when publishing was never more accessible, to find so few records of our lives. How sad it would be if an organisational archive were to hold decades of records from before the millennium, but nothing after. We have to take responsibility for the long-term life of our content.

This post is long enough, so I'll save my thoughts on how we might approach these problems for another night.