Here's a video which summarises some data visualisation I've been doing recently. The rest of this post explains the story behind it.

Background [edit]

I'm doing some work with Andreas Diomedes Soteriades and Marc Metzger, looking at global responses to climate change. I can't be more specific until we've finished the paper, but the general overview is that we're taking current data, classifying it, and then trying to classify the results of a bunch of climate models.

We're working with gridded global data, at 5 minute resolution, so we have 2 million cells, covering the land surface of the globe. We have:

• 4 climatic variables for each cell
• a classification for each cell into one of 125 classes

These make up our training set, as the variables used here are not the same variables as were used to do the classification.

We also have:

• 4 climatic variables for each cell, for a number of future scenarios, based on a bunch of different climate models

It's these data that we now want to classify according to the original schema. So, we need to train a classifier on the training set, and then run the classifier over the future scenarios to generate our output. This is a fairly big dataset - 2M points, 125 classes - so we need something simple and fast.

We started with a Naive Bayes approach - it's fast and quite robust, and doesn't need tuning - and got 76% classification accuracy on current data. Not perfect, but good enough to be going on with. The rest of this post is about how we discovered that it was limited, and why.

When we looked at the initial outputs, it seemed fairly reasonable - classes moved around in the future in ways we were expecting. However, for some outputs, we realised that there were huge regions of a single class, in a place where we really wouldn't expect to find it:

Whole World Classification

What's all that blue stuff doing in the middle of Africa - that looks weird! Note: it only really looks weird if you have some idea what the classes are, but it's still the same colour over the middle of africa as over bits of Northern Europe, which is a bit worrying.

So, we needed to find out why this was happening, is it a problem with the classification, or a natural effect of the data? It's DataViz time!

Setup [edit]

For this, I used three main tools:

• R, the statistical language
• ggplot2, a really lovely graph package
• ggobi, a multidimensional visualisation program, which can talk to R

If you're following along at home I'm going to assume that you've installed them all, and done library(ggplot2);library(rggobi), so it's all ready to go. I will put up links to data files once it's published, so for now you'll just have to follow the code and use your own data.

First up, I read the original data and the future scenario data into data.frames, and made sure the format matched:

> class <- read.csv("data/output.csv")
> head( origA )
 V1 V2 V3 V4 LON LAT Class ID
1 9927 1707 279 11967 -33.37559 83.62503 11 11
2 9927 1707 279 11959 -33.20892 83.62503 11 13
3 9896 1708 279 11965 -33.12559 83.62503 11 14

Now, since the datasets are very large, I needed to create a subset to explore. To get a good range of points, I took:

• 100 points from each class, randomly scattered over the globe
• all the points from a region around Africa where weird things were happening
• another 1000 points sampled from the class which was looking strange (class 80)

(see snippet at bottom)

I did this both for the baseline data and the classified future data, so I could compare the samples. Each one looked a bit like this:

> clasSamp <- getRegionAndStrata( class )
> ggplot( clasSamp, aes( x=LON,y=LAT,color=Class ) ) + geom_point(size=0.8)

World Stratified Sample

Then, I added a column to say which set each point was from, and combined them:

> clasSamp$Scenario <- "2080"
> basSamp$Scenario <- "Baseline"
> d <- rbind( clasSamp, basSamp )

Initial Exploration [edit]

Next, the data goes into GGobi. GGobi is designed to help with visualising and understanding large, multivariate datasets. There are a bunch of different tools it has for displaying the data, but to start with I used a 2D tour, along with some automatic brushing to highlight certain points. The 2D tour takes multidimensional data, and projects it down into 2D, but it does it by mapping a combination of input variables onto X and Y, and then changes that combination over time. As ggobi moves through different mappings of each variable, you get a constantly changing view of what's happening, as if you were rotating an object and looking at it from different angles. If that's confusing, have a look at the video further down the page...

Using the R package makes it all very easy:

gg <- ggobi( d )

This opens up a new GGobi window, with the points loaded in. It might look a bit like this: Ggobi Initial

It's not very helpful to start with. The first thing I did is chose a new colour scheme. I'm going to use 6 different classes, so I need one with at least 6 different colours, and for this I'm using "dark2 6":

GGobi Choose Colour

Now, we can go back to R, and get it to colour the points according to their class. (See snippet at the bottom)

This makes the default colour 1, and other colours depending on whether the point is in the region, whether is is in the problem class, and whether it is baseline or predicted data. The last 3 lines hide all the data except those in the selected region or in the problem class. It looks like this for me:

GGobi Coloured Hidden

Now, we're starting to see a bit of structure. The key is:

• Orange points are in the region
• Mauve/purple points are predicted points in the problem class
• Magenta points are predicted points in the problem class, in the selected region - the most likely wierdos
• Green points are the problem class in baseline
• Yellow points are the problem class in baseline in the region (I don't think there are any of these)

Now, I choose Display -> 2D Tour, and select the input variables I want to explore. All of a sudden, everything starts moving!

GGobi Tour Start

What I'm interested in is the problem points in the predicted set compared to the baseline, so I can hide everything else for now:

 shadowed(gg[1]) <- T
 shadowed(gg[1])[d$Class == 80] <- F

And after a bit more playing, watching the tour go round, here's the most telling picture: GGobi Class Separation Remember: Green is the problem class in baseline data (class 80), and mauve+magenta are the predicted points classified as class 80.

We can see that:

• the classifier is assigning a bunch of points to the problem class which have very different values for their variables than the original baseline data: there's a bunch of points which are totally disjoint from the original set
• the disjoint set is right at the edge of the cloud of points: it has very extreme values of the variables, quite possibly values which were not seen in the input set.
• The problem class has quite a high spread (variance) - in this particular view, it has points covering about half the range of the whole set. (This is important in a minute)

In fact, if we recolour the same view so that baseline data is orange and predicted data is grey, we can see that the predicted data is indeed more extreme than the input data: GGobi Baseline Separation

NOTE: I've "shadowed" the *future* data, so that it gets plotted behind the baseline. Otherwise, being more extreme, it would mostly cover it

Putting all of this together gave the video at the top of this page.

Understanding [edit]

OK so we know what's happening, but now it would be nice to understand *why*, and what we need to do about it. First, we need to have a think about how a Naive Bayes classified works. It's a very simple classifier:

• for the training set, extract the mean and variance of each variable for each class and create a Normal distribution to model the class.
• when classifying a new point, sample each distribution, weighted by the number of occurences of that class in baseline
• choose the distribution with the highest probability, and assign the new point to that class.

A picture probably helps: NaiveBayes Graphs

Here, we've got a bunch of one-dimensional points, which are classified as either circles or crosses (and there's the same number of each). We can fit a Normal distribution to each of the classes. When we want to classify a new point, we look at the value of each distribution at that point, and choose the higher.

This means that if our new point is <x1, it will be a circle. If it's between x1 and x2, it'll be a cross, and over x2, it'll be a circle again. That seems a little bit weird! There's a clear line between the two classes, so shouldn't everything above x1 be a cross?

Probably, it should - we can't necessarily know, because we don't have any data in that range. But, it seems intuitive that in this case, everything over x1 should be a cross. What's happening here is that because the crosses are much more tightly grouped, the distribution for them tails off a lot faster, so for really extreme x values, the distribution of circles is higher.

And this is exactly what's happened in our data - a class with a large variance is assigned to points with extreme values. In fact, as the values get more and more extreme, eventually whichever class has the highest variance will take over (assuming equal weighting).

This is an artefact of using Naive Bayes classifiers on data which is not in the range of the training set - it's an issue with classifiers, and models in general: how do they behave outside of the data they have been trained on. In this case, we don't like the behaviour, and want to find an alternative - that's another story, but it's always something to be aware of.

The take home points are:

• we saw some funny behaviour
• we looked at how this related to expected data
• this showed it had to do with extreme data values
• we understood that this is an artefact of the classifier we used, so we might have to look at different approaches to classification.

Code Snippets [edit]

Create a sample with 100 points from each class, all the points in a region, and extra points of the problem class

getRegionAndStrata <- function( d, MINLAT=-5, MAXLAT=25, MINLON=-20, MAXLON=30, nsamples=100, extra80=1000 )
{
 stratIndex <- tapply(1:dim(d)[1], d$Class, sample, size=nsamples,replace=TRUE)
 stratified <- d[ melt( unlist(stratIndex) )$value,]
 stratified$Type <- "Stratified"
 region <- d[d$LON > MINLON & d$LON < MAXLON & d$LAT < MAXLAT & d$LAT > MINLAT,]
 region$Type <- "Region"
 c80 <- d[d$Class==80,]
 e80 <- c80[sample( nrow(c80), size=extra80, replace=TRUE),]
 e80$Type <- "Extra80"
 res = rbind( region, stratified, e80 )
 res$Type <- factor( res$Type )
 res
}

Colouring GGobi plots by class. gg is the ggobi object created from "gg <- ggobi(d)", and d is the original data.

 glyph_colour(gg[1]) <- 1
 glyph_colour(gg[1])[d$Type == "Region"] <- 2
 glyph_colour(gg[1])[d$Class == 80 & d$Scenario == "2080"] <- 3
 glyph_colour(gg[1])[d$Type == "Region" & d$Class == 80 & d$Scenario == "2080"] <- 4
 glyph_colour(gg[1])[d$Class == 80 & d$Scenario == "Baseline"] <- 5
 glyph_colour(gg[1])[d$Type == "Region" & d$Class == 80 & d$Scenario == "Baseline"] <- 6
 
 shadowed(gg[1]) <- T
 shadowed(gg[1])[d$Type == "Region"] <- F
 shadowed(gg[1])[d$Class == 80] <- F