The first two accusations are reasonable – there are a lot of stupid comments out there, and they are very repetitive.  They are so hated by some that someone has developed a Firefox Add-on that will strip comments out of the page before you can even be offended by them.  On the other hand, though, people are very stupid and repetitive.  There’s often a surprising honesty to the statements that people make on youtube. It is suggested that the monumental scale of the youtube ‘community’ means that individuals are effectively anonymous, which liberates users from any self-censorship that would occur from fear of shame or other punishment.  It’s not all honesty, since people are more likely to be offensive if they know they can get away with it, but it’s a refreshingly different place to look for views and opinions in.

So while they may be stupid, they’re still worth looking at.  The issue of quantity is more of a practical issue, though.  For example, I keep tabs on  a video called “The Most Pathetic Baby Panda Ever”. It has almost 5 million views and 15,500 comments.   Even if you only dedicated 5 seconds to each comment, you have to spend over 21 hours studying them. The Evolution Of Dance has almost 250,000 comments -   almost two weeks’ worth of study.  While they’re interesting, they’re not that interesting.

Conveniently, though, this is exactly the kind of thing that the study of data mining is supposed to deal with.  In addition, the fact that the comments appear to be stupid (or simple,) and highly repetitive works in our favour. The first question I would try to ask of these comments is “what’s a typical comment?”  and toward that end, I have created the Youtuber.

Most of these factoids are interesting only to myself, sometimes not even that. Being able to dig into my financial transactions gives me the opportunity to do a number of things, though – I can construct an “average” day. If I place my expenditures on a city map, I can draw my route over a day (or a week, or a month), see when I buy my morning coffee, and the average radius of my lunchtime wanderings. I can even use it to retrace my steps, and find the name of that great Japanese restaurant I went to in Auckland last year.

You might still think that these are unimportant, but this ignores two issues. First, we don’t know what’s insignificant until we can see it. There may be important trends in my buying habits – if I buy a coffee in the morning, I may end up working late because I forget to go home. Second, it’s important to somebody – it might not be front-page news if I go to a restaurant, but it’s nice to have the name on hand so I can tell a friend whether to give it a miss or not.

Most of our interaction with computers (as digital cameras , cellphones or point-of-sale devices) is being recorded. The sum total of this recorded data is referred to as our digital footprint. While many people find the presence – or even the idea - of this record threatening, I think it gives us an opportunity to answer important questions about who we are, and what we do.

There are at least 80 million videos on Youtube [wikipedia], and if the average length is at least one minute, then it would take about 160 years to watch them all (with no waiting for loading).

There are at least 1.5 million CCTV cameras deployed in public places in the Greater London area [also wikipedia], meaning that if a person could watch 16 cameras at once to monitor them, you’d still need almost 100,000 people, or around one out of seventy living in London employed to watch over the other 69. (If we broke it into reasonable chunks, you’d want 4 full-time shifts of people, pulling the number up to 400k and the ratio down to about 1:18!

Project Gutenberg, wonderfully, has over 100,000 books on offer for download. If we assume that the average book is 100,000 words (a novel is generally supposed to be about 230,000 or so; the English translation of Les Miserables is 650,000) then it would take 90 years (of constant reading at a pace of 240wpm) to get through it all.

There are probably hundreds (or thousands, or millions) more examples of where the quantity of information we have at our disposal is so far beyond our perceivable reach that it becomes effectively unusable. This is why we need to find a better way to work with large quantities of information – a way of “Data Mining”, if you will pardon the phrase.

It’s true, datamining is becoming big business, and all of the big IT companies are doing it – but when was the last time that you did it? Unless you work at one of those big IT companies, probably never.

That’s something I want to change.  One of the programs I’ve been making recently, the Youtuber, lets individuals do some basic analysis on one of the most frivolous data sources out there – Youtube comments.  It is my hope if we make better tools for data analysis for the everyday user, they will learn something new and interesting about the world.

Not to mention what we might find when we turn this analysis back on ourselves! But that’s another story.