Visualization: Pixels, Degrees and lots of data!

My visualization project has lead me down a rabbit hole I never knew existed.  When I think of the world I think about the miles between location X and location Y, which can be easily translated into latitudes and longitudes.  Though I never knew the actual conversion I knew it was mathematically trivial and I had never really thought about how this works in something like Google Maps.

Mapping Coordinates and Pixels
Of course when dealing with mapping software the world can no longer be represented as a ellipsoid.  The standard way to project a globe onto a flat surface is the Mercator projection.  Using this projection, we can display the globe as a flat map.  There is some weirdness about the way the Mercator projection works, which you can read about here.  Once the world has been projected into this form we can easily display the map, but Google maps this coordinate system onto yet another coordinate system, their tile system.  This tile system is likely not a surprise to anyone who has ever used google maps, but the way that it works did surprise me a little bit.  Google has a predefined tile size (which is 256 by standard).  As you zoom the entire world is broken up into more and more tiles, but the viewport has the same number in it.  For example, at zoom level 2 this is a 2x2 grid, but at the 19th zoom level it's 2097151 x 2097151.  That's a lot of tiles!

Why does it matter?
Now the question is, where does this fit into my visualization project?  What I need is the ability to map a lat/long (or group of lat/longs) into a particular tile at every zoom level.  The initial scope of this project is to gather about a week's worth of data and allow the user to view this data at all zoom levels.  The data is currently being gathered at about 6 tweets/second.  6 * 60 * 60 gives us 21,600 tweets per hour which is 3,628,800 (call it 4 million) for the week.  The obvious (and bad) solution to this problem would be to simply create a database of the 4 million tweets at each zoom level and then when I wanted information for a particular tile (for a particular time) I would calculate the number of tweets in that tile/time, make a color out of that number and color the tile.  Obviously this solution is very bad and we can do much better.  The initial plan is to do a bunch of data preprocessing that will allow the data to grouped by both location and time, but that is a topic for another day.

--Colt

Visualization Project and Google Maps

This semester I've been tasked with finding a reasonable project for my advanced graphics course.  A small part of me wanted to use a basic physics engine, get some nice graphics and make my own version of angry birds or something, but I decided I'd go with something else.  Maybe something to do with the social world...

I switched my plan to a visualization of individual tweets on top of something like Nasa World Wind, but as I began to investigate it, I found it very useless. A visualization of an individual tweet isn't nearly as cool as a visualization of many tweets over time, which brought me to my final idea (with the help of Tim)...

I think I'm going to do a visualization of twitter geographical data on top of google maps (or google earth).  This way I'm aggregating the data before showing it to the user, which should be much easier to understand.  My plan is as follows:

1. Divide the map up into a grid.  For starters I will just do this on a single major city.
2. At different zoom levels the data will be regrouped.  This way when a user is looking at the entire city it should show them a set of grids and as they zoom, the grids will become more detailed.
3. Provide (in tooltips or based on click events) details about the data in the grid they're selecting.
4. Provide some kind of coloring of each grid so that a user can tell where the most tweets are coming from.
5. Weight the tweets based on how old they are (e.g. a tweet from yesterday doesn't influence the coloring as much as a tweet from today).

This should provide an aggregation of data that shows where the most geo-coded tweets come from.  The plan is to architect this in a way that if it works well that I could expand my search to include a larger area.

--Colt