IP Night Light

[#1]

The previous post created a day/night map of the world with the day represented by a background bitmap and the night represented by hollow outlines.  This example will show how to add some interesting detail to the night map by plotting IP locations, creating the yellow haze in the night section of the map below.

click for bigger image...

The IP locations come from files the script downloads from MaxMind.  Their data is pretty cool and lets you map IP addresses to locations.  Not using it for that here.  Here's their notice:

This example includes GeoLite data created by MaxMind, available from http://www.maxmind.com.

Since the file is pretty large, this JSL will check to see if the data is already downloaded to your $temp directory.  If the TRY statement fails to open the table (first statement, before the comma), the catch part of the try (statements after the comma) will download and save a fresh copy.  The download uses two open() statements: the first fetches the zip archive, the second does the text import on a CSV member read from the zip archive.  Just before saving the table (with just the Lat,Lon columns), <<SetTableVariable adds the attribution notice as a table variable.  Take a close look at that zip file manipulation if you ever need to have JMP automatically extract something from a zip. The extra work with the member list is because the member names have a recent date embedded and can't be hard-coded.  The second open(), after the read of the archive, omits a bunch of columns that could be really useful.  You can remove the columns() specification and JMP will spend a little time figuring them out, then import the table and make a source script, similar to this one, with the columns already figured out (faster next time).  The other member of the archive can be used with this member to make a database of IP locations.

// download the maxmind zip file http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip
// it isn't required, but provides a beautiful nighttime glow around parts of the world with internet connections.
Try(
    cityloc = Open( "$temp/cityloc.jmp" ); // try to open an existing saved table
   , // if that fails, open a zip archive from the internet
    za = Open( "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip", "zip" );
    zipMemberList = za << dir;// there are two files in the zipped archive, with any luck, this is still the name...
    // this zip archive member has the Lat,Lon data
    citylocblob = za << read(
        zipMemberList[If( // the member name includes a date that changes.  this should pick the right member
            Ends With( zipMemberList[1], "GeoLiteCity-Location.csv" ), 1,
            Ends With( zipMemberList[2], "GeoLiteCity-Location.csv" ), 2,
            . // if this happens, perhaps there is an extra file in the zip?
        )],
        Format( blob )
    );
    cityloc = Open(
        citylocblob, // open understands a blob can hold CSV data
        columns( // there are only two cols we need
            Omitted Column( . ), Omitted Column( . ), Omitted Column( . ),
            Omitted Column( . ), Omitted Column( . ),
            Column( "latitude", Numeric, "Continuous", Format( "Best", 10 ) ),
            Column( "longitude", Numeric, "Continuous", Format( "Best", 10 ) ),
            Omitted Column( . ), Omitted Column( . )
        ),
        Import Settings( // runs fast when it doesn't have to guess
            End Of Line( CRLF, CR, LF ), End Of Field( Comma, CSV( 1 ) ),
            Strip Quotes( 1 ), Use Apostrophe as Quotation Mark( 0 ), Scan Whole File( 1 ),
            Treat empty columns as numeric( 0 ), CompressNumericColumns( 0 ),
            CompressCharacterColumns( 0 ), CompressAllowListCheck( 0 ), Labels( 1 ),
            Column Names Start( 2 ), Data Starts( 3 ), Lines To Read( "All" ), Year Rule( "20xx" )
        )
    );
    // add  Creative Commons Attribution-ShareAlike 3.0 Unported License  text MaxMind wants...
    cityloc << settablevariable( "MaxMind", "This table includes GeoLite data created by MaxMind, available from http://www.maxmind.com" );
    cityloc<<save("$temp/cityloc.jmp"); // save it for next time
);

You'll get about 3/4 million locations that will create the yellow-white areas inside the boundaries in the nighttime areas in the map at the top (and you'll get a different number of rows because they update the table frequently):

The next part of the script uses the Time Zone  function from a couple of posts back.  The GMT time is needed to position the map sun/shadow at the correct location, now.  The remainder of the script is pretty similar to the previous script, with just one additional script added to the GraphBox to draw the extra locations (and a little script to load the locations into lat,lon matrixes):

Try(
    IPdt = cityloc;
    lat = IPdt:latitude << getasmatrix;
    lon = IPdt:longitude << getasmatrix;
, // data not available?
    lat = [];
    lon = [];
);
g[framebox( 1 )] << addGraphicsScript(
    3, // before shape's new position, after the shadow
    transparency(.2);
    Marker( Color State( {0.8, 0.8, 0.7} ),lon, lat );
);

A side note: the Time Zone function will not return an integer if your computer's clock has drifted more than a few seconds.  My home computer was off by about 4 seconds and reported 4.999 rather than 5 (because the round function in the script requests three decimal places).  Rounding like this may affect results, depending how the number is used.

The try() makes sure Lat,Lon is a matrix, possibly empty if something went wrong.  The graphic script creates a marker from the Lat,Lon matrix and sets the RGB color to a light yellowish graph.  The transparency of .2 makes duplicate points show up more intensely than single points.  The yellow color makes the points less obvious in the daytime parts of the graph, and some of the map texture is lost under them.

Coming up: capturing twitter data:

InternetLights.jsl