Self-hosted GeoIP Lookup

The word at the DL 5-min mark was was so GeoIP wins the vote, I need to break this into two parts because… yeah.

One man’s journey through broken links, depreciated instructions and bad web design on a quest to find himself…

Self-hosted GeoIP Lookup: Part 1, The Downloadening

Debrief:

For this to work we need two things, a beefy database and a software solution to use it. Part 1 is about getting the database, part 2 is the easy part.

It appears the only game in town is Maxmind’s geolocation databases licensed under Creative Commons CC BY-SA 4.0. The project started ~18 years ago and if you’re checking them out i’d recommend sticking to the sitemap.

They use a custom binary format for the database to maximize lookup speed so you need a special reader to use it but they provide one for all popular languages and there’s packages in all popular Linux repos that cover all popular use cases (they’re very popular). Lookup data is also available in CSV if you want to send LibreOffice Calc into the next dimension.

There’s 3 database groups:

  1. GeoLite legacy, which will be discontinued May 2022
  2. GeoLite2, which is the new hotness
  3. Geo2, which is more accurate but pay-2-win

We’ll be using GeoLite2 which contrary to every guide on the entire Internet is behind a login wall.

It’s go time:

I tried guerrillamail.com for the email while using ProtonVPN. I couldn’t sign up till i’d dropped the VPN and used a “real” email.

image

You’ll get an email, use it to set a new password and login to the control panel.

Option 1: Create permalinks so you can wget the databases server-side

  • On the left menu under “Services” click “My License Key”
    • Click “Generate new license key”
    • Click the “No” radial for “Will this key be used for GeoIP Update?”
    • Click “Confirm”
    • Copy the license key and save it
  • On the left menu under “GeoIP2 / GeoLite2” click “Download Files”
    • Option A:
      • Use Ulfnic’s script below to download, checksum and rename every database
    • Option B.
      • Click “Get Permalinks” next to each download
      • wget each URL replacing YOUR_LICENSE_KEY with the one you saved earlier
      • Optional: checksum each database
      • Optional: Open the checksum file to know the dated filename to use for the database file (so you know if there’s an updated version in future).

Option 2. Download the files locally and upload them to the server

  • On the left menu under “GeoIP2 / GeoLite2” click “Download Files”
    • Click “Download” on each database
    • Optional: checksum each database
    • Upload to server

If ls shows you this, you’re ready for part 2:

image


Option 1A server-side download script:

#!/usr/bin/env sh

# This script will...
# - Ask for a Maxmind license key
# - Download every GeoLite2 database using that license key
# - Apply the correct, dated filename inside the checksum file to each download instead of a generic filename
# - Perform all checksums

echo -n "Enter your license key: "; read YOUR_LICENSE_KEY

URLS=( \
"https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key="$YOUR_LICENSE_KEY"&suffix=tar.gz" \
"https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN-CSV&license_key="$YOUR_LICENSE_KEY"&suffix=zip" \
"https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key="$YOUR_LICENSE_KEY"&suffix=tar.gz" \
"https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City-CSV&license_key="$YOUR_LICENSE_KEY"&suffix=zip" \
"https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country&license_key="$YOUR_LICENSE_KEY"&suffix=tar.gz" \
"https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key="$YOUR_LICENSE_KEY"&suffix=zip" \
)

for URL in "${URLS[@]}"; do
	echo === FETCHING ===
	echo $URL
	wget -O sha256 "$URL.sha256"
	FILENAME=$(cat sha256 | awk '{print $2}')
	mv sha256 "$FILENAME.sha256"
	wget -O "$FILENAME" "$URL"
	echo === CHECKSUM ===
	sha256sum -c "$FILENAME.sha256"
done

2 Likes

Self-hosted GeoIP Lookup: Part 2, “I never asked for this.” ~ Adam, DuesEx

Debrief:

Making it harder by trying to make it easier.

Digging through repos I discovered Maxmind GeoIP-GeoLite-data and GeoIP-GeoLite-data-extra packages in Fedora and CentOS but they haven’t been updated since 2018. I gave them an install and they’re GeoIP legacy (so yesterday).

Similar situation with Debian/Ubuntu’s geoip-database and geoip-database-extra.

I gave the GeoIP package a try which is on several distros but sadly it just pulls in the 2018 data above. For GeoIP2, you’ll need to follow Part 1.

image

Looking at downloads from part 1, what we’re working with:

  • GeoLite2-ASN contains ASN information
  • GeoLite2-City contains everything but ASN information
  • GeoLite2-Country is GeoLite2-City without map coordinates

For the purposes of Part 2 we won’t be using the GeoLite2-Country database as GeoLite2-City makes it redundant however it’ll still be installed as some use cases may only need identification by country which is considerably faster.

Thankfully the database reader has general availability but the NGINX module for geoip2 isn’t in repos for Fedora/CentOS/RHEL no matter how much NGINX thinks its is. That leaves 3 options…

  1. Compiling the module in with NGINX every update.
  2. Not using Fedora/CentOS/RHEL (as if.)
  3. Produce a solution that doesn’t require a special module.

Hold my beer, we’re doing 3

It’s go time:

Install the GeoLite2 databases:

# Navigate to the directory containing the downloaded databases in Part 1

# Decompress GeoLite2 databases
sudo tar xvfz GeoLite2-ASN_20210112.tar.gz
sudo tar xvfz GeoLite2-City_20210112.tar.gz
sudo tar xvfz GeoLite2-Country_20210112.tar.gz

# Move them to /usr/share/geoip2
sudo mkdir /usr/share/geoip2/
sudo mv ./GeoLite2-ASN_20210112 /usr/share/geoip2/GeoLite2-ASN
sudo mv ./GeoLite2-City_20210112 /usr/share/geoip2/GeoLite2-City
sudo mv ./GeoLite2-Country_20210112 /usr/share/geoip2/GeoLite2-Country

# See layout
ls -laR /usr/share/geoip2/

Install the Maxmind database reader:

# Fedora/CentOS/RHEL:
sudo dnf install libmaxminddb libmaxminddb-devel

# Debian/Ubuntu:
sudo apt install libmaxminddb0 libmaxminddb-dev mmdb-bin

Test the GeoLite2 databases by requesting information on 8.8.8.8, each one should output different information on Google.

mmdblookup --file /usr/share/geoip2/GeoLite2-ASN/GeoLite2-ASN.mmdb --ip 8.8.8.8
mmdblookup --file /usr/share/geoip2/GeoLite2-City/GeoLite2-City.mmdb --ip 8.8.8.8
mmdblookup --file /usr/share/geoip2/GeoLite2-Country/GeoLite2-Country.mmdb --ip 8.8.8.8

That’s all you need to begin geoip integration. Part 3 will apply this to a router providing a full solution for geoip lookup via GET request.

# Placeholder for part 3