raclette: Human-friendly monitoring of Internet delays

Raclette simplifies the analysis of numerous traceroute results by merging results and adding semantics. For example, one can use raclette to read millions traceroute results from RIPE Atlas API and then query raclette for delays between “Amsterdam, NL” and “AS15169” (Google) to obtain the median Round Trip Times (RTTs) between all vantage points in Amsterdam and all Google IP addresses found in traceroutes.

Requirements

• Python 3.9+
• uv (to install python libraries and run raclette)
• a C/C++ compiler with OpenMP support (to build the Cython extensions)

Install

Get the latest source files:

git clone git@github.com:InternetHealthReport/raclette.git

Install dependencies in a virtual environment, build the Cython extensions, and create the cache directory:

uv sync
uv run python setup.py build_ext --inplace
mkdir cache

uv sync creates the .venv directory, there is no need to activate it: every command below is run with uv run, which uses that environment.

IP to ASN and IP to IXP mappings

Router IP addresses are mapped to ASNs and IXPs with data from IYP: prefixes originated in BGP come from the BGPKIT pfx2as dataset (bgpkit.pfx2asn) and IXP peering LANs come from the PeeringDB dataset (peeringdb.ix). Prefixes are cached in cache/pfx2asn.json (refreshed daily) and cache/ixp_prefixes.json (refreshed weekly). IXPs are reported with negative AS numbers, the absolute value is the PeeringDB ID of the IXP.

This is configured in the [lib] section of the configuration file:

[lib]
ip2asn_source = iyp

Set ip2asn_source = file to use local databases instead (a RIB dump in pickle format and an IXP jsonl file, see ip2asn_db and ip2asn_ixp), for example to analyze past events with the mappings of that time.

The IYP instance used for all datasets can be changed with the IYP_URL environment variable:

export IYP_URL=neo4j://iyp-bolt.ihr.live:7687

IP geolocation

IP addresses are geolocated with an IPinfo database in mmdb format. By default raclette reads data/ipinfo_location.mmdb, a different database can be given with the IPINFO_DB environment variable:

export IPINFO_DB=/path/to/your/ipinfo_location.mmdb

The database is automatically downloaded if it is missing or more than one week old. This requires an IPinfo API token, given with the IPINFO_TOKEN environment variable:

export IPINFO_TOKEN=your_ipinfo_token

If no token is given, or if the download fails, raclette keeps using the local database (a warning is logged). The database can also be updated manually:

python -c "import sys; sys.path.append('raclette'); import tools; tools.download_ipinfo_db()"

Anycast IP addresses are hosted at multiple locations, so geolocating them is meaningless. Raclette therefore ignores IP addresses that fall in an anycast prefix. Anycast prefixes come from the LACeS anycast census (datasets utwente.laces_v4 and utwente.laces_v6), which is fetched from IYP and cached in cache/anycast_prefixes.json (refreshed weekly).

RIPE ipmap geolocations can still be used instead (see tools.IpmapGeolocation), in that case fetch the ipmap data first:

wget https://ftp.ripe.net/ripe/ipmap/geolocations-latest -O cache/geolocations_ipmap.csv.bz2 

Mapping a single IP address

The mapping modules can be run on their own, for example to check the ASN and the IXP of a few IP addresses:

uv run raclette/iyp2asn.py 8.8.8.8 80.81.192.1

Example

In this example we'll look at delay changes for SMW3 cable failures and ASC cable activation. For that we look at Atlas anchoring measurements from September 3rd to 5th, 2018.

Configuration file

The first step is to make a configuration file that will describe the start/end times, the measurement IDs, and the output directory. For this example the configuration file looks like this (also available in the source files):

[main]

[io]
# Options for the Atlas REST API
start = 2018-09-02T12:00
stop = 2018-09-06T12:00
# Fetch data by chunks of chunk_size seconds. Set a smaller value if you have
# memory problems
chunk_size = 1800
msm_ids =  1748022, 1748024, 11645084, 11645087, 2244316, 2244318, 2244316, 2244318, 2435592, 2435594, 1796567, 1796569, 2904335, 2904338, 1618360, 1618362, 7970886, 7970889, 7970886, 7970889, 6886972, 6886975, 12237261 
probe_ids = 

# Options for the output
results = results/ASC_start/results_%(start)s.sql
log = results/ASC_start/log_%(start)s.log

[timetrack]
converter = allin_cy

[tracksaggregator]
window_size = 1800
significance_level = 0.05
# ignore links visited by small number of tracks/traceroutes
min_tracks = 3 

[anomalydetector]
enable=1

[lib]
# Read local databases, this analysis needs IP to ASN mappings of 2018
ip2asn_source = file
ip2asn_directory = raclette/lib/
ip2asn_db = data/rib.20180701.pickle
ip2asn_ixp =

The most important options are:

• start, stop: The beginning and end time of the measurement period
• msm_ids: The atlas measurements for which we want traceroutes
• log: The log file used for this run
• results: All results are stored in this file. This a sqlite database.

For this example we save this file to conf/asc-start.conf.

Traceroute analysis

To download traceroute and compute delays just run the following command:

uv run raclette/raclette.py -C conf/asc-start.conf

This is going to take about 30 minutes, it downloads four days of traceroutes. You can follow the progress in the log file with the following command:

tail -F results/ASC_start/log_2018-09-02T12:00.log

Looking at the results

All results are stored in a sqlite database. The filename is the one you gave in the configuration file (io/results). Raclette comes with a script to easily plot graphs from that database. For example, to plot delays between Melbourne and Singapore use the following commands:

uv run raclette/plotter.py results/ASC_start/results_2018-09-02T12:00.sql  '%Melbourne%' '%Singapore%'

The following graph will be save in the fig/ directory: The graph shows delay changes on Sep.3rd due to a submarine cable cut (SMW3) between Singapore and Australia and the start of a new submarine cable (ASC) on Sep.5th.

Miscellaneous

If you have executed raclette.py several times with the same results file. The results for each instance are stored with a different experiment ID (cf. the field expid in the database). To plot with results from the third execution:

uv run raclette/plotter.py results/ASC_start/results_2018-09-02T12:00.sql  '%Melbourne%' '%Singapore%' 3

You can also directly query the database with a sqlite client. The table you want to look at is diffrtt:

sqlite3 results/ASC_start/results_2018-09-02T12:00.sql
sqlite> .schema diffrtt
CREATE TABLE diffrtt (ts integer, startpoint text, endpoint text, median real, confhigh real, conflow real, nbtracks integer, nbprobes integer, entropy real, hop integer, expid integer, foreign key(expid) references experiment(id));
[...]
sqlite> .headers on
sqlite> select * from diffrtt limit 10;
ts|startpoint|endpoint|median|confhigh|conflow|nbtracks|nbprobes|entropy|hop|expid
1535890500|PB20092|AS48441v4|0.597|0.743|0.503|4|1|0.0|1|3
1535890500|PB20092|IPv4|311.129|344.056|185.155|22|1|0.0|3|3
1535890500|PB20092|AS12389v4|167.621|172.017|2.027|4|1|0.0|2|3
1535890500|PB20092|IX438v4|184.429|185.155|181.18|4|1|0.0|3|3
1535890500|PB20092|AS4637v4|340.924|348.105|264.141|4|1|0.0|4|3
1535890500|AS48441v4|AS12389v4|165.556|167.145|1.635|4|1|0.0|1|3
1535890500|AS48441v4|IPv4|346.223|348.654|343.459|18|1|0.0|3|3
1535890500|IPv4|AS12389v4|22.067|22.677|1.644|6|2|0.918295834054489|1|3
1535890500|AS48441v4|IX438v4|182.547|184.018|180.735|4|1|0.0|2|3
1535890500|IPv4|IX438v4|14.885|15.534|14.144|669|51|0.96986633428238|1|3