Data Catalog
Open Internet measurement data released under CC0 — freely available for research, policy, and public interest analysis.
Overview
M-Lab as a platform collects test data from many experiments running across our community. If you're new here, you can browse datasets to find published and archival data from across M-Lab's many experiments and tests. If you'd like to learn more about the tests that drive M-Lab data collection, you can browse our current and historical tests which drive data into our various datasets in M-Lab.
Data Sets
M-Lab data is published in two forms: enriched data (parsed, annotated, and queryable via BigQuery or available as pre-computed summaries) and archival data (measurement logs on Google Cloud Storage for deeper analysis). Both are released under CC0 1.0 Universal with no restrictions on use. Keep scrolling to find them!
M-Lab Data Documentation
Our knowledge base covers how to query the various M-Lab archives and datasets. We provide ready-to-use datasets (available here) and documentation in the Knowledge Base. We also have a Getting Started Page for Researchers in the Knowledge Base that can help you getting started with learning about tests, dataset limitations, and advanced querying of M-Lab data. If you have questions, please contact us.
Data License and Citing M-Lab Data
All data collected by M-Lab tests are available to the public without restriction under a No Rights Reserved Creative Commons Zero Waiver. Please cite M-Lab data sets as follows: The M-Lab test name Data Set, date range used. *M-Lab test URL *
For example:The M-Lab NDT Data Set 2009-02-11–2015-12-21. [https://measurementlab.net/tests/ndtor.
Or in BibTeX format:
@misc{mlab, author="{Measurement Lab}", title="The {M}-{L}ab {NDT} Data Set", year="(2009-02-11 -- 2015-12-21)", howpublished="\url{https://measurementlab.net/tests/ndt}",
comment="Depending on if you used viz.measurementlab.net, bigquery, or the raw data, please use one of the following notes:", note="Bigquery table {\tt measurement-lab.ndt.download}", note1="Google cloud storage {\tt gs://archive-measurement-lab/ndt}", note2="Data visualization system \url{https://viz.measurementlab.net}",}Enriched Datasets
Start here. These datasets have been parsed, annotated, and structured for analysis — queryable via BigQuery or downloadable as pre-computed summaries. Most research questions can be answered without touching raw data.
Raw Datasets
Unprocessed measurement logs and archives as collected directly from M-Lab's infrastructure, stored in Google Cloud Storage. Use these when enriched datasets don't cover your specific analysis need.
NDT (Network Diagnostic Tool) Dataset
The NDT dataset contains measurements of TCP throughput and latency collected by the Network Diagnostic Tool (NDT). NDT is a single-stream performance measurement of a connection's capacity for bulk transport. Measurements include download speed, upload speed, round-trip time (RTT), and packet loss between end-user devices and M-Lab servers worldwide.
2009 – present
Packet Header Data Set
The M-Lab packet-headers service provides a binary which collects packet headers for all incoming TCP flows and saves each stream of packet captures into a per-stream .pcap file where the filename is the UUID of the TCP flow. It only saves the packet headers, and it supports (with a command-line flag) IP anonymity for the saved addresses.
2019 – present
Reverse Traceroute Data Set
Reverse traceroute measures the network path back to a user from selected network endpoints, and provides a rich source of information on network routing and topology. For every TCP connection to an M-Lab server, the Reverse Traceroute service collects network path information from the client back to the M-Lab server. Unlike standard traceroute, which measures the forward path from server to client, reverse traceroute reconstructs the return path without requiring control over the client device. Developed by the Reverse Traceroute group led by Ethan Katz-Bassett (Columbia University), RevTr uses a combination of techniques including Record-Route pings, spoofed probes from nearby vantage points, and atlas-based path stitching to reconstruct reverse paths. As of the current integration with M-Lab, roughly 25% of everyday NDT speed tests are paired with a reverse traceroute measurement.
2024 – present
TCP INFO Data Set
TCP measures the network as part of its normal operation. All transport protocols, including TCP, measure the network to determine how much data to send and when to optimally fill the network. TCP INFO exposes these built-in measurements for diagnostics and other applications. M-Lab uses TCP INFO to collect statistics about every TCP connection used by each hosted measurement service running on the M-Lab platform. TCP INFO runs as a 'side' addition to other tools, taking advantage of TCP connections generated by incoming tests to M-Lab.
2019 – present
Traceroute Data Set
For every TCP connection to an M-Lab server, the Traceroute core service collects network path information from the M-Lab server back to the client IP that initiated the connection. This traceroute collection is done by M-Lab's traceroute-caller which uses traceroute tools like Paris Traceroute and scamper (from the Center for Applied Internet Data Analysis) to collect the actual traceroute data between the M-Lab server and the client. For every TCP connection, traceroute-caller generates two files: traceroute data (scamper1 datatype) and hop annotations (hopannotation1 datatype), which are archived in Google Cloud Storage and parsed into Google BigQuery tables.
2013 – present
Wehe Dataset
The Wehe dataset contains measurements collected by the Wehe app, which detects traffic differentiation (throttling) by ISPs for specific application types such as video streaming, music streaming, and conferencing. Each test replays recorded traffic from popular applications and compares the throughput achieved using original versus anonymized traffic.
2018 – present