Universal Internet BigQuery Dataset
Data Collection
The Universal Internet Dataset from Censys is the most comprehensive Internet-wide scan data in the industry. This dataset supersedes Censys' legacy IPv4 and IPv4 Banners datasets.
Censys continuously walks the entire IPv4 space, detecting 107 protocols on over 3500 ports to produce a high-resolution map of the public Internet and give threat hunters, attack surface managers, and other security professionals visibility like never before.
A 'snapshot' of every service on the Internet is taken every single day and is added as a partition to a dataset available to Censys enterprise customers who wish to query this data using Google BigQuery.
Scan data in any snapshot is never more than 10 days old, and the data for a service is either structured, protocol-specific key-value pairs, or, if the service is utilizing an unknown protocol, banner data and/or TLS information.
Usage
BigQuery Dataset Names
-
The entire history of daily Internet snapshots:
censys-io.universal_internet_dataset_v2
-
Today’s snapshot of the live Internet:
censys-io.universal_internet_dataset_v2.current
-
A snapshot from a specific day:
censys-io.universal_internet_dataset_v2.{yyyyMMdd}
Data Model
Read this introduction to the host data model to learn how how Internet hosts are represented by Censys.
Schema
Explore the Universal Internet dataset schema for fields you can query.
Query Syntax
Use Standard SQL to query the Censys datasets, which is now the default for Google BigQuery.
Examples
Want to know the port distribution of SMTP services using the compromised TLSv1_0 encryption method on Jun. 9, 2021? Try this query:
SELECT tls.version_selected, service_name, port, COUNT(*) AS count FROM `censys-io.universal_internet_dataset.20210609`, UNNEST(services) WHERE tls.version_selected = 'TLSv1_0' AND service_name = 'SMTP' GROUP BY 1, 2, 3 ORDER BY 4 DESC
Want to see the software (in CPE format) reported by Telnet services across all ports today? Try running this query:
SELECT software.vendor, software.product, software.version, port, COUNT(*) AS count FROM `censys-io.universal_internet_dataset.current`, UNNEST(services), UNNEST(software) AS software WHERE service_name = 'TELNET' GROUP BY 1,2,3,4 ORDER BY 5 DESC
Or see a count of all the Kubernetes services across the Internet on May 25, 2021 grouped by the port they’re running on:
SELECT port, COUNT(*) AS count FROM `censys-io.universal_internet_dataset.20210525`, UNNEST(services) WHERE service_name = 'KUBERNETES' GROUP BY 1 ORDER BY 2 DESC
Comments
0 comments
Article is closed for comments.