1. Censys
  2. Censys Data
  3. Google BigQuery

Other Articles In This Section

In this article:

Universal Internet BigQuery Dataset

  • Updated

The Universal Internet Dataset from Censys is the most comprehensive Internet-wide scan data in the industry. This dataset supersedes Censys' legacy IPv4 and IPv4 Banners datasets.

Censys continuously walks the entire IPv4 space, detecting 107 protocols on over 3500 ports to produce a high-resolution map of the public Internet and give threat hunters, attack surface managers, and other security professionals visibility like never before.

A 'snapshot' of every service on the Internet is taken every single day and is added as a partition to a dataset available to Censys enterprise customers who wish to query this data using Google BigQuery.

Scan data in any snapshot is never more than 10 days old, and the data for a service is either structured, protocol-specific key-value pairs, or, if the service is utilizing an unknown protocol, banner data and/or TLS information.

Usage

BigQuery Dataset Names

  • The entire history of daily Internet snapshots:

    censys-io.universal_internet_dataset_v2

  • Today’s snapshot of the live Internet:

    censys-io.universal_internet_dataset_v2.current

  • A snapshot from a specific day:

    censys-io.universal_internet_dataset_v2.{yyyyMMdd}

Data Model

Read this introduction to the host data model to learn how how Internet hosts are represented by Censys.

Schema

Explore the Universal Internet dataset schema for fields you can query.

Query Syntax

Use Standard SQL to query the Censys datasets, which is now the default for Google BigQuery.

Examples

Want to know the port distribution of SMTP services using the compromised TLSv1_0 encryption method on Jun. 9, 2021? Try this query:

 SELECT
  tls.version_selected,
  service_name,
  port,
  COUNT(*) AS count
FROM
  `censys-io.universal_internet_dataset.20210609`,
  UNNEST(services)
WHERE
  tls.version_selected = 'TLSv1_0'
  AND service_name = 'SMTP'
GROUP BY
  1,
  2,
  3
ORDER BY
  4 DESC

Want to see the software (in CPE format) reported by Telnet services across all ports today? Try running this query:

 SELECT
  software.vendor,
  software.product,
  software.version,
  port,
  COUNT(*) AS count
FROM
  `censys-io.universal_internet_dataset.current`,
  UNNEST(services),
  UNNEST(software) AS software
WHERE
  service_name = 'TELNET'
GROUP BY
  1,2,3,4
ORDER BY
  5 DESC

Or see a count of all the Kubernetes services across the Internet on May 25, 2021 grouped by the port they’re running on:

 SELECT
  port,
  COUNT(*) AS count
FROM
  `censys-io.universal_internet_dataset.20210525`,
  UNNEST(services)
WHERE
  service_name = 'KUBERNETES'
GROUP BY
  1
ORDER BY
  2 DESC

Related articles

Comments

0 comments

Article is closed for comments.