Skip to main content

IPv4 Banners BigQuery Dataset

  • Updated

Data Collection

The IPv4 Banners (aka, Universal Internet Dataset) collection is composed of lightweight scan data.  Censys now continuously collects banners from each of the 2029 TCP ports we scan, and all active hosts we discover are refreshed weekly.  For each detected IP, this dataset contains:

  • Banners, including HTTP responses from a simple GET request if the host is HTTP
  • Any TLS certificates presented by the server

Differences between IPv4 Banners and Traditional Deep IPv4 scan dataset

The Censys IPv4 dataset provides deep, structured data about a subset of the ports and protocols available in the IPv4 Banners dataset, including services such as RDP, SMTP, and MySQL. You can use BigQuery to JOIN between the IPv4 Banners dataset and traditional IPv4 dataset to gain additional context for hosts. All IPs present in the traditional dataset are included in the banners dataset.

 

Usage

Information

BigQuery dataset: censys-io.ipv4_banners_public

The full IPv4 Banners dataset schema can be explored here [more information]

The full list of TCP ports that we scan can be explored here [more information]

Examples 

Simple Search for ElasticSearch Servers

We’ll just search for the handy "You Know, for Search" text that appears in root ES responses in JSON-like data.

SELECT ip, s.port_number, SAFE_CAST(s.banner AS String) AS banner
FROM `censys-io.ipv4_banners_public.current`, UNNEST(services) AS s
WHERE SAFE_CAST(s.banner AS String) LIKE '%You Know, for Search%'

What's the population of alt-port HTTP servers in country X?

We'll search for HTTP servers running on ports other than 80 and 443 in Germany.

SELECT COUNT(*) AS non_standard_http
FROM `censys-io.ipv4_banners_public.current`,
  UNNEST(services) AS s
WHERE (SAFE_CAST(s.banner AS String) LIKE '%Content-Type: %' 
  OR SAFE_CAST(s.banner AS String) LIKE '%Server: %')
AND s.port_number != 80 AND s.port_number != 443
  AND location.country_code = 'DE'

Which certificates are suggestive of fraud (e.g. account-paypal.com)?

We'll use a regex to search for certificates with names containing PayPal, and return a list of the certificate names, along with the IP the certificate is hosted on. 

SELECT cert_names, STRING_AGG(ip, '\n') AS ips
FROM `censys-io.ipv4_banners_public.current`, UNNEST(services) AS s,
 UNNEST(certificate.names) AS cert_names
WHERE REGEXP_CONTAINS(cert_names, r'^.*paypal[^.].+com$') # *paypal*.com
 OR REGEXP_CONTAINS(cert_names, r'^.*[^.]paypal\.com$') # *paypal.com
GROUP BY cert_names

Related articles