IPv4 Banners BigQuery Dataset
Data Collection
The IPv4 Banners collection is composed of lightweight scan data. Censys now continuously collects banners from each of the 2029 TCP ports we scan, and all active hosts we discover are refreshed weekly. For each detected IP, this dataset contains:
- Banners, including HTTP responses from a simple GET request if the host is HTTP
- Any TLS certificates presented by the server
Differences between IPv4 Banners and Traditional Deep IPv4 scan dataset
The Censys IPv4 dataset provides deep, structured data about a subset of the ports and protocols available in the IPv4 Banners dataset, including services such as RDP, SMTP, and MySQL. You can use BigQuery to JOIN
between the IPv4 Banners dataset and traditional IPv4 dataset to gain additional context for hosts. All IPs present in the traditional dataset are included in the banners dataset.
Usage
Information
BigQuery dataset: censys-io.ipv4_banners_public
The full IPv4 Banners dataset schema can be explored here [more information]
The full list of TCP ports that we scan can be explored here [more information]
Examples
Simple Search for ElasticSearch Servers
We’ll just search for the handy "You Know, for Search" text that appears in root ES responses in JSON-like data.
SELECT ip, s.port_number, SAFE_CAST(s.banner AS String) AS banner
FROM `censys-io.ipv4_banners_public.current`, UNNEST(services) AS s
WHERE SAFE_CAST(s.banner AS String) LIKE '%You Know, for Search%'
What's the population of alt-port HTTP servers in country X?
We'll search for HTTP servers running on ports other than 80 and 443 in Germany.
SELECT COUNT(*) AS non_standard_http
FROM `censys-io.ipv4_banners_public.current`,
UNNEST(services) AS s
WHERE (SAFE_CAST(s.banner AS String) LIKE '%Content-Type: %'
OR SAFE_CAST(s.banner AS String) LIKE '%Server: %')
AND s.port_number != 80 AND s.port_number != 443
AND location.country_code = 'DE'
Which certificates are suggestive of fraud (e.g. account-paypal.com)?
We'll use a regex to search for certificates with names containing PayPal, and return a list of the certificate names, along with the IP the certificate is hosted on.
SELECT cert_names, STRING_AGG(ip, '\n') AS ips
FROM `censys-io.ipv4_banners_public.current`, UNNEST(services) AS s,
UNNEST(certificate.names) AS cert_names
WHERE REGEXP_CONTAINS(cert_names, r'^.*paypal[^.].+com$') # *paypal*.com
OR REGEXP_CONTAINS(cert_names, r'^.*[^.]paypal\.com$') # *paypal.com
GROUP BY cert_names
Comments
0 comments
Article is closed for comments.