IPv4 Banners Dataset

We've added a new dataset that for simple banners collected from IPv4 hosts on more than 1000 ports. The dataset contains records for all IPv4 addresses Censys has observed with an open port.

For each detected IP, this dataset contains:

  • Banners, including HTTP responses from a simple GET request if the host is HTTP
  • Any TLS certificates presented by the server

These lightweight banner grabs are in addition to our existing dataset containing structured data from deep scanning of common protocols.

Censys collects banners from more than 1000 ports continuously and all active hosts are refreshed weekly

Use these broader, lightweight scans to identify exposed services that are public-facing on the Internet and rely on our more in-depth IPv4 data sets to drill down into any suspicious or unexpected banner or TLS certificate information you find in the banners dataset.

How to access these new broad lightweight port scans

To view the the new ipv4-banners data set schema:

The full list of TCP/IP ports we scan in this data set are listed here.  

Here's a sample of the new ports we've added to our dataset:

Port 9200 (Elasticsearch)

{
  "ip": "x.x.x.x",
  "port_number": "9200",
  "transport_protocol": "TCP",
  "banner": "HTTP/1.1 200 OK\r\ncontent-type: application/json; charset=UTF-8\r\ncontent-length: 431\r\n\r\n{\n  \"name\" : \"node-1\",\n  \"cluster_name\" : \"elasticsearch\",\n  \"cluster_uuid\" : \"xxxxxxxxxxx\",\n  \"version\" : {\n    \"number\" : \"6.1.4\",\n    \"build_hash\" : \"d838f2d\",\n    \"build_date\" : \"2018-03-14T08:28:22.470Z\",\n    \"build_snapshot\" : false,\n    \"lucene_version\" : \"7.1.0\",\n    \"minimum_wire_compatibility_version\" : \"5.6.0\",\n    \"minimum_index_compatibility_version\" : \"5.0.0\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}\n"
}

Port 2376 (Docker)

{
  "ip": "x.x.x.x",
  "port_number": "2376",
  "transport_protocol": "TCP",
  "banner": "\u0015\u0003\u0001\u0000\u0002\u0002\n"
}

Port 6379 (Redis)

{
    "ip": "x.x.x.x",
    "port_number": "6379",
    "protocol": null,
    "transport_protocol": "TCP",
    "banner": "-DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above thin"
  }

Port 5601 and 5602 (Kibana)

{
  "ip": "x.x.x.x",
  "port_number": "5601",
  "transport_protocol": "TCP",
  "banner": "HTTP/1.1 200 OK\r\nkbn-name: kibana\r\nkbn-version: 6.2.4\r\ncache-control: no-cache\r\ncontent-type: text/html; charset=utf-8\r\ncontent-length: 217\r\naccept-ranges: bytes\r\nDate: Wed, 13 Mar 2019 15:13:31 GMT\r\nConnection: keep-alive\r\n\r\n"
}

Differences between IPv4 Banners and Traditional Deep IPv4 scan dataset

The Censys IPv4 dataset provides deep, structured data about a subset of the ports and protocols available in the IPv4 Banners dataset, including services such as RDP, SMTP, and MySQL. You can use BigQuery to JOIN between the IPv4 Banners dataset and traditional IPv4 dataset to gain additional context for hosts. All IPs present in the traditional dataset are included in the banners dataset.

To view the IPv4 data set schema: 

Leveraging the certificates dataset

The certificates dataset contains all X.509 certificates observed by Censys, including structured data from parsed certificates. You can link between the banners dataset and the certificates dataset via the certificate fingerprint_sha256  field.

Example Queries

Before writing your first query, note that you'll need to use Standard SQL to query the Censys datasets. Unfortunately, Google BigQuery defaults to Legacy SQL and the option to change your query to Standard SQL is hidden. Once you are in the Compose Query screen click Show Options and uncheck Use Legacy SQL. 

Another option is to include the following command at the top of your SQL statement in the web interface as shown below:

#standardsql
<SQL GOES HERE>

For more information on constructing SQL statements, check out the BigQuery SQL Reference, and the schema definition for the IPv4 Banners dataset.

Simple Search for Elasticsearch Servers

We’ll just search for the handy "You Know, for Search" text that appears in root ES responses in JSON-like data.

SELECT
 ip,
 services.port_number,
 SAFE_CONVERT_BYTES_TO_STRING(services.banner) AS str_banner
FROM
  censys-io.ipv4_banners_public.20190306,
  UNNEST(services) AS services
WHERE SAFE_CONVERT_BYTES_TO_STRING(services.banner) LIKE '%You Know, for Search%';

What's the population of alt-port http servers in country X?

SELECT
  COUNT(*) AS non_standard_http
FROM
  censys-io.ipv4_banners_public.20190306,
  UNNEST(services) AS p
WHERE
  (SAFE_CONVERT_BYTES_TO_STRING(p.banner) LIKE '%Content-Type: %'
    OR SAFE_CONVERT_BYTES_TO_STRING(p.banner) LIKE '%Server: %')
  AND p.port_number != 80
  AND p.port_number != 443
  AND location.country_code = 'DE';

Which certificates are suggestive of fraud (e.g. account-paypal.com)?

SELECT
  cert_names,
  STRING_AGG(ip, '\n') AS ips
FROM
  censys-io.ipv4_banners_public.20190306,
  UNNEST(services) AS p,
  UNNEST(certificate.names) AS cert_names
WHERE
  REGEXP_CONTAINS(cert_names, r'^.*paypal[^.].+com$') # paypal*.com
  OR REGEXP_CONTAINS(cert_names, r'^.*[^.]paypal\.com$') # *paypal.com
GROUP BY
  cert_names;

New to BigQuery?

Our BigQuery Introduction provides an overview of Google BigQuery and how to  query and export Censys data.

Add the Censys IPv4 Dataset to your BigQuery account

Follow the instructions in our Adding Censys Datasets to BigQuery guide and verify that you see the censys-io project and ipv4_banners dataset in your Google BigQuery web interface.


Need more help? Reach out to support@censys.io and we'll help you through the process. If you'd like access to these new lightweight port scans but aren't yet an enterprise customer, talk to sales@censys.io about how to upgrade your account.

Did this answer your question?