1. Censys
  2. Censys Data
  3. Censys Universal Internet Dataset

Other Articles In This Section

In this article

Data Model for the Censys Universal Internet Dataset

The Censys Universal Internet Dataset provides accurate, up-to-date records about Internet-facing hosts: both information that applies to the host as a whole as well as information about the services running on the host.

Host information falls into 3 categories:

  1. Host-level data.

  2. Service summary data.

  3. Service data.

Host-level Data

Host-level data includes information applicable to the host as a whole:

  • IPv4 or IPv6 address

  • name, if applicable

  • geolocation

  • routing information

  • operating system

A snapshot date also provides a timestamp from Censys for the representation of the host.

Host-level fields are located at the root level of the entity.

Example Host-level Data Encoded in JSON

{
    "host_identifier": {
      "ipv4": "61.239.213.42",
      "ipv6": null,
      "name": null,
    },
    "ipv4_int": "1039127850",
    "ipv6_int": null,
    "snapshot_date": "2021-05-23 00:00:00 UTC",
    "location": {
      "continent": "Asia",
      "country": "Hong Kong",
      "country_code": "HK",
      "city": "Shatin",
      "postal_code": "",
      "timezone": "Asia/Hong_Kong",
      "province": "Sha Tin",
      "coordinates": {
        "latitude": "22.3829",
        "longitude": "114.1779"
      },
      "registered_country": "Hong Kong",
      "registered_country_code": "HK"
    },
    "autonomous_system": {
      "asn": "9269",
      "description": "HKBN-AS-AP Hong Kong Broadband Network Ltd.",
      "bgp_prefix": "61.239.192.0/18",
      "name": "HKBN-AS-AP Hong Kong Broadband Network Ltd.",
      "country_code": "HK",
      "organization": null
    },
    "operating_system": null
}

Service Summary Data

Summary data includes fields that inventory the servicesCensys global scanning engineCensys global scanning engine detected on the host, presented as 2 lists: 1 of open ports and 1 of service names.

Each index in the ports_list array is paired to the value with the same index in the service_names_list array.

Summary fields are located at the root level of the host entity.

Example Service Summary Data Encoded in JSON

{
    "ports_list": [
      "7547"
    ],
    "service_names_list": [
      "HTTP"
    ],
}

Service Data

Services is the bulk of host data, with a separate record to show each service scanned by Censys global scanning engine.

Identifying Information and Metadata

Within a service record, a few fields can act as a composite identifier:

  • Port number

  • Transport protocol (for example, TCP, UDP, QUIC).

  • Service name (or extended service name, if you prefer to see the TLS indicator on the name.

Service names usually reflect application-layer (OSI-model L7) protocols such as HTTP or AMQP, although they can also be app-specific services that run on top of HTTP, such as Kubernetes or CWMP.

Example Service-identifying Data Encoded in JSON*

    "services": [
      {
        "port": "7547",
        "transport": "TCP",
        "service_name": "HTTP",
        "extended_service_name": "HTTPS",
        ...
        }
    ]

Metadata

Metadata includes information about when and from where Censys global scanning engine observed the service:

  • Perspective: the name of the ISP that Censys global scanning engine peered with.

  • Source IP

  • Observation timestamp

  • Discovery method: the name of the method by which a service was first found.

Example Metadata Encoded in JSON

    "services": [
      {
        ...
        "perspective": "TELIA",
        "observed_at": "2021-05-23 16:11:27 UTC",
        "source_ip": "74.120.14.38",
        "discovery_method": "IPV4_WALK_FULL_PRIORITY_1"
        ...
      }
    ]

Service-specific Scan Data

Service-specific data includes a map of parsed scan data that corresponds to the service name, TLS data, a banner, if applicable, and software information.

Service Name

The value of the service_name field is a key for a record containing all of the service-specific data parsed from scan.

For example, an RDP service can have a service_name of RDP and a populated record whose key was rdp containing data obtained by the Censys RDP scanner.

TLS

TLS, because it can be enabled for many service types, is shown as a separate service-level record.

Software

Software is a repeated record that contains application, operating system, and/or hardware information presented according to the CPE version 2.3 standard.

Example Service Data

    "services": [
      {
        "tls": null,
        "banner": "SFRUUC8xLjEgNDA0IE5vdCBGb3VuZApDb250ZW50LVR5cGU6IHRleHQvaHRtbApTZXJ2ZXI6IFdlYlNlcnZlci8xLjAgVVBuUC8xLjAKRXh0OiA=",
        "http": {
          "request": {
            "method": "GET",
            "uri": "http://61.239.213.42:7547/",
            "headers": [
              {
                "key": "Accept",
                "value": {
                  "headers": [
                    "Ki8q"
                  ]
                }
              },
              {
                "key": "User-Agent",
                "value": {
                  "headers": [
                    "TW96aWxsYS81LjAgKGNvbXBhdGlibGU7IENlbnN5c0luc3BlY3QvMS4xOyAraHR0cHM6Ly9hYm91dC5jZW5zeXMuaW8vKQ=="
                  ]
                }
              }
            ],
            "body": null
          },
          "response": {
            "protocol": "HTTP/1.1",
            "status_code": "404",
            "status_reason": "Not Found",
            "headers": [
              {
                "key": "Content-Type",
                "value": {
                  "headers": [
                    "dGV4dC9odG1s"
                  ]
                }
              },
              {
                "key": "Server",
                "value": {
                  "headers": [
                    "V2ViU2VydmVyLzEuMCBVUG5QLzEuMA=="
                  ]
                }
              },
              {
                "key": "Ext",
                "value": {
                  "headers": [
                    ""
                  ]
                }
              }
            ],
            "html_tags": [
              "PHRpdGxlPk9iamVjdCBOb3QgRm91bmQ8L3RpdGxlPg=="
            ],
            "body_size": "189",
            "body": "PGh0bWw+CjxoZWFkPgo8dGl0bGU+T2JqZWN0IE5vdCBGb3VuZDwvdGl0bGU+PC9oZWFkPjxib2R5Pgo8aDE+T2JqZWN0IE5vdCBGb3VuZDwvaDE+VGhlIHJlcXVlc3RlZCBVUkwgJy8nIHdhcyBub3QgZm91bmQgb24gdGhlIFNlcnZlci48cD5SZXR1cm4gdG8gPEEgSFJFRj0iIj5sYXN0IHBhZ2U8L0E+PHA+CjwvYm9keT48L2h0bWw+"
          },
          "body_hash": "c2hhMTozZjdhODk0YWI0NDhkNWFmNTlmN2M2ODQxY2Y2YjZkMGZkOTg2M2Rm"
        },
        "software": [
            {
                "uniform_resource_identifier": "cpe:2.3:a:hikvision:hikvision_web_server:*:*:*:*:*:*:*:*",
                "part": "a",
                "vendor": "Hikvision",
                "product": "Hikvision Web Server",
                "version": null,
                "update": null,
                "sw_edition": null,
                "target_sw": null,
                "target_hw": null,
                "language": null,
                "component_uniform_resource_identifiers": [],
                "other": [],
                "edition": null,
                "source": "OSI_APPLICATION_LAYER"
            }
        ],
        "vnc": null,
        "rdp": null,
        "ssh": null,
        "mysql": null,
        "ipmi": null,
        "amqp": null,
        "elasticsearch": null,
        "kubernetes": null,
        "memcached": null,
        "mssql": null,
        "oracle": null,
        "prometheus": null,
        "redis": null,
        "snmp": null,
        "postgres": null,
        "mongodb": null,
        "bacnet": null,
        "dnp3": null,
        "dns": null,
        "ftp": null,
        "imap": null,
        "ipp": null,
        "modbus": null,
        "mqtt": null,
        "ntp": null,
        "pc_anywhere": null,
        "pop3": null,
        "s7": null,
        "smb": null,
        "smtp": null,
        "telnet": null,
        "fox": null,
        "openvpn": null,
        "coap": null,
        "truncated": false,
        "sip": null,
        "team_viewer": null,
        "x11": null,
        "skinny": null,
        "unknown": null,
        "cwmp": null
      }
    ]
  }

Related articles

Comments

0 comments

Article is closed for comments.