1. Censys
  2. Censys Data
  3. Universal Internet Data Set

Other Articles In This Section

In this article:

Data Model for the Universal Internet Data Set

  • Updated

The Universal Internet Data Set provides accurate, up-to-date records about Internet-facing hosts: both information that applies to the host as a whole as well as information about the services running on the host.

Host information falls into three categories:

  • Host-level data

  • Service summary data

  • Service data

Host-level Data

Host-level data presents information applicable to the host as a whole: its IPv4 or IPv6 address, its name, if applicable, its geolocation, routing information, and operating system.

A snapshot date also provides a timestamp from Censys for the representation of the host.

Host-level fields are located at the root level of the entity.

Example Host-level Data Encoded in JSON

{
    "host_identifier": {
      "ipv4": "61.239.213.42",
      "ipv6": null,
      "name": null,
    },
    "ipv4_int": "1039127850",
    "ipv6_int": null,
    "snapshot_date": "2021-05-23 00:00:00 UTC",
    "location": {
      "continent": "Asia",
      "country": "Hong Kong",
      "country_code": "HK",
      "city": "Shatin",
      "postal_code": "",
      "timezone": "Asia/Hong_Kong",
      "province": "Sha Tin",
      "coordinates": {
        "latitude": "22.3829",
        "longitude": "114.1779"
      },
      "registered_country": "Hong Kong",
      "registered_country_code": "HK"
    },
    "autonomous_system": {
      "asn": "9269",
      "description": "HKBN-AS-AP Hong Kong Broadband Network Ltd.",
      "bgp_prefix": "61.239.192.0/18",
      "name": "HKBN-AS-AP Hong Kong Broadband Network Ltd.",
      "country_code": "HK",
      "organization": null
    },
    "operating_system": null
}

Service Summary Data

Summary data includes fields that inventory the services Censys detected on the host, presented as two lists: one of open ports and one of service names.

Each index in the ports_list array is paired to the value with the same index in the service_names_list array.

Summary fields are located at the root level of the host entity.

Example Service Summary Data Encoded in JSON

{
    "ports_list": [
      "7547"
    ],
    "service_names_list": [
      "HTTP"
    ],
}

Service Data

Services comprise the bulk of host data, with a separate record to represent each service scanned by Censys.

Identifying Information and Metadata

Within a service record, there are a few fields that can act as a composite identifier:

  • Port number

  • Transport protocol (e.g., TCP, UDP, QUIC)

  • Service name (or extended service name, if you prefer to see the TLS indicator on the name)

Service names usually reflect application-layer (OSI-model L7) protocols such as HTTP or AMQP, although they can also be app-specific services that run on top of HTTP, such as Kubernetes or CWMP.

Example Service-Identifying Data Encoded in JSON*

    "services": [
      {
        "port": "7547",
        "transport": "TCP",
        "service_name": "HTTP",
        "extended_service_name": "HTTPS",
        ...
        }
    ]

Metadata

Metadata includes information about when and from where Censys observed the service:

  • Perspective (The name of the ISP that Censys peered with)

  • Source IP

  • Observation timestamp

  • Discovery method (The name of the method by which a service was first found)

Example Metadata Encoded in JSON

    "services": [
      {
        ...
        "perspective": "TELIA",
        "observed_at": "2021-05-23 16:11:27 UTC",
        "source_ip": "74.120.14.38",
        "discovery_method": "IPV4_WALK_FULL_PRIORITY_1"
        ...
      }
    ]

Service-Specific Scan Data

Service-specific data includes a map of parsed scan data that corresponds to the service name, TLS data, a banner, if applicable, and software information.

Service Name

The value of the service_name field is a key for a record containing all of the service-specific data parsed from scan.

For example, an RDP service would have a service_name of RDP and a populated record whose key was rdp containing data obtained by the Censys RDP scanner.

TLS

TLS, since it can be enabled for many service types, is presented as a separate service-level record.

Software

Finally, software is a repeated record that contains application, operating system, and/or hardware information presented according to the CPE version 2.3 standard.

Example Service Data

    "services": [
      {
        "tls": null,
        "banner": "SFRUUC8xLjEgNDA0IE5vdCBGb3VuZApDb250ZW50LVR5cGU6IHRleHQvaHRtbApTZXJ2ZXI6IFdlYlNlcnZlci8xLjAgVVBuUC8xLjAKRXh0OiA=",
        "http": {
          "request": {
            "method": "GET",
            "uri": "http://61.239.213.42:7547/",
            "headers": [
              {
                "key": "Accept",
                "value": {
                  "headers": [
                    "Ki8q"
                  ]
                }
              },
              {
                "key": "User-Agent",
                "value": {
                  "headers": [
                    "TW96aWxsYS81LjAgKGNvbXBhdGlibGU7IENlbnN5c0luc3BlY3QvMS4xOyAraHR0cHM6Ly9hYm91dC5jZW5zeXMuaW8vKQ=="
                  ]
                }
              }
            ],
            "body": null
          },
          "response": {
            "protocol": "HTTP/1.1",
            "status_code": "404",
            "status_reason": "Not Found",
            "headers": [
              {
                "key": "Content-Type",
                "value": {
                  "headers": [
                    "dGV4dC9odG1s"
                  ]
                }
              },
              {
                "key": "Server",
                "value": {
                  "headers": [
                    "V2ViU2VydmVyLzEuMCBVUG5QLzEuMA=="
                  ]
                }
              },
              {
                "key": "Ext",
                "value": {
                  "headers": [
                    ""
                  ]
                }
              }
            ],
            "html_tags": [
              "PHRpdGxlPk9iamVjdCBOb3QgRm91bmQ8L3RpdGxlPg=="
            ],
            "body_size": "189",
            "body": "PGh0bWw+CjxoZWFkPgo8dGl0bGU+T2JqZWN0IE5vdCBGb3VuZDwvdGl0bGU+PC9oZWFkPjxib2R5Pgo8aDE+T2JqZWN0IE5vdCBGb3VuZDwvaDE+VGhlIHJlcXVlc3RlZCBVUkwgJy8nIHdhcyBub3QgZm91bmQgb24gdGhlIFNlcnZlci48cD5SZXR1cm4gdG8gPEEgSFJFRj0iIj5sYXN0IHBhZ2U8L0E+PHA+CjwvYm9keT48L2h0bWw+"
          },
          "body_hash": "c2hhMTozZjdhODk0YWI0NDhkNWFmNTlmN2M2ODQxY2Y2YjZkMGZkOTg2M2Rm"
        },
        "software": [
            {
                "uniform_resource_identifier": "cpe:2.3:a:hikvision:hikvision_web_server:*:*:*:*:*:*:*:*",
                "part": "a",
                "vendor": "Hikvision",
                "product": "Hikvision Web Server",
                "version": null,
                "update": null,
                "sw_edition": null,
                "target_sw": null,
                "target_hw": null,
                "language": null,
                "component_uniform_resource_identifiers": [],
                "other": [],
                "edition": null,
                "source": "OSI_APPLICATION_LAYER"
            }
        ],
        "vnc": null,
        "rdp": null,
        "ssh": null,
        "mysql": null,
        "ipmi": null,
        "amqp": null,
        "elasticsearch": null,
        "kubernetes": null,
        "memcached": null,
        "mssql": null,
        "oracle": null,
        "prometheus": null,
        "redis": null,
        "snmp": null,
        "postgres": null,
        "mongodb": null,
        "bacnet": null,
        "dnp3": null,
        "dns": null,
        "ftp": null,
        "imap": null,
        "ipp": null,
        "modbus": null,
        "mqtt": null,
        "ntp": null,
        "pc_anywhere": null,
        "pop3": null,
        "s7": null,
        "smb": null,
        "smtp": null,
        "telnet": null,
        "fox": null,
        "openvpn": null,
        "coap": null,
        "truncated": false,
        "sip": null,
        "team_viewer": null,
        "x11": null,
        "skinny": null,
        "unknown": null,
        "cwmp": null
      }
    ]
  }

Related articles

Comments

0 comments

Article is closed for comments.