Downloading Certs 2.0 Data
The Censys global scanning engine certificate repository is the largest in the world, and growing every day.
Acertificate’s contents are immutable after issued, although accompanying data can change, such as:
-
which certificate transparency logs it has been submitted to
-
whether it has been revoked
-
whether it has ever been seen in a Censys global scanning engine scan of the Internet
Censys provides these changes to certificate records and new records representing newly seen certificates in a daily download.
Each day, an incremental dataset with just that day’s certificate record changes and new record additions is available to download. It contains new certificate records and diffs of existing certificate record’s metadata.
If you want to download certificate data for unlimited querying and use in custom workflows, complete a one-time download of the full certificate repository, followed by a daily download containing the day’s changes to be applied to the local copy of the full dataset.
A full snapshot of all of the certificates in the repository is available on the 1st of every month.
-
certificates-v2-full
Incremental downloads are available every day, including on the 1st of every month. In the event that a client runs fewer than 1 time per day, apply all changes in date and time order from every incremental update.
-
certificates-v2-incremental
The certificate dataset is large, both in terms of the overall storage space necessary to accommodate its daily growth, and the client requirements for downloading the incremental changes published daily.
The certificates dataset is continually growing.
From the beginning of the year 2023, the number of new certificates added by Censys global scanning engine to the repository each month is about 500,000,000.
As of summer 2023, the size of the certificates-v2-full
dataset is about ~12TB.
As of the summer 2023, the size of the certificates-v2-incremental
dataset is about 30-60GB.
The files containing the Certificates 2.0 datasets are serialized in Avro binary, which has the data schema stored within it.
To get started with Avro, visit the site documentation.
Comments
0 comments
Please sign in to leave a comment.