Certificate Transparency Log aggregation, parsing, and streaming service written in Elixir
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
Ryan 26dff905b2
Merge pull request #56 from CaliDog/ryan/1.6.0
2 months ago
config Layout project more conventionally and add DNS-only stream 2 months ago
frontend Add heartbeat to the frontend certstream demo 2 months ago
lib Greatly simplify the serialization process and remove chain from the main stream 2 months ago
test Bump deps and fix credo warnings 7 months ago
.dockerignore Fix up frontend and docker config 2 years ago
.gitattributes Fix .gitattributes 1 year ago
.gitignore Refactor frontend and add build tooling 1 year ago
Dockerfile Update Dockerfile 6 months ago
LICENSE Create LICENSE 2 years ago
README.md Simplify internals explanation in readme 6 months ago
mix.exs Bump version 2 months ago
mix.lock Add an explicit ssl version for requests and bump deps 3 months ago

README.md

CertStream-Server

Aggregate and broadcast SSL certs as they're issued live.

Certstream-server is a service written in elixir to aggregate, parse, and stream certificate data from multiple certificate transparency logs. It leverages the amazing power of elixir/erlang to achieve great network throughput and concurrency with very little resource usage.

This is a rewrite of the original version written in python, and is much more efficient than the original and currently ships millions of certificates a day on a single Hetzner dedicated server without issue (~250TB of data every month!).

Getting Started

Getting up and running is pretty easy (especially if you use Heroku, as we include a Dockerfile!).

First install elixir (assuming you're on OSX, otherwise follow instructions for your platform)

$ brew install elixir

Then fetch the dependencies

$ mix deps.get

From there you can run it

$ mix run --no-halt

Alternatively, you can run it in an iex session (so you can interact with it while it's running) for development

$ iex -S mix
Interactive Elixir (1.8.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :observer.start

This will open up an http/websocket server on port 4000 (override this by setting a PORT environment variable).

Usage

Once you have the server runing on some port, you can visit the server to be greeted with the index page.

If you point a websocket client at the root URL (/), it'll give you a data structure that doesn't bundle the DER-encoded certificate.

If you want the certificate data as well, you can point a websocket client to /full-stream and it'll also bundle the DER-encoded certificate along with each certificate.

Internals

The structure of the application is pretty simple. The dataflow basically looks like this:

               CertificateBuffer
                      |
HTTP Watcher \        |        / Websocket Connection Process
HTTP Watcher  - ClientManager -  Websocket Connection Process
HTTP Watcher /                 \ Websocket Connection Process

HTTP Watchers

First we spin up 1 process for every entry in the known CTL list from the CT project, with each of them being responsible for sleeping for 10 seconds and checking to see if the top of the merkle tree has changed. Once a difference has been found, they go out and download the certificate data, parsing it and coercing it to a hashmap structure using the EasySSL library and sending it to the ClientManager.

ClientManager

This agent is responsible for brokering communication between the CT watchers and the currently connected websocket clients. Certificates are broadcast to websocket connection processes through an erlang pobox in order to properly load-shed when a slow client isn't reading certificates fast enough. The ClientManager also sends a copy of every certificate received to the CertificateBuffer.

CertificateBuffer

This agent is responsible for keeping a ring-buffer in memory of the most recently seen 25 certificates, as well as counting the certificates processed by Certstream.

Websocket Connection Process

Under the hood we use the erlang library Cowboy to handle static content serving, the json APIs, and websocket connections. There's nothing too special about them other than they're assigned a paired pobox at the start of every connection.

HTTP Routes

/latest.json - Get the most recent 25 certificates CertStream has seen

/example.json - Get the most recent certificate CertStream has seen

Data Structure

Certificate Updates

{
    "message_type": "certificate_update",
    "data": {
        "update_type": "X509LogEntry",
        "leaf_cert": {
            "subject": {
                "aggregated": "/CN=e-zigarette-liquid-shop.de",
                "C": null,
                "ST": null,
                "L": null,
                "O": null,
                "OU": null,
                "CN": "e-zigarette-liquid-shop.de"
            },
            "extensions": {
                "keyUsage": "Digital Signature, Key Encipherment",
                "extendedKeyUsage": "TLS Web Server Authentication, TLS Web Client Authentication",
                "basicConstraints": "CA:FALSE",
                "subjectKeyIdentifier": "AC:4C:7B:3C:E9:C8:7F:CB:E2:7D:5D:64:F2:25:0C:89:C2:AE:F0:5E",
                "authorityKeyIdentifier": "keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1\n",
                "authorityInfoAccess": "OCSP - URI:http://ocsp.int-x3.letsencrypt.org\nCA Issuers - URI:http://cert.int-x3.letsencrypt.org/\n",
                "subjectAltName": "DNS:e-zigarette-liquid-shop.de, DNS:www.e-zigarette-liquid-shop.de",
                "certificatePolicies": "Policy: 2.23.140.1.2.1\nPolicy: 1.3.6.1.4.1.44947.1.1.1\n  CPS: http://cps.letsencrypt.org\n  User Notice:\n    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/\n"
            },
            "not_before": 1508123861.0,
            "not_after": 1515899861.0,
            "as_der": "::BASE64_DER_CERT::",
            "all_domains": [
                "e-zigarette-liquid-shop.de",
                "www.e-zigarette-liquid-shop.de"
            ]
        },
        "chain": [
            {
                "subject": {
                    "aggregated": "/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3",
                    "C": "US",
                    "ST": null,
                    "L": null,
                    "O": "Let's Encrypt",
                    "OU": null,
                    "CN": "Let's Encrypt Authority X3"
                },
                "extensions": {
                    "basicConstraints": "CA:TRUE, pathlen:0",
                    "keyUsage": "Digital Signature, Certificate Sign, CRL Sign",
                    "authorityInfoAccess": "OCSP - URI:http://isrg.trustid.ocsp.identrust.com\nCA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c\n",
                    "authorityKeyIdentifier": "keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10\n",
                    "certificatePolicies": "Policy: 2.23.140.1.2.1\nPolicy: 1.3.6.1.4.1.44947.1.1.1\n  CPS: http://cps.root-x1.letsencrypt.org\n",
                    "crlDistributionPoints": "\nFull Name:\n  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl\n",
                    "subjectKeyIdentifier": "A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1"
                },
                "not_before": 1458232846.0,
                "not_after": 1615999246.0,
                "as_der": "::BASE64_DER_CERT::"
            },
            {
                "subject": {
                    "aggregated": "/O=Digital Signature Trust Co./CN=DST Root CA X3",
                    "C": null,
                    "ST": null,
                    "L": null,
                    "O": "Digital Signature Trust Co.",
                    "OU": null,
                    "CN": "DST Root CA X3"
                },
                "extensions": {
                    "basicConstraints": "CA:TRUE",
                    "keyUsage": "Certificate Sign, CRL Sign",
                    "subjectKeyIdentifier": "C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10"
                },
                "not_before": 970348339.0,
                "not_after": 1633010475.0,
                "as_der": "::BASE64_DER_CERT::"
            }
        ],
        "cert_index": 19587936,
        "seen": 1508483726.8601687,
        "source": {
            "url": "mammoth.ct.comodo.com",
            "name": "Comodo 'Mammoth' CT log"
        }
    }
}