curl-impersonate: A special compilation of curl that makes it impersonate Chrome & Firefox
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
lwthiker 4e0ca423af Update README.md with up-to-date information 4 days ago
.github/workflows Upload precompiled libcurl impersonate on release 3 weeks ago
chrome Use portable shebangs in wrapper scripts (#79) 4 days ago
firefox Use portable shebangs in wrapper scripts (#79) 4 days ago
tests Handle curl_easy_reset() calls when impersonating (#44) 1 month ago
Dockerfile.template Use portable shebangs in wrapper scripts (#79) 4 days ago
INSTALL.md Add unzip to dependencies for make chrome-build 4 weeks ago
LICENSE Initial commit 4 months ago
Makefile.in Fix broken clang build on macOS 4 weeks ago
README.md Update README.md with up-to-date information 4 days ago
browsers.json Store the supported browsers list as a JSON file 4 days ago
configure.ac Fix build system to work on Red Hat-based systems 1 month ago
generate_dockerfiles.sh Add Alpine Linux build system 4 months ago

README.md

curl-impersonate

Build and test Docker images

A special compilation of curl that makes it impersonate real browsers. It can impersonate the four major browsers: Chrome, Edge, Safari & Firefox. This curl binary is able to perform a TLS handshake that is identical to that of a real browser.

Why?

When you use an HTTP client with a TLS website, it first performs a TLS handshake. The first message of that handshake is called Client Hello. The Client Hello message that curl produces differs drastically from that of a real browser. Compare the following Wireshark capture. Left is a regular curl, right is Firefox. curl-ff-before

Some web services therefore use the TLS handshake to fingerprint which HTTP client is accessing them. Notably, some bot protection platforms use this to identify curl and block it. With the modified curl in this repository, the Client Hello message looks exactly like that of a real browser. This tricks TLS fingerprinters to think that it is a real browser that is accessing them.

How?

The modifications that were needed to make this work:

  • Compiling curl with nss, the TLS library that Firefox uses, instead of OpenSSL. For the Chrome version, compiling with BoringSSL.
  • Modifying the way curl configures various TLS extensions and SSL options.
  • Adding support for new TLS extensions.
  • Running curl with some non-default flags, for example --ciphers, --curves and some -H headers.

The resulting curl looks, from a network perspective, identical to a real browser. Compare: (left is curl-impersonate, right is Firefox):

curl-ff-after

Read the full description in the blog post: part a, part b.

Supported browsers

The following browsers can be impersonated.

Browser Version Build OS Target name Wrapper script
Chrome 99 99.0.4844.51 Windows 10 chrome99 curl_chrome99
Chrome 100 100.0.4896.75 Windows 10 chrome100 curl_chrome100
Chrome 101 101.0.4951.67 Windows 10 chrome101 curl_chrome101
Chrome 99 99.0.4844.73 Android 12 chrome99_android curl_chrome99_android
Edge 99 99.0.1150.30 Windows 10 edge99 curl_edge99
Edge 101 101.0.1210.47 Windows 10 edge101 curl_edge101
Firefox 91 ESR 91.6.0esr Windows 10 ff91esr curl_ff91esr
Firefox 95 95.0.2 Windows 10 ff95 curl_ff95
Firefox 98 98.0 Windows 10 ff98 curl_ff98
Firefox 100 100.0 Windows 10 ff100 curl_ff100
Safari 15.3 16612.4.9.1.8 MacOS Big Sur safari15_3 curl_safari15_3

This list is also available in the browsers.json file.

Basic usage

For each supported browser there is a wrapper script that launches curl-impersonate with all the needed headers and flags. For example:

curl_chrome101 https://www.wikipedia.org

You can add command line flags and they will be passed on to curl. However, some flags change curl's TLS signature which may cause it to be detected.

See Advanced usage for more options.

Installation

There are two versions of curl-impersonate for technical reasons. The chrome version is used to impersonate Chrome, Edge and Safari. The firefox version is used to impersonate Firefox.

Pre-compiled binaries

Pre-compiled binaries for Linux and macOS (Intel) are available at the GitHub releases page. Before you use them you need to install nss (Firefox's TLS library):

  • Ubuntu - sudo apt install libnss3
  • Red Hat/Fedora/CentOS - yum install nss nss-pem
  • macOS - brew install nss

The pre-compiled binaries contain libcurl-impersonate and a statically compiled curl-impersonate for ease of use.

Building from source

See INSTALL.md.

Docker images

Docker images based on Alpine Linux with curl-impersonate compiled and ready to use are available on Docker Hub. The images contain the binary and all the wrapper scripts. Use like the following:

# Firefox version
docker pull lwthiker/curl-impersonate:0.5-ff
docker run --rm lwthiker/curl-impersonate:0.5-ff curl_ff100 https://www.wikipedia.org

# Chrome version
docker pull lwthiker/curl-impersonate:0.5-chrome
docker run --rm lwthiker/curl-impersonate:0.5-chrome curl_chrome101 https://www.wikipedia.org

Distro packages

AUR packages are available to Arch users: curl-impersonate-chrome, curl-impersonate-firefox.

Advanced usage

libcurl-impersonate

libcurl-impersonate.so is libcurl compiled with the same changes as the command line curl-impersonate. It has an additional API function:

CURLcode curl_easy_impersonate(struct Curl_easy *data, const char *target);

You can call it with the target names, e.g. chrome101, and it will internally set all the options and headers that are otherwise set by the wrapper scripts. Specifically it sets:

  • CURLOPT_HTTP_VERSION
  • CURLOPT_SSLVERSION, CURLOPT_SSL_CIPHER_LIST, CURLOPT_SSL_EC_CURVES, CURLOPT_SSL_ENABLE_NPN, CURLOPT_SSL_ENABLE_ALPN
  • CURLOPT_HTTPBASEHEADER, CURLOPT_HTTP2_PSEUDO_HEADERS_ORDER (non-standard HTTP options created for this project).
  • CURLOPT_SSL_ENABLE_ALPS, CURLOPT_SSL_SIG_HASH_ALGS, CURLOPT_SSL_CERT_COMPRESSION, CURLOPT_SSL_ENABLE_TICKET (non-standard TLS options created for this project).

Note that if you call curl_easy_setopt() later with one of the above it will override the options set by curl_easy_impersonate().

Using CURL_IMPERSONATE env var

Experimental: If your application uses libcurl already, you can replace the existing library at runtime with LD_PRELOAD (Linux only). You can then set the CURL_IMPERSONATE env var. For example:

LD_PRELOAD=/path/to/libcurl-impersonate.so CURL_IMPERSONATE=chrome101 my_app

The CURL_IMPERSONATE env var has two effects:

  • curl_easy_impersonate() is called automatically for any new curl handle created by curl_easy_init().
  • curl_easy_impersonate() is called automatically after any curl_easy_reset() call.

This means that all the options needed for impersonation will be automatically set for any curl handle.

Note that the above will NOT WORK for curl itself because the curl tool overrides the TLS settings. Use the wrapper scripts instead.

Contents

This repository contains two main folders:

  • chrome - Scripts and patches for building the Chrome version of curl-impersonate.
  • firefox - Scripts and patches for building the Firefox version of curl-impersonate.

The layout is similar for both. For example, the Firefox directory contains:

  • Dockerfile - Used to build curl-impersonate with all dependencies.
  • curl_ff91esr, curl_ff95, curl_ff98 - Wrapper scripts that launch curl-impersonate with the correct flags.
  • curl-impersonate.patch - The main patch that makes curl use the same TLS extensions as Firefox. Also makes curl compile statically with libnghttp2 and libnss.

Other files of interest:

Contributing

If you'd like to help, please check out the open issues. You can open a pull request with your changes.

This repository contains the build process for curl-impersonate. The actual patches to curl are maintained in a separate repository forked from the upstream curl. The changes are maintained in the impersonate-firefox and impersonate-chrome branches.