Changelog

1.1.0

ProxyHerd can follow the public address of its fingerprint service when a tunnel assigns it at start. A laptop or any machine behind NAT now runs from one Docker Compose file and one command, and a reboot or a restarted tunnel needs nothing from you. Installations with a fixed FINGERPRINT_URL are not affected; upgrading needs no configuration change.

Features

fingerprint: FINGERPRINT_URL_DISCOVERY points ProxyHerd at an endpoint that knows the public address of the fingerprint service, such as the metrics server of a Cloudflare quick tunnel (http://cloudflared:2000/quicktunnel). ProxyHerd reads it at start, again every FINGERPRINT_URL_DISCOVERY_DELAY milliseconds (10 seconds by default) and switches as soon as the address changes; checks wait for the first answer instead of marking proxies offline. FINGERPRINT_URL becomes optional when it is set. See ProxyHerd on a laptop or behind NAT and the environment variables

Documentation

get-started: two Compose files to pick from, Server with a public IP and Laptop or behind NAT; the laptop one has no address to copy. See Get Started
fingerprint: the laptop section is one complete Compose file and one command, and docker compose logs proxyherd | grep "Fingerprint address" shows the address in use

Dependencies and build

build: the dependency ranges of the ProxyHerd bundle match the lock file again (a stale passport-oauth2 range broke the pruned lock file), so the container image installs the locked versions

1.0.1

Maintenance release. Proxies are checked once a minute instead of every few seconds, the website and the user interface use the ProxyHerd colours, and the Get Started page runs ProxyHerd and its fingerprint service from a single Docker Compose file. Upgrading needs no configuration change.

Changes

refresh: proxies and Proxy List pool entries are fingerprinted once a minute by default instead of every second, about ten times fewer requests through each proxy. For free proxy lists, set PROXY_REFRESH_DELAY and FREEPROXY_REFRESH_DELAY back to 1000 to notice dead proxies sooner; see Sizing and proxy traffic
brand: the website and the user interface use the green and amber of the ProxyHerd logo instead of the red of Scrapoxy; the Proxy List connector icon follows
licence: the copyright lines credit Fabien Vauchelles for Scrapoxy up to version 4.17.4 and Rakesh Jana for ProxyHerd; the npm and PyPI metadata name the ProxyHerd maintainer

Fixes

frontend: the sidebar shows the logo drawn for light backgrounds, so the whole wordmark is visible
website: broken GitHub and container registry links, the leftover French locale and other rebranding leftovers

Documentation

get-started: one Docker Compose file runs ProxyHerd and its fingerprint service, with a check through a proxy and a warning about exFAT drives on macOS; see Get Started
fingerprint: deployment patterns for a server with a public IP address, a laptop behind NAT and the Cloudflare Worker, with Worker prerequisites and sizing figures
connectors: paid proxy lists such as Webshare through the Proxy List connector
website: the user interface and Proxy List screenshots show ProxyHerd

Dependencies and build

deps: minor and patch updates of the npm dependencies and a refreshed lock file resolve 274 Dependabot security alerts; the remaining ones need major framework upgrades, tracked in issue #14
docker: both images are built on Node.js 20.13.1
ci: re-running a release skips the npm packages already published, and the end-to-end tests run in three parallel shards

1.0.0

First release of ProxyHerd, the community fork of Scrapoxy 4.17.4 (MIT). Configuration, storage, the REST API and the wire identifiers scrapers rely on (X-Scrapoxy-* headers, the sticky-session cookie, cloud resource names) are unchanged, so a 4.17.4 installation upgrades in place.

BREAKING CHANGE

The hosted fingerprint service no longer exists. Deploy the ProxyHerd fingerprint service and set FINGERPRINT_URL; ProxyHerd refuses to start without it. The npm package, binary, container image (ghcr.io/proxyherd/proxyherd) and Helm charts are named proxyherd. The Python package is proxyherd (settings PROXYHERD_); the scrapoxy module and SCRAPOXY_ settings keep working as deprecated aliases.

Features

fingerprint: self-hosted fingerprint service (packages/fingerprint/app, image ghcr.io/proxyherd/fingerprint, npm @proxyherd/fingerprint) with DB-IP Lite by default, MaxMind GeoLite2 with a licence key, or IP-only mode; weekly database refresh
fingerprint: Cloudflare Worker variant (packages/fingerprint/worker) with no database to manage; prerequisites, request limits and sizing on the fingerprint page
fingerprint: protocol version 3 sends no installation id or traffic counters; opt back in with FINGERPRINT_SEND_STATS=true
charts: fingerprintUrl value wired into the commander and refresh pods
python: proxyherd package with PROXYHERD_* settings and a backward-compatible scrapoxy shim

Changes

brand: new name, logo and documentation site; MIT licence and the original copyright retained, provenance recorded in NOTICE
privacy: Google Analytics, Algolia search, Discord links, sponsorship pages and every affiliate or referral link removed; search is local
contrib: Developer Certificate of Origin replaces the contributor licence agreement
ci: releases publish to GitHub Container Registry, npm and OCI Helm charts; the documentation deploys to GitHub Pages
security: the original author's OAuth credentials were removed from the IDE run configuration
test: free-proxy key expectations updated to the hostname#hash format

4.17.4

This release doesn't include any code changes. Simply update the repository!

BREAKING CHANGE

Please update your Docker image from fabienvauchelles/scrapoxy to scrapoxy/scrapoxy

Repositories

github: change repository from fabienvauchelles/scrapoxy to scrapoxy/scrapoxy
dockerhub: change repository from fabienvauchelles/scrapoxy to scrapoxy/scrapoxy

4.17.3

Features

gcp: add smaller machine types (thanks runa!)

4.17.2

Bug fixes

iproyal: add migration for file storage
website: documentation adjustments

4.17.1

Features

connection: respect keep-alive timeout for MITM connections

Bug Fixes

iproyal: remove API dependency for residential connection

4.17.0

Features

geonode: add Geonode proxies support

Bug fixes

freeproxies: change the way of generating ID hash
misc: minor fixes

4.16.1

Bug fixes

iproyal: update connector to follow the new API

4.16.0

Features

zyte: add support of the Zyte API in Proxy Mode

4.15.2

Bug fixes

master: keep non-scrapoxy request cookies with MITM

4.15.1

Features

zyte: customize URL of the Zyte Smart Proxy Manager

Bug fixes

frontend: fix button alignment introduces with Safari support

4.15.0

Features

tls: add an option to shuffle the ciphersuite to avoid TLS fingerprint

4.14.0

Features

headers: maintain the order and duplication of HTTP headers for MITM mode

Bug fixes

privacy: remove all Scrapoxy headers and cookies from responses sent to the target website.
connectors: prevent unnecessary data sharing with the frontend.
refactor: clean a lot of code

4.13.0

Features

liveproxies: add Live Proxies support

Bug fixes

errors: add missing 557 HTTP status code.

4.12.0

BREAKING CHANGE

Possible breaking change on HTTP status code (see below)

Features

Possible breaking change:
errors: non-authentication errors now return a 557 HTTP status code.

Bug fixes

website: add documentation
idea: add run config for backend and frontend
frontend: fix card button for Safari browser

4.11.2

Bug fixes

gcp: upgrade template image to debian-12

4.11.1

Features

smartproxy: added support for Smartproxy datacenter pay per GB product

4.11.0

Features

smartproxy: added support for Smartproxy
task: enabled naming tasks for better identification

Bug fixes

test: fixed issues with fingerprints in tests.
misc: updated and corrected some documentation and naming conventions

4.10.2

Bug fixes

transport: fix usage of username & password for SOCKS proxy

4.10.1

Features

azure: add support for Spot Instance (thanks to Julien Maitrehenry)
fingerprint: use GET instead of POST method to access fingerprint server (to bypass firewall restrictions)

Bug fixes

azure: handle running deployment to avoid duplicate instances creation

4.10.0

Features

brightdata: add support for Residential and Mobile proxies for BrightData
fingerprint: use GET instead of POST method to access fingerprint server (to bypass firewall restrictions)

Bug fixes

website: misc fixes

4.9.0

Features

netnut: add NetNut connector

Bug fixes

sticky: correct cookie injection for sticky sessions

4.8.1

Bug fixes

ninjasproxy: update the API url

4.8.0

Features

success rate: add success rate metrics to proxies and project

Bug fixes

gcp: fix copy/paste with GCP private key

4.7.2

Bug fixes

freeproxy: include auth info in key generation

4.7.1

Bug fixes

frontend: correct proxy URL parsing
frontend: show proxies error message
transport: set transport type to proxy level

4.7.0

Features

brightdata: add BrightData connector for ISP and DC proxies
backend: add Swagger endpoint for scraper API
frontend: show coupon in marketplace

Bug fixes

misc: many small corrections

4.6.2

Features

frontend: add coupon field in marketplace
proxy-seller: add proxy-seller coupon

4.6.1

Features

master: add HTTPS support to master

4.6.0

Features

proxy-seller: add Proxy-Seller connector

Bug fixes

metrics: include request and response headers size in metrics
website: add stickies session documentation

4.5.2

Bug fixes

aws: use the correct architecture to install and create instances
log: don't throw error if there is no configuration file at startup

4.5.1

Features

frontend: auto-generate a free name at credential and connector creation
website: add scrapy-impersonate documentation to scrapy

Bug fixes

aws: add new instance type like t4g.nano
freeproxy: allow bracket in URLs
transport: add missing timeout on connections
log: add more log at startup
website: reorganize the documentation tree

4.5.0

Features

freeproxies: collect freeproxies from an URL at specified interval
proxies: make timeout configurable to detect online/offline status of proxies
proxies: make timeout configurable for removing offline proxies
freeproxies: add a configurable timeout to detect online/offine status of freeproxies
freeproxies: add a configurable timeout for removing offline freeproxies
api: add an API endpoint to manage freeproxies and sources

Refactor

connectors: merge all connectors into 3 fixed directory (common, frontend, backend)
storage: merge all storages into 2 fixed directory (common, backend)
auth: merge all authentications into 2 fixed directory (common, backend)
website: reorganize website and documentation structure for connectors
website: manage plural text in toasts

Bug fixes

linter: fix linter for HTML and CSS
frontend: correct pagination component
socks: allow socks in connect tunnel mode

4.4.2

Bug fixes

proxy-cheap: fix website url

4.4.1

Bug fixes

github: add script to build and publish docker image and npm package

4.4.0

Features

proxy-cheap: add Proxy-Cheap connectors
hypeproxy: add HypeProxy connector

4.3.0

Features

nimble: add Nimble connector with Nimble IP

Bug fixes

refactor: rename cloud to datacenter
frontend: fix clipboard issue with old browser
misc: correct some naming, documentation
storage: allow empty file for file storage

4.2.3

Bug fixes

commander: log commander http error on console
frontend: more stable clipboard copy

4.2.2

Bug fixes

standalone: force standalone mode when no option is selected

4.2.1

Features

fingerprint: add useragent for all http calls (fingerprint and api)
frontend: split token in username and password in project settings

Bug fixes

package.json: add missing socks package

4.2.0

Features

freeproxies: support HTTPS proxies
freeproxies: support SOCKS4/SOCKS5 proxies

Bug fixes

protocol: Accept only HTTP and HTTPS requests from master

4.1.6

Bug fixes

frontend: correct pagination component
doc: replace local search with algolia
doc: misc documentation fixes

4.1.5

Bug fixes

auth: secure flag for the authentication cookie is now optional
doc: usage of username:password is better explained

4.1.4

Bug fixes

fingerprint: better fingerprint

4.1.3

Bug fixes

doc: correct link to picture
backend-app: add version to package-lock.json in build

4.1.2

Bug fixes

doc: add warning for xproxy installation
doc: improve documentation and website
eslint: merge scrapoxy-linter repository into scrapoxy repository

4.1.1

Bug fixes

iproyal: add migration layer for file storage

4.1.0

Features

iproyal: add support for Royal Residential proxies
marketplace: add marketplace menu with static proxies, dynamic proxies, hardware, datacenter providers and other

Bug fixes

transport: factorize code for residential proxies

4.0.1

Bug fixes

master: handle writeHead error after aborted connection

4.0.0

Scrapoxy 4 represents a complete rewrite, now developed in Typescript. It harnesses the capabilities of NestJS and Angular modules to enhance its functionality and efficiency.

BREAKING CHANGE

Scrapoxy 4 is not compatible with previous versions.

Features

datacenter providers: it supports main datacenter providers like AWS, Azure, GCP and more;
proxy services: it is compatible with proxy provider like Zyte, IPRoyal, Rayobyte and more;
hardware materials: it orchestrates 4G proxy farms hardware types, like Proxidize or XProxy.io;
free proxy lists: it handles manual lists of HTTP/HTTPS proxies;
auto-scale: it autoscale up and down proxies to optimize costs;
auto-rotate: it rotates proxies to avoid bans;
sticky sessions: it supports sticky sessions to keep the same IP address for a scraping session or a browser
traffic interception: it intercepts HTTP and HTTPS traffic to inject consistent headers or cookies;
monitoring: it monitors proxies and provides many statistics.

3.1.1

Bug fixes

master: use correctly writeEnd in socket and request (thanks to Ben Lavalley)

3.1.0

Features

mitm: decrypt & encrypt SSL requests to add headers (like x-cache-proxyname). Compatible with HTTPS requests in PhantomJS.
domains: manage whitelist or blacklist for urls (idea from Jonathan Wiklund)
docs: add ami-485fbba5 with type t2.micro

Bug fixes

logs: correct export path of logs
docs: correct documentation
ssl: add servername in the TLS connect (bug with HELLO)
pinger: use reject instead of throw error (crash program). Thanks to Anis Gandoura.

3.0.1

Features

digitalocean: support Digital Ocean tags on Droplets. Thanks to Ben Lavalley.

Bug fixes

digitalocean: use new image size (s-1vcpu-1gb instead of 512mb)

3.0.0

BREAKING CHANGE

The configuration of providers changes.

Features

providers: uses multiple providers at a time
awsec2: provider removes instances in batch every second (and no longer makes thousands of queries)
ovhcloud: provider creates instances in batch (new API route used)

Bug fixes

maxRunningInstances: remove blocking parameters maxRunningInstances

2.4.3

Bug fixes

node: change minimum version of Node.js to 8
dependencies: upgrade dependencies to latest version

2.4.2

Bug fixes

useragent: set useragent at instance creation, not at startup
instance: force crashed instance to be removed

2.4.1

Bug fixes

instance: correctly remove instance when instance is removed. Thanks to Étienne Corbillé.

2.4.0

Features

provider: add VScale.io provider. Thanks to Hotrush.

Bug fixes

proxy: use a valid startup script for init.d. Thanks to Hotrush.
useragent: change useragents with a fresh list for 2017

2.3.10

Features

docs: add ami-06220275 with type t2.nano

Bug fixes

instance: remove listeners on instance alive status on instance removal. Thanks to Étienne Corbillé.

2.3.9

Features

digitalocean: update Digital Ocean documentation
digitalocean: view only instances from selected region
instances: remove random instances instead of the last ones
pm2: add kill_timeout option for PM2 (thanks to cp2587)

Bug fixes

digitalocean: limit the number of created instances at each API request
digitalocean: don't remove locked instances

2.3.8

Features

docker: create the Docker image scrapoxy/scrapoxy

Bug fixes

template: limit max instances to 2

2.3.7

Features

connect: scrapoxy accepts now full HTTPS CONNECT method. It is useful for browser like PhantomJS. Thanks to Anis Gandoura

2.3.6

Bug fixes

template: replace old AWS AMI by ami-c74d0db4

2.3.5

BREAKING CHANGE

Please rebuild instance image.

Features

instance: change Node.js version to 6.x
ping: use an HTTP ping instead a TCP ping.

2.3.4

Features

stats: monitor stop count history
stats: add 3 more scales: 5m, 10m and 1h
logs: normalize logs and add more informations
scaling: pop a message when maximum number of instances is reached in a provider
scaling: add quick scaling buttons
docs: explain why Scrapoxy doesn't accept CONNECT mode
docs: explain how User Agent is overwritten

Bug fixes

dependencies: upgrade dependencies
ovh: monitor DELETED status
docs: add example to test scrapoxy with credentials
commander: manage twice instance remove

2.3.3

Bug fixes

master: sanitize bad request headers
proxy: catch all socket errors in the proxy instance

2.3.2

Bug fixes

docs: fallback to markdown for README (because npmjs doesn't like retext)

2.3.1

Features

docs: add tutorials for Scrapy and Node.js

Bug fixes

digitalocean: convert Droplet id to string

2.3.0

Features

digitalocean: add support for DigitalOcean provider

2.2.1

Misc

config: rename my-config.json to conf.json
doc: migrate documentation to ReadTheDocs
doc: link to the new website Scrapoxy.io

2.2.0

BREAKING CHANGE

node minimum version is now 4.2.1, to support JS class

Features

all: migrate core and gui to ES6, with all best practices
api: replace Express by Koa

Bug fixes

test: correct core e2e test

2.1.2

Bug fixes

gui: correct token encoding for GUI

2.1.1

Bug fixes

main: add message when all instances are stopped (at end)
doc: correct misc stuff in doc

2.1.0

Features

ovh: add OVH provider with documentation
security: add basic auth to Scrapoxy (RFC2617)
stats: add flow stats
stats: add scale for stats (1m/1h/1d)
stats: store stats on server
stats: add globals stats
doc: split of the documentation in 3 parts: quick start, standard usage and advanced usage
doc: add tutorials for AWS / EC2
gui: add a scaling popup instead of direct edit (with integrity check)
gui: add update popup when the status of an instance changes.
gui: add error popup when GUI cannot retrieve data
logs: write logs to disk
instance: add cloud name
instance: show instance IP
instance: always terminate an instance when stopping (prefer terminate instead of stop/start)
test: allow more than 8 requests (max 1000)
ec2: force to terminate/recreate instance instead of stop/restart

Bug fixes

gui: emit event when scaling is changed by engine (before, event was triggered by GUI)
stability: correct a lot of behavior to prevent instance cycling
ec2: use status name instead of status code

2.0.1

Features

test: specify the count of requests with the test command
test: count the requests by IP in the test command
doc: add GUI documentation
doc: add API documentation
doc: explain awake/asleep mode in user manual
log: add human readable message at startup

2.0.0

BREAKING CHANGE

API routes are prefixed with /api

Features

gui: add GUI to control Scrapoxy
gui: add statistics to the GUI (count of requests / minute, average delay of requests / minute)
doc: add doc about HTTP headers

Changelog ​

1.1.0

Features

Documentation

Dependencies and build

1.0.1

Changes

Fixes

Documentation

Dependencies and build

1.0.0

Features

Changes

4.17.4

Repositories

4.17.3

Features

4.17.2

Bug fixes

4.17.1

Features

Bug Fixes

4.17.0

Features

Bug fixes

4.16.1

Bug fixes

4.16.0

Features

4.15.2

Bug fixes

4.15.1

Features

Bug fixes

4.15.0

Features

4.14.0

Features

Bug fixes

4.13.0

Features

Bug fixes

4.12.0

Features

Bug fixes

4.11.2

Bug fixes

4.11.1

Features

4.11.0

Features

Bug fixes

4.10.2

Bug fixes

4.10.1

Features

Bug fixes

4.10.0

Features

Bug fixes

4.9.0

Features

Bug fixes

4.8.1

Bug fixes

4.8.0

Features

Bug fixes

4.7.2

Bug fixes

4.7.1

Bug fixes

4.7.0

Features

Bug fixes

4.6.2

Features

4.6.1

Features

4.6.0

Changelog