diff --git a/IMPLEMENTATION_PRIORITY.md b/IMPLEMENTATION_PRIORITY.md deleted file mode 100644 index b176041..0000000 --- a/IMPLEMENTATION_PRIORITY.md +++ /dev/null @@ -1,272 +0,0 @@ -# Implementation Priority & Plan - -This document outlines the implementation plan for javascript-proxy-headers. - -## Architecture Overview - -``` -┌─────────────────────────────────────────────────────────────┐ -│ javascript-proxy-headers │ -├─────────────────────────────────────────────────────────────┤ -│ Library Wrappers (high-level, user-friendly APIs) │ -│ ┌─────────┐ ┌───────────┐ ┌─────┐ ┌────────┐ ┌──────────┐ │ -│ │ axios │ │node-fetch │ │ got │ │ undici │ │superagent│ │ -│ └────┬────┘ └─────┬─────┘ └──┬──┘ └────┬───┘ └────┬─────┘ │ -│ │ │ │ │ │ │ -│ └────────────┴────┬─────┴─────────┴──────────┘ │ -│ │ │ -│ ┌──────────────────────▼──────────────────────────────┐ │ -│ │ ProxyHeadersAgent (core) │ │ -│ │ - Manages CONNECT request with custom headers │ │ -│ │ - Captures CONNECT response headers │ │ -│ │ - Compatible with http.Agent interface │ │ -│ └─────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────┘ -``` - -## Implementation Phases - -### Phase 1: Core Agent (Week 1) -**Goal:** Create `ProxyHeadersAgent` - the foundation for all library wrappers - -**Files:** -- `lib/core/proxy-headers-agent.js` - Main agent implementation -- `lib/core/connect-parser.js` - HTTP CONNECT response parser -- `lib/core/utils.js` - Shared utilities - -**Features:** -- [x] Parse proxy URL (host, port, auth) -- [x] Send custom headers in CONNECT request -- [x] Parse CONNECT response headers -- [x] Support Basic auth -- [x] Expose headers via callback or property -- [x] Compatible with Node.js `http.Agent` interface - -**API:** -```javascript -import { ProxyHeadersAgent } from 'javascript-proxy-headers'; - -const agent = new ProxyHeadersAgent('http://proxy:8080', { - proxyHeaders: { - 'X-ProxyMesh-Country': 'US' - } -}); - -// After request, headers available on agent -console.log(agent.lastProxyHeaders); // Map of response headers -``` - ---- - -### Phase 2: Undici Extension (Week 1-2) -**Goal:** Native undici support with cleanest API - -**Files:** -- `lib/undici-proxy.js` - -**Features:** -- [ ] Custom `ProxyAgent` subclass or wrapper -- [ ] Support `proxyHeaders` option -- [ ] Expose CONNECT response headers on response -- [ ] Support async header callback - -**API:** -```javascript -import { request } from 'javascript-proxy-headers/undici'; - -const { statusCode, headers, body, proxyHeaders } = await request(url, { - proxy: 'http://proxy:8080', - proxyHeaders: { 'X-ProxyMesh-Country': 'US' } -}); - -console.log(proxyHeaders.get('x-proxymesh-ip')); -``` - ---- - -### Phase 3: node-fetch Extension (Week 2) -**Goal:** Drop-in fetch wrapper with proxy header support - -**Files:** -- `lib/node-fetch-proxy.js` - -**Features:** -- [ ] Wrapper function around fetch -- [ ] Uses ProxyHeadersAgent internally -- [ ] Extends Response to include proxyHeaders - -**API:** -```javascript -import { proxyFetch } from 'javascript-proxy-headers/node-fetch'; - -const response = await proxyFetch(url, { - proxy: 'http://proxy:8080', - proxyHeaders: { 'X-ProxyMesh-Country': 'US' } -}); - -console.log(response.proxyHeaders.get('x-proxymesh-ip')); -``` - ---- - -### Phase 4: Axios Extension (Week 2-3) -**Goal:** Axios adapter with proxy header support - -**Files:** -- `lib/axios-proxy.js` - -**Features:** -- [ ] Custom axios instance factory -- [ ] Request interceptor for agent setup -- [ ] Response interceptor to merge headers -- [ ] Support both sync and async access - -**API:** -```javascript -import { createProxyAxios } from 'javascript-proxy-headers/axios'; - -const client = createProxyAxios({ - proxy: 'http://proxy:8080', - proxyHeaders: { 'X-ProxyMesh-Country': 'US' } -}); - -const response = await client.get(url); -console.log(response.headers['x-proxymesh-ip']); // Merged into response -``` - ---- - -### Phase 5: Got Extension (Week 3) -**Goal:** Got instance with proxy header support - -**Files:** -- `lib/got-proxy.js` - -**Features:** -- [ ] Use got.extend() for clean integration -- [ ] Leverage got's hook system -- [ ] Merge proxy headers into response - -**API:** -```javascript -import { createProxyGot } from 'javascript-proxy-headers/got'; - -const client = createProxyGot({ - proxy: 'http://proxy:8080', - proxyHeaders: { 'X-ProxyMesh-Country': 'US' } -}); - -const response = await client(url); -console.log(response.headers['x-proxymesh-ip']); -``` - ---- - -### Phase 6: SuperAgent Extension (Week 3-4) -**Goal:** SuperAgent plugin for proxy headers - -**Files:** -- `lib/superagent-proxy.js` - -**Features:** -- [ ] Plugin function for superagent -- [ ] Chainable API - -**API:** -```javascript -import superagent from 'superagent'; -import { proxyPlugin } from 'javascript-proxy-headers/superagent'; - -const response = await superagent - .get(url) - .use(proxyPlugin({ - proxy: 'http://proxy:8080', - proxyHeaders: { 'X-ProxyMesh-Country': 'US' } - })); - -console.log(response.headers['x-proxymesh-ip']); -``` - ---- - -## Technical Challenges - -### 1. CONNECT Response Timing -The CONNECT response is received before the TLS handshake. Need to: -- Buffer the response -- Parse headers before upgrading to TLS -- Store headers for later retrieval - -### 2. Connection Pooling -When using keep-alive/connection pooling: -- Each new CONNECT will have different proxy headers -- Need to track which response corresponds to which request -- Consider disabling pooling for simplicity initially - -### 3. Async Header Access -Proxy headers are available before the target response: -- Option 1: Callback when CONNECT completes -- Option 2: Property on agent, accessed after request -- Option 3: Merge into final response headers - -### 4. TypeScript Support -- Provide `.d.ts` files for all modules -- Extend existing library types properly - ---- - -## File Structure - -``` -javascript-proxy-headers/ -├── package.json -├── README.md -├── LIBRARY_RESEARCH.md -├── IMPLEMENTATION_PRIORITY.md -├── index.js # Main entry point -├── lib/ -│ ├── core/ -│ │ ├── proxy-headers-agent.js -│ │ ├── connect-parser.js -│ │ └── utils.js -│ ├── axios-proxy.js -│ ├── node-fetch-proxy.js -│ ├── got-proxy.js -│ ├── undici-proxy.js -│ └── superagent-proxy.js -├── test/ -│ ├── run_tests.js -│ ├── core.test.js -│ ├── axios.test.js -│ ├── node-fetch.test.js -│ ├── got.test.js -│ ├── undici.test.js -│ └── superagent.test.js -└── types/ - └── index.d.ts -``` - ---- - -## Success Criteria - -1. **Core Agent**: Can send headers in CONNECT, receive headers from response -2. **All Libraries**: Consistent API pattern across all wrappers -3. **Tests**: Each library tested with real proxy -4. **Documentation**: Clear examples for each library -5. **TypeScript**: Full type definitions - ---- - -## Risks & Mitigations - -| Risk | Mitigation | -|------|------------| -| Node.js internals change | Pin minimum Node version, test across versions | -| Library API changes | Use peer dependencies with version ranges | -| Connection pooling complexity | Start without pooling, add later | -| Performance overhead | Benchmark against direct connections | - ---- - -*Plan created: February 28, 2026* diff --git a/LIBRARY_RESEARCH.md b/LIBRARY_RESEARCH.md deleted file mode 100644 index 6c0372c..0000000 --- a/LIBRARY_RESEARCH.md +++ /dev/null @@ -1,413 +0,0 @@ -# JavaScript Library Proxy Header Support Research - -This document analyzes JavaScript/Node.js HTTP and web scraping libraries for their proxy support and potential for javascript-proxy-headers extension modules. - -## Executive Summary - -After reviewing 15+ JavaScript libraries, **none of them natively support sending custom headers to proxies or receiving proxy response headers** during HTTPS CONNECT tunneling. This is because: - -1. Most libraries rely on `https-proxy-agent` or similar packages that don't expose CONNECT headers -2. Browser automation tools (Puppeteer, Playwright) delegate proxy handling to the browser -3. The underlying Node.js `http.Agent` doesn't provide hooks for CONNECT headers -4. Undici has the most promising architecture but still lacks easy access to CONNECT response headers - ---- - -## The Problem - -When making HTTPS requests through an HTTP proxy, the connection flow is: - -``` -Client → HTTP CONNECT request → Proxy → Target Server - (with proxy headers) ↓ -Client ← CONNECT response ←────────┘ - (with proxy response headers) -Client ← TLS Tunnel → Proxy ← TLS → Target Server -``` - -The problem is twofold: -1. **Sending headers to proxy**: Need to add custom headers (e.g., `X-ProxyMesh-Country`) to the CONNECT request -2. **Receiving headers from proxy**: Need to read custom headers (e.g., `X-ProxyMesh-IP`) from the CONNECT response - ---- - -## Library Analysis - -### 1. axios (107k stars) ⭐ HIGHEST PRIORITY -**GitHub:** https://github.com/axios/axios -**Weekly Downloads:** ~48 million -**Description:** Promise-based HTTP client for the browser and Node.js - -**Proxy Support:** -- Browser: Uses browser's proxy settings -- Node.js: Requires external agent (typically `https-proxy-agent`) -- No built-in proxy configuration - -**Custom Proxy Headers:** ❌ No -- Relies entirely on the provided agent for proxy handling -- No mechanism to inject headers into CONNECT request -- No access to CONNECT response headers - -**Extension Feasibility:** ✅ HIGH -- Can create custom agent wrapper that intercepts CONNECT -- Use axios interceptors to merge proxy response headers -- Clean API possible: `axios.get(url, { proxyHeaders: {...} })` - -**Technical Approach:** -```javascript -// Create custom ProxyAgent that accepts headers -class ProxyHeaderAgent extends Agent { - constructor(proxy, proxyHeaders) { - // Intercept CONNECT, add headers, capture response - } -} -``` - ---- - -### 2. node-fetch (8.8k stars) ⭐ HIGH PRIORITY -**GitHub:** https://github.com/node-fetch/node-fetch -**Weekly Downloads:** ~38 million -**Description:** A light-weight module that brings Fetch API to Node.js - -**Proxy Support:** -- Requires external agent (`https-proxy-agent`, `node-fetch-native`, etc.) -- Agent passed via `agent` option - -**Custom Proxy Headers:** ❌ No -- Depends entirely on the agent implementation -- `https-proxy-agent` doesn't support custom CONNECT headers -- No access to CONNECT response - -**Extension Feasibility:** ✅ HIGH -- Create custom agent that wraps proxy connection -- Can expose headers via response object extension -- Simple wrapper function possible - -**Technical Approach:** -```javascript -import { proxyFetch } from 'javascript-proxy-headers/node-fetch'; - -const response = await proxyFetch(url, { - proxy: 'http://proxy:8080', - proxyHeaders: { 'X-ProxyMesh-Country': 'US' } -}); -console.log(response.proxyHeaders.get('X-ProxyMesh-IP')); -``` - ---- - -### 3. got (14.3k stars) ⭐ HIGH PRIORITY -**GitHub:** https://github.com/sindresorhus/got -**Weekly Downloads:** ~13 million -**Description:** Human-friendly and powerful HTTP request library - -**Proxy Support:** -- Agent-based (`https-proxy-agent`) -- Has hook system for request/response lifecycle -- Good architecture for extension - -**Custom Proxy Headers:** ❌ No -- Agent handles CONNECT internally -- Hooks don't have access to proxy layer - -**Extension Feasibility:** ✅ HIGH -- Hook system allows clean integration -- Can create `got.extend()` instance with proxy header support -- Response hooks can merge proxy headers - -**Technical Approach:** -```javascript -import { createProxyGot } from 'javascript-proxy-headers/got'; - -const proxyGot = createProxyGot({ - proxy: 'http://proxy:8080', - proxyHeaders: { 'X-ProxyMesh-Country': 'US' } -}); - -const response = await proxyGot(url); -console.log(response.headers['x-proxymesh-ip']); -``` - ---- - -### 4. undici (4.4k stars) ⭐ MEDIUM-HIGH PRIORITY -**GitHub:** https://github.com/nodejs/undici -**Weekly Downloads:** ~17 million -**Description:** HTTP/1.1 client from Node.js core team (powers native `fetch`) - -**Proxy Support:** -- Built-in `ProxyAgent` class -- Most flexible architecture of all libraries -- Supports `requestTls` and other low-level options - -**Custom Proxy Headers:** ⚠️ Partial -- `ProxyAgent` accepts `headers` option but unclear if sent during CONNECT -- No obvious way to receive CONNECT response headers -- Internal architecture may support this with modifications - -**Extension Feasibility:** ✅ HIGH -- Best architecture for extension -- `ProxyAgent` is well-documented and extensible -- May be able to subclass `ProxyAgent` directly - -**Technical Approach:** -```javascript -import { ProxyHeaderAgent } from 'javascript-proxy-headers/undici'; - -const agent = new ProxyHeaderAgent('http://proxy:8080', { - proxyHeaders: { 'X-ProxyMesh-Country': 'US' }, - onProxyConnect: (headers) => { - console.log('Proxy IP:', headers['x-proxymesh-ip']); - } -}); -``` - ---- - -### 5. superagent (16.6k stars) -**GitHub:** https://github.com/ladjs/superagent -**Weekly Downloads:** ~7 million -**Description:** Ajax for Node.js and browsers - -**Proxy Support:** -- Agent-based (`https-proxy-agent`) -- Plugin architecture available - -**Custom Proxy Headers:** ❌ No -- Same limitations as other agent-based libraries - -**Extension Feasibility:** ⚠️ MEDIUM -- Plugin system could work -- Less modern than other options -- Lower priority due to declining usage - ---- - -### 6. needle (1.6k stars) -**GitHub:** https://github.com/tomas/needle -**Weekly Downloads:** ~7 million -**Description:** Nimble, streamable HTTP client - -**Proxy Support:** -- Built-in `proxy` option -- Uses internal proxy handling - -**Custom Proxy Headers:** ❌ No -- Internal proxy implementation doesn't expose headers -- Would require patching internals - -**Extension Feasibility:** ⚠️ MEDIUM -- Could potentially patch the proxy handling code -- Not as clean as agent-based solutions -- Lower priority - ---- - -### 7. https-proxy-agent (4.7k stars) ⭐ KEY DEPENDENCY -**GitHub:** https://github.com/TooTallNate/proxy-agents -**Weekly Downloads:** ~52 million -**Description:** HTTP(s) proxy Agent for Node.js - -**Custom Proxy Headers:** ❌ No -- Does not support custom CONNECT headers -- Does not expose CONNECT response headers -- This is THE library most others depend on - -**Extension Feasibility:** ✅ HIGH - CORE TARGET -- **This is where we should focus** -- Creating an enhanced `https-proxy-agent` that supports headers would benefit ALL libraries -- Could submit upstream PR or create fork/wrapper - -**Technical Approach:** -```javascript -// Core implementation - enhanced proxy agent -class ProxyHeadersAgent extends Agent { - constructor(proxy, options = {}) { - this.proxyHeaders = options.proxyHeaders || {}; - this.onProxyResponse = options.onProxyResponse; - } - - connect(req, options) { - // 1. Create socket to proxy - // 2. Send CONNECT with custom headers - // 3. Parse response, call onProxyResponse - // 4. Return socket for TLS upgrade - } -} -``` - ---- - -### 8. tunnel (2.3k stars) -**GitHub:** https://github.com/koichik/node-tunnel -**Weekly Downloads:** ~24 million -**Description:** Node.js HTTP/HTTPS tunneling proxies - -**Custom Proxy Headers:** ⚠️ Partial -- Supports `proxyHeaders` option! -- However, does NOT expose CONNECT response headers - -**Extension Feasibility:** ✅ HIGH -- Already supports sending headers -- Just need to add response header capture -- Could be base for our implementation - ---- - -### 9. puppeteer (90.2k stars) -**GitHub:** https://github.com/puppeteer/puppeteer -**Weekly Downloads:** ~3.5 million -**Description:** Headless Chrome Node.js API - -**Proxy Support:** -- `--proxy-server` Chrome flag -- `page.authenticate()` for proxy auth - -**Custom Proxy Headers:** ❌ No -- Browser handles CONNECT internally -- No programmatic access to proxy headers - -**Extension Feasibility:** ❌ LOW -- Would require Chrome DevTools Protocol extensions that don't exist -- Not practical for this project - ---- - -### 10. playwright (69.5k stars) -**GitHub:** https://github.com/microsoft/playwright -**Weekly Downloads:** ~3 million -**Description:** Browser automation library - -**Proxy Support:** -- Browser launch option -- Built-in auth support - -**Custom Proxy Headers:** ❌ No -- Same browser limitations as Puppeteer - -**Extension Feasibility:** ❌ LOW -- Browser-based, not practical - ---- - -### 11. cheerio (28.9k stars) -**GitHub:** https://github.com/cheeriojs/cheerio -**Description:** Fast, flexible HTML parser - -**Proxy Support:** -- N/A - parsing library only -- Depends on external HTTP client - -**Extension Feasibility:** N/A -- Not an HTTP library - ---- - -### 12. crawlee (17.4k stars) -**GitHub:** https://github.com/apify/crawlee -**Description:** Web scraping and browser automation - -**Proxy Support:** -- Built-in proxy rotation -- Uses got/undici for HTTP, Playwright for browser - -**Custom Proxy Headers:** ❌ No -- Depends on underlying libraries - -**Extension Feasibility:** ⚠️ MEDIUM -- Would benefit from our got/undici extensions -- Could create integration middleware - ---- - -### 13. scrape-it (1.1k stars) -**GitHub:** https://github.com/IonicaBizau/scrape-it -**Description:** Node.js scraper - -**Proxy Support:** -- Uses `got` under the hood - -**Extension Feasibility:** ⚠️ MEDIUM -- Would benefit from got extension - ---- - -### 14. x-ray (6k stars) -**GitHub:** https://github.com/matthewmueller/x-ray -**Description:** Web scraper with composable API - -**Proxy Support:** -- Pluggable drivers -- Default uses `request` (deprecated) - -**Extension Feasibility:** ⚠️ LOW -- Uses deprecated library -- Would need driver rewrite - ---- - -### 15. ky (12.6k stars) -**GitHub:** https://github.com/sindresorhus/ky -**Description:** Tiny HTTP client based on Fetch - -**Proxy Support:** -- Browser: browser proxy -- Node.js: needs fetch polyfill with agent - -**Extension Feasibility:** ⚠️ MEDIUM -- Would inherit from our fetch/undici work - ---- - -## Summary Table - -| Library | Stars | Downloads/wk | Proxy Headers | Extension Priority | -|---------|-------|--------------|---------------|-------------------| -| axios | 107k | 48M | ❌ | **HIGH** | -| https-proxy-agent | 4.7k | 52M | ❌ | **HIGH (CORE)** | -| node-fetch | 8.8k | 38M | ❌ | **HIGH** | -| tunnel | 2.3k | 24M | ⚠️ Send only | **HIGH** | -| undici | 4.4k | 17M | ⚠️ Partial | **HIGH** | -| got | 14.3k | 13M | ❌ | **HIGH** | -| superagent | 16.6k | 7M | ❌ | MEDIUM | -| needle | 1.6k | 7M | ❌ | LOW | -| puppeteer | 90.2k | 3.5M | ❌ | LOW (browser) | -| playwright | 69.5k | 3M | ❌ | LOW (browser) | -| crawlee | 17.4k | - | ❌ | MEDIUM | -| ky | 12.6k | - | ❌ | LOW | - ---- - -## Key Insights - -### 1. Core Dependency Strategy -The `https-proxy-agent` and `tunnel` packages are used by most other libraries. Creating an enhanced version that supports both **sending and receiving** proxy headers would benefit the entire ecosystem. - -### 2. Undici is the Future -As Node.js's official HTTP client, undici has the cleanest architecture. It already supports some proxy header features and is the best long-term investment. - -### 3. Browser Libraries Not Viable -Puppeteer, Playwright, and browser-based tools cannot support custom proxy headers because the browser handles CONNECT internally. - -### 4. Wrapper Functions vs. Agent Replacement -Two approaches: -- **Agent-level**: Create enhanced proxy agents that work with all libraries -- **Library-level**: Create wrapper functions for each library - -We should do both: core agent implementation + library-specific wrappers for better DX. - ---- - -## Recommended Implementation Order - -1. **Core: ProxyHeadersAgent** - Enhanced proxy agent that supports send + receive -2. **undici extension** - Most modern, cleanest architecture -3. **node-fetch extension** - High usage, simple API -4. **axios extension** - Highest usage -5. **got extension** - Good architecture, hook support -6. **superagent extension** - Lower priority - ---- - -*Research conducted: February 28, 2026* diff --git a/MARKETING_PLAN.md b/MARKETING_PLAN.md deleted file mode 100644 index 00cd3f4..0000000 --- a/MARKETING_PLAN.md +++ /dev/null @@ -1,452 +0,0 @@ -# Developer Marketing Plan: javascript-proxy-headers - -This document outlines a comprehensive marketing strategy to promote javascript-proxy-headers within the JavaScript/Node.js developer community. - -## Executive Summary - -**Goal:** Establish javascript-proxy-headers as the go-to solution for proxy header handling in JavaScript HTTP libraries. - -**Target Audience:** -- Developers building web scrapers -- Developers using proxy services (especially ProxyMesh customers) -- Library maintainers of HTTP clients -- DevOps engineers managing proxy infrastructure - -**Key Messages:** -1. First library to solve proxy header send/receive for JavaScript -2. Supports all major HTTP libraries (axios, got, undici, node-fetch, superagent) -3. Same proven approach as python-proxy-headers - ---- - -## Part 1: Package Registry Publishing - -### 1.1 npm Publishing - -**Status:** Ready to publish - -**Steps:** -1. Create npm account or use existing ProxyMesh account -2. Run `npm login` with credentials -3. Verify package.json metadata is complete -4. Run `npm publish --dry-run` to check package contents -5. Run `npm publish` to publish -6. Verify at https://www.npmjs.com/package/javascript-proxy-headers - -**Post-publish:** -- Add npm badge to README -- Set up npm provenance (GitHub Actions workflow) -- Consider scoped package: `@proxymesh/proxy-headers` - -### 1.2 GitHub Packages - -**Steps:** -1. Add GitHub Packages publish workflow to `.github/workflows/` -2. Configure package.json with GitHub registry -3. Publish on release/tag - -### 1.3 JSR (Deno Registry) - -**Steps:** -1. Add `jsr.json` configuration file -2. Ensure ESM compatibility -3. Publish via `npx jsr publish` -4. List at https://jsr.io/ - ---- - -## Part 2: Documentation Sites - -### 2.1 ReadTheDocs - -**Status:** Configuration complete, needs import - -**Steps:** -1. Go to https://readthedocs.org/dashboard/ -2. Click "Import a Project" -3. Connect GitHub account if not already -4. Select `proxymeshai/javascript-proxy-headers` -5. Build will auto-trigger -6. Configure custom domain if desired: `javascript-proxy-headers.readthedocs.io` - -### 2.2 Update README Badges - -Add badges for: -- npm version -- npm downloads -- ReadTheDocs build status -- License -- Node.js version - ---- - -## Part 3: Awesome Lists (PRs to Submit) - -### 3.1 awesome-nodejs -**Repository:** https://github.com/sindresorhus/awesome-nodejs -**Stars:** 65,103 -**Section:** HTTP (currently lists got, undici, node-fetch, axios, superagent) - -**Action:** Submit PR to add to HTTP section - -**Proposed addition:** -```markdown -- [javascript-proxy-headers](https://github.com/proxymeshai/javascript-proxy-headers) - Send and receive custom proxy headers during HTTPS CONNECT tunneling. Supports axios, got, undici, node-fetch, superagent. -``` - -**PR Steps:** -1. Fork repository -2. Edit `readme.md` -3. Add entry in alphabetical order within HTTP section -4. Submit PR with title: "Add javascript-proxy-headers to HTTP section" -5. Reference that it extends libraries already in the list - ---- - -### 3.2 awesome-web-scraping (JavaScript) -**Repository:** https://github.com/lorien/awesome-web-scraping -**Stars:** 7,787 -**File:** `javascript.md` -**Section:** Network OR new section "Proxy Server" - -**Action:** Submit PR to add to Network or Proxy Server section - -**Proposed addition to Network section:** -```markdown -* [javascript-proxy-headers](https://github.com/proxymeshai/javascript-proxy-headers) - Send and receive custom headers during HTTPS proxy CONNECT tunneling. Extends axios, got, undici, node-fetch, superagent. -``` - -**PR Steps:** -1. Fork repository -2. Edit `javascript.md` -3. Add to Network section after existing HTTP clients -4. Submit PR with title: "Add javascript-proxy-headers to JavaScript Network section" - ---- - -### 3.3 awesome-proxy (if exists/create) -**Action:** Search for or create a proxy-focused awesome list - -**Steps:** -1. Search GitHub for "awesome proxy" lists -2. If suitable list exists, submit PR -3. Consider creating `awesome-proxy-tools` if none exists - ---- - -## Part 4: Library Documentation PRs - -### 4.1 axios Ecosystem/Community - -**Repository:** https://github.com/axios/axios -**Opportunity:** axios has a community ecosystem page - -**Action:** Submit PR to add to ecosystem documentation - -**Steps:** -1. Check for `ECOSYSTEM.md` or community section -2. Fork axios repository -3. Add javascript-proxy-headers as "Proxy Support" extension -4. Submit PR referencing the proxy header limitation (axios/axios#3459) - -**Proposed content:** -```markdown -### Proxy Extensions -- [javascript-proxy-headers](https://github.com/proxymeshai/javascript-proxy-headers) - Send custom headers to proxies and receive proxy response headers during HTTPS CONNECT -``` - ---- - -### 4.2 got Ecosystem - -**Repository:** https://github.com/sindresorhus/got -**Opportunity:** got links to related packages in README - -**Action:** Open issue or PR suggesting addition to related packages - -**Steps:** -1. Check got's README for "Related" or "Ecosystem" section -2. Open issue: "Add javascript-proxy-headers to related packages" -3. Explain that it extends got with proxy header support -4. If welcomed, submit PR - ---- - -### 4.3 undici Documentation - -**Repository:** https://github.com/nodejs/undici -**Opportunity:** undici is official Node.js project, may accept ecosystem links - -**Action:** Open discussion about proxy header extensions - -**Steps:** -1. Check undici documentation for ecosystem/plugins section -2. Open GitHub Discussion (not issue): "Proxy header extension: javascript-proxy-headers" -3. Describe the use case and implementation -4. Ask about listing in documentation - ---- - -### 4.4 node-fetch Wiki/README - -**Repository:** https://github.com/node-fetch/node-fetch -**Opportunity:** node-fetch has a wiki with ecosystem info - -**Action:** Check wiki for community extensions - -**Steps:** -1. Review node-fetch wiki -2. If community section exists, request addition -3. Otherwise, open issue suggesting documentation improvement - ---- - -### 4.5 superagent Ecosystem - -**Repository:** https://github.com/ladjs/superagent -**Opportunity:** superagent lists plugins in README - -**Action:** Submit PR to add plugin - -**Steps:** -1. Review superagent README plugins section -2. Fork and add javascript-proxy-headers -3. Submit PR: "Add javascript-proxy-headers proxy plugin" - ---- - -## Part 5: Direct Maintainer Outreach - -### 5.1 Identify Key Maintainers - -| Library | Maintainer(s) | Contact Method | -|---------|--------------|----------------| -| axios | Matt Zabriskie, contributors | GitHub issues | -| got | Sindre Sorhus | GitHub, Twitter @sindresorhus | -| undici | Node.js team | GitHub Discussions | -| node-fetch | node-fetch org | GitHub issues | -| superagent | Lad team | GitHub issues | - -### 5.2 Outreach Template - -**Subject:** javascript-proxy-headers - Proxy extension for [library] - -**Message:** -``` -Hi, - -I've created javascript-proxy-headers, a library that extends [library] to support -sending custom headers to proxies and receiving proxy response headers during HTTPS -CONNECT tunneling. - -This solves a common pain point for developers using proxy services (like ProxyMesh) -that rely on custom headers for features like country selection or IP assignment. - -The library is at: https://github.com/proxymeshai/javascript-proxy-headers - -Would you be open to: -1. Mentioning it in [library]'s ecosystem/related projects? -2. Any feedback on the implementation approach? - -Thanks for maintaining such a great library! -``` - -### 5.3 Outreach Priority - -1. **superagent** - Most likely to accept plugin listing -2. **got** - Sindre is responsive, maintains awesome-nodejs -3. **axios** - Large community, high impact -4. **undici** - Official Node.js, good for credibility -5. **node-fetch** - Moderate activity - ---- - -## Part 6: Content Marketing - -### 6.1 Blog Posts - -**Topics to write:** -1. "How to Send Custom Headers to HTTPS Proxies in Node.js" -2. "Solving the Proxy Header Problem in JavaScript" -3. "Comparing Proxy Support Across JavaScript HTTP Libraries" - -**Where to publish:** -- Dev.to -- Medium (JavaScript/Node.js publications) -- Hashnode -- ProxyMesh blog - -### 6.2 Example Blog Post Outline - -**Title:** "How to Send Custom Headers to HTTPS Proxies in Node.js" - -**Sections:** -1. The problem: Why proxy headers don't work with HTTPS -2. How CONNECT tunneling works -3. The solution: javascript-proxy-headers -4. Examples for each library -5. Real-world use cases (country selection, session management) - ---- - -### 6.3 Video Content - -**Ideas:** -1. Short demo video (2-3 min) showing the problem and solution -2. Tutorial video walking through implementation -3. Conference talk proposal for Node.js conferences - ---- - -## Part 7: Community Engagement - -### 7.1 Stack Overflow - -**Action:** Answer proxy-related questions and mention the library - -**Steps:** -1. Search for questions about: - - "axios proxy headers" - - "node-fetch proxy custom headers" - - "https proxy CONNECT headers node" -2. Provide helpful answers -3. Mention javascript-proxy-headers as a solution where appropriate -4. Do NOT spam - only answer where genuinely helpful - -**Example questions to monitor:** -- https://stackoverflow.com/questions/tagged/axios+proxy -- https://stackoverflow.com/questions/tagged/node.js+proxy - -### 7.2 Reddit - -**Subreddits:** -- r/node -- r/javascript -- r/webdev -- r/webscraping - -**Action:** Share when appropriate, engage in proxy discussions - -### 7.3 Discord/Slack Communities - -**Communities to join:** -- Node.js Discord -- Reactiflux (Node.js channels) -- Various web scraping communities - ---- - -## Part 8: GitHub Presence - -### 8.1 Repository Optimization - -**Checklist:** -- [ ] Add topics/tags: `proxy`, `http`, `axios`, `node-fetch`, `got`, `undici`, `https`, `tunnel` -- [ ] Add social preview image -- [ ] Pin repository on ProxyMesh org profile -- [ ] Add "Used by" section once adopted - -### 8.2 GitHub Sponsors - -**Action:** Set up GitHub Sponsors for the project - -### 8.3 Cross-link Repositories - -**Update these repos to link to javascript-proxy-headers:** -- python-proxy-headers (Related Projects section) ✓ -- proxy-examples (JavaScript section) ✓ -- scrapy-proxy-headers (Related Projects) - ---- - -## Part 9: SEO & Discoverability - -### 9.1 npm Keywords - -Already included in package.json: -- proxy, http, https, connect, tunnel, headers -- axios, fetch, got, undici, superagent -- proxymesh, web-scraping, http-client - -### 9.2 GitHub Topics - -Add to repository: -- proxy -- http-client -- axios -- node-fetch -- got -- undici -- web-scraping -- https-proxy - ---- - -## Part 10: Metrics & Tracking - -### 10.1 Key Metrics to Track - -| Metric | Source | Target (6 months) | -|--------|--------|-------------------| -| npm weekly downloads | npm | 1,000 | -| GitHub stars | GitHub | 100 | -| Documentation views | ReadTheDocs | 500/month | -| Referral links from libraries | GitHub | 3 libraries | - -### 10.2 Tracking Tools - -- npm stats: https://npm-stat.com -- GitHub Insights -- ReadTheDocs analytics - ---- - -## Action Items Summary - -### Immediate (Agent can do with GitHub account) - -| # | Action | Repository/Platform | Type | -|---|--------|---------------------|------| -| 1 | Add GitHub topics to repo | proxymeshai/javascript-proxy-headers | Settings | -| 2 | Create PR for awesome-nodejs | sindresorhus/awesome-nodejs | PR | -| 3 | Create PR for awesome-web-scraping | lorien/awesome-web-scraping | PR | -| 4 | Create PR for superagent plugins | ladjs/superagent | PR | -| 5 | Open Discussion on undici | nodejs/undici | Discussion | -| 6 | Open issue on got | sindresorhus/got | Issue | -| 7 | Update python-proxy-headers README | proxymesh/python-proxy-headers | PR | -| 8 | Update scrapy-proxy-headers README | proxymesh/scrapy-proxy-headers | PR | - -### Requires Human Action - -| # | Action | Notes | -|---|--------|-------| -| 1 | npm publish | Requires npm login | -| 2 | ReadTheDocs import | Requires RTD account | -| 3 | Blog post writing | Content creation | -| 4 | Video creation | Content creation | -| 5 | Stack Overflow answers | Ongoing engagement | -| 6 | Maintainer outreach emails | Personal communication | - ---- - -## Timeline - -**Week 1:** -- Publish to npm -- Import to ReadTheDocs -- Add GitHub topics -- Submit awesome list PRs - -**Week 2-3:** -- Library documentation PRs -- Open discussions with maintainers -- Cross-link related repositories - -**Week 4+:** -- Content marketing (blog posts) -- Community engagement -- Monitor and respond to feedback - ---- - -*Plan created: February 28, 2026* diff --git a/README.md b/README.md index 8f062c6..5889e6d 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,16 @@ -[![npm version](https://badge.fury.io/js/javascript-proxy-headers.svg)](https://badge.fury.io/js/javascript-proxy-headers) # JavaScript Proxy Headers -Extensions for JavaScript HTTP libraries to support **sending and receiving custom proxy headers** during HTTPS CONNECT tunneling. +[![npm version](https://badge.fury.io/js/javascript-proxy-headers.svg)](https://badge.fury.io/js/javascript-proxy-headers) + +Extensions for JavaScript HTTP libraries that send custom **proxy headers** on HTTPS CONNECT and read **proxy response headers** back. Works with any proxy provider; works best with ProxyMesh. ## The Problem When making HTTPS requests through a proxy, the connection is established via a CONNECT tunnel. During this process: -1. **Sending headers to the proxy** - Most JavaScript HTTP libraries don't provide a way to send custom headers (like `X-ProxyMesh-Country`) to the proxy server during the CONNECT handshake. +1. **Sending headers to the proxy** - Most JavaScript HTTP libraries don't provide a way to send custom **proxy headers** (like `X-ProxyMesh-Country`) to the proxy server during the CONNECT handshake. -2. **Receiving headers from the proxy** - The proxy's response headers from the CONNECT request are typically discarded, making it impossible to read custom headers (like `X-ProxyMesh-IP`) that the proxy sends back. +2. **Receiving headers from the proxy** - The **proxy response headers** from the CONNECT request are typically discarded, making it impossible to read custom headers (like `X-ProxyMesh-IP`) that the proxy sends back. This library solves both problems for popular JavaScript HTTP libraries. @@ -28,7 +29,7 @@ This library solves both problems for popular JavaScript HTTP libraries. | [needle](https://github.com/tomas/needle) | `javascript-proxy-headers/needle` | Lean HTTP client | | [typed-rest-client](https://github.com/microsoft/typed-rest-client) | `javascript-proxy-headers/typed-rest-client` | Azure / DevOps–style REST client | -**urllib** is not integrated yet: it expects an [undici](https://undici.nodejs.org/) `Dispatcher`, not a Node `Agent`. See `notes/urllib-integration-deferred.md` in this repo for a possible approach. +**urllib** is not integrated yet: it expects an [undici](https://undici.nodejs.org/) `Dispatcher`, not a Node `Agent`. Until an adapter exists, use `javascript-proxy-headers/undici`. ## Installation @@ -40,7 +41,14 @@ Then install the HTTP client(s) you use (for example `axios`, `got`, `ky`, `wret > **Note:** This package has no runtime dependencies by default—install only the adapters you need. -## 0.3.0 breaking changes +Requirements: + +- Node.js >= 18.0.0 +- One or more supported HTTP libraries + +## Quick Start + +### 0.3.0 breaking changes CONNECT response headers are **not** copied onto origin `response.headers`. Read them from the per-request `proxyHeaders` Map so concurrent requests stay isolated and hop-by-hop CONNECT headers cannot impersonate origin headers (`Set-Cookie`, `Location`, and similar). @@ -54,8 +62,6 @@ response.proxyHeaders.get('x-proxymesh-ip') `agent.lastProxyHeaders` still exists as a last-write-wins snapshot. Prefer `response.proxyHeaders` (or `getProxyHeaders()`) under concurrency. -## Quick Start - ### axios ```javascript @@ -216,6 +222,12 @@ https.get('https://httpbin.org/ip', { agent }, (res) => { ## Testing +No-network unit checks: + +```bash +npm run test:unit && npm run test:types +``` + Integration tests need a real proxy (set `PROXY_URL` or `HTTPS_PROXY`): ```bash @@ -232,19 +244,20 @@ node test/test_proxy_headers.js -v axios ky Verbose (`-v`) prints captured header values. See `test/test_proxy_headers.js --help`. -## Requirements +## Documentation -- Node.js >= 18.0.0 -- One or more supported HTTP libraries +- [javascript-proxy-headers on Read the Docs](https://javascript-proxy-headers.readthedocs.io/) +- **[proxy-examples](https://github.com/proxymesh/proxy-examples)** - Example code for using proxies -## Related Projects +## Other languages - **[python-proxy-headers](https://github.com/proxymesh/python-proxy-headers)** - Same functionality for Python -- **[proxy-examples](https://github.com/proxymesh/proxy-examples)** - Example code for using proxies +- **[ruby-proxy-headers](https://github.com/proxymesh/ruby-proxy-headers)** - Same functionality for Ruby +- **[scrapy-proxy-headers](https://github.com/proxymesh/scrapy-proxy-headers)** - Proxy header support for Scrapy spiders -## About +## Why ProxyMesh -Created by [ProxyMesh](https://proxymesh.com) to help our customers use custom headers to control proxy behavior. Works with any proxy that supports custom headers. +[ProxyMesh](https://proxymesh.com) created this library to help customers use custom headers to control proxy behavior, as documented at [docs.proxymesh.com](https://docs.proxymesh.com/). It works with any proxy that supports custom headers on CONNECT. ## License diff --git a/notes/urllib-integration-deferred.md b/notes/urllib-integration-deferred.md deleted file mode 100644 index b81673a..0000000 --- a/notes/urllib-integration-deferred.md +++ /dev/null @@ -1,11 +0,0 @@ -# urllib integration (deferred) - -`urllib` v4 routes HTTPS through **undici** using a **`Dispatcher`** (`RequestOptions.dispatcher`, `HttpClient#setDispatcher`), not Node’s `http.Agent`. `ProxyHeadersAgent` is an `https.Agent` and cannot be passed as `dispatcher`. - -A first-class adapter should: - -1. Refactor shared HTTPS CONNECT logic from `lib/undici-proxy.js` into a reusable helper (tunnel + `undici.Client` with `connect: { socket }`). -2. Implement a minimal **undici `Dispatcher`** (or compose `undici.Client`) that performs that tunnel for each request, matching current HTTPS scope. -3. Wire `new HttpClient(...).setDispatcher(proxyDispatcher)` (or per-request `dispatcher`) and merge `proxyHeaders` onto the `HttpClientResponse` surface the same way other adapters do. - -Until that exists, consumers can call `javascript-proxy-headers/undici` or use `createProxyFetch` + `urllib`’s fetch path only if they expose a compatible hook (they do not today without a custom dispatcher).