Custom HTTP Headers: How to Control Request Headers in Browsers

Introduction

HTTP headers are part of every request your browser sends. Headers like User-Agent, Accept-Language, Sec-CH-UA, and Referer form a significant portion of your browser's network identity. When these headers are inconsistent with your fingerprint profile, or when you need to inject application-specific headers for authentication and routing, the standard browser APIs fall short.

BotBrowser provides engine-level header control through the --bot-custom-headers flag and the BotBrowser.setCustomHeaders CDP command. Unlike extension-based approaches that intercept requests after they are formed, BotBrowser modifies headers before the request leaves the browser's networking stack, covering all request types without timing gaps.

Privacy Impact

HTTP headers reveal a surprising amount of information about your browser environment. The User-Agent string identifies your browser version and operating system. Accept-Language reveals your language preferences and, by extension, your likely geographic region. Client Hints headers (Sec-CH-UA, Sec-CH-UA-Platform, Sec-CH-UA-Mobile) provide structured data about your browser brand, platform, and device type.

When these headers do not align with other aspects of your fingerprint, the inconsistency stands out. A browser reporting Accept-Language: de-DE but with a US timezone, or a Sec-CH-UA-Platform: "Windows" but with macOS-specific navigator properties, creates mismatches that tracking systems can correlate.

BotBrowser profiles automatically align all standard headers with the profile's identity. The User-Agent, all Sec-CH-UA-* headers, and Accept-Language are derived from the profile's browser, platform, and locale settings. Custom headers added through --bot-custom-headers extend this baseline without disrupting the profile's standard header set.

Technical Background

How Browsers Construct Request Headers

When a browser sends an HTTP request, it constructs headers from multiple sources:

1. Built-in headers: User-Agent, Accept, Accept-Encoding, Accept-Language are added by the browser engine based on internal settings.
2. Client Hints: Sec-CH-UA, Sec-CH-UA-Mobile, Sec-CH-UA-Platform are sent by default. High-entropy hints like Sec-CH-UA-Full-Version-List are sent only when the server requests them via Accept-CH.
3. Application headers: Headers set by JavaScript through fetch() or XMLHttpRequest.
4. Extension-injected headers: Headers added by browser extensions through the webRequest API.

The order and composition of these headers create a consistent pattern for each browser. Chrome, Edge, and Firefox each produce different header orderings and combinations.

Profile-Based Header Consistency

When BotBrowser loads a fingerprint profile, it configures all standard headers to match the profile's identity:

• User-Agent matches the profile's browser version and platform
• Sec-CH-UA contains the correct brand tokens in the proper order
• Sec-CH-UA-Platform matches the profile's operating system
• Accept-Language reflects the configured locale and language preferences

These headers are set at the engine level and apply to all requests, including the initial navigation request, subresource loads, and JavaScript-initiated requests. There is no timing gap where incorrect headers could be observed.

Custom Headers vs. Standard Headers

Custom headers (those starting with X- or application-specific names) serve a different purpose than standard headers. They carry application-specific data: authentication tokens, session identifiers, application version markers, or routing hints. BotBrowser's --bot-custom-headers flag adds these custom headers without modifying the profile-managed standard headers.

Common Approaches and Their Limitations

Extension-Based Header Modification

Browser extensions can modify headers through the webRequest.onBeforeSendHeaders API. This approach has several drawbacks:

• Timing issues: The first request (the navigation request) may fire before the extension's listener is fully registered
• Incomplete coverage: Service worker requests, preflight CORS requests, and some internal requests may not trigger the extension's listener
• Detectable artifacts: Extensions that modify headers leave traces in the browser's extension API surface
• Performance overhead: Each request passes through the extension's JavaScript handler, adding latency

CDP-Based Header Overrides

Puppeteer and Playwright provide header override APIs through CDP:

// Puppeteer - limited approach
await page.setExtraHTTPHeaders({ 'X-Custom': 'value' });

This works for page-level headers but has limitations:

• Headers set this way only apply to the specific page
• Service workers and shared workers may not receive the custom headers
• The CDP Network.setExtraHTTPHeaders command requires Network.enable, which can itself affect fingerprinting

JavaScript Fetch/XHR Headers

Setting headers on individual fetch() or XMLHttpRequest calls only covers JavaScript-initiated requests. Navigation requests, image loads, stylesheet fetches, and other browser-initiated requests are not covered.

BotBrowser's Approach

The --bot-custom-headers Flag

BotBrowser's --bot-custom-headers flag (PRO) injects custom headers at the browser engine level:

chrome --bot-profile="/path/to/profile.enc" \
       --bot-custom-headers='{"X-Custom-Header":"value","X-Auth-Token":"abc123"}' \
       --user-data-dir="$(mktemp -d)"

Because the modification happens inside the Chromium networking stack, custom headers are added to:

• Initial navigation requests
• Subresource requests (images, scripts, stylesheets, fonts)
• XHR and Fetch API requests
• WebSocket upgrade requests
• Service worker requests

There are no timing gaps and no missed request types.

The BotBrowser.setCustomHeaders CDP Command

For runtime header management, BotBrowser provides a CDP command that can be sent to the browser-level session:

Puppeteer:

const cdpSession = await browser.target().createCDPSession();
await cdpSession.send('BotBrowser.setCustomHeaders', {
  headers: { 'x-requested-with': 'com.example.app' }
});

Playwright:

const cdpSession = await browser.newBrowserCDPSession();
await cdpSession.send('BotBrowser.setCustomHeaders', {
  headers: { 'x-requested-with': 'com.example.app' }
});

Important: This command must be sent to the browser-level CDP session, not a page-level session. Sending it to a page target will return ProtocolError: 'BotBrowser.setCustomHeaders' wasn't found.

Profile-Level Configuration

Custom headers can also be set in the profile's configuration via configs.customHeaders. This is useful when you want the headers embedded in the profile itself rather than specified at launch time.

Configuration and Usage

CLI Setup with Custom Headers

chrome --bot-profile="/path/to/profile.enc" \
       --bot-custom-headers='{"X-App-Version":"2.1.0","X-Session-ID":"sess_abc123"}' \
       --proxy-server=socks5://user:pass@proxy:1080 \
       --user-data-dir="$(mktemp -d)"

Playwright Integration

const { chromium } = require('playwright-core');

(async () => {
  const browser = await chromium.launch({
    executablePath: '/path/to/botbrowser/chrome',
    args: [
      '--bot-profile=/path/to/profile.enc',
      '--bot-custom-headers={"X-Auth":"token123","X-Client":"webapp"}',
    ],
    headless: true,
  });

  const context = await browser.newContext();
  const page = await context.newPage();
  await page.goto('https://example.com');
  await browser.close();
})();

Puppeteer Integration

const puppeteer = require('puppeteer-core');

(async () => {
  const browser = await puppeteer.launch({
    executablePath: '/path/to/botbrowser/chrome',
    args: [
      '--bot-profile=/path/to/profile.enc',
      '--bot-custom-headers={"X-Auth":"token123"}',
    ],
    headless: true,
    defaultViewport: null,
  });

  const page = await browser.newPage();
  await page.goto('https://example.com');
  await browser.close();
})();

JavaScript Quoting

When passing --bot-custom-headers from JavaScript, do not include shell-style quotes inside the value:

// Correct
const headers = { 'X-Custom': 'value', 'X-Auth': 'token' };
args.push('--bot-custom-headers=' + JSON.stringify(headers));

// Wrong - single quotes become part of the JSON value
args.push(`--bot-custom-headers='${JSON.stringify(headers)}'`);

Runtime Header Updates via CDP

For workflows that need to change custom headers during the session:

const { chromium } = require('playwright-core');

(async () => {
  const browser = await chromium.launch({
    executablePath: '/path/to/botbrowser/chrome',
    args: [
      '--bot-profile=/path/to/profile.enc',
    ],
    headless: true,
  });

  const cdpSession = await browser.newBrowserCDPSession();

  // Set initial headers
  await cdpSession.send('BotBrowser.setCustomHeaders', {
    headers: { 'x-requested-with': 'com.example.app' }
  });

  const page = await (await browser.newContext()).newPage();
  await page.goto('https://example.com');

  // Update headers mid-session
  await cdpSession.send('BotBrowser.setCustomHeaders', {
    headers: { 'x-requested-with': 'com.example.app', 'x-session': 'refreshed' }
  });

  await page.goto('https://example.com/dashboard');
  await browser.close();
})();

Verification

After launching BotBrowser with custom headers, verify they are applied:

1. Open a page that echoes request headers (like httpbin.org/headers)
2. Confirm your custom headers appear in the response
3. Check the Network tab in DevTools to see headers on all request types
4. Verify that standard headers (User-Agent, Sec-CH-UA) still match the profile

const page = await context.newPage();
await page.goto('https://httpbin.org/headers');
const headersText = await page.textContent('body');
console.log('Request headers:', headersText);

Best Practices

1. Use --bot-custom-headers for static headers. If the headers do not change during the session, the CLI flag is simpler than CDP.

2. Use BotBrowser.setCustomHeaders for dynamic headers. When headers need to change during the session (e.g., rotating auth tokens), use the CDP command.

3. Do not override profile-managed headers. Custom headers should add new headers, not replace User-Agent or Sec-CH-UA. Overriding profile headers creates inconsistencies.

4. Avoid common extension header names. Headers like X-Forwarded-For or X-Real-IP may conflict with proxy infrastructure. Use application-specific names.

5. Test with httpbin.org/headers. This endpoint echoes all received headers, making it easy to verify your configuration.

6. Always use browser-level CDP session for setCustomHeaders. Page-level sessions will return an error because the command operates at the browser level.

Frequently Asked Questions

Do custom headers apply to all requests or just navigation? All requests. BotBrowser applies custom headers at the engine level, covering navigation, subresource, XHR, Fetch, WebSocket, and service worker requests.

Can I set different custom headers per context? The --bot-custom-headers flag applies to all contexts. For per-context headers, use the BotBrowser.setCustomHeaders CDP command, which can be updated between context operations.

Do custom headers affect the fingerprint? Custom headers that are not part of the standard browser header set (like X-Custom-Header) do not affect the browser fingerprint. They are simply additional headers in the request.

Can I remove a standard header? BotBrowser does not support removing standard headers through custom headers. Profile-managed headers are always included to maintain fingerprint consistency.

Does --bot-custom-headers work with --proxy-server? Yes. Custom headers are added to requests regardless of whether a proxy is configured. The proxy sees the full set of headers.

What is the maximum number of custom headers I can set? There is no specific limit. However, excessively large header sets can increase request size and may trigger server-side limits.

Can I use --bot-custom-headers for Cookie headers? For cookie management, use the dedicated --bot-cookies flag instead. It provides proper cookie handling with domain scoping and expiration.

Summary

HTTP header control is essential for maintaining a consistent browser identity. BotBrowser provides engine-level header management through profiles (for standard headers), the --bot-custom-headers flag (for static custom headers), and the BotBrowser.setCustomHeaders CDP command (for dynamic runtime headers). Together, these tools ensure every request carries consistent, complete header information.

For related network configuration, see Proxy Configuration and User Agent Control and Client Hints. For identity management across multiple accounts, see Multi-Account Browser Isolation.