chore(sync): merge upstream firecrawl (2026-08-14, 243 commits) - #17

…its (firecrawl#3903)

fireclaw's controller-level credit check read ACUC chunk.remaining_credits
to enforce the variable multi-play cost (plays * 100). ACUC is being
stripped of credit fields (v49 zeroes remaining_credits), so this check
would 402 every play once callers move to v49.

Switch the controller check to autumnService.checkCredits({ value:
totalCredits }) — Autumn is the source of truth for credits. The
route-level checkCreditsMiddleware(100) only verifies a single play, so
this is the only gate covering the full plays * 100 cost. Fails open on an
Autumn outage, matching checkCreditsMiddleware. The success response's
remaining_credits now comes from getTeamBalance() instead of a re-read of
ACUC. This removes the last credit-enforcement reader of ACUC, unblocking
the rpc.ts -> v49 migration.

Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com>
Co-authored-by: micahstairs <micah@sideguide.dev>

Switch authCreditUsageChunk / authCreditUsageChunkFromTeam to call
auth_credit_usage_chunk_49 / _49_from_team, and drop the now-removed
i_is_extract and tally_untallied_credits arguments (v49 takes only the
lookup key). The app-level is_extract flag is retained where it's still
needed (ACUC cache key, chunk.is_extract); it just no longer flows into
the SQL call. Updates the two call sites in auth.ts accordingly.

Mechanical migration only — does not change any credit-field readers
(those are zeroed by v49 and handled in follow-up PRs).

Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com>
Co-authored-by: micahstairs <micah@sideguide.dev>

Expose the `enterprise` search option across the JS, Python, Ruby, and
.NET SDKs so callers can select enterprise search modes (e.g. ["zdr"] or
["anon"]) on /v2/search.

- js: add `enterprise` to SearchRequest and forward it in the payload
- python: add `enterprise` to SearchRequest and the search() signature
- ruby: add `enterprise` to SearchOptions fields and serialization
- dotnet: fix SearchOptions.Enterprise type (bool -> List<string>)

Bumps the affected SDK package versions.

Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com>
Co-authored-by: micahstairs <micah@sideguide.dev>

These five SDKs received search monitor target support (#82973) without a
follow-up version bump; js/python/ruby/dotnet were already bumped (firecrawl#3919).
Minor bumps to match that wave:
- go 1.7.2 -> 1.8.0
- php 1.7.1 -> 1.8.0 (also releases the menu scrape format)
- rust 2.10.0 -> 2.11.0
- java 1.10.2 -> 1.11.0
- elixir 1.7.2 -> 1.8.0

…rawl#3923)

Sites that lazy-load images (e.g. happysocks.com.au) ship a tiny
placeholder in `src` (often `?width=5`) and keep the real image in
`data-srcset` / `data-src`, swapping it in via JS only once the image
scrolls into view. Below-the-fold images never get swapped during a
scrape, so the placeholder leaked into the markdown output.

The existing "pick the largest srcset image" logic only looked at the
`srcset` attribute, which these images don't have until the loader
fires. Extend it to fall back to `data-srcset` (fed through the same
largest-image picker) and then `data-src`, only when a real `srcset`
isn't present. Images that already have a `srcset`/`src` are untouched,
so behavior for other sites is unchanged.

Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com>
Co-authored-by: micahstairs <micah@sideguide.dev>

…crawl#3926)

Switch authCreditUsageChunk / authCreditUsageChunkFromTeam to
auth_credit_usage_chunk_50 so per-team rate_limits overrides are merged onto
the plan instead of replacing it. Requires firecrawl-db v50 migration deployed.

Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com>
Co-authored-by: micahstairs <micah@sideguide.dev>

chore(sdks): bump go/php/rust/java/elixir for search monitor support

fix(monitor): search-mon-updates

* Nick: slack app monitoring

* Delete slack.test.ts

* Create slack.test.ts

* Update slack.test.ts

* Update apps/api/src/services/monitoring/types.ts

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

* Nick:

* Nick:

* Update slack.ts

---------

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

…ecrawl#3930)

* fix(security): resolve pnpm audit failures

* chore(js-sdk): bump package version

Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>

Research endpoints are GETs, so the POST-body origin injection in
HttpClient never applied and requests were logged with origin="api".
Attach origin as a query param in the research methods instead
(accepted by the v2 research proxy's commonQuery schema). Bump to 4.29.2.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJ3prpziToFecBhLGRiX6t

…3952)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…firecrawl#3957)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…rawl#3960)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…rawl#3949)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…awl#3961)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…dget (firecrawl#4237)

…ge (firecrawl#4249)

* fix(api): restore the five labs routes dropped by the rate limit change

b3d7891 rewrote every route in labs.ts to swap RateLimiterMode.Search for
RateLimiterMode.Labs and lost five of them on the way: GET /search/packs,
PATCH /search/packs/:packId, POST /search/data/pages, PATCH /search/data/:sourceId
and POST /search/data/:sourceId/refresh. It also deleted the comment warning that
this list has no catch-all and must be kept in sync with the service by hand.
Production has answered those five paths with a 404 since the deploy, which is
why provider packs stopped loading in the dashboard while engines and data kept
working.

Express replies to an unclaimed path with an HTML "Cannot GET" page, which a
dashboard fetch parses into nothing and reports as a bare 404 — the same thing it
shows when the service says a pack does not exist. One is a deploy gap and the
other is a missing record, so answer the gap with JSON that names it.

LABS_ROUTES in the new test locks the list so the next rewrite of this file
cannot quietly shorten it.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Update labs.ts

* Update labs.routes.test.ts

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

…wl#4209)

* docs: align self-hosting guidance with the public quickstart

* docs: refine self-hosting reference voice

* docs: align self-hosting deployment references

… links (firecrawl#4250)

…#4256)

* fix(deps): bump js-yaml overrides past GHSA-5p4m-2wfm-xmqj

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* keep js-yaml 3.x consumers on the patched 3.x line

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* feat(api): safeSearch param on v2 /search

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* rename safeSearch param to safe

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* forward safe flag to searxng fallback

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

… prefetch

Co-Authored-By: gaurav <gauravchadha1676@gmail.com>

…ions-document-gate

fix(api): don't throw SCRAPE_ACTIONS_NOT_SUPPORTED after document/pdf prefetch

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

DNS resolution failures were the only failure class billed a base
credit — every other failed scrape bills 0. Align them: a scrape that
never resolved delivers no content and now bills nothing. Lockdown
cache misses keep their 1-credit charge.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Introduce an exchange proxy router that forwards /exchange/discover and /exchange/retrieve requests to an upstream FIRE_EXCHANGE_URL with timeouts and header forwarding. The router enforces auth, team-level exchangeRetrieve flag, and per-route timeouts/dispatchers. Register the router in the API entrypoint. Add TeamFlags.exchangeRetrieve to the v1 types so teams can be opted into this endpoint. Includes error handling for timeouts and upstream failures.

Stop gateway tests from requiring internal response fields.
Allow GitHub search results without the resultType field.

…metry (firecrawl#4289)

* docs(sdks): disambiguate the research paper index from the `research` search category

Firecrawl serves a research paper index (~43M abstracts, ~90% biomedical —
PubMed/bioRxiv/medRxiv — plus arXiv) at /v2/search/research. Separately,
`search(categories=["research"])` is only a ~14-domain website filter over
ordinary web search. The names collide and agents routinely pick the filter.
Neither SDK contained any biomedical vocabulary.

This is a documentation/metadata change only. No runtime behavior changes.

Python:
- v2/methods/research.py: add module + per-function docstrings naming the
  corpus, warning about the collision, and documenting that these responses are
  returned raw (camelCase keys, not snake_case-normalized like the rest of the
  SDK).
- v2/methods/aio/research.py: mirror the same docstrings.
- v2/client.py, v2/client_async.py: add wrapper docstrings — an agent calling
  `firecrawl.search_papers(...)` sees the wrapper's docstring, not the module
  function's.
- v2/types.py: `Category` docstring restated the collision as if correct
  ("Filter results to research papers and academic sites"). Rewritten to say it
  is a website/domain filter and to point at `search_papers()`.

JavaScript:
- v2/types.ts: `CategoryOption` had no doc comment; add one, plus a note on the
  `categories` field of `SearchRequest`.
- v2/methods/research.ts, v2/client.ts: both described the surface as
  "(arXiv papers + GitHub history/readmes)". Rewritten to name the real corpus.
- v2/types.ts: `IdMap` claimed "Currently only `arxiv` is populated" while
  `PaperResult.primaryId` documented `pmid:`/`doi:`. Removed the stale claim
  rather than inventing an exhaustive namespace list (the API is proxied, so the
  namespace set is not verifiable in-repo).

Both SDKs:
- READMEs documented neither paper search nor plain search. Add a "Search"
  section and a "Research / paper search" section with biomedical examples.
- Package metadata (pyproject.toml, setup.py, package.json) had no research
  vocabulary. Add descriptions, keywords, and Science/Research + Bio-Informatics
  + Medical Science trove classifiers.
- v1 deprecation notices said "/v1/deep-research is deprecated. Use /v2/search
  instead.", routing agents straight into the website filter. Now point at
  /v2/search for web research AND search_papers()/research.searchPapers() for
  literature. (12 occurrences in Python, 3 in JS.)

Tests:
- Python had zero tests for any research method. Add 39 unit tests mirroring the
  JS suite (mock the HttpClient, pin method names and query-string
  construction), sync + async.

Deliberately not changed (breaking, needs a product decision):
- The Python research return type stays `Dict` rather than typed models; the
  camelCase-vs-snake_case inconsistency is documented instead.
- `related_papers` (Python) vs `similarPapers` (JS) not renamed; cross-referenced
  in both docstrings instead.

* fix(python-sdk): send `origin` from inspect_paper like the other research methods

`search_papers`, `read_paper`, `related_papers` and `search_github` all append
`origin=python-sdk@<version>` via `_query()`. `inspect_paper` called `_get()`
bare, so paper-detail requests arrived with no SDK attribution — the one
research call the API could not attribute to the Python SDK.

The API accepts it: `paperSchema` in
apps/api/src/controllers/v2/research-proxy.ts spreads `commonQuery`
(`origin`, `integration`) into its strictObject, and the controller reads it
through `requestOrigin(params, req)`.

Mirrored in `v2/methods/aio/research.py`.

The previous behaviour was pinned by `test_sends_no_query_params`; that test
now pins the corrected behaviour (and the id-encoding tests compare the path
component rather than the whole URL).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(python-sdk): expose the five research methods on the unified clients

The README documents `firecrawl.search_papers(...)` on the client you get from
`from firecrawl import Firecrawl`. That class (firecrawl/client.py:210) did not
have the method:

    >>> Firecrawl(api_key="fc-...").search_papers("q")
    AttributeError: 'Firecrawl' object has no attribute 'search_papers'

Unlike the JS SDK — where `class Firecrawl extends FirecrawlClient` inherits
everything the v2 client gains — the Python unified client re-exports the v2
surface method by method in `__init__`. `search_papers`, `inspect_paper`,
`read_paper`, `related_papers` and `search_github` were added to
`v2/client.py` / `v2/client_async.py` only, so they were reachable at
`firecrawl.v2.search_papers(...)` (via `V2Proxy.__getattr__`) but not at the
top level the docs and the docstrings point at.

Add all five to `Firecrawl` and `AsyncFirecrawl` as thin delegations to the v2
implementations, defined as real methods (not `__init__` attribute
assignments) so the docstrings — the disambiguation surface an agent reads via
`help()` / hover — are visible on the class itself. Docstrings are copied from
the v2 wrappers.

Tests: pin that both unified clients expose all five, that each one reaches the
right `/v2/search/research/...` path with the right query params (v2 HTTP layer
mocked), that responses come back verbatim, and that the docstrings still carry
the `search(categories=["research"])` disambiguation. 445 -> 456 unit tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore(sdks): bump firecrawl-py to 4.35.0 and @mendable/firecrawl-js to 4.32.1

Python 4.34.0 -> 4.35.0 (minor): this branch adds five new public methods to
`Firecrawl` / `AsyncFirecrawl`, changes the wire request for `inspect_paper`
(now sends `origin`), and rewrites the package metadata (description,
keywords, Science/Research + Bio-Informatics + Medical Science classifiers).
New public API, backwards compatible.

Note on "keeping pyproject.toml and setup.py in sync": neither hardcodes a
version. `setup.py` reads `__version__` out of `firecrawl/__init__.py`, and
`pyproject.toml` declares `dynamic = ["version"]` so setuptools takes it from
setup.py. `firecrawl/__init__.py` is the single source of truth; verified via
PEP 517 `prepare_metadata_for_build_wheel` -> `firecrawl_py-4.35.0.dist-info`.
It is also what `v2/utils/get_version.py` stamps into the `origin` param.

JS 4.32.0 -> 4.32.1 (patch): the JS side of this branch is documentation and
metadata only — doc comments, README sections, package description/keywords.
The public surface is unchanged; `Firecrawl extends FirecrawlClient` already
exposed `.research`, so no new methods were needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(python-sdk): single source of truth for the research docstrings

The five research methods (search_papers, inspect_paper, read_paper,
related_papers, search_github) are surfaced at six layers: the sync and
async implementation functions, and the delegating methods on
FirecrawlClient, AsyncFirecrawlClient, Firecrawl and AsyncFirecrawl.
Each layer carried its own literal copy of the prose, so correcting a
factual claim — corpus size and composition, the
search(categories=["research"]) disambiguation, wire names, Args/Returns
— meant a six-way edit that would drift (Cubic P3).

Move the text to firecrawl/v2/methods/research_docs.py and apply it with
a doc() decorator that sets __doc__ without wrapping the function. Every
layer still exposes the complete docstring: help() and pydoc on
Firecrawl.search_papers render exactly what they rendered before, byte
for byte after cleandoc, at all six layers. No public API, signature,
method name or runtime behaviour changes.

The two async deltas (the client Arg wording, and the awaited example)
are substituted into the shared template rather than forking the prose,
so there is one copy of each claim, not six. 568 lines of duplicated
prose become 201.

Adds two guard tests that fail if a layer forks its own copy again or
ships an unresolved template placeholder.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ADHZoLJe28rAnirV1VU2Bg

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…et (firecrawl#4300)

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

…eveloper searches

…riptions for zero-credit research endpoints

billing: make research paper endpoints zero-credit (research index)

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>

Update the scrape PDF detectPdf/processPdf path to the current crates.io release.

Co-authored-by: Cursor <cursoragent@cursor.com>

…-sync-fe49e4c97

# Conflicts:
#	.github/workflows/deploy-image.yml
#	.github/workflows/npm-audit-claude-remediation.yml
#	.github/workflows/npm-audit.yml
#	.github/workflows/publish-php-sdk.yml
#	.github/workflows/test-server.yml
#	apps/api/.env.example
#	apps/api/src/__tests__/snips/v2/audio-routing.test.ts
#	docker-compose.yaml
#	examples/kubernetes/firecrawl-helm/README.md

GNU grep -q exits at the first match, SIGPIPE-killing the still-streaming
git status; under pipefail the successful match reads as a failed pipeline,
so real conflicts crashed the script at git commit (exit 128) with no
result emitted — escalation issues said result=<none> instead of conflict.
Reproduced against a scratch merge with GNU grep 3.11; git ls-files
--unmerged has no pipeline to break.