# Conventions across every tool The MCP and REST surfaces share one envelope, one error taxonomy, and the same rules for bounds, pagination, provenance, and licensing. ## The envelope Every payload is an `Envelope[DataT, MetaT: Meta]` (`conventions/envelope.py`) with three standardized regions: - **`data`** — the answer, plus the identity/echo context an agent needs to confirm what it got (`ticker` in canonical form, `cik`, `company_name`, requested `period_type`). Bounded (see below), newest-first for time-ordered results. - **`meta`** — provenance uniform for the whole result. The base `Meta` carries `source` (a stable connector identifier, e.g. `sec_edgar_companyfacts`) and `as_of` (the freshest value-level `as_of` on the page; `None` when the page is empty). Per-tool `Meta` subclasses add the *constant* attribution fields — `currency`, `unit`, `delay`, `non_redistributable` — where they hold for every value in the result. - **`pagination`** — a `Pagination` (`limit`, `has_more`, `next_cursor`), or `null` for scalar tools that return a single object rather than a bounded list. A concrete tool declares a **named subclass** so the schema keeps a stable model name for FastMCP output-schema and FastAPI response-model generation: ```python class IncomeStatementsResult(Envelope[IncomeStatementsData, Meta]): ... ``` Per-value attribution (`unit`/`currency`/`as_of`/provenance such as `accession_number`) still rides on the **individual values inside `data`** — `meta` carries only what is constant across the result. Numeric values remain `Decimal` serialized as **JSON strings** (exact, no float rounding; consumers parse them), and absence is a **missing key** — never zero, never a meaning-bearing null. ### Multi-connector canonical datasets Some tools serve a canonical dataset assembled from more than one connector. The first case is `corporate_actions`: `meta.source` is the stable dataset identifier (`corporate_actions`), while each value's `source` carries connector attribution and is the licensing authority. The #80 redistribution gate must evaluate per-value sources for these tools. `meta.non_redistributable` is true when any value on the page is restricted, matching the conservative direction used by `stocks_get_financial_metrics` (#25). Typed-event models serialize inapplicable fields as null, `FilingSummary`-style; the "absence is a missing key" rule governs the `lines`-dict numeric-value idiom, not typed-record fields. Example (`stocks_get_income_statements`): ```json { "data": { "ticker": "AAPL", "cik": 320193, "company_name": "Apple Inc.", "period_type": "annual", "statements": [ { "fiscal_year": 2024, "fiscal_period": "FY", "period_type": "annual", "period_start": "2023-10-01", "period_end": "2024-09-28", "lines": { "revenue": { "value": "391035000000.000000", "unit": "USD", "currency": "USD", "as_of": "2024-11-01T00:00:00Z", "accession_number": "0000320193-24-000123" } } } ] }, "meta": { "source": "sec_edgar_companyfacts", "as_of": "2024-11-01T00:00:00Z" }, "pagination": { "limit": 4, "has_more": true, "next_cursor": "eyJ2IjoxLCJrIjpbLi4uXX0" } } ``` ### Partial results for batch tools Batch tools that accept multiple entities resolve each input independently rather than discarding the whole response when one item is unknown. Unresolved items are reported inside `data` — for line-item search, as `unresolved_tickers` (including suggestions when available) and `unknown_line_items`. Even an all-unresolved call is a successful envelope with empty results and the complete per-item report (#256). ## Bounds and token-awareness Results are always bounded (CLAUDE.md: never dump unbounded data into an agent context). The policy is **reject an out-of-range `limit`, never clamp it**: a silent clamp lies to the agent about what it asked for and lets the two surfaces drift, whereas a rejection teaches the agent the real contract. Each service module declares `DEFAULT_LIMIT`/`MAX_LIMIT` constants sized to the payload, and the bound is enforced in **three layers**, deliberately: 1. the MCP tool schema (`ge`/`le` on the parameter), 2. the REST route signature (`Query(ge=..., le=...)` → 422), 3. the service function itself (a `BadParameterError` backstop for any future caller that skips a surface). Per-tool-class defaults and maxima (units are periods/bars/rows per page): | Tool class | Default | Max | Status | |---|---|---|---| | Statements (`stocks_get_income_statements`) | 4 periods | 12 periods | Enforced today | | Line-item search (`stocks_search_line_items`) | 4 periods per ticker×item (40-row page) | 12 periods; 20 tickers × 20 items | Enforced today (#256) | | Metrics (`stocks_get_financial_metrics`) | 4 periods | 12 periods | Enforced today (#25) | | Segmented financials (`stocks_get_segmented_financials`) | 4 fiscal-year frames | 12 fiscal-year frames | Enforced today (#40) | | Screener (`stocks_screen`) | 20 | 100 | Enforced today (#38) | | Peer comparison (`stocks_get_peers`) | 10 peers / 4 metrics | 20 peers / 10 metrics | Enforced today (unpaged: a nearest-N answer, so `pagination` is null and `data.peer_universe_size` discloses truncation). The metric max counts what is *served* per row, and `revenue` is always served because the ranking is measured on it — so the schemas bound the *requested* list at 10 (the achievable maximum, reached by naming `revenue` among them) and the service backstop rejects a request that would put an 11th metric on the row | | Prices (daily bars) | ≤1y | ≈260 bars | Target (#20/#26) | | Price snapshot (`stocks_get_price_snapshot`) | n/a (explicit tickers) | 50 tickers | Enforced today (#258) | | Filings | 100 | 1000 | Target (#27) | | Filing search (`stocks_search_filings`) | 10 | 50 | Enforced today (#110) | | Filing sections (`stocks_get_filing_section`) | 10,000 chars | 25,000 chars | Enforced today (#115) | | Recent filings (`stocks_get_recent_filings`) | 100 | 1000 | Enforced today (#120) | | Corporate actions (`stocks_get_corporate_actions`) | 100 | 1000 | Enforced today (#78) | | Beneficial owners (`stocks_get_beneficial_owners`) | 20 | 100 | Enforced today (#206) | | Ownership-filing search (`stocks_search_ownership_filings`) | 20 | 100 | Enforced today (#206) | | Insider trades (`stocks_get_insider_trades`) | 20 | 100 | Enforced today (#39) | | IPO search (`stocks_search_ipos`) | 20 | 100 | Enforced today (#207) | | Macro observations | 100 | 500 | Target (#33) | | Real-estate observations (`re_get_series`) | 100 | 500 | Enforced today (#89) | | Recent-updates feed (`macro_list_recent_updates`) | 100 | 500 | Enforced today (#121) | | Economic calendar (`macro_get_calendar`) | 50 events / 14-day window | 200 events / 93-day window | Enforced today (#44) | | Freshness (`platform_get_freshness`) | 50 | 100 | Enforced today (#119) | | Platform catalog (`platform_get_catalog`) | 50 | 100 | Enforced today (#92) | | Account usage history (`GET /account/usage/history`) | 100 | 500 | Enforced today (#311) | | Account usage events (`GET /account/usage/events`) | 50 | 200 | Enforced today | | Entity search | 10 | 50 | Target (#23) | | Geography resolution (`geo_resolve`) | 10 | 50 | Enforced today (#568) | | Geography containment/crosswalks (`geo_contains`, `geo_crosswalk`) | 50 | 200 | Enforced today (#568) | | Real-estate discovery (`re_search_geographies`, `re_search_series`) | 10 | 50 | Enforced today (#89) | | Bank entity search (`banks_search_institutions`) | 10 | 50 | Enforced today (#127) | | Bank financials (`banks_get_financials`) | 8 quarters | 40 quarters | Enforced today (#127) | | Recipient search (`gov_search_recipients`) | 10 | 50 | Enforced today (#171) | | Recipient awards (`gov_get_recipient_awards`) | 5 fiscal years | 20 fiscal years | Enforced today (#171) | | Fund search (`funds_search_funds`) | 10 | 50 | Enforced today (#148) | | Fund holdings (`funds_get_holdings`) | 50 rows | 200 rows | Enforced today (#148) | | CMBS deal search (`cmbs_search_deals`) | 10 | 50 | Enforced today (#219) | | CMBS loans (`cmbs_get_loans`) | 25 | 100 | Enforced today (#219) | | Fund holders (`stocks_get_fund_holders`) | 25 | 100 | Enforced today (#149) | | Fund proxy votes (`funds_get_proxy_votes`) | 25 rows | 100 rows | Enforced today (#238) | | Proxy-voting summary (`stocks_get_proxy_voting_summary`) | 2 meetings | 5 meetings | Enforced today (#239) | | Company events (`stocks_get_company_events`) | 20 | 100 | Enforced today (#41) | | Earnings dates (`stocks_get_earnings_dates`) | 8 | 40 | Enforced today (#79; unpaged bounded history, so `pagination` is null) | | Event search (`stocks_search_events`) | 20 | 100 | Enforced today (#41) | | Expected reports (`stocks_search_expected_reports`) | 20 rows / 7-day window | 100 rows / 30-day window | Enforced today (forward-ordered like `macro_get_calendar`; the per-company `stocks_get_expected_report` is scalar, so its `pagination` is null) | ## Pagination Paging is opaque, forward-only, cursor-based (`conventions/cursor.py`). - **Cursor anatomy.** A cursor is base64url of `{"v": 1, "k": []}`, where `k` is the sort-key tuple of the **last item on the previous page** — not a row id. Url-safe so it survives a query string untouched, versioned so the format can evolve, opaque so callers treat it as a token rather than parsing it. Ordering is **newest-first** (descending key); the next page is the items whose key sorts strictly *before* (older than) the cursor. - **The forward-ordered exception: calendars.** `macro_get_calendar` (#44) pages *oldest-first* over `(scheduled_at, id)` ascending, and its next page is the items sorting strictly *after* the cursor. A calendar is read forwards — the next release is the interesting one — so newest-first would put the far edge of a 93-day window on page one. `stocks_search_expected_reports` is the same shape for the same reason, over `(expected window earliest, cik)` ascending. Everything else about the cursor is unchanged, including that the key must be a total order. A forward-ordered surface uses a SQL keyset; `paginate_sequence` is descending-only by construction. - **Position, not row id.** Because the cursor is a position in a total order, a restatement that inserts or removes items between calls can neither error nor silently skip — the boundary just stays put. This is why the sort key must be a *total* order (e.g. statements break `period_end` ties with fiscal year then fiscal period), or limit-truncation would be nondeterministic. - **`has_more` / `next_cursor`.** `has_more` says whether the query matches more items past this page; `next_cursor` is the token for the next page, and is set when — and only when — more remain (`None` otherwise). - **Bad cursors are bad input.** A malformed, wrong-version, wrong-shape, or wrong-arity/wrong-type cursor raises `BadParameterError` (`bad_parameter`), never a silent empty page. An empty page *past the end* of a valid query is a normal success, not an error. Two ways to produce a page, one `Pagination` shape either way: - **`paginate_sequence(items, *, limit, cursor, key)`** over an already-sorted (newest-first) in-memory sequence — the right tool for small results the service assembles in Python (e.g. income statements grouped by fiscal frame). - **SQL keyset** (the `limit + 1` trick, ordering by the cursor key) for large tables where loading everything into memory is wrong. ## Point-in-time reads `as_known_at` is the cross-category parameter name for ingestion-time time travel: answer as if asked at instant T, using, for each observation, the newest vintage ingested at or before T. It is opt-in; omitting it preserves today's latest-value view unchanged. Point-in-time reads fail honestly when T predates the series' vintage-coverage start: `out_of_coverage` names the earliest supported instant rather than silently falling back to current values. Coverage starts when vintage capture began for the series: the migration-0009-style backfill instant for series that predate capture, or the first ingest for series added later. Values present at capture keep their original ingest `as_of`, so observations on a page can carry an `as_of` earlier than `vintage_coverage_start`. Revisions made before capture were overwritten and are unrecoverable, which is exactly why requests before the boundary are refused rather than answered from an incomplete log. A known series with no vintage rows fails point-in-time requests with `out_of_coverage` ("no vintage coverage yet"). In latest mode, an empty page for that same valid series remains a normal success. In point-in-time mode, `meta` sets `point_in_time: true`, echoes the effective `as_known_at`, and reports `vintage_coverage_start`. One platform point-in-time contract has two distinct axes: - `as_known_at` is the ingestion-knowledge axis: what *we* held at T, available to any category with vintage capture. - `as_reported` is the source-document axis, implemented on the three stocks statement surfaces: values as originally filed, anchored to an accession and before restatements. Set their `as_reported` parameter to opt in. Before the company's first successful vintage-era statement ingest, this mode fails with `out_of_coverage`; it never substitutes latest values. A tool may eventually offer both because they answer different questions. They MUST NOT be conflated or aliased. ## Errors Every tool-surface failure an agent can provoke is one of a small, fixed set of categories (`conventions/errors.py`), each with a stable machine `code` and an agent-actionable message. **The same exception hierarchy feeds both surfaces:** `conventions.mcp.tool_error_boundary` maps it to a FastMCP `ToolError`, and `conventions.http.register_error_handlers` maps it to a REST response, so an agent sees the same information whichever surface it called. | Exception | `code` | HTTP | MCP rendering | |---|---|---|---| | `UnauthorizedError` | `unauthorized` | 401 | emitted by auth middleware before tool dispatch | | `BadParameterError` | `bad_parameter` | 422 | `[bad_parameter] ` | | `UnknownEntityError` | `unknown_entity` | 404 | `[unknown_entity] ` (+ `Did you mean: …?` when `suggestions` set) | | `OutOfCoverageError` | `out_of_coverage` | 404 | `[out_of_coverage] ` | | `OutOfLicenseError` | `out_of_license` | 403 | `[out_of_license] ` | | `RateLimitedError` | `rate_limited` | 429 | emitted by auth middleware before tool dispatch (+ `Retry-After`) | | `QuotaExceededError` | `quota_exceeded` | 429 | emitted by auth middleware before tool dispatch (+ `Retry-After`, `reset_at`) | | `UpstreamStaleError` | `upstream_stale` | 503 | `[upstream_stale] ` | | `UpstreamRateLimitedError` | `upstream_rate_limited` | 503 | `[upstream_rate_limited] ` (+ `Retry after Ns.`) | | *(infra: `OperationalError`; replay-guard 503 emitted by auth middleware before tool dispatch)* | `unavailable` | 503 | `[unavailable] database unreachable — …` | Notes on the rows: - `unauthorized`, `rate_limited`, and `quota_exceeded` are emitted by the API-key auth middleware on both REST and hosted streamable-HTTP MCP requests before the request reaches either surface's handlers. The body still uses the canonical `{"error": {...}}` envelope because outer middleware is not covered by the normal FastAPI exception handlers. - `unknown_entity` is "well-formed identifier, nothing we hold" (wrong symbol); `out_of_coverage` is "entity known, but we don't ingest that *class* of data for it" — kept distinct so an agent can tell the two apart. `suggestions` (did-you-mean) lives on the base error and is reserved for entity search (#23). - `rate_limited` is our-surface throttling (per API key). It is distinct from `upstream_rate_limited`, which means a data provider throttled one of our connectors. - `quota_exceeded` is a per-account request budget across all owned API keys and attributed OAuth traffic, not a per-key burst limit. Its window is the UTC calendar month; `reset_at` is the first instant of the next UTC month. It is distinct from `rate_limited`, which controls short-lived per-key or per-subject request bursts. - `upstream_rate_limited` subclasses `upstream_stale` and carries `retry_after` (seconds); REST also emits a `Retry-After` header, MCP appends `Retry after Ns.` to the tool-error line. - `unavailable` has no exception class of its own. The REST handler stamps it on an infrastructure `OperationalError` when the database is unreachable, and the auth middleware emits it when the shell assertion replay guard cannot reach Redis. The message distinguishes those failures. Database loss can first surface as a query-time error because the stdio server and REST app both start without a database; replay-guard loss fails a shell assertion before dispatch so an unconsumed assertion is never admitted. The REST error body is `{"error": {code, message, suggestions?, retry_after?, reset_at?}}` (serialized with `exclude_none`, so an error only shows the optional fields it uses). MCP folds the **same** fields into the `ToolError` string via `render_tool_error` — the `code` in brackets, then the message, then any `suggestions`/`retry_after`/`reset_at` context. Two carve-outs are deliberate: - **FastAPI's own request-schema 422s** keep their native `{"detail": [...]}` shape. They are rejected by the protocol layer before our handlers ever run — symmetric with MCP's schema layer rejecting the same class of input. - **A service-raised `pydantic.ValidationError` gets no handler** and 500s loudly. A row failing its result model is a data-integrity failure, not caller error; the MCP boundary re-raises `ValidationError` *before* its `ValueError` arm could mask it (and the server's `mask_error_details=True` keeps internals out of the agent context). Integrity failures stay loud. An empty result set for a *valid* entity (e.g. no TTM rows yet) is a normal 200/success with an empty list — honest, not an error. ## Redistribution gate On the public surface, a payload is served only if every source attribution it carries is registered as publicly redistributable. Tools whose primary source is restricted refuse with `out_of_license`. Tools that mix restricted and clean sources degrade by excluding restricted-source values at query/build time; `meta.restricted_omitted` then lists the **distinct restricted source tags this request actually excluded** (`null` on the internal surface or when nothing was excluded). The envelope layer independently re-checks the final payload and refuses anything that still carries a restricted or unknown source — a degradation bug fails closed, never leaks. Restricted-derived values that carry no source attribution of their own (the company profile's price-coverage span) are gated in the service, the same degrade-and-disclose way. Fields literally named `source` are collected recursively and checked. A model whose `source` field is descriptive metadata about a source, rather than data from that source, may opt that field out via `SOURCE_TAG_EXCLUDE`; the first case is `DatasetFreshness.source` in the freshness registry, which also carries an explicit `non_redistributable` flag. The default remains fail-closed. API-key credentials declare their surface as `internal` or `public`. Local stdio MCP and direct service calls default to `internal`; hosted HTTP traffic runs **stateless** and takes the surface of the key on each request — tool calls never inherit another request's identity via session state.