Lighter — operations: signer, preflight…

How to operate with a separate signer, what to verify at startup, what health information to monitor, a runbook for common failures with primary sources, and a micro-cycle for validation before going live. Facts were verified on the robinhoodchain instance.

TL;DR

The signer is a separate process on loopback and a dependency of the trading process. The key lives only in the signer’s environment; the trading process calls it over HTTP with a bearer token.
The trading process waits for the signer for up to 90 seconds at startup (polling /health every 3 seconds) instead of exiting after the first failure.
One key — one signer — one process. Processes on the same L1 address share the write window; processes on the same key also share the nonce.
Startup preflight: metadata → signer (/health, check_client) → account owner (l1_address = expected address) → positions and account value. A mismatched key or someone else’s account aborts startup; an empty account is a warning.
An initial burst of HTTP 429 responses on reads (for about a minute) is a read limit, not an error.
Instrumentation must expose throttling: placements held by the limiter need a separate counter, not the failure counter. Otherwise, the exchange can have fewer orders than expected while the system reports no errors.

1. Process layout

[trading process, TS]  --loopback HTTP, bearer-->  [signer, Python + lighter-sdk]  --https-->  Lighter instance
        |                                                        ^
        +-- public reads (orderBookDetails, account) ------------|--> Lighter instance
        +-- accountActiveOrders with auth token (token from signer /auth)

The trading process knows the instance base URL, account_index, signer URL, its bearer token, and the expected L1 address. It does not have the Lighter key.
The signer knows the base URL, account_index, api_key_index, private API key, bearer token, and bind address/port. Everything comes from the environment; nothing is stored on disk.
The trading process depends on the signer: restart them together.
One key — one signer — one process. Different processes on the same account share the address’s write window and disrupt one another’s accounting; on the same key, they also share the nonce.

2. Health and observability

What must be visible so that it does not have to be investigated manually:

Metric	Why
`writeBudget: { used, limit, reserve }`	write-window occupancy at the time of the reading
placements held by the limiter — a separate counter	throttling is not a failure; mixing it with failures turns waiting for the window into an “incident”
signer: `/health` `{ ok, account_index, api_key_index, url }`	which account it is bound to and whether it responds
read-completeness signals (positions read, order list complete)	a truncated “empty” result must not pass as “empty”
write memory: sizes of `cancelled`, `placed`, and the 21734 rejection memory	diagnosing rollup echo

To verify whether all of your orders are open, read the exchange directly (positions + accountActiveOrders), not a summary from your own process. A “placed — canceled” flap within seconds is visible only in the exchange’s order history.

3. Failure runbook

Symptom	Primary source	Cause	Action
Many rejections without text; the text contains `23000` / `40 requests per 60 second`	exchange response through the signer	write window	limiter before sending, with a reserve; check whether all write types are counted (leverage, cancellations)
`21706` on orders that pass $10	exchange response	lot-based `min_base_amount`	gate before sending; for a full close, bump above both minimums
`21734` / `too far from the mark` on repeated attempts at one price	exchange response	limit order too far from the mark	remember the rejected price for 5 minutes; retry when the market reaches it
Initial burst of `HTTP 429` responses on reads	transport log	read limit	wait a minute; if it lasts longer, look for unnecessary reads
An order “will not cancel”; cancellation is OK, order remains in the book	`accountActiveOrders`: `order_id` ≠ `String(order_index)`	rounded identifier	cancel using the `order_id` string
Duplicate orders; the same order is canceled dozens of times	`accountActiveOrders` + cancellation log	rollup echo	keep confirmed-write memory for 45 seconds
Position does not change after an IoC; order is “REJECTED”	positions before/after	spread wider than the cross, or IoC not yet sequenced	measure the spread; use a 1.5% cross
The Lighter key fails format validation and the process does not start	startup log	the key is not EVM, but validation expects EVM format	do not validate a Lighter key as EVM; prove authorization through preflight (`check_client` + `l1_address`)
`check_client failed` when the signer starts	signer log	key does not match `account_index` / `api_key_index`	check the indexes and key; do not start
“sub-account belongs to another address”	`account.l1_address`	wrong `account_index`	fix configuration; do not trade
Margin/ROE is off by 100×, false protective trigger	exchange arithmetic: `Σ ntl × imf/100 = margin req`	fraction scales	`/100` for the account, `/10000` for metadata
More collateral than expected (L/2 times more at target leverage L)	positions: `initial_margin_fraction` `"50.00"`	leverage was not set	`update_leverage`

4. Micro-cycle before going live

Metadata: market count, market_id values for the required assets, decimals, and minimums.
Signer: /health → ok:true, with the correct account_index and api_key_index.
Owner: account.l1_address = expected address.
A GTT order slightly above $10 at a distant price → appears in accountActiveOrders (after a lag) → order_id ≠ String(order_index)? → canceled using the order_id string → disappears → total_order_count returns to its prior value.
Minimum-size IoC entry on a liquid market → position changes → resting reduceOnly larger than the position → position goes exactly to 0 and does not reverse (experiment from orders.md §3).
update_leverage on a market → the position/account initial_margin_fraction changes.

Pitfalls

What breaks	Why	Correct approach
Trading process exits when the signer restarts	exits after the first `/health` failure	wait for the signer at startup (up to 90 seconds)
Fewer orders than expected on the exchange, with no errors	throttling is invisible to instrumentation	separate counter for held placements

Open questions / not verified

The signer and lighter-sdk build on Windows and macOS have not been verified.
One signer for multiple account_index values (sub-accounts) of one address has not been tried; according to the error text, they share the write window.
The effect of clock skew has not been verified (the SDK nonce is a counter, not time).
Not verified live: codes other than 23000/21706/21734, GTT after 28 days, deduplication by client_order_index, and fees.

Facts verified through 2026-09-16.

© markpaper authors. Licensed under CC BY 4.0: when publishing or adapting this material, credit “markpaper — Lighter knowledge base” and link to the original and the license.