# Lighter — operations: signer, preflight, observability, failure runbook

How to operate with a separate signer, what to verify at startup, what health information to monitor, a runbook for common failures with primary sources, and a micro-cycle for validation before going live. Facts were verified on the robinhoodchain instance.

## TL;DR

1. **The signer is a separate process on loopback** and a dependency of the trading process. The key lives only in the signer’s environment; the trading process calls it over HTTP with a bearer token.
2. **The trading process waits for the signer** for up to 90 seconds at startup (polling `/health` every 3 seconds) instead of exiting after the first failure.
3. **One key — one signer — one process.** Processes on the same L1 address share the write window; processes on the same key also share the nonce.
4. **Startup preflight:** metadata → signer (`/health`, `check_client`) → account owner (`l1_address` = expected address) → positions and account value. A mismatched key or someone else’s account aborts startup; an empty account is a warning.
5. **An initial burst of `HTTP 429` responses on reads** (for about a minute) is a read limit, not an error.
6. **Instrumentation must expose throttling:** placements held by the limiter need a separate counter, not the failure counter. Otherwise, the exchange can have fewer orders than expected while the system reports no errors.

---

## 1. Process layout

```
[trading process, TS]  --loopback HTTP, bearer-->  [signer, Python + lighter-sdk]  --https-->  Lighter instance
        |                                                        ^
        +-- public reads (orderBookDetails, account) ------------|--> Lighter instance
        +-- accountActiveOrders with auth token (token from signer /auth)
```

- **The trading process** knows the instance base URL, `account_index`, signer URL, its bearer token, and the expected L1 address. It does not have the Lighter key.
- **The signer** knows the base URL, `account_index`, `api_key_index`, private API key, bearer token, and bind address/port. Everything comes from the environment; nothing is stored on disk.
- The trading process depends on the signer: restart them together.
- **One key — one signer — one process.** Different processes on the same account share the address’s write window and disrupt one another’s accounting; on the same key, they also share the nonce.

---

## 2. Health and observability

What must be visible so that it does not have to be investigated manually:

| Metric | Why |
|---|---|
| `writeBudget: { used, limit, reserve }` | write-window occupancy at the time of the reading |
| placements held by the limiter — a separate counter | throttling is not a failure; mixing it with failures turns waiting for the window into an “incident” |
| signer: `/health` `{ ok, account_index, api_key_index, url }` | which account it is bound to and whether it responds |
| read-completeness signals (positions read, order list complete) | a truncated “empty” result must not pass as “empty” |
| write memory: sizes of `cancelled`, `placed`, and the 21734 rejection memory | diagnosing rollup echo |

To verify whether all of your orders are open, read the exchange directly (positions + `accountActiveOrders`), not a summary from your own process. A “placed — canceled” flap within seconds is visible only in the exchange’s order history.

---

## 3. Failure runbook

| Symptom | Primary source | Cause | Action |
|---|---|---|---|
| Many rejections without text; the text contains `23000` / `40 requests per 60 second` | exchange response through the signer | write window | limiter before sending, with a reserve; check whether all write types are counted (leverage, cancellations) |
| `21706` on orders that pass $10 | exchange response | lot-based `min_base_amount` | gate before sending; for a full close, bump above both minimums |
| `21734` / `too far from the mark` on repeated attempts at one price | exchange response | limit order too far from the mark | remember the rejected price for 5 minutes; retry when the market reaches it |
| Initial burst of `HTTP 429` responses on reads | transport log | read limit | wait a minute; if it lasts longer, look for unnecessary reads |
| An order “will not cancel”; cancellation is OK, order remains in the book | `accountActiveOrders`: `order_id` ≠ `String(order_index)` | rounded identifier | cancel using the `order_id` string |
| Duplicate orders; the same order is canceled dozens of times | `accountActiveOrders` + cancellation log | rollup echo | keep confirmed-write memory for 45 seconds |
| Position does not change after an IoC; order is “REJECTED” | positions before/after | spread wider than the cross, or IoC not yet sequenced | measure the spread; use a 1.5% cross |
| The Lighter key fails format validation and the process does not start | startup log | the key is not EVM, but validation expects EVM format | do not validate a Lighter key as EVM; prove authorization through preflight (`check_client` + `l1_address`) |
| `check_client failed` when the signer starts | signer log | key does not match `account_index` / `api_key_index` | check the indexes and key; do not start |
| “sub-account belongs to another address” | `account.l1_address` | wrong `account_index` | fix configuration; do not trade |
| Margin/ROE is off by 100×, false protective trigger | exchange arithmetic: `Σ ntl × imf/100 = margin req` | fraction scales | `/100` for the account, `/10000` for metadata |
| More collateral than expected (L/2 times more at target leverage L) | positions: `initial_margin_fraction` `"50.00"` | leverage was not set | `update_leverage` |

---

## 4. Micro-cycle before going live

1. Metadata: market count, `market_id` values for the required assets, decimals, and minimums.
2. Signer: `/health` → `ok:true`, with the correct `account_index` and `api_key_index`.
3. Owner: `account.l1_address` = expected address.
4. A GTT order slightly above $10 at a distant price → appears in `accountActiveOrders` (after a lag) → `order_id` ≠ `String(order_index)`? → canceled using the `order_id` string → disappears → `total_order_count` returns to its prior value.
5. Minimum-size IoC entry on a liquid market → position changes → resting reduceOnly larger than the position → position goes exactly to 0 and does not reverse (experiment from `orders.md` §3).
6. `update_leverage` on a market → the position/account `initial_margin_fraction` changes.

---

## Pitfalls

| What breaks | Why | Correct approach |
|---|---|---|
| Trading process exits when the signer restarts | exits after the first `/health` failure | wait for the signer at startup (up to 90 seconds) |
| Fewer orders than expected on the exchange, with no errors | throttling is invisible to instrumentation | separate counter for held placements |

---

## Open questions / not verified

- The signer and `lighter-sdk` build on Windows and macOS have not been verified.
- One signer for multiple `account_index` values (sub-accounts) of one address has not been tried; according to the error text, they share the write window.
- The effect of clock skew has not been verified (the SDK nonce is a counter, not time).
- Not verified live: codes other than 23000/21706/21734, GTT after 28 days, deduplication by `client_order_index`, and fees.

---

Facts verified through 2026-09-16.

---

<!-- license-footer -->
_© markpaper authors. Licensed under [CC BY 4.0](LICENSE.md): when publishing or adapting this material, credit “markpaper — Lighter knowledge base” and link to the original and the license._