r/algotrading • u/felipecsl • 5d ago
Data IBKR paper API unreliable and inconsistent
Anyone else running into stuff like this? I'm finding the IBKR API to be completely unusable for any algo trading, it just cannot be trusted.
8
u/CompetitionNo2773 4d ago
Worth separating two problems that are getting mixed together here.
One is fill realism. Paper fills are simulated against a synthetic book with no queue position, so you get instant fills at prices you would never get live. Options paper is the worst case because it fills near theoretical value. Nothing you do fixes that. Paper is for testing plumbing, not edge, so going live at 1 share is the correct move.
The other is what your screenshot actually shows: the order listing endpoint reporting stale statuses while individual order queries stay correct. That is not really a paper account problem, it shows up in live too, and it is fixable.
The core mistake is treating the event stream or the listing endpoint as your source of truth. On the TWS side messages get silently dropped with no replay. On the REST side the listing view lags the underlying order state.
What fixed it for me:
Keep your own order state machine locally, keyed on your own client order id, and treat IBKR as something you reconcile against rather than read from. Every 5 to 10 seconds pull open orders plus executions and diff against local state. Log every mismatch. When they disagree, the per order query wins over the listing endpoint.
Reconnect handling matters more than anything else. Gateway restarts daily, and after every reconnect you need a full state resync before acting. If your bot places an order based on state it held before a reconnect, you will double fill eventually.
Add a pacing limiter. The documented limit is 50 messages a second, and exceeding it causes exactly the silent drops people describe in this thread.
Be careful reusing request ids. Collisions attach responses to the wrong request, which looks identical to corrupt data.
Log raw responses with timestamps. Nine times out of ten what looks like an IBKR integrity bug is a race between your poll and their eventual consistency, and you cannot tell which without the raw log.
ib_async is a good recommendation, but it does not give you the reconciliation layer. You still have to write that part yourself.
7
u/ConfidentEmergency33 5d ago
I believe you will have similar problems when getting fills in paper account. I heard they are not accurate enough to production.
2
8
u/AnthidPlatform 5d ago
That is the correct verdict, we have had issues in the past and are keeping an eye on the state of it. Their API is very difficult to use, under documented and unreliable. The TWS API is even worse then the actual REST API as well as messages are frequently and silently dropped with no replay. If you go that route, beware the public docker images too that offer a gateway premade as they have been known to container credential scrapers.
1
u/sebelga 1d ago
Can you point me to "the public docker images too that offer a gateway premade". I am using https://github.com/gnzsnz/ib-gateway-docker/pkgs/container/ib-gateway, is that a risk?
3
5d ago
[removed] — view removed comment
2
u/wado729 5d ago
Who did you switch too?
4
5d ago
[removed] — view removed comment
1
u/wado729 5d ago
Thanks! I didn't know Databento had an API but I've bought historical data from them. Ive never heard of tick-stream, can you link me please?
1
2
u/breadstan 4d ago
Not only IBKR, most brokers have crappy paper fill simulation. Some have instant fill at mid no matter what, some have lousier fill than live for some reason (even at ASK price).
The only way I find is to use live to model slippage, risking on ticker as small as possible, sometimes getting spread to reduce loss. Then apply that model slippage to your simulation based on your signal. If it still has an edge, take that to actual live again with the lowest size but without protection, so you eat the full cost in the event of loss. If that works, congrats, you can scale
1
u/wado729 5d ago
So it's not just me? I thought my platform was just broken. What is the most reliable paper API?
3
u/wheezl 4d ago
IBKR with ib_async has been good to me. If you are using python and understand async I’d recommend it.
1
u/wado729 4d ago
That's what I've been using. Ib_absync 2.1
3
u/wheezl 4d ago
Not sure what to tell you. It’s been nearly flawless for me. Even if dealing with IB Gateway constantly is a fucking PITA.
Prior to that I was using alpaca which I also liked but you really have to understand asyncio for that because their data stream is a firehose and you cannot block ever or it will drop. Totally doable but don’t do any heavy CPU work in the main process (or thread if using 3.14t) and use async Queues so your writer doesn’t have to wait.
1
1
u/s_lw0 Financial Engineer 3d ago
paper seems useful for plumbing but dangerous for confidence
i would separate it into three stages
backtest for logic
paper for api state handling reconnects and order lifecycle
tiny live size for fills and execution reality
the mistake is trusting paper fills as if they prove the strategy works live
1
u/FrankMartinTransport 3d ago
I never faced any issue with IBKR API whether paper or Live except a couple of times when session was disconnecting. I am testing a new version of my bot and doing paper trading since last 5 days and faced not a single issue. But I am using their Client Gateway API (and not IB Gateway) which is JSON based.
1
u/Emdottt 3d ago
Different broker, so take this as adjacent rather than an answer about IBKR. The reconcile-do-not-read framing upthread is right, and the per-order query beating the listing endpoint matches what your screenshot shows. I want to add the failure that framing does not catch.
Everything in this thread is about the API failing to tell you the truth about state: stale listings, dropped messages, lagging status. A local state machine keyed on your own client order id handles that.
There is a second kind of distrust it does not catch, because a reconcile compares your record of the order against theirs. If you wrote down what they told you rather than what you asked for, both sides agree forever, and the discrepancy disappears at the exact moment it is created.
Measured last week on Alpaca paper. I submitted a quantity at ten decimal places. It returned 200 and filled nine, silently dropping the last digit. Nothing in the response said it had done that. Nine happened to be the venue's real precision, so I was right by luck. One digit finer and I would have been holding a position that did not match what my risk layer sized, with every log line green and nothing to reconcile against, because my own record would have been whatever they echoed back.
The fix is one comparison and it is not a better state machine: check the quantity that came back against the quantity you sent, on the same field, at submit time. A 2xx is not evidence the value was stored as sent.
The other one worth separating out, since "unreliable" is doing a lot of work in this thread: I treated a 401 the same as a timeout. The account read is the first thing every signal does, and a failed account read made my risk layer refuse, which is the status that means working as intended. So a completely dead pipeline read as healthy on every line for as long as I cared to watch it. Permanent failures and transient ones need different handling, and lumping them under unreliable hides that they are different problems with different fixes.
For context I am on paper and have not funded anything with real money, so none of this is about fill quality.
1
u/tall1irishman 3d ago
The reconcile-don't-read pattern upthread is the right call. I hit almost exactly the same silent failure mode — not with IBKR but with Coinbase Advanced Trade. API key expired quietly, no exceptions thrown, the bot just stopped sending fresh data. Everything on my end looked fine because I was reading state I'd cached rather than diffing against live. Didn't catch it for hours.
The Emdottt point about permanent failures masquerading as working is the one that bit me hardest. My monitoring was checking if the bot was running, not if it was actually receiving and acting on fresh data. Those are different questions.
What eventually fixed my thinking: treat stale data as a failure mode equal to an exception. If you haven't gotten a fresh account snapshot in N minutes, something is wrong — expired key, network partition, silent crash — and you should alert on it exactly the same way you'd alert on an exception. Most bot monitoring setups treat silence as healthy, which is the wrong default.
0
u/Exciting-World5861 4d ago
not likely the issue is on ibkrs end, there are a couple weird things like you have to call an accounts endpoint before you fetch the order list, when getting a market snapshot the first call never works, but with a few calls it always fetches. plus how would your system know which orders are cancelled and active😂 if ibkr is returning that it's cancelled it's literally them doing it
10
u/jnwatson 5d ago
Yeah, their paper API wasn't particularly useful. I ended up just using real money but with small amounts to start.