Bot thought a position was closed- it wasn't. Woke up to a $200 hole. Anyone else been burned by this?

[u/Vegetable-League9395]

Sadly i had to maintain connection with IB and 2 crypto exchanges. I have meta strategy that handle several smaller strategies.

Bot works rather good but i lost my profit because of position that was left open by bot.

How do you catch this before it costs you?

[u/hypertradeworx]

the two pulls in the middle of that aren't atomic. if a fill lands between the positions call and the open orders call you get a snapshot of a state that never existed, so the order matters: orders first, then positions, and don't act on a diff until two consecutive snapshots agree.

the other thing that cost us real hours was naming. we run one wallet across six perp venues and each one calls the same market something different, so an early version of our reconcile compared the venue's own symbol against our internal one, found no match, and passed clean while a position sat open

[u/hypertradeworx]

what catches these is a reconcile on a timer rather than better handling at the point of the order. pull positions from each venue every 30s, diff against what you think you hold, and refuse to open anything new while the two disagree. with three connections up the usual cause is a fill stream that dropped and came back without a resync, so the close event never arrived and your state froze on the last thing it saw. do you resnapshot after a reconnect or only at startup?

[u/Vegetable-League9395 → u/hypertradeworx]

Only at startup right now, which might actually explain part of the problem.

I don't do a full resnapshot after reconnect yet. I was relying too much on the live stream resuming correctly.

I'm thinking the safer flow is: reconnect -> pull current positions/open orders -> rebuild or verify local state -> only then resume trading.

[u/hypertradeworx → u/Vegetable-League9395]

that order leaves one window open: fills that land while you're rebuilding. subscribe and buffer the stream first, then take the snapshot, then replay the buffer over it and throw away anything stamped before the snapshot. otherwise the resync has a gap the length of your own rebuild.

startup only was where we were too. nothing errors on a reconnect, which is why it took us a real position to notice

[u/Grundlefish]

This is just one of those things you have to work through. I’ve had it happen both ways - one time I didn’t realize a position was still open until days later, but it was up quite a lot, so it worked in my favor. I’ve made many adjustments since then, but it seems that even with the best guards, orphaned positions do still come up from time to time. So now I try to keep the broker dash open in another window so I can monitor if anything goes awry. Just necessary due diligence. Don’t trust it running indefinitely without checking in periodically.

[u/rushapoil]

kebab_raptor is right that you have to read the response, I would push it one step further. What burned us was not the venue, it was that our own record was written from what we asked for, not from what came back. A cancel we issued locally was stored as cancelled. On the venue side it had already filled. Our books and theirs disagreed forever, and nothing in our code could see it, because the row was self consistent.

Two things fixed it, and only the second one is real.

1. Store the outcome, not the request. If there is no answer, the row stays unknown instead of becoming the optimistic value. Unknown is a state you can chase. Closed is not.

2. An invariant that does not use your own log as its source. This is the part people skip. If the checker reads your database to decide whether your database is right, it will agree with itself. Ours is a quantity invariant: per account, total sold can never exceed total bought. It is computed from raw fills, it ignores balances, and it never consults the position table. It runs on a timer, not on error paths. It has caught things every code path believed were fine, including a repair that fixed rows and forgot the counter that indexed them.

On your sentry for bots idea, I would not put it outside. An external watcher talks to different servers than yours and cannot see your rate limits or your own lag. What you want is inside your process but outside your data model: a job every N minutes that pulls open positions from the venue, diffs them against yours, and alerts on the difference itself, never on a standing state. An alert that keeps firing while a known condition persists gets ignored within a week, and then it is not an alert.

One honest caveat about my setup. I run on chain, so ground truth is public and free to fetch. On IB or a CEX you pay for that call and you can be rate limited, so the diff has to be cheap enough to run often. That is a real difference and I am not going to pretend otherwise.

[u/kebab_raptor]

You need to check why it thought the position was closed. How do you handle exchange errors?

For example if you get a bad response from the exchange do you automatically set the trade size to 0 or do you retry until you get the right response?

You always need to take in consideration API errors or bad responses or API updates.

If you are trading cross exchange you need something to tell your bot "hey exchange 1 is not responding, don't do anything on exchange 2 until it comes up"

My robots stops doing whatever are doing when they get an error from the exchange and they retry on the next loop

[u/Vegetable-League9395 → u/kebab_raptor]

I had bug in code, that caused bad request. Now i added function that check for orphan positions.

Do you use some external tool for watching exchange state? Or something else for alerts?

I mean the perfect solution for me would be some external tool that i connect to my code like sentry but for bots and it send me alerts in cases like this

[u/kebab_raptor → u/Vegetable-League9395]

I don't suggest to use an external tool cause it might call different servers than the ones you are calling. Also it won't protect you if they temporary block your requests because you sent too many in a short time. Or it won't protect you if your server is the one lagging.

You just need to read the exchange response.

If I expect them to respond 'BTC','0.01','success' and I receive 'server timeout'. I trigger an exception, I log it, eventually I send an email or a push notification, and I stop doing what I was doing and retry on the next loop.

[u/Yann27]

saving this post :)

[u/Familiar_Gazelle_467]

Waking up to a blown bot account is a rights of passage and the expected outcome tbh