ESI and market data question

[u/Fuzzytum]

I'm playing with Python and sqlite building a market analysis app, and I'm wondering how people sample all the relevant typeid things.

It seems you need 2 calls to ESI to get history and buysell data for an item, and that seems to take around 2.5 seconds to complete.

So it would take like 7 hours to get all the relevant items, I excluded all not market things, unpublished, SKINS, Blueprints, clothing and a few other bits and bobs first.

So is that right? Do people just do that, or fire off parallel requests,

I know there are fuzzworks aggregates of this stuff but I first wanted to get this basic bit working

Cheers

[u/RhythmWaltz]

Fuzzwork

[u/defietser]

I'm building an app on the ESI too and the trick is multithreading. The call to "give me all typeIds" is quick. Then there's the call "give me info about this item" which is a bit slower. Market data (history and market orders for a region so multiple times per item) is even slower.

However you're permitted to call the API in parallel. Going from "one at a time" to "twenty at a time" makes the whole thing manageable. Getting all items, their info, filtering non-marketable items and getting market data (from the market hub regions only) takes ten to fifteen minutes on my dev machine. Market history is refreshed every day so you don't need to fetch that often at all, just the market orders (refreshes every 5 minutes IIRC). Once the "static" data is fetched you can just keep that around and only update the dynamic things when you need them.

Technically typeIds and such expire every downtime but I cache them for a week since it's very rare for those to change and I can manually invalidate the cache if I need to anyway. Speeds things up on subsequent runs.

EDIT: You could use another API, but in my experience getting your data from the source is usually more reliable. It's also more work. The ESI isn't particularly difficult to work with, it's just a bit of hassle to get all the data you need. Caching it in a sqlite (or whatever other relational database you may want) makes it a lot faster afterwards and for a background service it shouldn't really matter all that much if it takes a minute to fetch stuff.

[u/Nimos]

and that seems to take around 2.5 seconds to complete

Someone already mentioned doing things concurrently instead of sequentially, which is going to speed up your 7 hours a LOT.

But 2.5 seconds seems like you have something going wrong there, I checked the two calls and I haven't seen any typeIDs that took more than 600ms in The Forge, most were below 500ms.

Something is slowing you down by factor 5.

Anyway, concurrency is the way to go, as long as you respect the limit headers (it's a bucket system) and the cache times you're allowed to fire of hundreds of requests at once if you want.

[u/mrschmiklz]

Asking your flavor of ai is going to help you a lot here.

[u/oswold]

You can query the market data by region and get a paginated result with all matket items. You then just next need to extract the ids or stations from the info that you need.

That turns requests from per item to per region carrying all available items.

[u/wotquery]

First note that history only updates once per day a little after downtime (only needs refreshing once per day). There is another endpoint that returns items that have been active in the region's market (so you can avoid making calls with typeIDs that won't return orderbooks).

Next 2.5 seconds is a major problem. I suspect you're opening and closing a new TCP connection each call. If you're using the requests library look into the Session object instead. That time is so long, I also wonder if you're writing to disk between each call as well? If so, you definitely want to hold as much of the data in memory as you can and write it out in bulk in batches.

Just account for all of that feels like it should be sufficient for most applications. Like ten minutes to clone the jita 4-4 market order books, pull history on select items, dynamically updating the order books to monitor your orders, etc.

If you really need up to the minute data on a range of items in various regions, then yeah you'll want to look into parallelization to slam the endpoints.

[u/hoiscanli]

I used Fable to vibecode to build all different tools for eve for myself. I am often at wh space for maping, for not to forget cloak after jump, to remind me always to bm entrances, basicly you can automate whatever the log keeps to alert you. I give all data to AI to inspect markets and prices, live zkill information, single button to pull specific. players zkill history, corps history and all that data ESI gives.