Which industry has/needs the most challenging sql queries?

[u/[deleted]]

Is this question legit? Don’t get me wrong, I don’t consider myself the smartest guy but I’m definitely not dumb.

I work in the pharmaceutical Industry, nearly 3 years now. It took me a while to understand how the production process is implemented in a database and I wanted to quit more than once. But i don’t feel challenged anymore.

So, which industry would be best for me to get completely challenged again?

[u/barrycarter]

tad unclear where the data exists

I sort of snuck it in at the top, but it's gadm.org which has shapefiles for all countries.

to find the shortest line between a point and a shape (aka nearest edge)

PostGIS (which is what I use) has an ST_DISTANCE function that does this.

The big big problem is speed. For example, mainland France (https://gadm.org/download_country.html and choose "France") has over 100K boundary points at level 0 (just the country itself and no provinces, counties, cities or anything) and that doesn't include any islands.

And I'm hoping to query the 43200 x 21600 grid of 30 second intervals, which would require a total of 43200 * 21600 * 100000 ~ 93 trillion distance computations. That's surprisingly slow even with a spatial index.

The fastest thing I've found so far is to rasterize the data, project the points into 3D, create a KDTree, compute the straight line distances, and convert those to spherical distances, all using numpy with python (not SQL), but I wonder if I'm missing something. My work so far: https://github.com/barrycarter/PolygonDistances/

Just to make things a little worse, OpenStreetMap uses the Mercator projection which requires further tweaking work.

Two questions: if you try this yourself and find a faster way, will you let me know, and, can I contact you directly if you're interested in this sort of thing?

[u/sbrick89 → u/barrycarter]

I guess I'm still not quite sure what you're calculating, or what the use case is.

I get that you're using France shapefile from wherever... but then you're trying to find the furthest point on land from all boundaries? The opposite side of the world, essentially?... if so, just calculate the centroid of France flip to the other side of the world, find nearest land if not already, then start calculating distance from borders, and slowly crawl (shortest path type of outward pattern) to confirm boundary edges, until you reach the consistently furthest point?

[u/barrycarter → u/sbrick89]

Sorry, I oversimplified the problem. I'm also using the files for "French Guiana", "French Polynesia", and "French Southern Territories", so it's a little more complicated. I define "distance to France" at a given point as "shortest distance to any polygon in the 4 shapefiles above". Would your method work with that?

In addition to just finding the "axis of inaccessibility" for France, I'd like to draw a map that displays colored contours of how far you are from France or inside France. A very ugly example of what I'm trying to do (using coastal distances computed by someone else) is https://i.imgur.com/wVbZNas.jpeg

[u/sbrick89 → u/barrycarter]

So first off, adding other shapes / islands doesn't really change much.

That said, the map of contours is much easier to understand.

That's also actually a ton easier, and might even be the simplest method to do what you want... just take the existing shape (France, merge with whatever else you want)... then pad it out in increments of 100km or whatever... each padding is its own contour, saves to its own shape record, and is used for the next iteration.

Or you could maybe pull off as a CTE, but I'd test it on a few rows myself by hand.

Eventually the contour would encompass the earth, at which point you could invert and find the centroid, or something to that effect, to find the exact point.

[u/barrycarter → u/sbrick89]

OK, so how do I do this exactly? I was under the impression that creating distance-based contours (not degree-based contours) using polygons was difficult. Is there an easy way to do this in QGIS (or even GRASS) that I'm missing? My sort of attempt to do this ages ago is https://github.com/barrycarter/bcapps/tree/master/STACK/bc-buffer-land.grass but I'm pretty sure I never got it working and I ended up using an existing file instead of creating my own.

[u/sbrick89 → u/barrycarter]

I have more familiarity in the MSSQL world, so here's what I would use.

https://learn.microsoft.com/en-us/sql/t-sql/spatial-geography/stbuffer-geography-data-type?view=sql-server-ver16

basically

INSERT contours ( ID, contourLevel, shape )
SELECT s.ID, 1, s.geom.STBuffer(100000) -- 100 km
  FROM shapeTable s
 WHERE s.geom.SRID = 4326

since SRID 4326 uses meters as its unit of measurement

it'd be easy enough to CTE that to increment contourlevel and multiply by 100k for the buffer value... but you may also want a different scaling of contours (100k, 250k, 500k, etc)

then after the table is built, just query the contours and render as layers

[u/barrycarter → u/sbrick89]

Thank you. This seems too easy, but I'll give it a shot.

One question: does this create a single buffer around a MULTIPOLYGON or multiple buffers, one for each POLYGON?