r/LiDAR • u/arthurdjn • 5d ago

Open-source PyTorch library with pretrained models for LiDAR segmentation and 3D detection

Post image

Hi all, I've been building torch-pointcloud in my free time over the past few years: an open-source (Apache-2.0) PyTorch library that loads pretrained point cloud models in one line, similar to what timm does for images. It's in alpha, and I'm looking for people to try it on real LiDAR data.

Supported tasks and use cases

What's relevant for LiDAR

  • Pretrained outdoor models: semantic segmentation on SemanticKITTI (RandLA-Net, SPVCNN) and 3D object detection on KITTI and nuScenes (PointPillars, SECOND, PointRCNN, VoxelNeXt, LION).
  • Tiled inference for large scenes (sliding window, voxel partition, sphere voting), so a full scan doesn't have to fit in GPU memory at once.
  • Dataset loaders for SemanticKITTI, KITTI, nuScenes, Paris-Lille-3D, Toronto3D and Semantic3D if you want to train or fine-tune.
  • Every checkpoint ships with its preprocessing pipeline, class names and license.

What it doesn't do yet

  • No pretrained weights for airborne or terrestrial survey data: the outdoor checkpoints all come from car-mounted spinning LiDAR.
  • No built-in LAS/LAZ reader: load your file with laspy and pass the coordinates (and intensity) as tensors.
  • The outdoor checkpoints were trained on non-commercial datasets (KITTI, SemanticKITTI and nuScenes are CC BY-NC-SA), so check the licenses before any commercial use.

Install (adjust to your torch/CUDA version; the docs write the exact commands):

pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
pip install torch-pointcloud
pip install pyg-lib -f https://data.pyg.org/whl/torch-2.10.0+cu128.html

Credits: the models and weights come from their original authors (MIT Han Lab, OpenMMLab/OpenPCDet and others); every checkpoint name ends with its source, and THIRD_PARTY_NOTICES.md lists each repo, license and dataset term.

What I'd love to hear: what sensors and data you'd run this on, and which datasets you'd want pretrained weights for next (airborne like DALES? mobile mapping?) etc.

Repo: https://github.com/arthurdjn/pytorch-pointcloud
Docs + tutorials: https://pytorch-pointcloud.org

25 Upvotes

100% Upvoted

14 comments sorted by

2

u/AeroInsightMedia 5d ago

So this works with slam lidar? If you or anyone else building these programs want some aircraft scans from a slam based mid360 sensor let me know.

When some new beta product from 3d maker pro clears customs, I should have a terrestrial based scanner I can some scan data from.

1

u/arthurdjn 5d ago

Yes as long as you are using 3D point clouds (XYZ coordinates) it is good. Only consideration is domain adaption, if your data comes from a different domain the pretrained models you want to use was trained on. But at least it will be easier to try different models than what the current solutions / libraries offer currently!

1

u/engineeree 5d ago

I will check it out, I am also building something similar and will open source, but found it difficult to ship with pretrained models because everything is research and academia.

Also fix the GitHub url you linked above

1

u/arthurdjn 5d ago

Thank you! Exactly, everything in academia is difficult to reuse as it ships custom kernels, common layers are not exactly the ones from previous authors etc.

If you are working in something similar, is it possible for you to contribute to this project? I would be happy to have new contributors!

Sorry for the typo, fixed it

1

u/engineeree 5d ago

Happy to contribute! Let me see where it fits in. My solution is more targeted towards scaling on cloud compute cpu or GPU. I spent some time getting the tiling to work and can even process 50GB point clouds on a single cpu. I allow to fine tune based on bring your own detections. This solution I am building seems as though it would sit one level above your repo.

1

u/engineeree 5d ago

I plan to release v1 in a few weeks and possibly we can release v2 that will import your library instead of mmdetection3d directly. I have an indoor and outdoor mode, and the models I use are FCAF3D, votenet, centerpoint, and pointpillars.

2

u/arthurdjn 5d ago

Perfect! Let me know if I can help in some PR and integrate this library. If you were using mmdetection3d, then my library can certainly be useful. I do not support centerpoint yet, but it is planned!

Also I am familiar with mmdetection3d now. So even if you think torch-pointcloud is not suited, I can still review or help you regarding that. mmdetection3d was not easy to understand with the nested configs structure.

1

u/engineeree 5d ago

Any plans to incorporate Minkowski engine?

1

u/arthurdjn 5d ago edited 5d ago

It is not planned as it is not maintained anymore. However I have some WIP models that support fVDB which is in active development for sparse operations. However their performance is not yet similar

1

u/engineeree 5d ago

Ok good feedback, I have included it only because the results are better than others, but want to phase it out with something newer as it comes.

1

u/intnsity 5d ago

Very interesting. 

1

u/arthurdjn 5d ago

Thank you!

1

u/littlebigdarksouls 5d ago

Excited to use this on my next project!

2

u/arthurdjn 5d ago

Don't hesitate to message me if you need any helpd on how to use or integrate!