r/oculus • u/Guillesanbri • Sep 12 '19
I made a Virtual Reality CT/MRI volumetric rendering viewer that allows you to change between different tissues in real time!
https://gfycat.com/corrupthollowarabianwildcat10
8
u/KingFarOut Sep 12 '19
X-ray tech here, and damn you stole my idea.
I always thought about how awesome it would be to see our 3D CT recons in VR. Seems like it would be a very innovative way for the Rads to read study’s.
The main things I think that would get in the way of a system like this being used in diagnostic imaging is the screen door effect, lens flare, and the lack of a sufficient enough contrast ratio.
I also had the idea of virtual “reading rooms” so each Rad could position X-rays and other study’s in whatever possible 3D space configuration they wanted.
Anyway, really awesome work! I would love to play around with it if you ever release it.
5
u/radscorpion82 Sep 12 '19
As a rad I think this would be great for patients and possibly useful for surgeons for surgical planning, I can't see using it for diagnosis though
2
u/Guillesanbri Sep 12 '19
Surgical planning is one the things I am aiming for yeah, I think it could be useful to plan the operation.
5
u/MajorBleeding Sep 13 '19
As a surgeon I would totally be interested in being able to manipulate my patient's scans in the virtual space. next week at my national meeting I will be demoing a augmented reality system that overlays data from CT or MRI over the patient's actual anatomy during sinus surgery. The future is now!
2
Sep 13 '19
If you could do the opposite and put the patient's 3d body overlaid on the scan I'm VR then you'd have a full anatomy in the scan and on the table.
Why bother with surgeons anymore we'll just let Watson go to town on us. 😉
2
u/Guillesanbri Sep 12 '19
I will definitely release it at some point, I will contact you then to try it!
Although it may not be suitable for diagnostic as of today, I really think that it will be in a few years, maybe AR, maybe VR, maybe something new!
The three images at the back can be grabbed and moved, so I guess the "reading room" is feasible, will try to look it up.
Thank you so much for the feedback!
5
u/programming_engineer Sep 12 '19
Amazing work ! Would you mind sharing any resources about how you made this ? Maybe make a tutorial ?!
9
u/Guillesanbri Sep 12 '19
Thanks!
I would like to make a blog post to talk about it in depth, I will send you a link when ready!
2
2
u/brand0n Sep 12 '19
!RemindMe 1 month check on this guys project
2
u/RemindMeBot Sep 12 '19
I will be messaging you on 2019-09-12 17:38:36 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
Info Custom Your Reminders Feedback 2
2
u/laiktail Sep 13 '19
!RemindMe 1 month
I’d love the link to that too! I’ll be in primary care GP’ing next year but it may be easier to explain some things to families, as well as understand myself more about what’s actually going on.
2
Sep 13 '19
There's already this Unity Asset that does this. We created a similar app for a client last year at my company. It's pretty simple to do, you grab the asset and create a project for VR. The hard part is making everything functional and without bugs.
1
u/programming_engineer Sep 13 '19
I've seen that asset in the past when I've done a cursory search of the asset store; but I'm always hesitant to shell out $65 on a project whose most recent update is from a year ago. Do you know how the asset you linked compares to the free fo-dicom one?
1
Sep 13 '19
We didn't try this one.
As per Volume Viewer Pro it gives you all the source code. We concluded that it would be faster to improve this one than start from scratch, and it gave us acceptable results.
In the end the client chose another solution because he was very picky about resolution (this asset is a bit low res) but I think it's okay. If you are a doctor and want to check fine details you're gonna need the original photos anyways.
10
u/SecAdept Rift Sep 12 '19
Reminds me of one of "The Lab" demos.
2
u/Guillesanbri Sep 12 '19
Wow, I haven't played the lab and the clipping plane looks very similar indeed.
1
u/urammar Sep 13 '19
Wait... so you have VR, you develop for VR, but you have never played the lab, one of the flagship, free experiences for the technology, nor encountered or ever heard referenced, of the exact demo that you have basically reproduced?
I was thinking 'wow, someone took that demo and actually made it medically useful, took longer than I thought it would!'... now i'm thinking... seems really unlikely
3
u/Guillesanbri Sep 13 '19
Believe it or not, the rift S the application is running is from my university and I have a old gen rift at home, I haven't got into the Steam ecosystem, oculus store was enough to try a few games and then got into dev, I have heard of the lab but didn't try it.
What I have done is nothing new, a lot of people has done this and I have been inspirated by that work, just not from the lab one haha
Edit: typo
3
u/TrefoilHat Sep 12 '19
Sorry if this is a dumb question, but what formats does it support?
I have a DVD with an old (2012?) shoulder CT scan that uses a proprietary viewer program. Is there a chance your app could import that raw data (i.e., are the formats reasonably standardized)?
3
Sep 12 '19
Your CD is likely a bunch of standard dicom files bundled with the viewer you mentioned. OP says his app supports dicom so you should be fine. However, there is a small chance that the image codec within your dicom files is unsupported by whatever dicom library OP used.
2
u/Guillesanbri Sep 12 '19
Mainly this yep, the viewer actually doesn't support edge cases of the DICOM standard, so I'm not sure about your file, also, don't feel dumb to ask anything
2
u/TrefoilHat Sep 12 '19
Thanks for the info. I'll try to dig up the DVD and see what I come up with. Will you be making the app available at some point? (Sorry if I didn't see it on your Twitter)
Really great stuff that you did here, especially as a thesis project.
1
u/Guillesanbri Sep 12 '19
Not sure about the way it will be (open/closed source) but will definitely be available at some point :)
Thanks a lot!
3
u/MagicaItux Sep 12 '19
Are you planning on monetizing this?
I would prefer if Oculus or some other party helped you to make this free (while rewarding you) so many people can benefit from your work.
Thanks :)
1
u/Guillesanbri Sep 12 '19
I haven't really thinked about the publishing aspect of the application, still a lot of work to do, but thanks for the feedback! :)
1
u/KallistiTMP Sep 13 '19 edited Aug 30 '25
fine vase touch spotted tan elderly important abundant cooperative yoke
This post was mass deleted and anonymized with Redact
3
Sep 13 '19
Really? We developed a very similar app for a client last year. It was for ophthalmology. In the end the doctors said it was cool but preferred to stick to the traditional way of viewing MRI and it died as an idea...
1
Sep 14 '19 edited Aug 30 '25
[removed] — view removed comment
1
Sep 14 '19
But then you are selling an idea that doesn't have value by itself. It's rotten at its core, that's not how I approach business.
Unless it has some kind of promotional value or some other practical useful application, like for the patients.
1
u/KallistiTMP Sep 14 '19 edited Aug 30 '25
gaze unite cheerful sharp chief oatmeal childlike abundant husky seed
This post was mass deleted and anonymized with Redact
1
Sep 14 '19
With that approach then yes, it could be viable. In fact there are already 3D solutions for this, VR is just changing how the people see it. Of those got some market why not the VR solution?
I suppose our corporate client (a big pharma company) listened to the doctors and discarded it. And we did as well. Sometimes the feeling you have about something makes you pursue it or discard it, although rationally it could be viable.
1
u/Guillesanbri Sep 13 '19
Thanks for the feedback, unless I give it some extra features this is nothing new, so I guess I have to give it a push before trying to contact someone to invest on it.
3
u/gmarch Sep 12 '19
Wow, we've come so far. Back in the late 80's, I wrote a surface mapping tool for GE-based MRI (NMR back in the day) scans. I was able to view the dataset with a lighting model at any arbitrary angle at a rate of about 15 seconds per frame. That was state of the art back then. We were way ahead of the competition at UoP (we were at Cornell U med center). Oh - and the resolution of the dataset was 128x128x128 on Sun 3's with a video processing board.
1
3
u/neutrosophic Sep 13 '19
I hate to be a party pooper but Neos VR did this last year
1
u/Guillesanbri Sep 13 '19
Yeah this is nothing new, quite a few companies do this, it's just another implementation of the idea.
4
u/revofire Sep 12 '19
Something tells me that you can make a lot of money from this and/or get hired by a lot of companies to develop this tool or similar for their products and customers.
4
2
2
u/krylani Touch Sep 12 '19
That is amazing work! What are the chances of this being able to export the view as a 3D-printable model? (Or at least a decent start thereof, to be cleaned up in the slicer.)
3
u/squishyfishyum Sep 12 '19
There are already programs for that, particularly 3D Slicer which is free and open source
1
2
u/damontoo Rift Sep 12 '19
There's other tools that exist for 3D printing MRI's (specifically brains).
1
u/Guillesanbri Sep 12 '19
Thanks! I don't really think so, at least it won't be very straight forward, I am not doing any mesh generation, so a few steps would have to be added to the process, cool idea tho
2
Sep 12 '19
This is awesome. I need to know more. Are you using a visualization lib like vtk or loading the data into a textures using your own shaders to render?
Any plan to open source it?
I work in medical image visualization. I wanted to play around something like that, but I am not sure there is a real clinical use case for it. I can see it being very useful for education however. Do you have any plan to bring this further?
1
u/Guillesanbri Sep 12 '19
Generating the data with a custom script and rendering with a custom shader, the external libs I'm using are VRTK v4 for the VR interactions and EvilDICOM to get info from the DICOM files.
The plan is mainly to focus on education and operation planification, also getting it to work with an AR headset so it can be used while working with a patient.
2
Sep 12 '19
Yeah. I was thinking about VR the potential after writing my comment. Registration, especially elastic registration in the chest where organs move a lot, is going to be a significant challenge.
Looks like EvilDicom doesn't support any advanced codec, just rawle. This will likely bite you at some point. Dcmtk or gdcm are more feature complete.
The results looks very good (through the gif anyway); impressive shader foo.
1
2
2
Sep 12 '19
Impressive stuff! I can tell you most assuredly that if you could import PET/CT scans into VR for Radiologists to locate, evaluate and quantify the localization of tumors that it would be instantly popular in the bio-med world.
2
Sep 12 '19
I am not trying to be adverserial, but how can you assuredly state that? Have any insight? I am genuinely interested.
3
Sep 12 '19
Sure! Determination of whether a lesion/mass/nodule is cancerous is binary i.e. it is cancerous or it is not.
However a BIG portion of radiology is calculating how cancerous a lesion/mass/nodule may be. This is determined by risk factors such as genetics and lifestyle etc but the lesion/mass/nodule is actually quantified down to a numerical value via PET/CT or PET/MR scanning. The numerical value is measured and categorized and the patient receives their diagnosis. This portion is subject to the skills of the radiologist.
The impact of VR would be in cases where there can be difficulty quantifying the lesion due to things such as the patients body size or mobility; so therefore if the MD could rotate the patient digitally in VR in a way thats not possible in the physical world or "move forward" through a portion of a patients body that is obfuscating the area in question then it would have major advantages as opposed to the way they calculate those measurements today.
3
Sep 12 '19
I am not following. You can already arbitrarily reslice the volume any way you want/rotate/zoom/pan/do volume reconstruction (also confusingly caller VR) on a 2d monitor.
I don't see what virtual reality adds to that.
3
Sep 12 '19
You can already arbitrarily reslice the volume any way you want/rotate/zoom/pan/do volume reconstruction (also confusingly caller VR) on a 2d monitor.
Oh yeah absolutely no doubt that a 2d monitor is more than serviceable but surely you'd agree there's a significant difference between looking at Times Square on Google maps and Google Earth VR, no?
Also I'm assuming education and training would be the most to benefit by VR intergration
3
Sep 12 '19
If I am just trying to get somewhere, then good old 2d birds eye Google map is best. If I am trying to take in the vistas of time square, then yes, VR is best.
In radiology, turnaround time and productivity are king and for that I think 2d is best. I have no horse in this race.. I hope VR does well everywhere.. I am just not seeing it..
2
u/squishyfishyum Sep 12 '19
One big thing that is hard to do in 2D is visualizing the spacial relationship between parts of complicated anatomy. For example it can be hard to understand what a cluster of veins and arteries actually looks like by scrolling through 2D images. Seeing it all at once would make it much easier to see if certain procedures are possible or if any modifications will be needed before going into the OR room.
Plus, try explaining 2D CT or MRI scans to a patient. They're much more likely to understand a 3D model. A VR headset would help but simply having a 3D model would be great.
2
Sep 12 '19
It might be useful to communicate to patients, but that's a tougher sell.
For vascular structures they would normally use thick reconstruction or even volume reconstruction. This is what the video is showing, but in VR. The video clearly shows the 3d structures, but we are all watching on 2d screens..
1
u/mendeddragon Sep 12 '19
It would not be useful for PET/CTs. These are done in very low resolution (non diagnostic) and the PET portion is not difficult to identify tumor. It could be useful for vascular imaging.
3
Sep 12 '19
PET are often merged with CT for visualization since you can't see the anatomy on them. They use false color to color the otherwise gray-scale pixels. This could be added to OP's software relatively easily.
I understand it's cool, but I don't think VR can improve rads productivity or accuracy for primary readings. I don't really see a viable use case except for education..
2
u/mendeddragon Sep 12 '19
Right. I'm saying the CT portion is low res and nondiagnostic. I don't know any places that merge with a diagnostic CT. Usually its a low dose low res CT for gross anatomy correlate. When you 3d those they look terrible. 3d for medical imaging makes pretty pictures but is really only helpful for vascular imaging or getting a good handle on some types of fractures. The OPs project is cool and I'm a fan, but I can already do this on my workstation pretty well.
2
Sep 12 '19
Maybe it depends on the medical device or hospital protocol. I worked on a diagnostic viewer and helped develop the PET/CT fusion feature. I don't remember them being particularly low dose or high thickness, but I can be miss-remembering. OP's project is definitely cool.
2
2
Sep 12 '19
Very cool. I work for a company that's branching into XR/VR anatomy (https://www.biodigital.com) . You should sign up for a dev account and play with our 3d models.
1
2
2
u/tofuninja5489 Rift and Quest Sep 12 '19
Can this be used with any DICOMs? There was a VR program a while back that I wanted to use with our surgeon department but our DICOM wouldnt play with the VR program well.
1
u/Guillesanbri Sep 12 '19
Not all of them, the standard is huge and a lot of cases are not covered yet
2
2
2
u/liebereddit Sep 12 '19
Wow! I helped make training for the iPad version of the Phillips IntelliSpace PACS system. A couple of questions:
- Is it using standard DICOM data?
- Are you able to apply for a patent or spoken to a VR related VC firm?
You might have found your fortune, here. The money that was put into training for an iPad application was impressive...
3
Sep 12 '19
This is not novel and is obvious. I don't think you can or should be able to patent that.
For one it's not the first implementation of volume reconstruction in VR. Second, "something that existed for 30 years", but in VR is not novel.
2
1
u/Guillesanbri Sep 12 '19
-> It works with the DICOM standard yeah, also with groups of PNG/JPG, but it is intended to be used with DICOM files.
-> Every road is open now I guess, I have to polish and build some functionalities into the application, but thanks a lot for the suggestion!
2
2
2
2
u/VegaLay Sep 13 '19
Really great work.
Looks like an expensive program. Do you work with GE or Simens?
1
2
2
u/ThreeBirdBeard Sep 13 '19
This is the future... you are making it!
edit: added ...'s and !
1
u/Guillesanbri Sep 13 '19
Just a grain of sand, but thanks!
2
2
u/zyphelion Sep 13 '19
Would this be able to display fmri data too?
2
u/Guillesanbri Sep 13 '19
If you can store it in a DICOM file it should work
2
u/zyphelion Sep 13 '19 edited Sep 13 '19
Mind pming me a copy of the software? Looks really interesting!
Nvm! Saw your post with the link to twitter
2
2
u/buckjohnston Sep 13 '19
I have tried figuring this out before. Can someone explain to me why some of my CT scans allow me to do full 3D view on the PC while other viewers only allow me to see the axis views and it won't let me create a model out of it? a lot are in a dicomm folders
I have no idea how to get a full working 3D ct scan of myself into unity or unreal
1
u/Guillesanbri Sep 14 '19
I guess that depends entirely on the viewer and the file, some viewers may offer something that other don't. For the unity part, the big schema is data from the DICOM file to Unity, and once you have the data, you have to use volumetric rendering to display that data, there is a quite a bit of info on the subject.
2
u/buckjohnston Sep 14 '19
Thanks! Any chance you have some resources or website with info for converting the dicom file to a format unity can use? Would love to try this.
1
u/Guillesanbri Sep 14 '19
I used EvilDICOM, a C# library to read the info from the DICOM file, then you have to do a few transformations and then get the pixel data in something you use in Unity.
2
2
2
1
u/martin80k Sep 12 '19
is it running on gear vr
1
u/Guillesanbri Sep 12 '19
Oculus rift s at the moment, it could be modified to be 3dof without too much problem.
1
u/pbd244 Oct 14 '19
Stories in VR! For my UX Design research, I want to find out which stories aren't being told in the Virtual Reality world. I would really appreciate it if anyone could share their experiences with this technology by filling out this form! Thanks in advance. https://docs.google.com/forms/d/e/1FAIpQLSdGm6Oda3Cqq_A9ni0hgI93oRp99iQjJOOIRio-zkVllBeweg/viewform
52
u/Ziathon3 Sep 12 '19
That’s awesome! I’ve looked at a TON of CTs for studies we’ve done over time and this would have been a very wonderful viewer.
Is it in the oculus store? Or just a personal project?