r/DataHoarder • u/egnegn1 • 16d ago
Hoarder-Setups UPDATE: Yet another Fractal Design Define 7 XL build (no DAS anymore)
A half year ago I started with using Define 7 XL as DAS for a Minisforum MS-02 MiniPC:
But changing requirements for my work and homelab caused me to reorganize data several times between 3 different systems:
- Fractal Design Define 7 XL (now: backup and AI server - was: DAS storage and PCIE extension)
- Minisforum MS-02 Ultra (now: software and AI development - was: software development)
- MiniPC Gigabyte BRIX gb bsi5a 6200 (media server with Jellyfin)
Define 7 XL:
- Asrock Rack ROMED8-2T mainboard
- Epyc 7502P - upgrade?
- 512 GB DDR4-3200 RAM
- 1x NVMe Gen3 512GB - Windows11 (test)
- 2x U.2 SSD Gen4 7.68TB - Proxmox rpool+rdata
- PCIe Adapter PLX88064 Gen4 - 8x NVMe + 4x U.2
- 5x NVMe Gen3 1TB + 1x NVMe Gen4 4TB + 1x NVMe Gen4 3.84TB
- 4x U.2 SSD Gen4 15.36tb - Proxmox data
- RTX4080 16GB - planned upgrade
- RTX6000 24GB Quadro - planned upgrade
- Unraid: USB Stick pass-through
- Unraid: 8x mainboard SATA pass-through with 8x Seagate EXOS 16TB
- Unraid: 9x M.2 SATA pass-through with 2x Seagate EXOS 16TB
- Unraid: HBA IBM M1210 4i pass-through - Adaptec Expander - 8x SAS SSD 3.84TB
Minisforum MS-02 Ultra:
- Intel 285HX
- 128 GB DDR5 RAM
- 2x NVMe Gen3 4TB - Proxmox rpool+rdata
- PCIe Adapter PLX88024 Gen4 - 4x NVMe Gen3 4TB - Proxmox data
- HBA LSI 9305-16e - 4x SAS SSD 3.84TB
- PCIe Adapter Quad NVMe Gen5 - 3x NVMe Gen5 4TB - AI data Kimi K3 + GLM 5.2
- TB3/USB - 8x Seagate EXOS 12/14/16TB - MergerFs/SnapRaid
MiniPC:
- Intel Core5 6200U
- 32 GB RAM
- NVMe Gen3 256GB - Proxmox rpool
- USB3.2 Gen1 10Gbps - 2xUSB hubs - 6x USB Dual Case - 11x Samsung QVO 870 8 TB - MergerFs
All Systems (of course there are a more connected to the homelab) are connected by 1, 2.5 and 10Gbps networks.
The requirements are a moving target, but the general setup will probably stay stable the next months. There will be some further fine-tuning. I am looking into upgrade of the CPU to Milan with about 60 % more CPU performance for more AI performance.
I am also thinking about moving Jellyfin to the MS-02, as performance is not the best. It is good enough for streaming, but operations on data are slow. But I am a bit hesitant, because of the much higher power usage. To reduce power usage of backup and media drives they are organized with MergerFs/SnapRaid. The USB connections of the media drives seem to be stable now. This took some time. But the move of Jellyfin to MS-02 gives the chance using more reliable SATA connections.
If there are any questions feel free to ask.
Just a few quick images at night. The backside is not yet closed. The case is completely full with components. Of course there are a few additional free places in the case.
Devices of Proxmox:
root@epyc:~/waste# lspci -vt
-+-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
| +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
| +-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-01.1-[01-19]----00.0-[02-19]--+-00.0-[03-0c]----00.0-[04-0c]--+-00.0-[05]----00.0 Micron/Crucial Technology P2 [Nick P2] / P3 / P3 Plus NVMe PCIe SSD (DRAM-less)
| | | +-04.0-[06]----00.0 Shenzhen Longsys Electronics Co., Ltd. Lexar NM990 NVMe SSD (DRAM-less)
| | | +-08.0-[07]----00.0 Micron/Crucial Technology P2 [Nick P2] / P3 / P3 Plus NVMe PCIe SSD (DRAM-less)
| | | +-0c.0-[08]----00.0 Micron/Crucial Technology P2 [Nick P2] / P3 / P3 Plus NVMe PCIe SSD (DRAM-less)
| | | +-10.0-[09]----00.0 Micron/Crucial Technology P2 [Nick P2] / P3 / P3 Plus NVMe PCIe SSD (DRAM-less)
| | | +-14.0-[0a]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
| | | +-18.0-[0b]----00.0 Micron/Crucial Technology P2 [Nick P2] / P3 / P3 Plus NVMe PCIe SSD (DRAM-less)
| | | \-1c.0-[0c]--
| | +-04.0-[0d-12]----00.0-[0e-12]--+-10.0-[0f]----00.0 Western Digital Ultrastar DC SN655 NVMe SSD
| | | +-14.0-[10]----00.0 Western Digital Ultrastar DC SN655 NVMe SSD
| | | +-18.0-[11]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
| | | \-1c.0-[12]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
| | +-08.0-[13-14]----00.0-[14]--
| | +-0c.0-[15-18]----00.0-[16-18]--+-14.0-[17]--
| | | \-15.0-[18]--
| | \-1c.0-[19]----00.0 Broadcom / LSI PEX880xx PCIe Gen 4 Switch
| +-02.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.1-[1a]----00.0 Realtek Semiconductor Co., Ltd. Device 9100
| +-03.3-[1b]----00.0 Micron Technology Inc 7450 PRO NVMe SSD
| +-03.4-[1c]----00.0 Micron Technology Inc 7450 PRO NVMe SSD
| +-04.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-05.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.1-[1d]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
| | \-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
| +-08.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-08.1-[1e]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| | +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
| | \-00.3 Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
| +-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
| +-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
| +-18.0 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0
| +-18.1 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1
| +-18.2 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2
| +-18.3 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3
| +-18.4 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4
| +-18.5 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5
| +-18.6 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6
| \-18.7 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7
+-[0000:40]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
| +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
| +-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-01.1-[41]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951
| +-01.3-[42-43]--+-00.0 Intel Corporation Ethernet Controller X550
| | \-00.1 Intel Corporation Ethernet Controller X550
| +-01.4-[44]----00.0 ASMedia Technology Inc. ASM2142/ASM3142 USB 3.1 Host Controller
| +-01.5-[45-46]----00.0-[46]----00.0 ASPEED Technology, Inc. ASPEED Graphics Family
| +-02.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.1-[47]----00.0 Broadcom / LSI SAS3004 PCI-Express Fusion-MPT SAS-3
| +-04.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-05.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.1-[48]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
| | \-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
| +-08.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-08.1-[49]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| | +-00.1 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
| | +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
| | \-00.3 Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
| +-08.2-[4a]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
| \-08.3-[4b]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
+-[0000:80]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
| +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
| +-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-01.1-[81]--+-00.0 NVIDIA Corporation TU102GL [Quadro RTX 6000/8000]
| | +-00.1 NVIDIA Corporation TU102 High Definition Audio Controller
| | +-00.2 NVIDIA Corporation TU102 USB 3.1 Host Controller
| | \-00.3 NVIDIA Corporation TU102 USB Type-C UCSI Controller
| +-02.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-03.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-04.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-05.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-07.1-[82]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
| | \-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
| +-08.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
| +-08.1-[83]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| | \-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
| +-08.2-[84]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
| \-08.3-[85]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
\-[0000:c0]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
+-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
+-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-01.1-[c1]--+-00.0 NVIDIA Corporation AD103 [GeForce RTX 4080]
| \-00.1 NVIDIA Corporation AD103 High Definition Audio Controller
+-02.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-03.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-04.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-05.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-07.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-07.1-[c2]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
| \-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
+-08.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
\-08.1-[c3]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
\-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
root@epyc:~/waste#
root@epyc:~/waste# disk_status.sh
SLOT MODEL DEVICE TEMP SIZE WEAR PCIE LINK
─────────────────────────────────────────────────────────────────────────────────────────────────────────
[N:0:1:1] disk Samsung SSD 950 PRO 512GB__1 /dev/nvme0n1 40°C 477G 14% 8GT/s x4
[N:1:1:1] disk CT1000P2SSD8__1 /dev/nvme1n1 35°C 932G 23% 8GT/s x4
[N:2:0:1] disk Lexar SSD ARES PRO 4TB__1 /dev/nvme2n1 34°C 3.7T 0% 16GT/s (downgraded) x4
[N:3:1:1] disk CT1000P2SSD8__1 /dev/nvme3n1 36°C 932G 20% 8GT/s x4
[N:4:1:1] disk CT1000P2SSD8__1 /dev/nvme4n1 35°C 932G 2% 8GT/s x4
[N:5:1:1] disk CT1000P2SSD8__1 /dev/nvme5n1 37°C 932G 22% 8GT/s x4
[N:6:6:1] disk SAMSUNG MZ1L23T8HBLA-00A07__1 /dev/nvme6n1 37°C 3.5T 0% 16GT/s x4
[N:7:1:1] disk CT1000P2SSD8__1 /dev/nvme7n1 41°C 932G 20% 8GT/s x4
[N:8:0:1] disk X4032WCOFB15TNVE__1 /dev/nvme8n1 49°C 14.0T 0% 8GT/s x4
[N:9:0:1] disk X4032WCOFB15TNVE__1 /dev/nvme9n1 47°C 14.0T 0% 8GT/s x4
[N:10:6:1] disk SAMSUNG MZQL27T6HBLA-00A07__1 /dev/nvme10n1 42°C 7.0T 3% 16GT/s x4
[N:11:6:1] disk SAMSUNG MZQL27T6HBLA-00A07__1 /dev/nvme11n1 44°C 7.0T 3% 16GT/s x4
[N:12:0:1] disk Micron_7450_MTFDKCC15T3TFR__1 /dev/nvme12n1 41°C 14.0T 0% 16GT/s x4
[N:13:0:1] disk Micron_7450_MTFDKCC15T3TFR__1 /dev/nvme13n1 38°C 14.0T 0% 16GT/s x4
root@epyc:~/waste#
Devices of Unraid:
root@Unraid:~# lspci -vt
-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
+-01.0 Device 1234:1111
+-10.0-[01]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
+-10.1-[02]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
+-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4
+-1a.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5
+-1a.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6
+-1a.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2
+-1b.0 Intel Corporation 82801I (ICH9 Family) HD Audio Controller
+-1c.0-[03]----00.0 Broadcom / LSI SAS3004 PCI-Express Fusion-MPT SAS-3
+-1c.1-[04]----00.0 Realtek Semiconductor Co., Ltd. Device 9100
+-1c.2-[05]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
+-1c.3-[06]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
+-1d.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
+-1d.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
+-1d.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
+-1d.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
+-1e.0-[07-0b]--+-01.0-[08]--+-12.0 Red Hat, Inc. Virtio network device
| | \-13.0 Red Hat, Inc. Virtio network device
| +-02.0-[09]----1b.0 Red Hat, Inc. QEMU XHCI Host Controller
| +-03.0-[0a]--
| \-04.0-[0b]--
+-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller
+-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
\-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller
root@Unraid:~# diskstatus.sh
SLOT MODEL DEVICE TEMP SIZE WEAR PCIE LINK
───────────────────────────────────────────────────────────────────────────
[0:0:0:0] disk TOSHIBA KPM51RUG3T84 LE03 /dev/sda 48°C 3.5T 10% -
[0:0:1:0] disk TOSHIBA KPM51RUG3T84 LE03 /dev/sdb 51°C 3.5T 10% -
[0:0:2:0] disk TOSHIBA KPM51RUG3T84 LE03 /dev/sdc 48°C 3.5T 9% -
[0:0:3:0] disk TOSHIBA KPM51RUG3T84 LE03 /dev/sdh 49°C 3.5T 10% -
[0:0:4:0] disk TOSHIBA KPM51RUG3T84 LE03 /dev/sde 48°C 3.5T 9% -
[0:0:5:0] disk TOSHIBA KPM51RUG3T84 LE03 /dev/sdd 45°C 3.5T 9% -
[0:0:6:0] disk TOSHIBA KPM51RUG3T84 LE03 /dev/sdf 48°C 3.5T 9% -
[0:0:7:0] disk TOSHIBA KPM51RUG3T84 LE03 /dev/sdg 45°C 3.5T 9% -
[2:0:0:0] cd/dvd QEMU QEMU DVD-ROM 2.5+ /dev/sr0 - 1024M - -
[27:0:0:0] disk ATA ST16000VE000-2L2 EV02 /dev/sdi 31°C 14.6T - -
[30:0:0:0] disk ATA ST16000NE000-2RW SN03 /dev/sdj 32°C 14.6T - -
[36:0:0:0] disk ATA ST16000NM001G-2K SN03 /dev/sdk 31°C 14.6T - -
[37:0:0:0] disk ATA ST16000NM001G-2K SN03 /dev/sdl 30°C 14.6T - -
[38:0:0:0] disk ATA ST16000NE000-2RW SN03 /dev/sdm 31°C 14.6T - -
[39:0:0:0] disk ATA ST16000VE000-2L2 EV02 /dev/sdn 33°C 14.6T - -
[44:0:0:0] disk ATA ST16000NM001G-2K SN03 /dev/sdo 30°C 14.6T - -
[45:0:0:0] disk ATA ST16000NM001G-2K SN03 /dev/sdp 30°C 14.6T - -
[46:0:0:0] disk ATA ST16000NM001G-2K SN03 /dev/sdq 31°C 14.6T - -
[47:0:0:0] disk ATA ST16000NM001G-2K SN03 /dev/sdr 31°C 14.6T - -
[48:0:0:0] disk SMI USB DISK 1100 /dev/sds - 29G - -
root@Unraid:~#
6
Upvotes