Simple 3-Node Lustre Cluster Tutorial

This tutorial sets up a basic 3-node Lustre cluster for testing: Node 1 (MGS/MDT), Node 2 (OSS with 2 OSTs), Node 3 (client). Assumes RHEL 9.x (adjust for other distros). Use dedicated block devices (e.g., /dev/sdb for MDT /dev/sdc, /dev/sdd for OSTs). For production, add HA (e.g., Pacemaker) and RAID. Based on Lustre 2.17.0 (January 2026). Not for production, use with caution.

Prerequisites

TCP/IP Configuration (All Nodes)

# Load LNet
modprobe lnet
lnetctl lnet configure

# Add network (adjust IP/subnet)
lnetctl net add --net tcp0 --if eth0

# Verify NIDs
lctl list_nids  # e.g., 192.168.1.1@tcp0

Node 1: MGS/MDT Setup

# Format combined MGS/MDT
mkfs.lustre --fsname=testfs --mgs --mdt --index=0 /dev/sdb

# Mount
mkdir /mnt/testfs-mdt0
mount.lustre /dev/sdb /mnt/testfs-mdt0

# Verify
lfs df -h

Node 2: OSS with 2 OSTs Setup

# Format OST1 (use Node1 NID)
mkfs.lustre --fsname=testfs --ost --index=0 --mgsnode=192.168.1.1@tcp0 /dev/sdc

# Format OST2
mkfs.lustre --fsname=testfs --ost --index=1 --mgsnode=192.168.1.1@tcp0 /dev/sdd

# Mount OSTs
mkdir /mnt/testfs-ost0 /mnt/testfs-ost1
mount.lustre /dev/sdc /mnt/testfs-ost0
mount.lustre /dev/sdd /mnt/testfs-ost1

# Verify
lfs df -h  # From Node1 or Node3 after client mount

Node 3: Client Setup

# Mount (use Node1 NID)
mkdir /mnt/testfs
mount -t lustre 192.168.1.1@tcp0:/testfs /mnt/testfs

# Verify
lfs df -h
df -h /mnt/lustre

Usage and Testing

# On client: Create file
dd if=/dev/zero of=/mnt/testfs/testfile bs=1M count=100

# Check striping
lfs getstripe /mnt/testfs/testfile

# Cleanup (unmount client first, then OSTs, then MDT)
umount /mnt/testfs  # Node3
umount /mnt/testfs-ost0 /mnt/testfs-ost1  # Node2
umount /mnt/testfs-mdt0  # Node1

InfiniBand/RoCE Instructions

Warnings and Best Practices