Lustre Features

A comprehensive list of all features in the Lustre file system, including explanations, usage, and availability by version. Based on the latest information as of January 2026, with Lustre 2.17.0 as the current stable release and 2.18.0 under development.

Core Components | Performance Features | Security Features | High Availability | Scalability | Monitoring | Interoperability | Advanced Features

Introduction

Lustre is a high-performance, parallel distributed file system designed for large-scale cluster computing. This document provides a detailed list of its features, drawn from the official Lustre manual (updated for 2.x series), release notes from lustre.org (up to 2.17.0 released in late 2025), and recent JIRA tickets from jira.whamcloud.com for ongoing developments. Note that the manual may lag behind; we've incorporated updates from recent releases and roadmap items.

For the most accurate information, refer to lustre.org and jira.whamcloud.com.

1. Core Components

Feature Description Usage Version Available
Management Server (MGS) Central configuration store for all Lustre file systems in a cluster. Provides configuration to targets and clients. Stores config info. Should have dedicated storage (can share with MDS). Clients and targets contact MGS for server info and persistent tunable parameters. 1.x+
Metadata Servers (MDS) Manage metadata (names, directories, permissions) stored in MDTs. Handle network requests for one or more MDTs. Each MDS manages local MDTs. Supports multiple MDTs via DNE. 1.x+
Metadata Targets (MDT) Physical storage for metadata. Each filesystem has at least one MDT (root directory). Supports multiple MDTs with DNE and Striped Directories. Stores filenames, dirs, permissions, file layout. Distributed via DNE. Striped directories distribute files across MDTs. 1.x+ (DNE remote directories: 2.5+ Striped Directories: 2.8+)
Object Storage Servers (OSS) Node that provides file I/O and network handling for OSTs. Typically serve 2-8 OSTs. File data I/O service. Network request handling. 1.x+
Object Storage Targets (OST) Store user file data in objects. Number of objects per file is configurable. Up to 1 PiB each. Provides most of storage capacity. Configurable object count per file. 1.x+
Lustre Clients Compute nodes running client software to mount Lustre. Include MGC, MDCs, OSCs, LOV, LMV. Interface between VFS and servers. MGC: management; MDC: metadata, one per MDT; OSC: one per OST; LOV: aggregates OSCs; LMV: aggregates MDCs. 1.x+
Lustre Networking (LNet) Custom networking API for metadata and I/O communication. Supports InfiniBand, IP (RDMA), routing. Connectionless, asynchronous. Supports multiple network types. RDMA for high performance. Multi-rail, routing, health monitoring. 1.x+ (Multi-rail: 2.10; Dynamic Peer Discovery: 2.11; Asymmetrical Routes: 2.13)

2. Performance Features

Feature Description Usage Version Available
ldiskfs (enhanced ext4) High-performance backend filesystem for data/metadata. Used for OST/MDT storage. ZFS also supported. 1.x+
POSIX Compliance Full POSIX compliance with limited exceptions. Supports mmap(), atomic operations. Most operations atomic. mmap() supported. 1.x+
High-Performance Networking RDMA over InfiniBand, OmniPath, IPoIB. Multi-network bridging via routing. OFED for InfiniBand. Lustre routing for multi-network. 1.x+
Byte-Granular Locking (LDLM) Fine-grained locking for concurrent access. MDT LDLM for metadata, OST LDLM for file stripes. Scales locking performance. Prevents stale data. 1.x+
Controlled Striping Per-file, directory, or filesystem striping. RAID-0, configurable stripe_count/size. Default: stripe_count=1, size=1MB. Use 'lfs setstripe'. Up to 2000 objects/file (31.25 PiB with ldiskfs). 1.x+
Data Integrity Checksum all data in transit. Client → OSS checksum. Protects against corruption. 1.x+
MPI I/O Optimized parallel I/O via ADIO layer. Matches FS architecture. 1.x+
Progressive File Layout (PFL) Composite files with dynamic extents and striping. On-demand object instantiation. 'lfs setstripe --component-end'. Add/delete components. 2.10+
Self-Extending Layout (SEL) Dynamically extends layouts on low-space OSTs. Splits components. Spillover to new OSTs. 2.13+
Foreign Layout Files point to external objects outside Lustre namespace. 'lfs setstripe --foreign'. Extended attributes. 2.13+
Data on MDT (DoM) Stores small file data directly on MDT for performance. 'lfs setstripe --layout mdt'. Max 1MB stripe size. Disable via 'dom_stripesize'. 2.11+
Lazy Size on MDT (LSoM) Delays size updates on MDT for small files. Optimizes space allocation. 2.12+
File Level Redundancy (FLR) Mirrors files/directories for reliability. 'lfs mirror create/extend/split/resync/verify'. 'lfs find --mirror-count'. 2.11+
Persistent Client Cache (PCC) Caches files on client SSDs persistently. Uses HSM for data synchronization. HSM-based. 'lctl pcc add/detach'. 'lfs pcc state'. 2.13+
Large Bulk I/O (16MB RPC) Larger RPCs for bulk transfers. Up to 16MB. 'brw_size' on OST. 2.9+
Server-Side Advice/Hinting Hints for optimizing I/O patterns. 'lfs ladvise'. 'LU_LADVISE_WILLREAD'. 2.9+
Lockless I/O Tunables Reduces locking overhead. 'contended_locks', 'max_nolock_bytes'. 2.x+
Network Request Scheduler (NRS) Policies: FIFO, CRR-N, ORR, TRR, TBF, Delay. 'lctl set_param osc.*.nrs=crr_n'. TBF: 2.6+; Delay: 2.10+
Portal Round-Robin Distributes requests across portals. 'portal_rotor=RR_RT'. 2.x+
LNet Peer Health Monitors peer/router health. Health values: local/no-resend. 'lnetctl set health_sensitivity'. 2.12+
Asymmetrical Routes Non-symmetric routing. 'lnetctl set drop_asym_route=1'. 2.13+
Dynamic Peer Discovery Auto-discovers peers. 'lnetctl set discovery=1'. 2.11+
Multi-Rail Software Multiple interfaces per node. 'lnetctl net add --net tcp2 --if eth0,eth1'. YAML import/export. 2.10+
Self-Extending Layout (SEL) An extension of PFL that allows the MDS to dynamically extend file layouts as files grow, handling OST space issues or tiered storage spillover automatically. Use lfs setstripe -E -1 -c 1 --extension-size 1G /path/file for dynamic extension; integrates with PFL and FLR for automatic repetition or spillover. 2.13+
Overstriping Extends striping to allow more stripes than available OSTs by reusing OSTs, improving performance for highly concurrent workloads on smaller systems. Set with lfs setstripe -C -2 (overstripe factor of 2); useful for testing or small clusters. 2.10+ (enhanced in later versions)
Lazy Size on MDT (LSoM) Stores file size and block counts lazily on the MDT to reduce metadata overhead for large striped files, improving stat() performance.   2.13+
Network Request Scheduler (NRS) with Token Bucket Filter (TBF) Manages and prioritizes RPC requests to prevent overload, with TBF for rate limiting based on job ID, user, or other policies. Enable with lctl set_param ost.OSS.ost_io.nrs_policies=tbf; configure rules like lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start high_prio {nid:*.@tcp} {opc: {1 2}}". 2.9+ (TBF in 2.5+)
Adaptive Timeouts Dynamically adjusts RPC timeouts based on network conditions and load to improve resiliency and reduce unnecessary evictions. Set lctl set_param at_min=5 at_max=600; monitors service times automatically. 2.1+
ldiskfs HDD Optimizations
  • Large, aligned client RPCs to maximize RAID/HDD bandwidth and minimize IOPS overhead.
  • Large, aligned block allocation to prevent RAID read-modify-write penalties.
  • Configurable hybrid NVMe+HDD OSTs: metadata stored on fast NVMe, bulk data on capacity HDDs.
  • Uncompressed multi-threaded writes allocated sparsely, enabling later in-fill of holes for contiguous sequential reads.
  • Compressed chunks in ldiskfs (variable size) allocated densely to avoid read-modify-write due to data holes.
  • Dynamic read/write RAM cache tuned by I/O pattern and file size:
    • Large read/write operations use a dedicated page pool to bypass the page cache and achieve full HDD bandwidth.
    • Small read/write operations are cached in RAM to minimize IOPS to the HDD backend.
 Automatic for ldiskfs OSTs on rotational media. Tune hybrid NVMe+HDD via LVM setup (metadata at start of device on NVMe). Control cache behavior via lctl set_param parameters (e.g., osd-ldiskfs.*.read_cache_enable, write_cache_*). Sparse allocation enabled by default for uncompressed writes; dense for compressed. 2.10+ (many in 2.15+ / 2.17+; hybrid LVM in recent versions)
ldiskfs NVMe/Flash Optimizations
  • Automatic detection of non-rotational (SSD/NVMe) OSTs to apply flash-friendly defaults.
  • File layout can explicitly target NVMe or HDD OSTs via OST Pools.
    • Allows mixing flash + HDD within a single file (e.g., NVMe for initial/index portions, HDD for bulk).
  • For mirrored files (FLR), client read/write prefers flash OSTs over HDD.
  • Smaller RPC sizes possible on flash OSTs to improve IOPS.
  • Client I/O path optimizations to reduce CPU overhead per I/O operation.
  • Read/write RAM cache disabled by default on NVMe OSTs (flash has low latency; avoids unnecessary buffering).
  • Large, aligned block allocation tuned specifically for QLC flash (low DWPD endurance).
 Auto-detected for non-rotational devices. Use lfs setstripe --pool to target specific OST pools (e.g., flash vs. HDD). Prefer flash in FLR layouts via layout configuration. Tune RPC size via osc.*.max_rpcs_in_flight and related params. Disable cache explicitly if needed via osd-ldiskfs.*.read_cache_enable=0. 2.12+ (auto-detection and pool-based mixing); many in 2.15+ / 2.17+

3. Security Features

Feature Description Usage Version Available
Default Security TCP connections from privileged ports only. UNIX group verification on MDS. Basic access control. 1.x+
ACLs POSIX ACLs with root squash. setfacl, getfacl. Mount option acl. 1.8+ (default)
Root Squash Maps root to unprivileged user. Via nodemaps. 2.x+
Sub-directory Isolation Restricts clients to sub-trees. Fileset feature. 2.18+
SELinux Enforcement Enforces SELinux policies on clients. lctl nodemap_set_sepol --name NODEMAP --sepol POLICY ... 2.13+
Client-side Encryption Encrypts files/directories using fscrypt. 'fscrypt setup/encrypt/lock'. AES-256-XTS. 2.14+ (content), 2.15+ (filename)
Kerberos (KRB) Secure authentication with GSS. 'mgssec=krb5i'. Keytabs, principals. 2.10+ (secure MGS)
Shared-Secret Key (SSK) Secures RPCs with HMAC/SHA, AES. 'lgss_sk' for key generation. 'lctl conf_param srpc.flavor'. 2.9+
Nodemap for UID/GID Mapping Maps UIDs/GIDs across nodes. 'lctl nodemap_add/range'. Supports projid, offsets. 2.8+ (core), 2.15+ (projid), 2.18+ (offsets)
Nodemaps Allows mapping clients by NID ranges for administrative controls like export restrictions, squash/rewrite UID/GID, fine grained feature access controls for clients, even while running as root, and feature enablement. Create with lctl nodemap_add --name client_group --range 10.0.0.[1-100]@tcp; set squash lctl nodemap_modify --name client_group --squash_uid=99. 2.7+ (enhanced in 2.15+ with project IDs and offsets)

4. High Availability and Failover

Feature Description Usage Version Available
Active/Active Failover Failover for MDTs/OSTs using shared storage. No single point of failure. HA managers. 1.x+
Multiple-Mount Protection (MMP) Prevents concurrent mounts of the same physical storage device on multiple servers use tune2fs -O mmp or tune2fs -O ^mmp to enable or disable MMP on an existing file system, use --backfsoptions='-O mmp' to enable for ldiskfs. This feature is usually enabled by default if mkfs.lustre --servicenode or mkfs.lustre --failnode are used during formatting 2.6.9+
Imperative Recovery Shortens recovery via MGS notifications. 'ir_factor', 'noir' mount option. 2.x+
Version-Based Recovery (VBR) Recovers without data loss on gaps. Inode versions validated. Before 2.8+
Commit on Share (COS) Commits transactions before sharing. 'mdt.commit_on_sharing=1'. Default in 2.x+
Reply Reconstruction Reconstructs dropped replies. Multiple replies per client. 2.8+
Metadata Replay Replays client requests on restart. Uses XIDs, transactions. 1.x+
Failout/Failover Modes Failout returns errors immediately. 'mkfs.lustre --failover.mode=failout'. 2.x+
Set MDT Read-Only Configures MDT as read-only. 'lctl set_param mdt.*.read_only=1'. 2.13+
Version-Based Recovery (VBR) Ensures recovery from failures by tracking object versions, allowing partial recovery and skipping unaffected clients. Enabled by default; monitor with lctl get_param *.recovery_status. 2.0+
Imperative Recovery (IR) Allows servers to proactively notify clients of recovery needs, speeding up the process after failovers. Enable with lctl set_param mgs.MGS.ir_notify_clients=1; automatic in HA setups. 2.4+

5. Scalability and Capacity

Feature Description Usage Version Available
Scalability 10–100,000 clients, 50 TB/s I/O, 225M IOPS. OSS: 1,000 nodes/4,000 OSTs. MDS: 256 nodes/256 MDTs. 1.x+
Capacity Growth Add OSTs/MDTs without interruption. Dynamic expansion. 1.x+
OST Pools Groups OSTs for balanced usage. 'lctl pool_new/add/remove'. 'lfs setstripe --pool'. 1.x+
File System Limits Max file: 32 PiB (ldiskfs), 8 EiB (ZFS). Max objects/OST: 1,000M. Max OST size: 4,096 TiB. 1.x+
Multiple File Systems Run multiple FS on same cluster. Unique 'fsname:NID'. 1.x+
Distributed Namespace (DNE) Distributes metadata across multiple MDTs for scalability and performance. Remote directories on different MDTs. Stripe single large directories across MDTs. Automatic MDT space balance. 2.6+ (Remote Directories), 2.8+ (Striped Directories), 2.14+ (Space Balanced Directories)
Distributed Namespace (DNE) Distributes directory metadata across multiple MDTs for scalability, with DNE2 for striped directories and DNE3 for automated balancing. Create striped dir: lfs mkdir -i -1 -c 4 /dir; enable auto-striping on FS root. 2.3+ (DNE1), 2.8+ (DNE2), 2.16+ (DNE3)
User/Group Quotas Enforces disk and inode usage limits per user or group across the filesystem. Enable lctl set_param -P fs.quota.ost=ug; set lfs setquota -u user -b 1T -i 1M /fs. 1.8+ (core feature)
Pool Quotas Enforces quotas on specific OST pools, allowing fine-grained space management across storage tiers. Enable lctl set_param -P fs.quota.ost=ugp; set lfs setquota -p poolname -u user 1T /fs. 2.13+
Project Quotas Accounts and optionally enforces quota limits based on Project IDs, which are independent of file access controls (as User and Group Quotas are). Project Quotas are useful for tracking space usage in a directory tree, and in particular for isolated subdirectory mounts used in multi-tenant environments Enable lctl set_param -P fs.quota.ost=ugp; assign default project limit with lfs setquota -P 1T /fs or assign a limit for a specified project with lfs setquota -p 100 1T /fs 2.5+ (enhanced in 2.15+)

6. Monitoring and Management

Feature Description Usage Version Available
Lustre Changelogs Track changes for monitoring/auditing. 'lctl changelog_register'. 'lfs changelog'. 2.x+ (HLINK: 2.11+)
Lustre Jobstats Monitors I/O per JobID. 'lctl set_param -P jobid_var=procname_uid'. 'lljobstat -c 10'. 2.4+ (core), 2.15+ (lljobstat)
Lustre Monitoring Tool (LMT) Top-like display for MDS/OSS. Python-based, server-side. 2.x+
CollectL Collects stats to file for analysis. Runs on MDS/OSS/Clients. 2.x+
Other Tools lltop, tacc_stats, xltop, scripting. Batch integration. 2.x+
Tuning Parameters mkfs.lustre, tunefs.lustre, lctl. 'lctl set_param -P' for persistence. 1.x+
HSM (Hierarchical Storage Management) Tiered storage integration. 'lctl set_param mdt.*.hsm_control=enabled'. 'lhsmtool_posix'. 2.5+
ZFS Snapshots Snapshots for recovery/versioning. 'lctl snapshot_create/destroy'. 2.10+
ChangeLogs Records filesystem events like creates, deletes, and attribute changes for auditing or replication. Register user: lfs changelog_register; view: lfs changelog fs; clear: lfs changelog_clear. 2.0+
Jobstats / JobID Tracks I/O and metadata operations per job, user, or process for performance analysis and billing. Enable lctl set_param jobid_var=procname_uid; view lctl get_param job_stats; use lljobstat for summaries. 2.4+ (enhanced in 2.14+ with session-based)
LNet Discovery Automatically discovers peer NIDs in multi-rail setups for dynamic configuration. Enable lnetctl set discovery 1; manual lnetctl discover <nid>. 2.10+

7. Interoperability and Recovery

Feature Description Usage Version Available
Interoperability Mixed architectures, endianness, versions. Successive major releases. 1.x+
Object-Based Architecture Isolates clients from on-disk format. Enables upgrades. 1.x+
Disaster Recovery Online LFSCK for consistency. Runs while FS in use. 2.x+
Backups lustre_rsync, device-level, filesystem-level. Changelog-based replication. 2.x+
Foreign Layouts Supports non-native layouts for integration with external systems like HSM or other filesystems. Use with PFL/SEL for custom extensions; inspect via lfs getstripe --foreign. 2.13+

8. Advanced and Upcoming Features

Feature Description Usage Version Available
Striping Distributes data across OSTs for parallel I/O. 'lfs setstripe' for custom striping patterns. 1.x+
Erasure Coding Adds redundancy using parity components for large files, allowing recovery from OST failures without full mirroring. Similar to RAID-4. Configure parity on files, divide into RAID sets (e.g., 8+2). Use for redundancy in large filesystems. Under development for 2.18+ (tracked in JIRA LU-10911)
64-bit Time API Migration Moves to standard kernel time APIs for 2038 safety and better handling on tickless machines. Internal; ensures long-term compatibility. 2.14+
RHEL 9.x/10 Support Support for newer Red Hat Enterprise Linux versions, including container tools and data visualization enhancements. Use on RHEL 8.5+ servers/clients. 2.15+ (RHEL 9.6), 2.18+ (RHEL 10)
Client-Side Data Compression (CSDC) Provides a transparent mechanism to compress and decompress data on Lustre clients, reducing network bandwidth and storage usage/cost. Transparent to users; configured by administrators. Compresses data in client RAM before sending RPCs to OSTs. 2.18+
Trash Can/Undelete (TCU) Moves deleted files into a temporary ".Trash" holding directory to allow user recovery after accidental or malicious deletion. Automatic on delete; recover via tools like 'lfs trash unrm' or browsing .Trash directory. Under development for 2.18 (tracked in JIRA LU-18456)
Lustre File System Check (LFSCK) Online consistency checker for repairing MDT-OST inconsistencies, namespace issues, and layouts. Start lctl lfsck_start -t layout -A; monitor lctl get_param mdd.*.lfsck_layout. 2.3+
YAML LNet Configuration Allows declarative configuration of LNet networks, peers, and routes via YAML files for dynamic management. Import lnetctl import config.yaml; export lnetctl export current.yaml. 2.10+
LNet Self-Test Tool for testing LNet bandwidth, latency, and health in isolation. Create session lnetctl self_test create; run lnetctl self_test run --src <nid> --dst <nid>. 2.0+ (enhanced in 2.10+)
Dynamic NID Configuration Runtime addition/removal of NIDs without restarting Lustre, supporting multi-rail and failover. Use lnetctl net add --net tcp1 --if eth1; C-API for programmatic changes. 2.17+