FLR Failover Integration Examples

File Level Redundancy (FLR) integrates with Lustre failover by providing file-level mirroring, allowing continued access during OST failures or failovers without full data loss. FLR complements server failover (active/passive/active/active) by enabling reads from available mirrors and resync post-recovery. This guide provides examples for Lustre 2.17.0 (January 2026), based on the Lustre Operations Manual (updated 2025) and FLR architecture. FLR is phased: Phase 1 (delayed mirroring) in 2.11+; full integration requires shared storage and HA tools like Pacemaker.

Prerequisites

• Failover setup with shared storage (e.g., SAN/RAID) for OSTs/MDTs.
• OST pools for fault domain separation (e.g., pool1 for primary, pool2 for mirrors).
• HA software (Pacemaker) for automatic failover detection and switching.
• Clients 2.11+ for FLR support.

Basic FLR Setup with Failover

Create mirrored files with layouts separated by fault domains (e.g., different OSSs).

# Create OST pools for separation
lctl pool_new testfs.primary
lctl pool_add testfs.primary OST[0-4]
lctl pool_new testfs.mirror
lctl pool_add testfs.mirror OST[5-9]

# Create mirrored file with pools
lfs mirror create -N -c 2 -p primary -N -c 2 -p mirror /mnt/lustre/file1

# Verify layout
lfs getstripe -v /mnt/lustre/file1

PFL/SEL Integration with FLR for Failover

Use PFL/SEL for dynamic layouts in mirrored files, ensuring redundancy during growth or failover.

# PFL mirrored with failover pools
lfs mirror create -N -E 4M -c 1 -p primary --flags=prefer -E eof -c 2 -N -E 16M -c 4 -p mirror -E eof -c -1 /mnt/lustre/pfl_file

# SEL mirrored for extension during failover
lfs mirror create -N -E 1G -c 1 -z 64M -p primary -E -1 -c 4 -z 256M -N -c -1 -p mirror /mnt/lustre/sel_file

Resync After OST Failover

During OST failover (e.g., OSS1 fails, OSS2 takes over), FLR allows reads from mirrors. Post-failover, resync stale mirrors.

# Simulate failure: Mark OST degraded
lctl set_param obdfilter.testfs-OST0000.degraded=1

# Find stale files and resync them
lfs find --mirror-state=^ro -0 /mnt/lustre | xargs -0 lfs mirror resync



# After resync, reactivate OST
lctl set_param obdfilter.testfs-OST0000.degraded=0

Automated Resync with HA Tools

Integrate with Pacemaker for post-failover resync scripts.

# Pacemaker resource for resync (custom script example)
cat <<EOF > /usr/local/bin/lustre_resync.sh
#!/bin/bash
lfs find --mirror-state=^ro -0 /mnt/lustre | xargs -0 lfs mirror resync | xargs lfs mirror resync
EOF

chmod +x /usr/local/bin/lustre_resync.sh

# Add to Pacemaker
pcs resource create resync ocf:heartbeat:Dummy op start timeout=60s op stop timeout=60s op monitor interval=120s timeout=30s
pcs resource meta resync target-role="Started"

Best Practices

• Use separate pools/racks for mirrors to avoid correlated failures.
• Set prefer flags on SSD mirrors for performance during normal operation.
• Schedule periodic verify/resync to maintain redundancy.
• Combine with RAID for device-level protection.
• Monitor with ChangeLog for stale events.
• Avoid immediate resync for read-mostly files to reduce overhead.

For advanced: Use policy engines (e.g., RobinHood) for automated resync. No major FLR-failover changes in 2.17.