|
1 |
| -# Lustre Network Driver (LND) ko2iblnd configuration |
| 1 | +# Configure ko2iblnd Lustre Network Driver (LND) for Soft-RoCE performance |
2 | 2 |
|
3 |
| -The ko2iblnd.ko changes are needed for better Soft-RoCE performance on LNDs. |
| 3 | +The ko2iblnd.ko module requires modifications to optimize Soft-RoCE performance. |
| 4 | +If your setup does not involve Soft-RoCE connections, this section does not apply. |
| 5 | + |
| 6 | +## Prerequisites |
| 7 | + |
| 8 | +Ensure that Lustre is installed with the ko2iblnd module built for the Soft-RoCE driver (RXE) by specifying `--with-o2ib=yes` for `/.configure` or `rpmbuild`. |
| 9 | +If this option is not specified, the build process will attempt to automatically detect external OFED installations or internal o2ib support. |
| 10 | +If neither is detected, the ko2iblnd module will not be built. |
| 11 | + |
| 12 | +For detailed instructions, see the _Cray ClusterStor Lustre Client Build Configuration Guide S-9100_. |
4 | 13 |
|
5 | 14 | ## Compute Node tuning for Soft-RoCE
|
6 | 15 |
|
@@ -49,7 +58,7 @@ Tuning on compute node can be achieved in two ways. Follow the steps that work b
|
49 | 58 | /sys/module/ko2iblnd/parameters/cksum:0
|
50 | 59 | /sys/module/ko2iblnd/parameters/concurrent_sends:84
|
51 | 60 | /sys/module/ko2iblnd/parameters/conns_per_peer:4
|
52 |
| - /sys/module/ko2iblnd/parameters/credits:84 |
| 61 | + /sys/module/ko2iblnd/parameters/credits:1024 |
53 | 62 | /sys/module/ko2iblnd/parameters/dev_failover:1
|
54 | 63 | /sys/module/ko2iblnd/parameters/fmr_cache:1
|
55 | 64 | /sys/module/ko2iblnd/parameters/fmr_flush_trigger:384
|
@@ -112,7 +121,7 @@ NOTE: These changes will not persist on file system upgrade and should be reappl
|
112 | 121 | ```console
|
113 | 122 | [root@hpelus1n01 ~]# vim /mnt/nfsdata/images/$(nodeattr -UV ver)/appliance.x86_64/etc/modprobe.d/ko2iblnd.conf
|
114 | 123 |
|
115 |
| - options ko2iblnd conns_per_peer=4 ntx=2048 peer_credits=42 peer_credits_hiw=64 concurrent_sends=256 credits=1024 map_on_demand=1 |
| 124 | + options ko2iblnd conns_per_peer=4 ntx=2048 peer_credits=42 peer_credits_hiw=64 concurrent_sends=84 credits=1024 map_on_demand=1 |
116 | 125 | ```
|
117 | 126 |
|
118 | 127 | 7. Recreate the SquashFS image after updating the `ko2iblnd.conf` file in the image.
|
@@ -163,7 +172,7 @@ NOTE: These changes will not persist on file system upgrade and should be reappl
|
163 | 172 | /sys/module/ko2iblnd/parameters/cksum:0
|
164 | 173 | /sys/module/ko2iblnd/parameters/concurrent_sends:84
|
165 | 174 | /sys/module/ko2iblnd/parameters/conns_per_peer:4
|
166 |
| - /sys/module/ko2iblnd/parameters/credits:84 |
| 175 | + /sys/module/ko2iblnd/parameters/credits:1024 |
167 | 176 | /sys/module/ko2iblnd/parameters/dev_failover:1
|
168 | 177 | /sys/module/ko2iblnd/parameters/fmr_cache:1
|
169 | 178 | /sys/module/ko2iblnd/parameters/fmr_flush_trigger:384
|
|
0 commit comments