NVMe over Fabrics Explained: NVMe/TCP, RDMA, Discovery and Security

NVMe over Fabrics (NVMe-oF) carries the NVMe storage command set across a network instead of a PCIe bus, so a server can use a remote flash namespace almost as if it were a local drive. It runs over three transport families: RDMA (RoCEv2, InfiniBand, iWARP), Fibre Channel and plain TCP. A secure deployment combines per-host access lists, in-band authentication (DH-HMAC-CHAP, renamed KX-HMAC-CHAP in 2026), TLS 1.3 for NVMe/TCP, and a storage network that is isolated from everything else.

The protocol itself does none of that by default. A Linux target exported over TCP with allow_any_host switched on will hand a block device to anyone who can reach port 4420 and knows, or discovers, the subsystem name. This post goes one level deeper than our iSCSI explainer. It covers how the specifications evolved, the object model (NQNs, subsystems, controllers, namespaces, queues and capsules), how the transports differ, how discovery works with discovery controllers, centralised discovery controllers and mDNS, what the security features actually do, which Linux kernels support them, and a lab that builds an NVMe/TCP target, connects to it, then adds authentication and TLS.

Key takeaways

• NVMe-oF 1.0 was released in June 2016 (the published PDF is dated 5 June 2016) with RDMA and Fibre Channel transports. The NVMe/TCP transport binding was ratified and published in November 2018, and NVMe-oF 1.1 (dated 22 October 2019) folded TCP in. The refactored NVMe 2.0 family in 2021 moved fabrics into the Base Specification and split each transport into its own document; the TCP Transport Specification 1.0 is dated 2 June 2021.
• As of 4 August 2026 the current set is NVMe Base Specification 2.4 plus separate PCIe (1.4), RDMA (1.3) and TCP (1.3) transport specifications. Base 2.4 renamed the DH-HMAC-CHAP authentication protocol to KX-HMAC-CHAP because it now covers post-quantum key encapsulation as well as Diffie-Hellman.
• Every host and every NVM subsystem has an NVMe Qualified Name (NQN), up to 223 bytes, either nqn.yyyy-mm.reverse.domain:string or nqn.2014-08.org.nvmexpress:uuid:<uuid>. Access control, authentication and TLS identities are all keyed on NQNs.
• IANA assigned TCP port 4420 to NVMe over Fabrics and TCP port 8009 to NVMe-oF discovery. The TCP transport specification says 8009 is the default discovery port and that I/O controllers have no default port, although 4420 is the common choice.
• Discovery works through discovery controllers that return a discovery log page. TP8010 added Centralized Discovery Controllers (CDCs) that hosts and Direct Discovery Controllers register with, and TP8009 added DNS-SD and mDNS discovery under the _nvme-disc service name.
• NVMe/TCP TLS is PSK based. Implementations that support TLS must support TLS 1.3 and the TLS_AES_128_GCM_SHA256 suite, should support TLS_AES_256_GCM_SHA384, and must support PSK with (EC)DHE. TLS 1.2 should not be used, and versions before 1.2 must not be.
• Linux gained the RDMA host and target in kernel 4.8, Fibre Channel in 4.10, NVMe/TCP in 5.0, in-band authentication on both host and target in 6.0 (released 2 October 2022) and TLS for NVMe/TCP on both sides in 6.7 (released 7 January 2024), with the TLS handshake done in user space by tlshd.

What NVMe over Fabrics is, and how the specifications evolved

NVMe started as a way for host software to talk to flash over PCIe with many deep queues and very little per-command overhead. The NVM Express consortium describes its specifications as defining "how host software communicates with non-volatile memory across multiple transports". PCIe was the first transport. Fabrics are everything else.

The NVM Express page on the original fabrics specification says NVMe-oF 1.0 was released in June 2016 and extended NVMe "to additional transports beyond PCIe, such as Ethernet, Fibre Channel, Infiniband and RDMA". The same page says NVMe-oF 1.1, released in 2019, added finer grained I/O resource management, end-to-end flow control, NVMe/TCP and improved fabric communication. The consortium keeps the standalone fabrics document only as a historical reference now, because the 2.0 refactor moved its content elsewhere.

NVMe/TCP arrived first as a technical proposal. An NVM Express blog post published on 15 November 2018 announced that the ratified "NVMe/TCP Transport Binding specification" was available for public download and described it as mapping NVMe queues, capsules and data delivery onto TCP. The transport was then incorporated into NVMe-oF 1.1.

In 2021 the consortium refactored everything into the NVMe 2.0 family. Its "Changes in NVM Express Revision 2.0" page lists the new structure: a Base Specification, separate command set specifications, and three transport specifications (PCIe, RDMA and TCP). The theory of operation for fabrics, discovery and authentication now lives in the Base Specification, and each transport document only describes how capsules map onto that wire.

Dated timeline of the fabrics specifications

The kernel version rows come from reading the drivers/nvme/host/Kconfig and drivers/nvme/target/Kconfig files at each release tag, which shows exactly when an option such as NVME_TCP, NVME_TARGET_AUTH or NVME_TCP_TLS first exists. Distribution kernels often backport these features, so check your vendor's kernel rather than the version number alone.

The model: hosts, subsystems, controllers, namespaces, NQNs, queues and capsules

NVMe-oF reuses the vocabulary of local NVMe, and most confusion comes from mixing up the layers. From the outside in:

Host. The machine that issues commands. It has a host NQN, usually stored in /etc/nvme/hostnqn on Linux, and a host identifier in /etc/nvme/hostid.

NVM subsystem. The unit you connect to and the unit access control applies to. It has a subsystem NQN, contains one or more controllers, and owns one or more namespaces. On a storage array a subsystem is often one tenant or one application's volume group.

Controller. The per-association interface a host talks to. On a fabric, controllers are usually created dynamically: when a host connects to a subsystem, the target allocates a controller ID for that association. A host can have controllers to the same subsystem through several ports, which is how native NVMe multipathing works.

Namespace. The actual block storage, roughly what SCSI called a LUN. A subsystem exposes namespaces by namespace ID (NSID), and Linux names the resulting block device /dev/nvmeXnY.

NQN. The Base Specification defines the NVMe Qualified Name as a UTF-8 string with a maximum of 223 bytes, null terminated, in one of two formats. The first is for organisations that own a domain: nqn., a yyyy-mm date when the naming authority owned the domain, the reversed domain, then a colon and an owner chosen suffix, for example nqn.2014-08.com.example:nvme:nvm-subsystem-sn-d78432. The second needs no naming authority: nqn.2014-08.org.nvmexpress:uuid: followed by a 128-bit UUID. The spec says NQNs are compared byte for byte, without locale or Unicode specific logic. NQNs are identifiers, not secrets: they appear in discovery log pages, in Connect commands and in TLS PSK identities.

Discovery NQN. The well known name nqn.2014-08.org.nvmexpress.discovery identifies a discovery subsystem. Newer specifications also allow a discovery subsystem to have a unique NQN, which matters for bidirectional authentication because a host cannot authenticate a controller whose name it does not know.

Queues. Each association has one Admin Queue and one or more I/O Queues, each a submission and completion pair. On PCIe these are ring buffers in host memory. On a fabric each queue maps onto a transport connection: for NVMe/TCP, each queue is its own TCP connection. The host creates the Admin Queue with a Fabrics Connect command, then creates I/O queues with further Connect commands, one per queue.

Capsules. The Base Specification defines a capsule as "an NVMe unit of information exchange used in NVMe over Fabrics". A command capsule carries a 64 byte submission queue entry and may carry data or scatter gather lists in-line. A response capsule carries the 16 byte completion. Data that does not fit in-line moves separately: RDMA uses remote memory reads and writes, NVMe/TCP uses dedicated data PDUs, and Fibre Channel uses its own information units.

Fabrics commands. Beyond the normal Admin and I/O commands, fabrics adds Connect, Property Get and Property Set (replacing the PCIe controller registers), Disconnect, and Authentication Send and Receive for in-band authentication.

Put together, a connection looks like this: the host contacts a discovery controller, reads a discovery log page that lists subsystems and their transport addresses, opens an Admin Queue connection to the chosen subsystem with Connect (sending its host NQN and host ID), optionally authenticates, reads controller properties, enables the controller, identifies namespaces, then opens I/O queues.

How the transports compare: RDMA, Fibre Channel and TCP

The command set is identical on every fabric. What changes is how capsules and data cross the wire, what hardware you need and how the network has to behave.

NVMe/TCP

The TCP Transport Specification maps each NVMe queue onto one TCP connection and frames everything as Protocol Data Units (PDUs). A connection opens with an Initialize Connection Request (ICReq) PDU from the host and an Initialize Connection Response (ICResp) from the controller, which negotiate options such as data alignment and digests. After that, command capsules travel in CapsuleCmd PDUs, responses in CapsuleResp PDUs, and bulk data in H2CData and C2HData PDUs, with Ready to Transfer (R2T) PDUs used for flow control on writes.

Two optional digests protect against corruption. The host asks for a header digest (HDGST) or data digest (DDGST) in ICReq, and each is on only if both sides set the bit. The specification computes both with CRC32C: they catch bit errors, but they are not cryptographic integrity. The spec treats a header digest error as fatal, because the receiver can no longer trust the PDU length and loses its place in the byte stream, and a data digest error as non-fatal.

On ports, the specification says TCP port 4420 is assigned to NVMe over Fabrics and 8009 to NVMe-oF discovery, that 8009 is the default discovery port, and that there is no default port for I/O controllers. I/O controllers may use 4420 or the dynamic and private range 49152 to 65535 and should not use 8009. Port 4420 must not be used for both NVMe/iWARP and NVMe/TCP on the same IP address on the same network, which is a useful detail if you mix transports on one subnet.

NVMe/TCP's appeal is that it runs on ordinary Ethernet with ordinary NICs and needs no lossless network. The NVM Express page on the transport says it lets data centres "utilize their existing ubiquitous Ethernet infrastructure with multi-layered switch topologies and traditional Ethernet network adapters". The cost is CPU: the host and target run the full TCP stack and copy data, so per-I/O CPU use and tail latency are higher than RDMA unless the NIC offloads some of the work.

NVMe/RDMA: RoCEv2, InfiniBand and iWARP

RDMA lets one machine read and write registered memory on another without involving the remote CPU in each transfer. NVMe/RDMA puts command and response capsules in RDMA SEND messages and moves data with RDMA READ and WRITE operations against buffers the host advertises in the command. That keeps the target CPU out of the data path and gives the lowest latency of the three families.

There are three RDMA fabrics. InfiniBand is its own network with its own switches and subnet manager. RoCEv2 runs RDMA over UDP and IP on Ethernet; IANA lists UDP port 4791 as "IP Routable RocE", registered by the InfiniBand Trade Association. iWARP runs RDMA over TCP. RoCEv2 is the common choice on Ethernet, and in practice operators configure the network to avoid drops for RDMA traffic (priority flow control and explicit congestion notification are the usual tools), because RoCE's transport recovers poorly from loss. That tuning across every switch in the path is the main operational cost.

NVMe/FC

FC-NVMe carries NVMe capsules over Fibre Channel, using the same switches, host bus adapters and zoning model as a Fibre Channel SCSI SAN. Fibre Channel is already a lossless, credit based fabric designed for storage, and many arrays and HBAs can run SCSI and NVMe at the same time on the same ports, which makes FC-NVMe the least disruptive path for shops that already run Fibre Channel. Its security model is the Fibre Channel one: fabric login, zoning on switches, masking on arrays, and Fibre Channel security protocols where deployed. NIST SP 800-209 lists switch and host authentication (its example is FC-SP-2 AUTH-A) as a recommendation for FC SAN and NVMe-oF networks.

Transport summary

Latency figures depend so heavily on NICs, CPUs, queue depth and media that any single number would mislead. The ordering most operators see is RDMA lowest, then FC, then TCP, with TCP closing the gap as offloads improve. Measure on your own hardware before choosing.

How discovery works

A host needs three things before it can connect: the transport type, the transport address and port, and the subsystem NQN. Discovery is how it learns them.

Discovery controllers and the discovery log page

A discovery controller is a special controller that exposes no namespaces. A host connects to it (by default, to the well known discovery NQN), reads the Discovery Log Page and disconnects or stays connected for change notifications. Each log page entry gives a transport type (TRTYPE), address family, transport address (TRADDR), transport service identifier (TRSVCID, the TCP port for NVMe/TCP), the subsystem NQN, and a Transport Specific Address Subtype (TSAS). For TCP, the TSAS carries a SECTYPE field: 0 for no security, 1 for TLS 1.2 (described only by the obsolete fabrics specification and not to be used), 2 for TLS 1.3. A Transport Requirements (TREQ) field says whether a secure channel is required, not required or not specified.

Two things follow. First, the discovery log page is a map of your storage: every subsystem name and address the controller is willing to list. Second, it tells hosts whether TLS is required, so a target that requires TLS should say so in TREQ and a host should honour it.

Direct and centralised discovery controllers (TP8010)

The Base Specification defines two kinds of discovery controller. A Direct Discovery Controller (DDC) lives on a storage subsystem and can register its discovery information with a Centralized Discovery Controller (CDC). A CDC "reports discovery information registered by Direct Discovery controllers and hosts". It is the NVMe/TCP answer to the Fibre Channel name server: one place where storage registers what it offers, hosts register who they are, and an administrator defines which hosts may see which subsystems.

Registration uses the Discovery Information Management command, which registers, deregisters or updates entries. A DDC can push its information to a CDC (push registration) or ask the CDC to pull it (pull registration, triggered with a Kickstart Discovery Request). Hosts register too, which lets the CDC enforce zoning: a host that queries a CDC sees only the subsystems that are zoned to it. The nvme-stas project documents this as TP8010, the "NVMe-oF Centralized Discovery Controller" proposal, and implements "Fabric Zoning-aware automated connection management".

mDNS and DNS-SD (TP8009)

The Base Specification's section on automated discovery for IP fabrics says a host learns the IP address of a discovery controller by administrative configuration, by DNS-SD (RFC 6763), or by some other means. DNS-SD records can come from multicast DNS (RFC 6762) or from a normal DNS server. Service names take the form _nvme-disc._tcp.local for NVMe/TCP, with _udp for RoCE and _tcp again for iWARP, and _cdc._sub._nvme-disc to find a CDC specifically. The spec notes that mDNS only reaches one broadcast domain, so crossing subnets needs an RFC 8766 discovery proxy or an "mDNS NVMe-oF proxy". The nvme-stas README identifies this as TP8009, "Automated Discovery of NVMe-oF Discovery Controllers for IP Networks".

Zero configuration discovery is convenient on a dedicated storage VLAN. On a shared network it means any device that can send multicast can advertise itself as a discovery controller, so hosts that act on mDNS should only do so on interfaces dedicated to storage, and the result should be cross checked against an allow list or a CDC. nvme-stas leaves zeroconf disabled by default; you turn it on with zeroconf=enabled in stafd.conf.

What nvme-cli and nvme-stas do

nvme-cli is the standard Linux command line tool for NVMe, built on libnvme; its README says the pair supports local drives and "NVMe-oF fabrics (TCP, RDMA, FC)". For fabrics the important subcommands are nvme discover (connect to a discovery controller and print the log page), nvme connect (create a controller for one subsystem), nvme connect-all (discover and connect to everything listed), nvme disconnect, nvme list and nvme list-subsys. A JSON configuration file under /etc/nvme/ can make connections persistent.

nvme-cli is stateless: it runs, connects and exits. The nvme-stas README explains why that is not enough for TP8009 and TP8010, which need something that watches mDNS continuously and reacts to change notifications from a CDC. nvme-stas provides two systemd daemons: stafd finds discovery controllers (through Avahi for mDNS or from manual configuration), connects to them and caches their log pages, and stacd connects to the I/O controllers those log pages list.

One version note: nvme-cli 3.0 reorganised key handling under nvme keys. The old nvme gen-dhchap-key, nvme gen-tls-key and nvme check-tls-key still exist as deprecated aliases for nvme keys gen-kxchap-secret, nvme keys gen-tls-psk and nvme keys check-tls-psk, and the --dhchap-secret connect option is marked deprecated in favour of --kxchap-secret. Most distributions still ship 2.x, so the lab below uses the 2.x names and shows the 3.x names alongside.

What security the specification provides

NVMe-oF has four layers of protection you can turn on. None of them is on by default in a fresh Linux target.

Access control by host NQN

The simplest control is an allow list. A subsystem lists the host NQNs that may connect; Connect commands from any other host NQN are rejected. On Linux this is the allowed_hosts directory under each subsystem, plus the attr_allow_any_host switch that, when set to 1, disables the check entirely.

On its own this is weak. The host NQN is self asserted in the Connect command and visible to anyone who can see the traffic or read a host's /etc/nvme/hostnqn. It stops mistakes, such as a test server mounting a production volume, but not an attacker on the storage network. That is what authentication is for.

In-band authentication: DH-HMAC-CHAP, now KX-HMAC-CHAP

In-band authentication runs inside the NVMe association, after Connect and before the host is allowed to do anything else, using the Authentication Send and Authentication Receive commands. The Base Specification describes the protocol as CHAP (RFC 1994) "enhanced to use the Hashed Message Authentication Code (HMAC) mechanism" with stronger hash functions and an optional key exchange.

The exchange has four or five messages. The host sends AUTH_Negotiate listing the hash functions and key exchange groups it accepts. The controller replies with a Challenge naming the chosen hash and group, a sequence number, a random challenge and its key exchange value. The host answers with a Reply containing its HMAC response, its own key exchange value and, for bidirectional authentication, a challenge of its own. The controller sends Success1, including its response to the host's challenge when bidirectional authentication is in use, and the host confirms with Success2.

Key details from Base 2.4:

• Each host and subsystem has a secret associated with its NQN. The spec warns that two entities with the same key can impersonate each other, so keys must differ per entity.
• SHA-256 and SHA-384 must be supported; SHA-512 is defined. The spec notes that some security policies outside NVMe prohibit SHA-256.
• The finite field groups ffdhe2048, ffdhe3072, ffdhe4096, ffdhe6144 and ffdhe8192 from RFC 7919 are defined, and Base 2.4 adds elliptic curve and ML-KEM options. That is why it renamed the protocol: "This authentication protocol was known as DH-HMAC-CHAP in the NVM Express Base Specification revision 2.3 and prior." KX-HMAC-CHAP stays backward compatible when an FFDHE group or no key exchange is used.
• Without a key exchange the protocol is called HMAC-CHAP. The spec says that with insufficiently random keys a passive eavesdropper may recover the key with an off-line dictionary attack, and that an active attacker can downgrade the exchange to HMAC-CHAP. Every implementation must be configurable to require a key exchange.
• Secrets are exchanged in a text representation that starts DHHC-1:, followed by a two digit transform indicator, the base64 secret and a CRC. The transform indicator (00 none, 01 SHA-256, 02 SHA-384, 03 SHA-512) tells the consumer to transform the secret with the NQN before use.

Authentication proves who is on each end. It does not encrypt the data. On its own, a successful authentication followed by clear text I/O still exposes every block to anyone who can capture the traffic, and an active attacker on the path could in principle tamper with I/O after authentication. That is why the specification links authentication to TLS for NVMe/TCP.

TLS 1.3 for NVMe/TCP

The TCP Transport Specification 1.3 states the rules plainly. TLS is optional for NVMe/TCP. TLS versions before 1.2 and all versions of SSL must not be used, TLS 1.2 should not be used, and implementations that support TLS must support TLS 1.3. TLS for NVMe/TCP is based on pre-shared keys rather than certificates. Implementations must support TLS_AES_128_GCM_SHA256 and should support TLS_AES_256_GCM_SHA384, must support PSK with (EC)DHE key exchange so that a leaked PSK does not expose recorded traffic, and must let administrators disable PSK-only key exchange. The secp384r1 group is mandatory, and Revision 1.3 recommends the SecP384r1MLKEM1024 hybrid group for environments whose policies require post-quantum key exchange.

The PSK handling is precise:

• A configured PSK is what an administrator provisions on both host and subsystem, in an interchange format beginning NVMeTLSkey-1:. Subsystems should support a different configured PSK per host.
• A retained PSK is derived from the configured PSK with HKDF, using the host NQN as context, and the configured PSK "shall be destroyed as soon as the retained PSK is generated and stored".
• A generated PSK comes out of a KX-HMAC-CHAP exchange instead of being provisioned.
• The TLS PSK used in the handshake is derived from the retained or generated PSK, with a PSK identity string built from NVMe, a version digit, R or G for retained or generated, a hash indicator, the host NQN, the subsystem NQN and a base64 HMAC digest of the PSK.

The discovery log page entry's SECTYPE value of 2 announces that a port speaks TLS 1.3.

Secure channel concatenation

Base 2.4 lets AUTH_Negotiate carry a Secure Channel Concatenation (SC_C) field that ties authentication to TLS. With SC_C set to NEWTLSPSK, a KX-HMAC-CHAP exchange on an Admin Queue over plain TCP generates a new PSK, and the host then sets up TLS using that generated PSK. With REPLACETLSPSK, an exchange on an Admin Queue that is already inside TLS replaces the association's PSK. NOSC means authentication only and is allowed on any queue. That lets a site provision only authentication secrets and have TLS keys created per association. nvme-cli exposes this as --concat. The spec prohibits concatenation over a secure channel that was set up directly under a transport specification, such as an ordinary NVMe/TCP TLS connection, although replacing that channel's PSK is permitted.

Zoning, masking and isolation

The NVMe protocol cannot protect a network it does not control. NIST SP 800-209, Security Guidelines for Storage Infrastructure (October 2020), treats storage networks as a separate security domain and gives recommendations that map directly onto NVMe-oF:

• AC-SS-R24: restrict which hosts can reach which block devices through zoning and masking "to the minimum required access".
• AC-SS-R25: apply the same restriction to snapshots, replicas and point in time copies, which are often forgotten.
• AC-SS-R26: set the default zone to "deny all".
• AC-SS-R27: separate environments (development, test, production) and traffic types (data, management, replication, backup).
• NC-SS-R1: give every host and switch a unique identity and authenticate it before it joins the network.
• NC-SS-R3 and NC-SS-R4: blend host, switch and storage based zoning, and mask "as close to the data as possible".

For NVMe/TCP the equivalents are a dedicated VLAN or VRF for storage traffic, firewall rules that allow ports 4420 and 8009 only from host subnets, per-host allow lists on each subsystem, CDC zoning where you run a CDC, and management interfaces on a separate network again.

Linux kernel support by feature

The TLS options in Kconfig say they enable "TLS encryption for NVMe TCP using the netlink handshake API" and point to the ktls-utils project for the handshake daemon. The kernel does record encryption with kTLS, while tlshd in user space performs the handshake and looks up PSKs in the kernel keyring (the .nvme keyring). The ktls-utils README says the kernel needs net/handshake support, which arrived in 6.5, plus CONFIG_TLS, CONFIG_KEYS and CONFIG_KEYS_REQUEST_CACHE.

One target side warning: the NVME_TARGET_AUTH_DEBUG option's help text says it adds debug messages "including the generated DH-HMAC-CHAP secrets". Never enable it on a production target.

What goes wrong on an unprotected fabric

The failure modes are conceptual and well known from iSCSI and Fibre Channel. None needs a software bug.

• Open subsystems. A Linux subsystem with attr_allow_any_host set to 1, or an array volume mapped to "all hosts", lets any machine that can reach the port connect and read or write the namespace. A writable block device in the wrong hands means data theft, or file system corruption that ends up looking like a disk failure.
• Discovery as reconnaissance. A discovery controller answering on 8009 or 4420 lists subsystem NQNs and addresses to whoever asks. That is by design, so the discovery port should only be reachable from hosts that need it.
• Host NQN spoofing. Without authentication, an allow list only checks a name the host chooses to send.
• Clear text data. Without TLS, NVMe/TCP carries every block in the clear. The CRC32C digests detect accidental corruption, not deliberate modification.
• Rogue discovery. With mDNS discovery on a shared segment, a device can advertise itself as a discovery controller and steer hosts to it.
• Key reuse. Using one DH-HMAC-CHAP secret for every host defeats the point; the Base Specification itself warns that entities sharing a key can impersonate each other.
• Snapshot exposure. Hosts get a production volume locked down while snapshots and replicas stay mapped widely, the gap NIST's AC-SS-R25 addresses.

NVMe/TCP vs iSCSI, and what a migration looks like

iSCSI (consolidated in RFC 7143, April 2014) carries SCSI commands over TCP. NVMe/TCP carries NVMe commands over TCP. Both run on standard Ethernet and standard NICs, both use TCP for reliability, and both name endpoints with qualified names (IQNs for iSCSI, NQNs for NVMe). IANA registers TCP 3260 for iSCSI and 4420 and 8009 for NVMe-oF.

The differences are in the command model and the queueing. iSCSI translates block I/O into SCSI commands, and a session typically funnels through one connection per path. NVMe/TCP keeps NVMe's multi-queue model, with one TCP connection per I/O queue, usually one queue per CPU core, so each core can submit and complete I/O without sharing locks. That is where most of the latency and CPU per I/O advantage comes from, rather than from the wire format.

Security looks similar on paper and differs in detail. iSCSI authentication is CHAP inside the login phase; RFC 7143 defines CHAP with MD5 for that exchange, and iSCSI relies on IPsec for confidentiality. NVMe/TCP authentication is HMAC-CHAP with SHA-256 or stronger and an optional Diffie-Hellman exchange, and its confidentiality is TLS 1.3 defined in its own transport specification. In practice, IPsec for iSCSI was rarely deployed, and TLS for NVMe/TCP is newer and still not universally supported by arrays, so check both ends before planning around it.

A migration usually runs in this order:

1. Inventory. List each iSCSI initiator, target portal, IQN, LUN, multipath configuration and file system. Note which hosts use CHAP and which rely on network isolation only.
2. Check support. Confirm the array exports NVMe/TCP, which features it supports (in-band authentication, TLS, CDC registration) and how it handles multipathing (Asymmetric Namespace Access, ANA). Confirm host kernels and nvme-cli versions.
3. Plan names and keys. Generate a stable host NQN per host (nvme gen-hostnqn), decide subsystem naming, and generate one authentication secret per host.
4. Network. Put NVMe/TCP on the storage VLAN, allow 4420 (and 8009 if you use a discovery port) only from host subnets, and check MTU end to end if you use jumbo frames.
5. Pilot. Present a new namespace over NVMe/TCP alongside the existing iSCSI LUN, copy data at the block or file level, and test failover by pulling paths.
6. Cut over. Switch the mount or volume group to the NVMe device, keep the iSCSI LUN read only for rollback, then remove it.
7. Multipath. Linux NVMe uses native multipathing in the kernel by default on most distributions rather than device-mapper multipath, so remove or adjust old multipath rules for these devices.
8. Decommission. Remove the iSCSI targets, CHAP secrets and firewall rules so nothing reachable is left behind.

Step by step: build a lab NVMe/TCP target, connect, then add authentication and TLS

This lab uses two Linux machines (or VMs) on an isolated network: a target at 192.0.2.10 and a host at 192.0.2.20, both running a kernel of 6.7 or later for the TLS step (6.0 or later is enough for authentication). It uses the kernel's own nvmet target through configfs, so you can see each object the spec describes. Run it only on a lab network.

Warning: this lab exports a block device. Anything the host writes to the exported namespace overwrites what is on that device. Use a file backed loop device created for the purpose, never a disk holding data, and check the device path before each export step.

1. Load modules and create a scratch backing device on the target

# On the target
sudo modprobe nvmet
sudo modprobe nvmet-tcp
mountpoint -q /sys/kernel/config || sudo mount -t configfs none /sys/kernel/config

# A 1 GiB scratch file, attached to the next free loop device
truncate -s 1G /var/tmp/nvmet-lab.img
LOOPDEV=$(sudo losetup --find --show /var/tmp/nvmet-lab.img)
echo "Lab backing device: $LOOPDEV"

Check the device path before going further. losetup -l should show $LOOPDEV pointing at /var/tmp/nvmet-lab.img, and lsblk "$LOOPDEV" should show a 1 GiB device with no mount points.

losetup -l "$LOOPDEV"
lsblk "$LOOPDEV"

2. Create the subsystem and namespace

SUBNQN=nqn.2026-10.lab.example:nvme:lab1
cd /sys/kernel/config/nvmet

sudo mkdir subsystems/$SUBNQN
# Do NOT allow any host: we will add an explicit allow list
echo 0 | sudo tee subsystems/$SUBNQN/attr_allow_any_host

sudo mkdir subsystems/$SUBNQN/namespaces/1
# Check: $LOOPDEV must be the scratch loop device from step 1
echo -n "$LOOPDEV" | sudo tee subsystems/$SUBNQN/namespaces/1/device_path
echo 1 | sudo tee subsystems/$SUBNQN/namespaces/1/enable

Each mkdir creates a spec object: the subsystem directory is the NVM subsystem named by SUBNQN, and namespaces/1 is namespace ID 1.

3. Create a TCP port and bind the subsystem to it

sudo mkdir ports/1
echo tcp        | sudo tee ports/1/addr_trtype
echo ipv4       | sudo tee ports/1/addr_adrfam
echo 192.0.2.10 | sudo tee ports/1/addr_traddr
echo 4420       | sudo tee ports/1/addr_trsvcid
sudo ln -s /sys/kernel/config/nvmet/subsystems/$SUBNQN ports/1/subsystems/$SUBNQN

The addr_* attributes are the same fields a discovery log page entry reports: transport type, address family, address and service identifier. Linking the subsystem into the port is what makes it reachable there.

4. Allow exactly one host

On the host, find or create its NQN:

# On the host
sudo modprobe nvme-tcp
cat /etc/nvme/hostnqn 2>/dev/null || nvme gen-hostnqn | sudo tee /etc/nvme/hostnqn

Back on the target, create a host object with that exact string and link it into the subsystem's allow list:

# On the target; paste the host's NQN
HOSTNQN=nqn.2014-08.org.nvmexpress:uuid:REPLACE-WITH-HOST-UUID
cd /sys/kernel/config/nvmet
sudo mkdir hosts/$HOSTNQN
sudo ln -s /sys/kernel/config/nvmet/hosts/$HOSTNQN subsystems/$SUBNQN/allowed_hosts/$HOSTNQN

Firewall the port at the same time. With nftables, for example, accept TCP 4420 only from the host's address on the storage interface and drop the rest.

5. Discover and connect from the host

# On the host
nvme discover -t tcp -a 192.0.2.10 -s 4420
nvme connect  -t tcp -a 192.0.2.10 -s 4420 -n nqn.2026-10.lab.example:nvme:lab1
nvme list
nvme list-subsys

The discovery output lists one entry with trtype: tcp, trsvcid: 4420, subnqn set to the lab NQN and a sectype of none. The Linux target answers discovery on the ports you configure, which is why the lab passes -s 4420 explicitly instead of relying on a default discovery port. nvme list shows a new /dev/nvmeXnY of 1 GiB. Disconnect before the next step:

nvme disconnect -n nqn.2026-10.lab.example:nvme:lab1

6. Enable DH-HMAC-CHAP, then make it bidirectional

Generate one secret for the host and one for the controller. With nvme-cli 2.x:

# On the host (nvme-cli 2.x). nvme-cli 3.x: nvme keys gen-kxchap-secret -m 1
HOSTKEY=$(nvme gen-dhchap-key -m 1 -n "$(cat /etc/nvme/hostnqn)")
CTRLKEY=$(nvme gen-dhchap-key -m 1 -n nqn.2026-10.lab.example:nvme:lab1)
echo "$HOSTKEY"   # DHHC-1:01:...:

The 01 after DHHC-1: says the secret is to be transformed with SHA-256. Move the two secrets to the target over a channel you trust (not chat, not a ticket) and configure the host object:

# On the target
H=/sys/kernel/config/nvmet/hosts/$HOSTNQN
echo -n "DHHC-1:01:PASTE-HOST-SECRET:"       | sudo tee $H/dhchap_key
echo -n "DHHC-1:01:PASTE-CONTROLLER-SECRET:" | sudo tee $H/dhchap_ctrl_key
echo "hmac(sha384)" | sudo tee $H/dhchap_hash
echo ffdhe3072      | sudo tee $H/dhchap_dhgroup

dhchap_key is what the host must prove it knows. dhchap_ctrl_key is what the controller proves it knows when the host asks for bidirectional authentication. Setting a DH group other than null keeps the exchange from falling back to HMAC-CHAP without key exchange, which the spec says is open to off-line dictionary attacks with weak secrets.

Connect with both secrets (with nvme-cli 3.x the options are --kxchap-secret and --kxchap-ctrl-secret):

# On the host
nvme connect -t tcp -a 192.0.2.10 -s 4420 \
  -n nqn.2026-10.lab.example:nvme:lab1 \
  --dhchap-secret="$HOSTKEY" \
  --dhchap-ctrl-secret="$CTRLKEY"

Passing secrets on the command line puts them in shell history and the process list. For anything beyond a lab, put them in the nvme-cli JSON configuration with tight file permissions.

7. Check the result, including a negative test

# On the host
nvme list-subsys
dmesg | grep -i -E 'nvme.*(auth|dhchap)' | tail
ls /sys/class/nvme/nvme*/dhchap_secret

# Negative test: no secret must fail
nvme disconnect -n nqn.2026-10.lab.example:nvme:lab1
nvme connect -t tcp -a 192.0.2.10 -s 4420 -n nqn.2026-10.lab.example:nvme:lab1 \
  || echo "rejected as expected"

The kernel log on both sides normally records whether authentication succeeded. A connection with no secret, or with the wrong one, must fail. If it succeeds, check that attr_allow_any_host is 0 and that the host object has dhchap_key set.

8. Optional: add TLS 1.3 on NVMe/TCP

TLS needs kernel 6.7 or later built with NVME_TCP_TLS and NVME_TARGET_TCP_TLS, and the tlshd daemon from ktls-utils running on both machines with access to the .nvme keyring (check tlshd.conf(5) on your distribution). Generate a configured PSK and insert the derived TLS PSK on each side:

# On the host (nvme-cli 2.x). nvme-cli 3.x: nvme keys gen-tls-psk / nvme keys insert-tls-psk
nvme gen-tls-key --hostnqn="$(cat /etc/nvme/hostnqn)" \
  --subsysnqn=nqn.2026-10.lab.example:nvme:lab1 --hmac=1 --insert

Run the matching insert on the target with the same configured PSK, host NQN and subsystem NQN, then mark the port as TLS only. The port must be disabled while you change it, which in practice means unlinking the subsystem first:

# On the target
cd /sys/kernel/config/nvmet
sudo rm ports/1/subsystems/$SUBNQN
echo tls1.3 | sudo tee ports/1/addr_tsas
cat ports/1/addr_treq          # should now report a secure channel as required
sudo ln -s /sys/kernel/config/nvmet/subsystems/$SUBNQN ports/1/subsystems/$SUBNQN

The kernel refuses tls1.3 if TLS target support is not built in or if no keyring is available, and when TLS is enabled it sets the port's transport requirement to "required" by default. Connect with --tls:

# On the host
nvme discover -t tcp -a 192.0.2.10 -s 4420   # sectype should now read tls13
nvme connect  -t tcp -a 192.0.2.10 -s 4420 -n nqn.2026-10.lab.example:nvme:lab1 --tls
journalctl -u tlshd --since "5 min ago"

A capture on the storage interface should show TLS records and no NVMe/TCP PDUs in the clear after the handshake. Once both authentication and TLS work, consider --concat, which derives the TLS PSK from the authentication exchange so you provision only authentication secrets.

9. Tear down

# On the host
nvme disconnect -n nqn.2026-10.lab.example:nvme:lab1
# On the target: unlink, disable, remove in reverse order
cd /sys/kernel/config/nvmet
sudo rm ports/1/subsystems/$SUBNQN
sudo rmdir ports/1
echo 0 | sudo tee subsystems/$SUBNQN/namespaces/1/enable
sudo rmdir subsystems/$SUBNQN/namespaces/1
sudo rm subsystems/$SUBNQN/allowed_hosts/$HOSTNQN
sudo rmdir subsystems/$SUBNQN hosts/$HOSTNQN
# Check the path before detaching: it must be the lab loop device
losetup -l "$LOOPDEV" && sudo losetup -d "$LOOPDEV"

For a reusable target, the nvmetcli tool can save and restore this configfs tree as JSON, which is easier to review in version control than a shell history.

Comparison: iSCSI, NVMe/TCP, NVMe/RDMA and NVMe/FC

A hardening checklist that maps to the spec

Where Safeguard fits

Safeguard does not configure storage arrays, switches or Linux targets, does not inspect storage traffic, and does not analyse array, NIC or HBA firmware. Nothing in the lab above depends on it. The fit is with the software around the fabric. Teams run storage controllers, CSI drivers, provisioning scripts, nvme-cli and nvme-stas packaging, and monitoring exporters as code and container images built from open source packages. Safeguard scans repositories, container images and SBOM files (safeguard scan --dir ., --image <name>, --sbom <file>), shows which projects carry a given component and version, matches them against vulnerability data, ranks findings with reachability analysis and applies policies and gates. When a fix ships for a library inside your storage automation or a containerised storage agent, that tells you which builds still carry the vulnerable version. Array operating systems and drive or adapter firmware come from their vendors, and their fixes still have to come from those vendors.

Common mistakes

• Leaving attr_allow_any_host at 1. It is the quickest way to get a lab working and the most common reason a namespace is reachable by hosts that should never see it.
• Treating the allow list as authentication. Host NQNs are self asserted. Without KX-HMAC-CHAP, anyone who learns an allowed NQN can present it.
• One secret for every host. The Base Specification warns that entities sharing a key can impersonate each other. Generate one per host, and a separate controller secret for bidirectional authentication.
• Authentication without key exchange. A null DH group leaves HMAC-CHAP, which the spec says can be attacked off-line if secrets are weak and can be forced by an active attacker. Require a key exchange group.
• Assuming digests are security. HDGST and DDGST are CRC32C checksums. They catch bit flips, not tampering.
• Authenticating but not encrypting. Authentication proves identity at connect time; it does not hide or protect the blocks that follow. Use TLS on NVMe/TCP or a physically isolated fabric.
• Running mDNS discovery on a shared network. Zero configuration is convenient on a dedicated storage VLAN and risky anywhere else.
• Exposing 4420 and 8009 beyond the storage network. Discovery tells anyone who asks what you have.
• Forgetting snapshots and replicas. The production volume is locked down while a nightly snapshot is mapped to a wide host group.
• Secrets in shell history. --dhchap-secret on the command line ends up in history files and process listings. Use configuration files with restrictive permissions or the keyring.
• Following old nvme-cli examples blindly. nvme-cli 3.0 renamed key commands and changed how --secret is encoded; check nvme version and the man page you actually have.
• Leaving debug options on. NVME_TARGET_AUTH_DEBUG logs generated secrets.

FAQ

What is NVMe over Fabrics?

NVMe over Fabrics (NVMe-oF) is the part of the NVMe specifications that lets a host send NVMe commands to storage across a network instead of a local PCIe bus. Commands and responses travel in capsules over RDMA (RoCEv2, InfiniBand, iWARP), Fibre Channel or TCP. The first NVMe-oF specification was released in June 2016.

What is the difference between NVMe/TCP and NVMe/RDMA?

Both carry the same NVMe commands. NVMe/TCP runs over ordinary TCP on any IP network with standard NICs and needs no lossless Ethernet, at the cost of more CPU per I/O. NVMe/RDMA uses RDMA reads and writes so the target CPU stays out of the data path, giving lower latency but needing RDMA capable NICs and, for RoCEv2, a network tuned to avoid drops.

What port does NVMe/TCP use?

IANA assigns TCP 4420 to NVMe over Fabrics and TCP 8009 to NVMe-oF discovery. The TCP transport specification makes 8009 the default discovery port and gives I/O controllers no default, though 4420 is the usual choice; the discovery log page tells the host which port to use.

What is an NQN in NVMe?

An NVMe Qualified Name is the unique name of a host or NVM subsystem, up to 223 bytes. It is either nqn.yyyy-mm.reversed.domain:suffix for organisations that own a domain, or nqn.2014-08.org.nvmexpress:uuid: followed by a UUID. NQNs are identifiers, not secrets.

Is NVMe over TCP encrypted?

Not by default. The TCP transport specification makes TLS optional. Where it is used it must be TLS 1.3 with pre-shared keys, the TLS_AES_128_GCM_SHA256 suite at minimum and PSK with (EC)DHE support. Linux supports it from kernel 6.7 with the tlshd handshake daemon.

What is DH-HMAC-CHAP?

DH-HMAC-CHAP is NVMe's in-band authentication protocol: CHAP with HMAC-SHA-256 or stronger and an optional finite field Diffie-Hellman exchange, run with Authentication Send and Receive commands after Connect. NVMe Base Specification 2.4 renamed it KX-HMAC-CHAP because it now also supports elliptic curve and post-quantum key exchange. Linux supports it from kernel 6.0.

What is a centralized discovery controller?

A Centralized Discovery Controller (CDC), defined through TP8010, is a discovery service that storage subsystems and hosts register with. It returns to each host only the subsystems zoned to it, playing the role a name server plays in a Fibre Channel SAN. On Linux, nvme-stas acts as the host side client.

How does NVMe-oF discovery work with mDNS?

Discovery controllers can advertise themselves with DNS-SD under service names such as _nvme-disc._tcp.local, and hosts can find them with multicast DNS or a normal DNS server. This was added through TP8009. mDNS reaches only one broadcast domain, so crossing subnets needs a discovery proxy.

Should I move from iSCSI to NVMe/TCP?

If your array and host kernels support it, NVMe/TCP usually gives lower latency and lower CPU per I/O on the same Ethernet, because it keeps NVMe's per-core queues. Plan names and per-host secrets first, pilot alongside existing LUNs, and confirm whether both ends support in-band authentication and TLS before you rely on them.

How do I secure an NVMe over Fabrics network?

Isolate storage traffic on its own VLAN or fabric, firewall ports 4420 and 8009 to host subnets, disable "allow any host" and use per-host allow lists, require bidirectional KX-HMAC-CHAP with a key exchange group and unique secrets, require TLS 1.3 on NVMe/TCP, keep mDNS discovery off shared networks, and mask snapshots and replicas as tightly as production volumes.

Fix it without waiting for upstream

Software supply chain hygiene still matters on a storage fabric, because the targets, discovery daemons, CSI drivers, key handling tools and monitoring agents that run it are built from open source packages and shipped as images that get rebuilt and redeployed. The fit is indirect here: the array firmware and adapter firmware come from vendors, but the open source code your team builds around the fabric is yours to keep current.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

The first demo is a real session with the Safeguard CLI. The assistant reads a finding and its advisory, asks before each command, upgrades a dependency, runs the tests and pushes nothing. The same flow applies when a library inside a storage provisioning tool, a CSI driver you build in house or a containerised discovery agent needs an upgrade.

Click through it at your own pace, or open it full screen.

The second demo sets up the Safeguard runner on your own machine, approves it with a device code and then scans from the terminal. Storage automation repositories and target images often live on internal hosts next to the storage network, and the runner lets you scan them from a machine you control instead of opening that network up.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and lets you choose which tools Claude can use. Granting a narrow set of tools is the same habit as a subsystem allow list with attr_allow_any_host set to 0: each party gets only what it needs.

Click through it at your own pace, or open it full screen.

Download the apps at safeguard.sh/download and read the CLI docs at docs.safeguard.sh/docs/cli.

Quick check

Sources

• NVM Express: NVMe Set of Specifications (current revisions as of 4 August 2026)
• NVM Express: NVMe over TCP Transport Specification page and revision history
• NVM Express: NVMe over TCP Transport Specification, Revision 1.3 (ratified 31 July 2026)
• NVM Express: NVMe Base Specification, Revision 2.4 (31 July 2026)
• NVM Express: NVMe over Fabrics specification, historical reference (1.0, 1.0a, 1.1, 1.1a)
• NVM Express: Changes in NVM Express Revision 2.0
• NVM Express blog: Welcome NVMe/TCP to the NVMe-oF Family of Transports (15 November 2018)
• IANA: Service Name and Transport Protocol Port Number Registry (nvme 4420, nvme-disc 8009, roce 4791, iscsi-target 3260)
• NIST SP 800-209: Security Guidelines for Storage Infrastructure (October 2020)
• RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3 (August 2018)
• RFC 7143: Internet Small Computer System Interface (iSCSI) Protocol (Consolidated) (April 2014)
• Linux kernel: drivers/nvme/host/Kconfig
• Linux kernel: drivers/nvme/target/Kconfig
• Linux kernel: drivers/nvme/target/configfs.c
• Linux kernel documentation: NVMe subsystem
• Kernelnewbies: Linux 4.8, Linux 4.10, Linux 5.0, Linux 6.0 and Linux 6.7 release dates
• nvme-cli and libnvme README
• nvme-cli documentation: nvme-connect, fabrics options, nvme-discover
• nvme-cli documentation: nvme-gen-dhchap-key, nvme-keys-gen-kxchap-secret, nvme-gen-tls-key, nvme-keys-insert-tls-psk, nvme-check-tls-key
• libnvme
• nvme-stas: NVMe STorage Appliance Services (TP8009 and TP8010)
• ktls-utils: TLS handshake agent (tlshd) for in-kernel TLS consumers

Related reading: iSCSI explained, NVMe and ATA sanitize commands explained, How an SSD is made and How a rack server is made.

Last reviewed: 24 October 2026. NVMe specifications, nvme-cli option names and kernel features change between releases; check the specification revision, kernel configuration and tool versions you actually run before relying on a specific behaviour.