PennyLane
Install
Install
  1. Blog/
  2. High-Performance Computing/
  3. There and back again: a 1.74 µs tale with Backline

November 16, 2026

There and back again: a 1.74 µs tale with Backline

Hong-Sheng Zheng

Hong-Sheng Zheng

Ali Farazdaghi

Ali Farazdaghi

Joseph Lee

Joseph Lee

Lee O'Riordan

Lee O'Riordan

Josh Izaac

Josh Izaac

What happens to Backline's round trip latency if you change one block of logic on the FPGA? We added support for NVIDIA's open-source Holoscan Sensor Bridge, kept the same workstation, network card and cable, and the round trip latency from the FPGA to a CPU and back fell from 2.47 µs to 1.74 µs. This is, to our knowledge, the lowest published for quantum controllers between an FPGA and a CPU over a switched network protocol.

Time spent moving data is time the decoder can't use: the shorter the round trip latency, the more room there is for decoding work.

In September, together with AMD, we announced Backline, an open platform for compiling quantum programs and running them across FPGAs, CPUs, and GPUs with low latency, all from Python. In October, we showed how to bring your own decoder to it. Today, we're opening up the other end of the link: the RDMA engine inside the FPGA itself.

An RDMA engine is the block of FPGA logic that turns data into network packets and back. In addition to AMD ERNIC™ support, we've added NVIDIA's open-source Holoscan Sensor Bridge (HSB), and measured the two on the same AMD Ryzen™ Threadripper™ PRO workstation, with the same network card and the same cable.

With HSB, the round trip latency from the FPGA to the CPU and back takes 1.74 µs, down from 2.47 µs with ERNIC. To our knowledge, that's the lowest published round trip for quantum control between a CPU and an FPGA over a switched network protocol. To an AMD Instinct™ MI210 GPU, it fell from 4.56 µs to 3.75 µs.

Read on for what changed, how these numbers compare with other published results, and how to choose an RDMA engine from PennyLane.

What's new?

Backline connects the control FPGA sitting next to the quantum hardware with the CPUs and GPUs running your decoders, using RDMA over Converged Ethernet (RoCE v2).

On the FPGA, the piece of logic that speaks RoCE (pronounced 'rocky') is the RDMA engine. Until now, Backline has used AMD's ERNIC, a full-featured, queue-based RDMA engine that is already part of many AMD Versal designs. With this update, you can now also use NVIDIA's Holoscan Sensor Bridge. HSB comes from NVIDIA's Holoscan platform, where it streams camera, radar, and RF data from FPGAs directly into GPU memory. It's lightweight, open source under the Apache 2.0 license, and a natural fit for a stream of syndrome data. (If HSB sounds familiar, that might be because it's also the FPGA IP behind NVQLink, NVIDIA's reference architecture for connecting quantum control systems to GPUs.)

From PennyLane, switching to HSB takes two settings. Simply load the HSB bitstream (the FPGA design) onto the board, pass backend="hsb_hwhs" to the controller, and add peer=hsb to the coprocessor's config:

import pennylane as qp

FPGA = qp.Controller(
    name="fpga-controller",
    remote=True,
    hardware="fpga",
    backend="hsb_hwhs",  # HSB; backend="hwhs" is ERNIC
    executor_options={**FPGA_SERVER, "port": 9811},
    init_args={"config": "reply_to_us=1000000"},
)

GPU = qp.Coprocessor(
    name="gpu-coproc",
    coprocessor_fn="gpu_steane_launcher",
    remote=True,
    hardware="gpu",  # the Instinct MI210
    endpoint=qp.Endpoint(WORKSTATION_ADDRESS, 7760),  # an address the FPGA board can reach
    executor_options={**WORKSTATION_SERVER, "port": 9813},
    init_args={"config": "dev=mlx5_1;gid=5;peer=hsb"},  # HSB; peer=catalyst is ERNIC
)

Here, FPGA_SERVER and WORKSTATION_SERVER hold the connection details for your FPGA board and workstation. Your QNodes, compiler passes, and decoders all stay the same.

To see what HSB brings to the table compared to our previous Backline benchmarking experiments, we kept everything else exactly the same — the same AMD Versal VPK120 FPGA, connected by a single cable to the same AMD Threadripper PRO workstation.

The setup: an AMD Versal VPK120 FPGA connected to a Threadripper PRO workstation by a direct attach cable
Backline's hardware handshake and NVIDIA's Holoscan Sensor Bridge on the AMD Versal™ VPK120 FPGA, connected by a single cable at 100 Gbps to an NVIDIA ConnectX®-7 network card in the Threadripper PRO workstation. The CPU or the Instinct MI210 GPU answers each round.

A new floor: 1.74 µs

On every workload we measured, HSB comes in 0.7 to 0.8 µs lower than ERNIC — both at the median, and out in the tail.

HSB against ERNIC: median and 99.9th percentile round trips for four workloads
HSB against ERNIC on the same workstation, a million rounds each. Filled dots are medians, hollow dots the 99.9th percentile.

The CPU results are where HSB really shines. The median round trip latency to the CPU is just 1.74 µs, and the 99.9th percentile sits only 28 ns above it. The two RDMA engines don't overlap at all — across a million rounds of the CPU echo each, HSB's slowest round (1.96 µs) was still faster than ERNIC's fastest (2.45 µs)!

Histograms of a million CPU echo round trips for HSB and ERNIC, which do not overlap
Every round of the CPU echo, a million per RDMA engine.

Including a decoder in the workflow doesn't change the picture. The Steane decoder is light enough that it adds nothing we could measure on the CPU, allowing HSB's savings to carry straight through. Even with a much heavier qLDPC decoder on the GPU, where decoding takes up most of the 32 µs round trip, HSB still saves 0.8 µs. This makes sense; the change is entirely on the FPGA side, so every workload benefits.

CoprocessorWorkloadHSB median (µs)ERNIC median (µs)HSB 99.9th pct. (µs)ERNIC 99.9th pct. (µs)
Threadripper PRO CPUEcho1.7362.4701.7642.610
Threadripper PRO CPUSteane decoder1.7362.4701.7642.610
Instinct MI210 GPUEcho3.7484.5554.4285.250
Instinct MI210 GPUSteane decoder3.8524.5504.5885.285
Instinct MI210 GPUqLDPC decoder31.8432.6532.5633.41
Round trip latencies from the VPK120 FPGA to the Threadripper PRO workstation and back, a million rounds each, with the first dropped. The Steane decoder workflow uses a compiled Steane decoder coprocessing function (for example, as per demo 3 from the paper) whereas the qLDPC demo is utilizing a Triton compiled GPU kernel.

For each measurement, we ran a million rounds (dropping the first as a warm-up). Logic in the FPGA fabric issued each round as soon as the previous reply arrived; we call this a hardware loop, run back to back. The timer runs on the FPGA itself, from sending the request to receiving the reply, allowing it to cover the entire path: the RDMA engine in both directions, the cable, the network card, and the CPU or GPU. We also tried issuing a round every 10 µs with HSB, rather than back to back, and saw the same median to within a few nanoseconds.

Going back to the latency tiers from our September post, this puts both the CPU and the GPU in the synchronous co-processing tier (1 to 100 µs) used for tasks like backup QEC decoding — but now with an extra 0.7 to 0.8 µs of every round available to the decoder itself.

How does this compare?

To put these numbers in context, we went looking for every published round trip latencies we could find between a quantum control FPGA and a CPU or GPU, across papers, technical blogs, and vendor announcements.

But first, what do we mean by "over a switched network protocol"? Our FPGA is network-attached. It has its own network port, and talks to the host's network card using a protocol built to run through network switches (RoCE v2, in our case), rather than sitting on a PCIe card inside the host or sharing a chip with the CPU. Among published round trip latencies between an open protocol network-attached FPGA and a CPU for quantum control, 1.74 µs is, to our knowledge, the lowest.

Published median round trips from a network-attached FPGA to a CPU or GPU and back
Published round trip latencies between a network-attached FPGA and a CPU or GPU (medians). Ours used a direct attach cable, as did NVIDIA's NVQLink and DOCA GPUNetIO results; the hardware and network path differ from result to result.

Note that we measured over a single direct attach cable between the VPK120 and the ConnectX-7, with no switch in between, comparable to NVIDIA's NVQLink and DOCA GPUNetIO results. A real installation may add switches, and each one adds its own latency in both directions (THQLink reports 0.13 µs for each extra switch hop on its network).

For CPUs, the lowest result we found from anyone else is THQLink, which connects an FPGA to an Intel Xeon server over the Tianhe supercomputer's proprietary interconnect. THQLink reports a median of 2.87 µs, 1.1 µs above Backline with HSB. NVIDIA also ships a CPU path for HSB, but we haven't yet seen a round trip latency published for it.

For GPUs, it's a different story. NVIDIA reported 3.84 µs for NVQLink in October 2025, and our 3.75 µs to the MI210 is on par with that. Since then, however, NVIDIA has brought its own HSB-based round trip latency down to 2.7 µs with DOCA GPUNetIO and 2.92 µs with cudaq-realtime, both on Blackwell GPUs — about 1 µs lower than ours.

We have a good idea of where most of that gap comes from. In NVIDIA's setup, the GPU kernel posts each reply to the network card itself, while in ours, the GPU kernel hands its reply off to a CPU thread to post. We believe that the extra 2 µs our GPU path takes over the CPU path is mostly spent on the host side, between the network card and the GPU kernel and back. (The hardware differs too, of course.) Letting the MI210 post its own replies could close some of that gap, and swapping the GPU may also yield better performance as well.

Under the hood: queues versus streams

So why is HSB faster? The answer lies in how each RDMA engine expects to be handed work.

On the FPGA, Backline's own controller logic — the hardware handshake, or hwhs — issues each round, numbers it, times it, and matches up the reply when it arrives. It does the same job whether ERNIC or HSB sits behind it, but isn’t identical. Each bitstream carries one RDMA engine, along with a hardware handshake built to fit it. So where does the difference come from?

  • ERNIC is queue-based, much like the RDMA network cards found in data-center servers. To send a message, the controller writes the payload into memory, builds a work request describing it, and rings a doorbell to let ERNIC know that work is waiting. ERNIC then fetches both from memory before it can build and send the packet. This approach is very general, and supports many connections, reliable delivery, and large transfers — which is what most RDMA applications need.

  • HSB, on the other hand, is a streaming bridge. The controller hands it a single 64-byte word, and HSB builds the RoCE packet immediately, with the reply coming back the same way. There are no work requests, no doorbells, and no memory reads on the FPGA side before anything is sent.

This shorter path is where most of the gain comes from. (HSB also runs at a faster clock on our board, 250 MHz compared to ERNIC's 200 MHz.)

It's worth noting that these design choices are intentional, and your RDMA engine of choice may differ depending on your needs. A queue-based design is the right choice for general-purpose RDMA; however for specific jobs — for example, sending a small syndrome as fast as possible — a streaming design removes overhead.

Best of all, you don't have to pick just one. The results above came from an AMD CPU, GPU, and FPGA, running NVIDIA's RDMA engine and network card — and mixing and matching like this is exactly what Backline was built for. Since everything else stays the same whichever RDMA engine sits on the FPGA, you can choose the one that best fits each experiment with a simple configuration change.

Which RDMA engine should I use?

Each RDMA engine makes a different trade-off, so you can pick the one that suits your experiment:

If you need...UseController backendCoprocessor peer
The shortest round trip over a direct attach cableNVIDIA HSB (Apache 2.0)hsb_hwhspeer=hsb
Reliable connections or many producers, with AMD IP already in many Versal designsAMD ERNIChwhs (default)peer=catalyst (default)

It's also worth knowing that HSB uses RDMA's unreliable connected mode, as NVQLink does. On a direct attach cable between the FPGA and the network card, this keeps acknowledgement traffic off the critical path. However, it also means there's no retransmission at the RDMA layer, and no RDMA reads. (HSB does use one-sided RDMA writes, though; that's how each request reaches the CPU or GPU in our setup.) If you need retransmission, ERNIC uses reliable connections.

What's next?

Changing just one block of logic on the FPGA brought the round trip latency to the CPU down from 2.47 µs to 1.74 µs — to our knowledge, the lowest published for quantum control between a CPU and an FPGA over a switched network protocol. To the GPU, it fell from 4.56 µs to 3.75 µs. That's 0.7 to 0.8 µs back in every single round, on every workload we measured, for the decoder to use.

But there's still plenty more to explore. On the GPU side, for example, having the GPU post its replies to the network card directly, rather than handing them to a CPU thread, could bring the round trip latency down even further. And with more of every round now free for decoding, we're excited to see which decoders you bring to Backline with our bring-your-own-decoder workflow.

Try it out and get involved

Ready to try HSB with Backline? Here's where to start:

  • Get the code: The HSB design for the VPK120, including its driver and the test coprocessors, is available in the backline-vpk120 repository. HSB itself comes from NVIDIA's Holoscan Sensor Bridge project.

  • Explore the technology: Read the Backline paper for a technical overview of the architecture and its original measurements.

  • New to Backline? Read the announcement, and see how to go from local CPU prototyping to low-latency FPGA-GPU communication in our demo. Then bring your own decoder.

Have an RDMA engine you'd like to see running with Backline? Let us know! Head over to our GitHub repository to check out the ongoing work and join in on the development, or pop over to the PennyLane discussion forum to tell us how you are using PennyLane, Backline, and Catalyst in your workflows.

And if you are as excited as we are, make sure to sign up for the PennyLane newsletter for the latest PennyLane features and updates!

Instinct, ERNIC, Ryzen, Threadripper, and Versal are trademarks of Advanced Micro Devices, Inc. NVIDIA, Holoscan, and ConnectX are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries.

About the authors

Hong-Sheng Zheng
Hong-Sheng Zheng

Hong-Sheng Zheng

Hong-Sheng Zheng is a compiler engineer on the Compilation team at Xanadu.

Ali Farazdaghi
Ali Farazdaghi

Ali Farazdaghi

Ali is a systems software developer focused on performance, low-latency communication, GPUs and compute accelerators.

Joseph Lee
Joseph Lee

Joseph Lee

Joseph is a physicist and software developer. At Xanadu, he leads the performance team and makes PennyLane go zoom.

Lee O'Riordan
Lee O'Riordan

Lee O'Riordan

Physicist, purveyor of angular momentum, GPUs, pointy guitars, and computational things. Working on quantum stuff.

Josh Izaac
Josh Izaac

Josh Izaac

Josh is a theoretical physicist, software tinkerer, and occasional baker. At Xanadu, he contributes to the development and growth of Xanadu’s open-source quantum software products.

Last modified: November 16, 2026

Related Blog Posts

Never miss a milestone

Get the latest quantum updates delivered to your inbox.

Join the list
PennyLane

PennyLane is an open-source quantum software platform for quantum computing, quantum machine learning, and quantum chemistry. Create meaningful quantum algorithms, from inspiration to implementation.

Created with ❤️ by Xanadu.

Research

  • Research

  • Performance

  • Hardware and simulators

  • Demos library

  • Compilation hub

  • Quantum datasets

Education

  • Teach

  • Learn

  • Codebook

  • Coding challenges

  • Videos

  • Glossary

Software

  • Install

  • Features

  • PennyLane documentation

  • Catalyst documentation

  • Development guide

  • How-to guides

  • API

  • GitHub


Xanadu

© Copyright 2026 | Xanadu | All rights reserved

TensorFlow, the TensorFlow logo and any related marks are trademarks of Google Inc.

Privacy policyTerms of serviceCookies policyCode of conduct