01 · Architecture
One packet, four ownership domains
Linux owns the packet first. XNIC maps the packet buffer for DMA, publishes a descriptor, and rings an MMIO doorbell. The emulated device owns that descriptor until it writes completion. The driver then orders its reads, unmaps the buffer, and returns ownership to Linux.
Deliberate scope
One RX ring, one TX ring, 64 descriptors by default, no jumbo frames, offloads, multiqueue, RSS, MSI-X, SR-IOV, or production-family compatibility. Small scope makes ownership and recovery defensible rather than hidden behind features.
02 · PCI device contract
Registers, descriptors, and ordering
The driver binds only to PCI ID 8086:100e, maps BAR0, and accesses registers through ordered MMIO accessors. It negotiates coherent descriptor DMA separately from streaming packet-buffer mappings.
| Register group | Implemented responsibility |
|---|---|
| CTRL / STATUS | Global reset, set-link-up, posted-write flush, and link observation. |
| ICR / IMS / IMC / ITR | Interrupt cause acknowledgement, masking, re-enable, and moderation. |
| RDBAL/H · RDLEN · RDH · RDT | One coherent legacy receive ring. |
| TDBAL/H · TDLEN · TDH · TDT | One coherent legacy transmit ring and doorbell. |
| RAL / RAH / EERD | Station MAC and QEMU-compatible EEPROM fallback. |
TX invariant
- Reserve one descriptor so full and empty never alias.
- Map the skb for
DMA_TO_DEVICE. - Publish descriptor contents with
dma_wmb(). - Write
TDT; hardware owns the entry. - After descriptor-done, use
dma_rmb(), unmap, free, and advance cleanup.
RX invariant
- Map an empty skb for
DMA_FROM_DEVICE. - Return the descriptor through
RDT. - Validate completion, EOP, errors, and frame length.
- Install a replacement before returning the slot.
- Refill failure withholds ownership and schedules recovery.
next = (current + 1) & (ring_count - 1)
empty: next_to_use == next_to_clean
full: next_to_use + 1 == next_to_clean (mod ring_count)03 · Concurrency and recovery
The interrupt/NAPI handoff
The hard interrupt reads and acknowledges ICR, masks causes, and schedules NAPI. NAPI cleans TX and consumes RX only up to its budget. Interrupts stay masked while budget is exhausted. On completion, NAPI changes its state before causes are re-enabled, closing the lost-wakeup window.
| Path | Context | Serialization |
|---|---|---|
| Transmit enqueue / completion | Networking caller / NAPI softirq | tx_lock |
| Receive completion | NAPI softirq | Single NAPI instance |
| Interrupt | Hard IRQ | Mask plus NAPI state |
| Open / close | Process context | RTNL plus reset_lock |
| Recovery | Workqueue | RTNL plus reset_lock |
Important lock-order result: close must not synchronously cancel a reset worker while holding RTNL; the worker may be waiting for RTNL, producing a circular wait. Removal cancels work only after unregister_netdev().
04 · Qualification evidence
Expected versus observed
The recorded environment was QEMU 11.0.3 with Apple HVF, Ubuntu ARM64, Linux 6.8.0-136, GCC 13, sparse, tcpdump, and decoded PCAP inspection.
| Scenario | Observed result |
|---|---|
| ICMP and ring wrap | Pass 646,400/646,400 replies, zero loss, 10,100 RX and TX wrap deltas. |
| UDP and TCP | Pass Host received the UDP payload; a 64 MiB TCP stream completed. |
| NAPI budget | Pass Below-budget completion and 328 exact-budget exhaustions under an above-budget stream. |
| Ring-full recovery | Pass Deterministic 64-entry doorbell stall, queue stop, release, wake, and traffic recovery. |
| Fault recovery | Pass RX allocation failure, TX watchdog, malformed descriptors, and probe stages 1–5. |
| Reset under traffic | Pass 100 resets with automatic recovery and no module reload. |
| Lifecycle | Pass 1,000 interface cycles and 100 complete rebind cycles. |
| Mixed concurrency | Pass 30-minute run, 95 iterations, 107 cumulative resets, no driver error counters. |
| KFENCE | Pass Enabled with no report in the observed runs. |
| KASAN / lockdep | Unavailable Not enabled by the guest kernel; no pass is claimed. |
| MSI execution | Not observed Allocation is implemented; QEMU selected legacy INTx. |
The public repository retains raw logs, PCAPs, counters, environment versions, and exact commands for independent inspection.
05 · Debugging diary
Observed failures and corrected hypotheses
Runtime bug: ring-full transition was invisible
- Symptom
- A stalled TX ring stopped Linux’s queue, but the
tx_ring_fulldiagnostic did not reliably advance. - Wrong hypothesis
- Completion cleanup was racing the deliberate doorbell stall.
- Corrected evidence
- Flood ping generated too little deterministic pressure. A 4,096-frame AF_PACKET sender reproduced ownership independently of ping flow control.
- Root cause
- Full state was diagnosed only on a future transmit call. Linux had already honored the queue stop and was not required to call again.
- Fix
- Detect the transition immediately after consuming the final usable descriptor while holding the same lock as completion cleanup.
Build bugs: current kernel APIs moved
Clean Linux 6.17 CI rejected the obsolete PCI_IRQ_LEGACY name; current upstream uses PCI_IRQ_INTX. The W5500 module also included an unaligned-access header that moved between Linux 6.8 and 6.17. Explicit two-byte big-endian helpers removed that cross-version dependency. Both fixes were then compiled with GCC and sparse against ARM64 6.8 and x86-64 6.17 headers.
W5500 metadata bug: the obvious ID was still wrong
The SPI core warned that no ID matched xnic,w5500-lab. Adding the complete compatible string did not fix it. Inspection showed the core strips the vendor prefix and matches w5500-lab; adding that suffix produced 100 silent load/unload cycles.
06 · W5500 physical-lab track
A second driver designed for real wires
The W5500 driver uses Socket 0 in MACRAW mode with all 16 KiB TX and RX memory assigned to it. Synchronous SPI can sleep, so TX runs in a work item and RX in a threaded interrupt—not NAPI or hard-IRQ context.
Software complete
- SPI mode 0 VDM register and buffer transfers
- 16 KiB boundary split and stable size-register reads
- TX backpressure, threaded RX, link polling, ethtool counters
- Serialized reset and deterministic
force_resettrigger - ARM64 Linux 6.8 GCC/sparse build, ten contract tests, overlay build
- 100 unbound module lifecycles with zero kernel messages
Physical evidence pending
- 3.3 V rail and reset timing on an oscilloscope
VERSIONR == 0x04and SPI framing on a logic analyzer- ARP, ICMP, UDP, TCP, and broadcast PCAPs
- Buffer wrap, saturation, 100 link cycles, 100 resets
- KASAN/KFENCE/lockdep on the board kernel
- 30 minutes of mixed physical traffic
Raspberry Pi wiring contract
| Pi pin | BCM signal | WIZ850io |
|---|---|---|
| 17 | 3.3 V | 3.3 V |
| 20 | Ground | Ground |
| 19 | GPIO10 / MOSI | MOSI |
| 21 | GPIO9 / MISO | MISO |
| 23 | GPIO11 / SCLK | SCLK |
| 24 | GPIO8 / CE0 | SCSn |
| 22 | GPIO25 | INTn |
| 18 | GPIO24 | RSTn |
Current truth: no board is connected. This is implemented, software-qualified preparation for physical bring-up—not evidence that physical bring-up has happened.
07 · DPDK and cloud gates
Completed mechanics, pending hardware
DPDK result that exists
A compact DPDK 23.11 rte_ethdev forwarder initializes one port and queue, uses mempool-backed bursts, performs L2 forwarding, frees unsent mbufs after partial TX, counts drops, and shuts down cleanly on signals. The virtual net_pcap PMD executed 20 RX and 20 TX with zero drops; deterministic partial-TX and SIGTERM paths passed.
ENA result that does not exist yet
Real ENA validation requires a dedicated DPDK interface while the management interface remains kernel-owned. Infrastructure must pass testpmd first, then preserve topology, hugepages, IOMMU groups, frame sizes, offered load, xstats, loss, and termination evidence.
EFA / RDMA result that does not exist yet
Two EFA nodes must share a subnet and Availability Zone. The gate requires fi_info, fi_pingpong, and fi_rdm_bw with CPU affinity, message sizes, repetitions, tail latency, bandwidth, errors, and a TCP baseline.
fi_info -p efa -t FI_EP_RDM
fi_pingpong -p efa <peer-private-address>
fi_rdm_bw -p efa <peer-private-address>Cloud status: no ENA or EFA runtime evidence has been published. The preflight is read-only and creates no resources.
08 · Validation boundaries
What the evidence establishes
Validated
- Clean-room Linux PCI Ethernet driver in C against QEMU’s 82540EM-compatible interface
- DMA rings, descriptor ownership, MMIO ordering, interrupt/NAPI synchronization, and recovery
- Fault injection, tcpdump/PCAP investigation, GCC, sparse, and cross-kernel CI
- Functional DPDK application mechanics through a virtual PMD
- W5500 SPI driver implementation and software preflight
Not validated
- Physical-silicon or board bring-up
- Oscilloscope, logic-analyzer, or JTAG validation
- Production or upstream driver ownership
- Real-NIC DPDK performance or ENA execution
- RDMA, EFA, ConnectX, or RoCE execution
09 · Reproduce XNIC
From clean checkout to a bound driver
The recorded environment is reproducible on an Apple Silicon Mac with Homebrew, QEMU/HVF, an Ed25519 SSH key, 4 GiB available RAM, and roughly 8 GiB free storage. TCG is the slower behavioral fallback.
git clone https://github.com/blackdragoon26/xnic-v1.git
cd xnic-v1
./scripts/host/bootstrap-macos.sh
./scripts/host/fetch-guest.sh
./scripts/host/run-qemu.shWith QEMU running, use a second terminal:
./scripts/host/sync-to-guest.sh
ssh -p 2222 xnic@127.0.0.1
cd ~/xnic-v1
sudo ./scripts/guest/setup.sh
make
sudo ./scripts/guest/bind-driver.sh
sudo ./scripts/guest/qualification-suite.shThe suite writes new timestamped evidence rather than overwriting the baseline. Compare its raw output with the expected-versus-observed matrix before calling the run successful.