SANSAR: Cryptominers on Hijacked AI Clusters Forge NVIDIA's Telemetry Library So GPUs Report 0% While They Mine

Attackers entered GPU hosts through an exposed Ray job agent, swapped libnvidia-ml.so.1 to fake idle GPUs, and hid behind a kernel rootkit, Tor C2 and an Ethereum dead drop.
Overview
Most cryptojacking is caught the same way: someone notices a GPU pinned at 100% when it should be idle, or a power bill that no longer fits the workload. SANSAR is built to beat exactly that check. In a Linux intrusion that researchers track as REF3364, the attackers replaced NVIDIA's own management library on compromised GPU servers. Every tool that asked the driver "how busy is this card?" got back the answer "0%", while the GPUs were in fact mining for someone else.
The victims were AI compute hosts. The way in was not an NVIDIA flaw but an internet-reachable Ray job agent, the distributed Python framework widely used for model training and inference. Once inside, the operators deployed a modular toolkit: a Rust orchestrator controlled over Tor, off-the-shelf CPU and GPU miners, a kernel rootkit, a falsified NVML library, and a second backdoor that finds its controller through an Ethereum smart contract. The toolkit reached three GPU hosts, and mining was confirmed on two.
The Campaign
The weak point was a familiar one in AI infrastructure. On the head node, the Ray dashboard listened only on loopback, which looks safe. But the Ray job agent was reachable from the internet and accepted job submissions without authentication. In Ray, submitting a job means running arbitrary Python on the cluster, so an exposed job endpoint amounts to remote code execution by design. Ray's own security guidance says it must be reachable only from trusted networks. A wave of attacks on exposed Ray clusters was already seen in 2024, and this case shows the lesson still has not fully landed.
The SANSAR toolkit is staged in /opt/vadi_/ and consists of:
- sansar (Rust, v0.1.3): the orchestrator. It talks to an onion-service C2 through the embedded Arti Tor library, optionally via Lyrebird bridges (obfs4/WebTunnel), and supervises the miners. It also opens interactive shells and can launch raw TCP SYN floods on command.
- kurt (XMRig 6.25.0) and tilki (SRBMiner-MULTI 3.6.9): unmodified public CPU and GPU miners. Pool and wallet are pushed from the C2 at runtime, so the binaries contain neither.
- libnvidia-ml.so.1: the falsified NVIDIA Management Library (NVML).
- acpi_thermal_mgr.ko: an ftrace kernel rootkit disguised as a thermal driver.
- netns: an independent backdoor that runs as a systemd service.
Timeline
- 22 October 2026: external probes appear in the victim's Ray logs.
- 24 October 2026: a submitted job runs
nvidia-smiand sends the host's GPU inventory to an external server. The attackers were choosing which hardware was worth taking. - 25 October 2026: a Python reverse-shell job connects out. The actor adds an SSH key to the Ray session account's
authorized_keys, later gets root, and deploys the toolkit to three GPU hosts. - 12 November 2026: the researchers publish their analysis.
Tradecraft
Lying to the dashboard. The NVML trick is the most original part of the operation. It does not use LD_PRELOAD, which defenders already look for. Instead it overwrites the system's libnvidia-ml.so.1 and hides the original as /usr/lib/.libnvidia-ml.so.1.backup. The wrapper passes every call to the real library, then edits the results: utilisation is always 0%, temperature a random 36–46 °C, power a random 100–200 W. The numbers look healthy and slightly noisy. nvidia-smi, monitoring agents and exporters built on NVML all inherit the lie, because they read through the same library.
Hiding at the kernel level. The acpi_thermal_mgr rootkit hooks directory listing (getdents/getdents64), file opens, kill, and the kernel's TCP/UDP socket listings. Hidden processes (by default kurt and lyrebird) vanish from /proc, return "no such process" to PID lookups, and can only be signalled by the sansar manager itself. It is built through DKMS and wired into the initramfs, so it survives reboots and kernel updates. It can also remove itself from the module list, although that option was off by default.
C2 that is hard to take down. SANSAR registers over Tor with an HMAC-signed handshake and a custom binary protocol, and wipes shell history on every session (HISTFILE=/dev/null). The netns backdoor uses a different route: it makes read-only eth_call requests to public Ethereum RPC endpoints, decodes an IPv4 address stored in a smart contract, and connects to it on TCP 8080. To move their infrastructure, the operators only have to update the contract. There is no domain to seize.
Turkish fingerprints. The component names are Turkish words (kurt = wolf, tilki = fox, vadi = valley), and the rootkit source left in /usr/src/ has Turkish comments. That points to the developers' working language, not to a nation-state sponsor, and nothing here is tied to a named group. The motive is plainly financial, with a DDoS feature on the side.
Indicators of Compromise
SHA-256:
00c9ac44291f83c72d54302e99d155a3e26d02225a3e516764d5833a99657c9e(sansar orchestrator)675c24728f87f7aa0fa07527027c95c24c246049c65eaef89270ab04ea50a180(libnvidia-ml.so.1 wrapper)6ba1aa275e0e2826e8b48b36473539ca56cbdc41f429e539a45f8532b084d7a4(acpi_thermal_mgr.ko rootkit)df8ab6fd699e6ae90e63e9696bae8f01342708bafc2ed88af414337e4f237194(netns backdoor)96fc528ca5e7d1c2b3add5e31b8797cb126f704976c8fbeaecdbf0aa4309ad46(kurt / XMRig)acc706afc9322a3eae1ef49049749e0a7c9f1a123895ac56dc8a289beb2fb5b7(tilki / SRBMiner-MULTI, official build)cc1ad7f14777a098ccc91fd4e8f6143da14418c2b6fdf23b7f8f6b3d1b0b7f1d(lyrebird 0.8.1)
Network and blockchain:
- Onion C2:
efxisnqldelyr5ra25oqfrsut54b34g26e7zrsiuupsoekjof2gho2id[.]onion:9051 - Ethereum dead-drop contract:
0x43eb291f56688c63c93e8a93c4591fc997cf6349(selector0x6ded96e7) - netns controller: resolved IPv4 on TCP 8080
Host artefacts:
/opt/vadi_/,/opt/vadi_/nvml-hook//usr/lib/.libnvidia-ml.so.1.backup/usr/src/acpi_thermal_mgr-1.0/, kernel moduleacpi_thermal_mgrarti_data/,bridgesand.client_idnext to the orchestrator binary
Detection
The hard part is that the compromised host is the witness, and its testimony has been doctored. Detection has to rely on evidence the attacker does not control:
- Measure power out-of-band. Compare the GPU power reported by NVML with what the server's BMC (IPMI/Redfish) or the rack PDU measures. A node that "idles at 150 W" per
nvidia-smibut draws several kilowatts at the wall is mining. This is the single most reliable signal against SANSAR. - Verify the NVML library from outside. Hash
libnvidia-ml.so.1and compare it with the NVIDIA driver package or golden image, using tooling run from trusted media rather than from the suspect host. Look for a.libnvidia-ml.so.1.backupfile next to it. - Hunt for the rootkit through DKMS. Run
dkms statusand check/usr/src/and the initramfs for anacpi_thermal_mgrmodule. Real ACPI thermal drivers don't arrive by DKMS. - Watch egress from GPU nodes. Training hosts have no reason to speak Tor, run obfs4/WebTunnel transports, call public Ethereum RPC endpoints, or open outbound TCP 8080. Look up the host's public IP via ipify, ifconfig.me, icanhazip and checkip.amazonaws.com in sequence. Because the rootkit hides sockets, base this on network flow logs, not on
ssrun on the host. - Audit Ray. Review job-submission history for unfamiliar jobs running
nvidia-smi, reverse shells, or outbound connections, and checkauthorized_keyson Ray service accounts. - Use the published YARA rules
Linux_Trojan_Sansar_11eea454andLinux_Trojan_Netns_c94a4b17on disk images.
Recommended Actions
- Never expose Ray job agents, dashboards or GCS ports to the internet. Put them behind a VPN or an authenticated proxy, and limit who can submit jobs.
- Treat any confirmed host as fully compromised. A kernel rootkit with initramfs persistence means rebuilding from a clean image, not cleaning up in place.
- Add BMC/PDU power telemetry to your GPU monitoring and alert when it diverges from in-band utilisation.
- Enable Secure Boot with kernel module signature enforcement on GPU nodes so that DKMS-built unsigned modules cannot load.
- Keep NVIDIA driver files under file-integrity monitoring and rotate any credentials, tokens and SSH keys that were present on Ray nodes.
Analyst Assessment
SANSAR matters less for its miners, which are commodity, than for what it says about where attackers are going. GPU clusters are now worth enough that criminals invest in tooling built specifically to hide from GPU observability. Faking NVML output is a cheap trick with an outsized effect, because almost every GPU monitoring stack trusts that one library. We expect the technique to be reused, including on rented cloud GPUs where the victim sees only the dashboard and the invoice. Teams running AI infrastructure should assume that in-band telemetry can be forged and anchor their monitoring in signals the host cannot fake. Severity: High for organisations running internet-reachable Ray or similar job-execution frameworks on GPU hardware.
