Process spawning performance in Rust.
As part of my PhD studies, I’m working on a distributed task runtime called HyperQueue. Its goal is to provide an ergonomic and efficient way to execute task graphs on High-Performance Computing (HPC) distributed clusters, and one of its duties is to be able to spawn a large amount of Linux processes efficiently. HyperQueue is of course written in Rust1, and it uses the standard library’s Command API to spawn processes2. When I was benchmarking how quickly it can spawn processes on an HPC cluster, I found a few surprising performance bottlenecks, which I will describe in this post. Even though most of these bottlenecks are only significant if you literally spawn thousands of processes per second, which is not a very common use-case, I think that it’s still interesting to understand what causes them. Before we knew Rust, we were developing distributed systems in our lab in C++. Suffice to say… it was not a good idea.Well, it actually uses tokio::process::Command, but that’s just an async wrapper over std::process::Command anyway.
Read in full here:
This thread was posted by one of our members via one of our news source trackers.