Concurrent Shell Command Queues翻译背景介绍How to run shell commands in parallel while limiting their concurrency.I had a use case recently where I wanted to speed up some Bash processing that I was doing, but still keep concurrency constrained. So I wondered: is there a straightforward way to create thread pools in Bash? The answer is: yes, but it takes some work to be semi-resilient and portable.翻译结构俚语You may ask: why do this in Bash at all? It seems silly to use a language with poor concurrency controls. You'd be right to ask this question, and my answer is: because I have a fondness for Bash, and I wanted the challenge. But you're still probably better off using another language, such as Python.翻译结构俚语Here is code that creates a command queue that accepts shell commands to execute. You do not need to know all the commands that you want executed up-front, you can submit commands to it over time.翻译结构俚语bashqueue_start initializes the queue (managed by mkfifo) and starts the executor (concurrency managed by xargs -P), queue_submit submits commands to the queue, and queue_wait waits for the queue to drain before cleaning up. Calling queue_wait is optional, command processing will start as soon as you queue it with queue_submit.翻译结构俚语You can copy and paste this code into a script, or it's also safe to put in your dotfiles, as it only sets some of the exit traps for interactive shells. It works in Bash and Zsh, and it has as much defensive programming as I know how to add, but I'm confident it has failure modes that are difficult to predict or avoid.翻译结构俚语Example usageMy original use case had to do with making a large number of web requests, so here's an example using Postman Echo :● ● ●翻译结构俚语shellA more deterministic scenario demonstrating that concurrency is respected, only starting command three after command two has completed:● ● ●翻译结构俚语shellCommand queues are inherited by subshells:● ● ●翻译结构俚语shellDifferent subshells can have their own command queue:● ● ●翻译结构俚语shellQueueing each line of a command's output as a command to execute:● ● ●翻译结构俚语shellQueueing each line of a file as a command:● ● ●翻译结构俚语shellKnown issuesHere are some reasons why you might just want a different programming language:A shell can only have one queue.The amount of additional code needed to support multiple is complicated and likely isn't needed in most cases. Subshells can still have their own queues.翻译结构俚语Command output is intentionally not streamed.This prevents interleaving the output of most commands. Output larger than PIPE_BUF may still get interleaved.queue_wait doesn't return a value indicating any job failure.**This would require a temporary file or a second file descriptor and some complicated code.翻译结构俚语queue_wait has to wait for running processes started after queue_start.Processes started after queue_start will inherit the queue write file descriptor (which is what allows subshells to work), and xargs won't see the end of the queue until every copy of the write descriptor is closed.翻译结构俚语_queue_stop indiscriminately terminates descendants.Any other backgrounded jobs will be terminated, including ones started before queue_start.Why not just xargs -P?If you know all commands that you want to execute up-front, then the solution gets far simpler:翻译结构俚语● ● ●shellor翻译结构俚语● ● ●shellGNU parallel works similarly, though macOS doesn't ship with it:翻译结构俚语● ● ●shell翻译结构俚语