Python Concurrency: Multi-threading and Multi-processing

In Python, ThreadPoolExecutor and ProcessPoolExecutor are part of the concurrent.futures module, which provides a high-level interface for asynchronously executing callables. These executors are designed to simplify the use of threads and processes for parallel execution of tasks.

ThreadPoolExecutor Example

The ThreadPoolExecutor is used for executing tasks in a pool of threads. It’s suitable for I/O-bound tasks and when you want to limit the number of threads that run concurrently.

import concurrent.futures

def task(n):
    return f"Task {n} is executed"

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    futures = [executor.submit(task, i) for i in range(10)]
    for future in concurrent.futures.as_completed(futures):
        print(future.result())

In this example:

• We define a simple function task that simulates a task.
• We use ThreadPoolExecutor with a maximum of 5 worker threads.
• We submit 10 tasks to the executor.
• We use as_completed to retrieve the results as they are completed.

ProcessPoolExecutor Example

The ProcessPoolExecutor is ideal for CPU-bound tasks and leverages multiple processes, bypassing the Global Interpreter Lock (GIL) in CPython.

import concurrent.futures

def compute(n):
    # Simulate a CPU-bound computation
    result = 0
    for i in range(1000000):
        result += n * i
    return f"Result of computation {n}: {result}"

with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:
    futures = [executor.submit(compute, i) for i in range(4)]
    for future in concurrent.futures.as_completed(futures):
        print(future.result())

In this example:

• We define a compute function that performs a CPU-intensive task.
• We create a ProcessPoolExecutor with 4 worker processes.
• We submit 4 compute tasks to the executor.
• We collect and print the results as they are completed.

Points to Note

• The max_workers parameter defines the maximum number of threads or processes that the executor can use. Adjust it based on your task and the available resources.
• The submit method schedules the callable to be executed and returns a Future object, representing the execution’s eventual result.
• It’s important to use the with statement to ensure that the resources are properly cleaned up after the execution.
• For ProcessPoolExecutor, keep in mind that each process has its own memory space, so communication between processes is more costly compared to threads.