Topic 17118 min readStructured Tutorial

Process Communication in Python

Quick Answer

Process Communication explains process communication is essential when multiple processes need to exchange data or coordinate actions.

Learning Objectives

Explain the purpose of Process Communication in a practical learning context.
Identify the main ideas, terms, and decisions involved in Process Communication.
Apply Process Communication in a simple real-world scenario or practice task.

Introduction

Process communication is essential when multiple processes need to exchange data or coordinate actions.

Python provides several mechanisms to enable communication between processes, especially in the multiprocessing module.

Effective communication between processes is key to building efficient concurrent applications.

Understanding Process Communication

In multiprocessing, processes run independently with separate memory spaces. To share data or synchronize, they must communicate explicitly.

Python offers multiple communication methods such as pipes, queues, shared memory, and managers.

Processes do not share memory by default.
Communication mechanisms help exchange data safely.
Choosing the right method depends on the use case.

Communication Methods in Python Multiprocessing

Let's explore the main communication methods available in Python's multiprocessing module.

Comparison of Python Process Communication Methods
Method	Description	Use Case
Pipe	A two-way communication channel between two processes.	Simple two-process communication.
Queue	A thread and process-safe FIFO queue.	Multiple producers and consumers.
Shared Memory	Memory segments accessible by multiple processes.	High-performance data sharing.
Manager	Server process managing shared objects.	Complex shared state across processes.

Using Pipes

Pipes provide a connection between two processes allowing them to send and receive data.

They are simple but limited to two endpoints.

Practical Examples

Let's look at simple examples demonstrating process communication using pipes and queues.

Practical Example

This example creates a pipe between two processes. The sender process sends a message, and the receiver process receives and prints it.

This example demonstrates a producer putting items into a queue and a consumer retrieving them. The consumer stops when it receives None.

Examples

Example: Using Pipe for Process Communication

from multiprocessing import Process, Pipe

def sender(conn):
    conn.send('Hello from sender')
    conn.close()

def receiver(conn):
    msg = conn.recv()
    print('Received:', msg)
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p1 = Process(target=sender, args=(parent_conn,))
    p2 = Process(target=receiver, args=(child_conn,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

This example creates a pipe between two processes. The sender process sends a message, and the receiver process receives and prints it.

Example: Using Queue for Multiple Producers and Consumers

from multiprocessing import Process, Queue
import time

def producer(q):
    for i in range(5):
        q.put(f'item {i}')
        time.sleep(0.1)

def consumer(q):
    while True:
        item = q.get()
        if item is None:
            break
        print('Consumed:', item)

if __name__ == '__main__':
    q = Queue()
    p1 = Process(target=producer, args=(q,))
    p2 = Process(target=consumer, args=(q,))
    p1.start()
    p2.start()
    p1.join()
    q.put(None)  # Signal consumer to exit
    p2.join()

This example demonstrates a producer putting items into a queue and a consumer retrieving them. The consumer stops when it receives None.

Best Practices

Choose the simplest communication method that fits your needs.
Avoid sharing complex objects directly; use serialization if needed.
Close connections and clean up resources properly.
Use queues for multiple producers and consumers to avoid race conditions.
Test inter-process communication thoroughly to avoid deadlocks.

Common Mistakes

Assuming processes share memory by default.
Not closing pipes or queues leading to resource leaks.
Using non-thread-safe data structures for communication.
Ignoring synchronization when accessing shared memory.
Overcomplicating communication when simpler methods suffice.

Hands-on Exercise

Implement Producer-Consumer with Queue

Write a Python program using multiprocessing.Queue where one process produces numbers and another consumes and prints them.

Expected output: Consumer prints all produced numbers and exits cleanly.

Hint: Use put() to add items and get() to retrieve them. Use a sentinel value to signal completion.

Experiment with Shared Memory

Create two processes that share an array using multiprocessing.shared_memory and modify it concurrently.

Expected output: Both processes can read and write to the shared array.

Hint: Use SharedMemory and numpy arrays for easy manipulation.

Interview Questions

What is the difference between a pipe and a queue in Python multiprocessing?

Interview

A pipe provides a two-way communication channel between two processes, while a queue is a thread and process-safe FIFO structure that supports multiple producers and consumers.

How does shared memory improve process communication?

Interview

Shared memory allows multiple processes to access the same memory block directly, reducing serialization overhead and improving performance for large data sharing.

What is Process Communication, and why is it useful?

Beginner

Process communication is essential when multiple processes need to exchange data or coordinate actions.