Parallel Python

Overview

Bypass the GIL with Worker Processes

Parallel Python (pp) provides a clean, robust framework for executing Python code concurrently across multi-core systems (SMP) and networked compute clusters. It is lightweight, zero-dependency, and designed to integrate smoothly into any Python application.

While multi-threading is standard for I/O-bound tasks in Python, CPython's Global Interpreter Lock (GIL) prevents CPU-bound threads from executing in true parallel fashion. Because only one bytecode instruction runs at any given moment, multi-threaded computational workloads remain serialized regardless of how many CPU cores your hardware provides.

Parallel Python breaks through the GIL by orchestrating isolated worker processes over high-performance inter-process communication (IPC). All low-level process lifecycles, bidirectional pipes, network sockets, serialization, and scheduling are handled automatically. Your application simply submits jobs and retrieves results. The exact same tasks run interchangeably on local CPU cores or on remote ppserver cluster nodes, with intelligent, dynamic load balancing across all available resources.

Architecture

How It Works

Your application interacts with a single pp.Server instance. Under the hood, the server manages an in-memory job queue and an intelligent FIFO/load-balanced scheduler, dispatching tasks across local worker subprocesses (via stdio IPC pipes) and remote cluster nodes (via TCP sockets to ppserver) without exposing any low-level networking or process plumbing to your code.

• Local Workers (SMP): Local workers run as dedicated Python subprocesses (python -m pp._worker) communicating over high-throughput standard I/O pipes. Defaulting to one worker per logical CPU core, local workers operate with zero network overhead and no socket port contention.
• Remote Cluster Nodes: Remote workers run under the ppserver daemon on external physical or virtual machines. Clients establish TCP connections (default port 60000) using an 8-byte length-framed binary protocol with content-addressed caching and SHA-1 challenge-response authentication.
• Content-Addressed Code Shipping: Functions, user-defined classes, and dependency modules are extracted and transmitted on first invocation. Remote and local nodes cache the compiled code by cryptographic content hash, ensuring subsequent submissions transfer only argument payloads.
• Mutual Challenge-Response Authentication: Remote cluster communication is protected by a shared secret passphrase. The server issues a cryptographic challenge verified with a salted SHA-1 digest before any remote command or code execution is permitted.
• Dynamic Resource Auto-Discovery: When ppserver -a is enabled, nodes broadcast their presence over UDP. Clients configured with ppservers=("*",) detect active nodes in real time, automatically expanding the compute pool and pruning unreachable hosts.
• Standard Output Redirection: Worker processes capture standard output during task execution. When a job completes, its return value and captured stdout are returned to the client, reproducing console output faithfully on the host application.
• Fine-Grained Execution Control: Dynamically scale worker pools with set_ncpus(), synchronize task batches by group tag with wait("group_name"), attach asynchronous completion callbacks, or leverage pp.Template for maximum batch submission throughput.

Quick Start

Get Started in Five Lines

pip install pp

Requires Python 3.10+ — pure Python, zero dependencies. Automatically provides the pp module and the ppserver command-line executable.

1. Multi-Core Execution on One Machine (SMP)

import pp

def square(x):
    return x * x

# Initialize the server (defaults to 1 worker per CPU core)
job_server = pp.Server()

# Submit tasks asynchronously for parallel execution
f1 = job_server.submit(square, (10,))
f2 = job_server.submit(square, (20,))
f3 = job_server.submit(square, (30,))

# Retrieve results (calling f() blocks until the task completes)
print(f1(), f2(), f3())

# Clean up worker processes and pipes when finished
job_server.destroy()

2. Distributed Execution on a Cluster

Launch the ppserver daemon on each remote computational node:

node-1$ ppserver
node-2$ ppserver
node-3$ ppserver

Connect from your client script by supplying the node hostnames or IP addresses:

import pp

ppservers = ("node-1", "node-2", "node-3:60000")
job_server = pp.Server(ppservers=ppservers)

# Submit jobs with helper functions and imported modules
f1 = job_server.submit(func1, args1, depfuncs=(helper1,), modules=("math",))
f2 = job_server.submit(func2, args2, depfuncs=(helper2,), modules=("numpy",))

# Retrieve results from whichever nodes finish first
print(f1(), f2())

job_server.destroy()

3. Cluster with UDP Auto-Discovery

Instead of hardcoding a static list of IP addresses, let cluster nodes announce themselves automatically. Start each node with -a and use a wildcard on the client:

ppserver -a

import pp

# Auto-discover all available nodes on the local network
job_server = pp.Server(ppservers=("*",))

# On subnets where broadcast is restricted, specify the broadcast target:
# job_server = pp.Server(ppservers=("*",), broadcast="192.168.1.255")

4. High-Throughput Batch Submissions with Templates

Use pp.Template to compile and package a function and its dependencies once, eliminating redundant serialization overhead when submitting large volumes of tasks:

import pp

def multiply(a, b):
    return a * b

job_server = pp.Server()
template = pp.Template(job_server, multiply)

# Submit 1,000 tasks with minimal serialization overhead
tasks = [template.submit(i, i * 2) for i in range(1000)]
results = [t() for t in tasks]
print("Sum of results:", sum(results))

job_server.destroy()

5. Asynchronous Callbacks and Job Groups

import pp

job_server = pp.Server()

def on_complete(job_id, result):
    print(f"[{job_id}] completed with result: {result}")

# The callback is invoked as on_complete(*callbackargs, result)
job = job_server.submit(my_task, (data,),
                        callback=on_complete,
                        callbackargs=("job-42",),
                        group="batch-1")

# Wait synchronously for all jobs in a specific group
job_server.wait("batch-1")
job_server.destroy()

Reference

Module API Reference

pp.Server

Instantiates a Parallel Python execution server managing local worker processes and/or remote cluster connections.

pp.Server Methods

pp.Template

Pre-compiles and serializes a function definition, dependency hierarchy, and import list once. Subsequent invocations via template.submit(*args) dispatch tasks with minimal overhead, maximizing throughput when fanning out large batches of identical computations.

pp.DestroyedServerError

Exception raised when attempting to retrieve the result of a task (task()) if the underlying pp.Server instance was destroyed before the computation finished.

Job Semantics & Best Practices

• Return Values & Output: Invoking job() blocks and returns the evaluated result. Standard output written by the worker during execution is captured and mirrored on the client console.
• Fault Resilience: If a worker subprocess exits unexpectedly or a remote node disconnects mid-computation, the server logs the incident, recycles the worker, and automatically reschedules uncompleted tasks.
• Named Lambdas & Functions: Functions defined inline as named assignments (e.g. f = lambda x: x * 2) and top-level functions are supported. Unnamed inline lambdas (e.g. submit(lambda x: x, ...)) cannot be resolved by name and raise an immediate ValueError.
• Serialization: All task arguments and return values are serialized with standard Python pickle. Arguments should be free of non-serializable objects (such as open file descriptors, network sockets, or mutex locks).

Remote Compute

ppserver Command-Line Daemon

The ppserver command turns any machine into a high-performance Parallel Python compute node. Start it on remote hosts, then reference them directly via pp.Server(ppservers=...) or let UDP auto-discovery connect them automatically.

# Start with 8 workers and UDP auto-discovery enabled
ppserver -w 8 -a

# Listen on a specific network interface and port with a custom secret
ppserver -i 192.168.1.50 -p 60000 -s "SuperSecretPassphrase"

# Write a PID file and inspect real-time stats using SIGUSR1
ppserver -P /var/run/ppserver.pid
kill -USR1 "$(cat /var/run/ppserver.pid)"

On Unix-like systems, sending SIGUSR1 to the server process (kill -USR1 <pid>) prints real-time job execution statistics to standard output on the next event loop iteration without interrupting active connections.

Examples

A Complete Distributed Example

This example calculates the sum of prime numbers below multiple thresholds concurrently. The helper function isprime is passed via depfuncs, and the math module is imported in worker namespaces via modules:

import math
import pp

def isprime(n):
    """Returns True if n is prime, False otherwise."""
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False
    for i in range(3, int(math.isqrt(n)) + 1, 2):
        if n % i == 0:
            return False
    return True

def sum_primes(n):
    """Calculates the sum of all prime numbers below n."""
    return sum(x for x in range(2, n) if isprime(x))

# Start the execution server with one worker per CPU core
job_server = pp.Server()

# Submit one job per input; isprime and math are shipped automatically
inputs = (100_000, 100_100, 100_200, 100_300)
jobs = [
    job_server.submit(sum_primes, (i,), depfuncs=(isprime,), modules=("math",))
    for i in inputs
]

# Collect and display results as they complete
for i, job in zip(inputs, jobs):
    print(f"Sum of primes below {i:,} is {job():,}")

# Print execution breakdown across all workers
job_server.print_stats()
job_server.destroy()

The examples/ directory in the source repository contains complete runnable scripts covering prime summation, brute-force MD5 cracking, dynamic worker pool resizing, asynchronous completion callbacks, automatic differentiation, and cluster throughput benchmarking.

Compatibility

Tested Platforms & Runtimes

Parallel Python is pure Python with zero external dependencies and no compiled C-extensions. Built strictly on standard library primitives (subprocess, socket, pickle, threading, and struct), it installs effortlessly and runs reliably across an exceptionally broad spectrum of Python versions, operating systems, hardware architectures, and endiannesses.

1. Python Interpreters & Runtimes

2. Operating Systems & C Libraries

3. Hardware Architectures & Endianness