Lightweight parallelization library for Python.

OpenGRIS Parfun is a lightweight library making it easy to write and run Python in parallel and distributed systems.

The main feature of the library is its @parallel decorator that transparently executes standard Python functions in parallel following the map-reduce pattern:

from typing import List

import parfun as pf


@pf.parallel(
    # parallelize by chunking the argument list (map)
    split=pf.per_argument(
        values=pf.py_list.by_chunk
    ),

    # merge the output by concatenating the results (reduce)
    combine_with=pf.py_list.concat,
)
def list_pow(values: List[float], factor: float) -> List[float]:
    """compute powers of a list of numbers"""
    return [v**factor for v in values]


if __name__ == "__main__":
    with pf.set_parallel_backend_context("local_multiprocessing"):  # use a local pool of processes
        print(list_pow([1, 2, 3], 2))  # runs in parallel, prints [1, 4, 9]

Features

• Provides significant speedups to existing Python functions.
• Only requires basic understanding of parallel and distributed computing systems.
• Automatically estimates the optimal sub-task splitting strategy (the partition size).
• Transparently handles data transmission, caching, and synchronization.
• Supports various distributed computing backends:
  • Python's built-in multiprocessing module.
  • Scaler.
  • Dask.

Quick Start

Install Parfun directly from PyPI:

pip install parfun
pip install "parfun[pandas,scaler,dask]"  # with optional dependencies

The official documentation is available at citi.github.io/parfun/.

Take a look at our documentation's quickstart tutorial to get more examples and a deeper overview of the library.

Alternatively, you can build the documentation from source:

cd docs
pip install -r requirements.txt
make html

The documentation's main page can then be found at docs/build/html/index.html.

Benchmarks

Parfun effectively parallelizes even short-duration functions.

For example, when running a short 0.28-second machine learning function on an AMD Epyc 7313 16-Core Processor, we found that Parfun provided an impressive 7.4x speedup. Source code for this experiment here.

Contributing

Your contributions are at the core of making this a true open source project. Any contributions you make are greatly appreciated.

We welcome you to:

• Fix typos or touch up documentation
• Share your opinions on existing issues
• Help expand and improve our library by opening a new issue

Please review functional contribution guidelines to get started 👍.

NOTE: Commits and pull requests to FINOS repositories will only be accepted from those contributors with an active, executed Individual Contributor License Agreement (ICLA) with FINOS OR contributors who are covered under an existing and active Corporate Contribution License Agreement (CCLA) executed with FINOS. Commits from individuals not covered under an ICLA or CCLA will be flagged and blocked by the (EasyCLA) tool. Please note that some CCLAs require individuals/employees to be explicitly named on the CCLA.

Need an ICLA? Unsure if you are covered under an existing CCLA? Email [email protected]

Code of Conduct

Please see the FINOS Community Code of Conduct.

License

Copyright 2023 Citigroup, Inc.

This project is distributed under the Apache-2.0 License. See LICENSE for more information.

SPDX-License-Identifier: Apache-2.0.

Contact

If you have a query or require support with this project, raise an issue. Otherwise, reach out to [email protected].