Tutorial

Python Multiprocessing Example

Published on August 3, 2022
Default avatar

By Pankaj

Python Multiprocessing Example

While we believe that this content benefits our community, we have not yet thoroughly reviewed it. If you have any suggestions for improvements, please let us know by clicking the “report an issue“ button at the bottom of the tutorial.

In our previous tutorial, we learned about Python CSV Example. In this tutorial we are going to learn Python Multiprocessing with examples.

Python Multiprocessing

Parallel processing is getting more attention nowadays. If you still don’t know about the parallel processing, learn from wikipedia. As CPU manufacturers start adding more and more cores to their processors, creating parallel code is a great way to improve performance. Python introduced multiprocessing module to let us write parallel code. To understand the main motivation of this module, we have to know some basics about parallel programming. After reading this article, we hope that, you would be able to gather some knowledge on this topic.

Python Multiprocessing Process, Queue and Locks

There are plenty of classes in python multiprocessing module for building a parallel program. Among them, three basic classes are Process, Queue and Lock. These classes will help you to build a parallel program. But before describing about those, let us initiate this topic with simple code. To make a parallel program useful, you have to know how many cores are there in you pc. Python Multiprocessing module enables you to know that. The following simple code will print the number of cores in your pc.

import multiprocessing

print("Number of cpu : ", multiprocessing.cpu_count())

The following output may vary for your pc. For me, number of cores is 8. python multiprocessing example cpu count

Python multiprocessing Process class

Python multiprocessing Process class is an abstraction that sets up another Python process, provides it to run code and a way for the parent application to control execution. There are two important functions that belongs to the Process class - start() and join() function. At first, we need to write a function, that will be run by the process. Then, we need to instantiate a process object. If we create a process object, nothing will happen until we tell it to start processing via start() function. Then, the process will run and return its result. After that we tell the process to complete via join() function. Without join() function call, process will remain idle and won’t terminate. So if you create many processes and don’t terminate them, you may face scarcity of resources. Then you may need to kill them manually. One important thing is, if you want to pass any argument through the process you need to use args keyword argument. The following code will be helpful to understand the usage of Process class.

from multiprocessing import Process


def print_func(continent='Asia'):
    print('The name of continent is : ', continent)

if __name__ == "__main__":  # confirms that the code is under main function
    names = ['America', 'Europe', 'Africa']
    procs = []
    proc = Process(target=print_func)  # instantiating without any argument
    procs.append(proc)
    proc.start()

    # instantiating process with arguments
    for name in names:
        # print(name)
        proc = Process(target=print_func, args=(name,))
        procs.append(proc)
        proc.start()

    # complete the processes
    for proc in procs:
        proc.join()

The output of the following code will be: python multiprocessing example process class

Python multiprocessing Queue class

You have basic knowledge about computer data-structure, you probably know about Queue. Python Multiprocessing modules provides Queue class that is exactly a First-In-First-Out data structure. They can store any pickle Python object (though simple ones are best) and are extremely useful for sharing data between processes. Queues are specially useful when passed as a parameter to a Process’ target function to enable the Process to consume data. By using put() function we can insert data to then queue and using get() we can get items from queues. See the following code for a quick example.

from multiprocessing import Queue

colors = ['red', 'green', 'blue', 'black']
cnt = 1
# instantiating a queue object
queue = Queue()
print('pushing items to queue:')
for color in colors:
    print('item no: ', cnt, ' ', color)
    queue.put(color)
    cnt += 1

print('\npopping items from queue:')
cnt = 0
while not queue.empty():
    print('item no: ', cnt, ' ', queue.get())
    cnt += 1

python multiprocessing queue

Python multiprocessing Lock Class

The task of Lock class is quite simple. It allows code to claim lock so that no other process can execute the similar code until the lock has be released. So the task of Lock class is mainly two. One is to claim lock and other is to release the lock. To claim lock the, acquire() function is used and to release lock release() function is used.

Python multiprocessing example

In this Python multiprocessing example, we will merge all our knowledge together. Suppose we have some tasks to accomplish. To get that task done, we will use several processes. So, we will maintain two queue. One will contain the tasks and the other will contain the log of completed task. Then we instantiate the processes to complete the task. Note that the python Queue class is already synchronized. That means, we don’t need to use the Lock class to block multiple process to access the same queue object. That’s why, we don’t need to use Lock class in this case. Below is the implementation where we are adding tasks to the queue, then creating processes and starting them, then using join() to complete the processes. Finally we are printing the log from the second queue.

from multiprocessing import Lock, Process, Queue, current_process
import time
import queue # imported for using queue.Empty exception


def do_job(tasks_to_accomplish, tasks_that_are_done):
    while True:
        try:
            '''
                try to get task from the queue. get_nowait() function will 
                raise queue.Empty exception if the queue is empty. 
                queue(False) function would do the same task also.
            '''
            task = tasks_to_accomplish.get_nowait()
        except queue.Empty:

            break
        else:
            '''
                if no exception has been raised, add the task completion 
                message to task_that_are_done queue
            '''
            print(task)
            tasks_that_are_done.put(task + ' is done by ' + current_process().name)
            time.sleep(.5)
    return True


def main():
    number_of_task = 10
    number_of_processes = 4
    tasks_to_accomplish = Queue()
    tasks_that_are_done = Queue()
    processes = []

    for i in range(number_of_task):
        tasks_to_accomplish.put("Task no " + str(i))

    # creating processes
    for w in range(number_of_processes):
        p = Process(target=do_job, args=(tasks_to_accomplish, tasks_that_are_done))
        processes.append(p)
        p.start()

    # completing process
    for p in processes:
        p.join()

    # print the output
    while not tasks_that_are_done.empty():
        print(tasks_that_are_done.get())

    return True


if __name__ == '__main__':
    main()

Depending on the number of task, the code will take some time to show you the output. The output of the following code will vary from time to time. python multiprocessing example

Python multiprocessing Pool

Python multiprocessing Pool can be used for parallel execution of a function across multiple input values, distributing the input data across processes (data parallelism). Below is a simple Python multiprocessing Pool example.

from multiprocessing import Pool

import time

work = (["A", 5], ["B", 2], ["C", 1], ["D", 3])


def work_log(work_data):
    print(" Process %s waiting %s seconds" % (work_data[0], work_data[1]))
    time.sleep(int(work_data[1]))
    print(" Process %s Finished." % work_data[0])


def pool_handler():
    p = Pool(2)
    p.map(work_log, work)


if __name__ == '__main__':
    pool_handler()

Below image shows the output of the above program. Notice that pool size is 2, so two executions of work_log function is happening in parallel. When one of the function processing finishes, it picks the next argument and so on. python multiprocessing pool So, that’s all for python multiprocessing module. Reference: Official Documentation

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about us


About the authors
Default avatar
Pankaj

author

Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
JournalDev
DigitalOcean Employee
DigitalOcean Employee badge
March 15, 2021

Hi All, i want to launch multiple chrome browser instance(100) in same time…using selenium python anybody help can will be grateful.

- Anand Naraboli

    JournalDev
    DigitalOcean Employee
    DigitalOcean Employee badge
    June 6, 2020

    Please help me to resolve this issue https://stackoverflow.com/q/62237516/13193575

    - Lovely

      JournalDev
      DigitalOcean Employee
      DigitalOcean Employee badge
      May 8, 2020

      Thanks was helpful as always. The simple explanation is the strong point.

      - Sukesh Suvarna

        JournalDev
        DigitalOcean Employee
        DigitalOcean Employee badge
        February 5, 2020

        Thanks, your two way communication example helped me finally close in on my bug. One that your code will have as well if number or size of messages in the queue fills the buffer used by it. ‘.get()’ needs to be called on the Queue being used to send messages back from the sub-process (child) to the main process (parent) before the ‘.join()’ call that blocks main until the sub-process has completed. What is not seen from glancing at the Python code is that underneath, the Queue depends on a limited size buffer, even if no maximum count is set. Because of that, if the get() and join() are out of order a deadlock can happen where the sub-process cannot exit because it cannot finish putting things in the queue, and the main process cannot continue to where it removes things from the queue because it is waiting on the sub-process to exit. “…you need to make sure that all items which have been put on the queue will eventually be removed before the process is joined.” - https://docs.python.org/3.8/library/multiprocessing.html#programming-guidelines

        - Kevin Whalen

          JournalDev
          DigitalOcean Employee
          DigitalOcean Employee badge
          July 12, 2019

          Very easy to understand and informative! Thanks a lot man!

          - Kenny

            JournalDev
            DigitalOcean Employee
            DigitalOcean Employee badge
            June 20, 2019

            Great article man ! Simple and very informative. Even official python documentation for multiprocessing is not this simple and easy to read.

            - Ashish Dhiman

              JournalDev
              DigitalOcean Employee
              DigitalOcean Employee badge
              June 11, 2019

              How to Kill the Proc? is Proc.terminate() works best

              - Senthilkumar Rajendran

                JournalDev
                DigitalOcean Employee
                DigitalOcean Employee badge
                February 11, 2019

                Thank you but I think there isn’t enough explanation.

                - Emre ATAKLI

                  JournalDev
                  DigitalOcean Employee
                  DigitalOcean Employee badge
                  February 7, 2019

                  In the do_job function ,if there other method to replace time.sleep? For some function ,you may need not to know the possible time to sleep.

                  - RUI ZHANG

                    JournalDev
                    DigitalOcean Employee
                    DigitalOcean Employee badge
                    September 27, 2018

                    Why does the start() must be called within main function ?

                    - xram

                      Try DigitalOcean for free

                      Click below to sign up and get $200 of credit to try our products over 60 days!

                      Sign up

                      Join the Tech Talk
                      Success! Thank you! Please check your email for further details.

                      Please complete your information!

                      Get our biweekly newsletter

                      Sign up for Infrastructure as a Newsletter.

                      Hollie's Hub for Good

                      Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

                      Become a contributor

                      Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

                      Welcome to the developer cloud

                      DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

                      Learn more
                      DigitalOcean Cloud Control Panel