python中的多cpu并行编程_Python

多cpu并行编程

python多线程只能算并发，因为它智能使用一个cpu内核
python下pp包支持多cpu并行计算

安装

				?

									pip install pp

使用

				?

									#-*- coding: UTF-8 -*-

									import math, sys, time

									import pp

									def IsPrime(n):

									    """返回n是否是素数"""

									    if not isinstance(n, int):

									        raise TypeError("argument passed to is_prime is not of 'int' type")

									    if n < 2:

									        return False

									    if n == 2:

									        return True

									    max = int(math.ceil(math.sqrt(n)))

									    i = 2

									    while i <= max:

									        if n % i == 0:

									            return False

									        i += 1

									    return True

									def SumPrimes(n):

									    for i in xrange(15):

									        sum([x for x in xrange(2,n) if IsPrime(x)])

									    """计算从2-n之间的所有素数之和"""

									    return sum([x for x in xrange(2,n) if IsPrime(x)])

									inputs = (100000, 100100, 100200, 100300, 100400, 100500, 100600, 100700)

									# start_time = time.time()

									# for input in inputs:

									#     print SumPrimes(input)

									# print '单线程执行，总耗时', time.time() - start_time, 's'

									# # tuple of all parallel python servers to connect with

									ppservers = ()

									#ppservers = ("10.0.0.1",)

									if len(sys.argv) > 1:

									    ncpus = int(sys.argv[1])

									    # Creates jobserver with ncpus workers

									    job_server = pp.Server(ncpus, ppservers=ppservers)

									else:

									    # Creates jobserver with automatically detected number of workers

									    job_server = pp.Server(ppservers=ppservers)

									print "pp 可以用的工作核心线程数", job_server.get_ncpus(), "workers"

									start_time = time.time()

									jobs = [(input, job_server.submit(SumPrimes,(input,), (IsPrime,), ("math",))) for input in inputs]#submit提交任务

									for input, job in jobs:

									    print "Sum of primes below", input, "is", job()# job()获取方法执行结果

									print "多线程下执行耗时: ", time.time() - start_time, "s"

									job_server.print_stats()#输出执行信息

执行结果

pp 可以用的工作核心线程数 4 workers
Sum of primes below 100000 is 454396537
Sum of primes below 100100 is 454996777
Sum of primes below 100200 is 455898156
Sum of primes below 100300 is 456700218
Sum of primes below 100400 is 457603451
Sum of primes below 100500 is 458407033
Sum of primes below 100600 is 459412387
Sum of primes below 100700 is 460217613
多线程下执行耗时: 15.4971420765 s
Job execution statistics:
job count | % of all jobs | job time sum | time per job | job server
8 | 100.00 | 60.9828 | 7.622844 | local
Time elapsed since server creation 15.4972219467
0 active tasks, 4 cores

submit 函数定义

				?

									def submit(self, func, args=(), depfuncs=(), modules=(),

									        callback=None, callbackargs=(), group='default', globals=None):

									    """Submits function to the execution queue

									        func - function to be executed  执行的方法

									        args - tuple with arguments of the 'func' 方法传入的参数，使用元组

									        depfuncs - tuple with functions which might be called from 'func' 传入自己写的方法 ，元组

									        modules - tuple with module names to import  传入 模块

									        callback - callback function which will be called with argument

									                list equal to callbackargs+(result,)

									                as soon as calculation is done

									        callbackargs - additional arguments for callback function

									        group - job group, is used when wait(group) is called to wait for

									        jobs in a given group to finish

									        globals - dictionary from which all modules, functions and classes

									        will be imported, for instance: globals=globals()

									    """

多核cpu并行计算

多进程实现并行计算的简单示例
这里我们开两个进程，计算任务也简洁明了

				?

									# 多进程

									import multiprocessing as mp

									def job(q, a, b):

									    print('aaa')

									    q.put(a**1000+b*1000)  # 把计算结果放到队列

									# 多进程

									if __name__ == '__main__':

									    q = mp.Queue()  # 一个队列

									    p1 = mp.Process(target=job, args=(q, 100, 200))

									    p2 = mp.Process(target=job, args=(q, 100, 200))

									    p1.start()

									    p1.join()

									    # print(p1.ident)

									    p2.start()

									    p2.join()

									    res = q.get() + q.get()  # 读取队列，这里面保存了两次计算得到的结果

									    print('result:', res)