概述
原文转自:https://abcdabcd987.com/python-multiprocessing/
Meta" style="font-size:14px;color:rgb(130,130,130);">Nov 2,2016 ?
vs
Python 自带的库又全又好用,这是我特别喜欢 Python 的原因之一。Python 里面有
和 这两个用来实现并行的库。用线程应该是很自然的想法,毕竟(直觉上)开销小,还有共享内存的福利,而且在其他语言里面线程用的确实是非常频繁。然而,我可以很负责任的说,如果你用的是 cpython 实现,那么用了
Highlighter-rouge" style="font-family:Monaco,232);">threading
就等同于和并行计算说再见了(实际上,甚至会比单线程更慢),除非这是个IO密集型的任务。
指的是 提供的 Python 实现。是的,Python 是一门语言,它有各种不同的实现,比如 , , 等等……我们用的最多的就是 CPython,它几乎就和 Python 画上了等号。
CPython 的实现中,使用了 即全局锁,来简化解释器的实现,使得解释器每次只执行一个线程中的字节码。也就是说,除非是在等待IO操作,否则 cpython 的多线程就是彻底的谎言!
有关 GIL 下面两个资料写的挺好的:
指的是 提供的 Python 实现。是的,Python 是一门语言,它有各种不同的实现,比如 , , 等等……我们用的最多的就是 CPython,它几乎就和 Python 画上了等号。
CPython 的实现中,使用了 即全局锁,来简化解释器的实现,使得解释器每次只执行一个线程中的字节码。也就是说,除非是在等待IO操作,否则 cpython 的多线程就是彻底的谎言!
有关 GIL 下面两个资料写的挺好的:
指的是 提供的 Python 实现。是的,Python 是一门语言,它有各种不同的实现,比如 , , 等等……我们用的最多的就是 CPython,它几乎就和 Python 画上了等号。
CPython 的实现中,使用了 即全局锁,来简化解释器的实现,使得解释器每次只执行一个线程中的字节码。也就是说,除非是在等待IO操作,否则 cpython 的多线程就是彻底的谎言!
有关 GIL 下面两个资料写的挺好的:
因为 GIL 的缘故
Highlighter
-rouge" style="font-family:Monaco,232);">threading
不能用,那么我们就好好研究研究 。(当然,如果你说你不用 cpython,没有 GIL 的问题,那也是极佳的。)
首先介绍一个简单粗暴,非常实用的工具,就是 。如果你的任务能用
Highlighter-rouge" style="font-family:Monaco,232);">ys = map(f,xs)
来解决,大家可能都知道,这样的形式天生就是最容易并行的,那么在 Python 里面并行计算这个任务真是再简单不过了。举个例子,把每个数都平方:
<span class="k" style="font-weight:bold;">def <span class="nf" style="color:rgb(153,0);font-weight:bold;">f<span class="p">(<span class="n">x<span class="p">):
<span class="k" style="font-weight:bold;">return <span class="n">x <span class="o" style="font-weight:bold;">* <span class="n">x
<span class="n">cores <span class="o" style="font-weight:bold;">= <span class="n">multiprocessing<span class="o" style="font-weight:bold;">.<span class="n">cpu_count<span class="p">()
<span class="n">pool <span class="o" style="font-weight:bold;">= <span class="n">multiprocessing<span class="o" style="font-weight:bold;">.<span class="n">Pool<span class="p">(<span class="n">processes<span class="o" style="font-weight:bold;">=<span class="n">cores<span class="p">)
<span class="n">xs <span class="o" style="font-weight:bold;">= <span class="nb" style="color:rgb(0,134,179);">range<span class="p">(<span class="mi" style="color:rgb(0,153,153);">5<span class="p">)
<span class="c" style="color:rgb(153,136);font-style:italic;"># method 1: map
<span class="k" style="font-weight:bold;">print <span class="n">pool<span class="o" style="font-weight:bold;">.<span class="nb" style="color:rgb(0,179);">map<span class="p">(<span class="n">f<span class="p">, <span class="n">xs<span class="p">) <span class="c" style="color:rgb(153,136);font-style:italic;"># prints [0,1,4,9,16]
<span class="c" style="color:rgb(153,136);font-style:italic;"># method 2: imap
<span class="k" style="font-weight:bold;">for <span class="n">y <span class="ow" style="font-weight:bold;">in <span class="n">pool<span class="o" style="font-weight:bold;">.<span class="n">imap<span class="p">(<span class="n">f<span class="p">, <span class="n">xs<span class="p">):
<span class="k" style="font-weight:bold;">print <span class="n">y <span class="c" style="color:rgb(153,136);font-style:italic;"># 0,16,respectively
<span class="c" style="color:rgb(153,136);font-style:italic;"># method 3: imap_unordered
<span class="k" style="font-weight:bold;">for <span class="n">y <span class="ow" style="font-weight:bold;">in <span class="n">pool<span class="o" style="font-weight:bold;">.<span class="n">imap_unordered<span class="p">(<span class="n">f<span class="p">, <span class="n">xs<span class="p">):
<span class="k" style="font-weight:bold;">print<span class="p">(<span class="n">y<span class="p">) <span class="c" style="color:rgb(153,136);font-style:italic;"># may be in any order
<span class="k" style="font-weight:bold;">def <span class="nf" style="color:rgb(153,0);font-weight:bold;">f<span class="p">(<span class="n">x<span class="p">):
<span class="k" style="font-weight:bold;">return <span class="n">x <span class="o" style="font-weight:bold;">* <span class="n">x
<span class="n">cores <span class="o" style="font-weight:bold;">= <span class="n">multiprocessing<span class="o" style="font-weight:bold;">.<span class="n">cpu_count<span class="p">()
<span class="n">pool <span class="o" style="font-weight:bold;">= <span class="n">multiprocessing<span class="o" style="font-weight:bold;">.<span class="n">Pool<span class="p">(<span class="n">processes<span class="o" style="font-weight:bold;">=<span class="n">cores<span class="p">)
<span class="n">xs <span class="o" style="font-weight:bold;">= <span class="nb" style="color:rgb(0,134,179);">range<span class="p">(<span class="mi" style="color:rgb(0,153,153);">5<span class="p">)
<span class="c" style="color:rgb(153,136);font-style:italic;"># method 1: map
<span class="k" style="font-weight:bold;">print <span class="n">pool<span class="o" style="font-weight:bold;">.<span class="nb" style="color:rgb(0,179);">map<span class="p">(<span class="n">f<span class="p">, <span class="n">xs<span class="p">) <span class="c" style="color:rgb(153,136);font-style:italic;"># prints [0,1,4,9,16]
<span class="c" style="color:rgb(153,136);font-style:italic;"># method 2: imap
<span class="k" style="font-weight:bold;">for <span class="n">y <span class="ow" style="font-weight:bold;">in <span class="n">pool<span class="o" style="font-weight:bold;">.<span class="n">imap<span class="p">(<span class="n">f<span class="p">, <span class="n">xs<span class="p">):
<span class="k" style="font-weight:bold;">print <span class="n">y <span class="c" style="color:rgb(153,136);font-style:italic;"># 0,16,respectively
<span class="c" style="color:rgb(153,136);font-style:italic;"># method 3: imap_unordered
<span class="k" style="font-weight:bold;">for <span class="n">y <span class="ow" style="font-weight:bold;">in <span class="n">pool<span class="o" style="font-weight:bold;">.<span class="n">imap_unordered<span class="p">(<span class="n">f<span class="p">, <span class="n">xs<span class="p">):
<span class="k" style="font-weight:bold;">print<span class="p">(<span class="n">y<span class="p">) <span class="c" style="color:rgb(153,136);font-style:italic;"># may be in any order
<span class="k" style="font-weight:bold;">def <span class="nf" style="color:rgb(153,0);font-weight:bold;">f<span class="p">(<span class="n">x<span class="p">):
<span class="k" style="font-weight:bold;">return <span class="n">x <span class="o" style="font-weight:bold;">* <span class="n">x
<span class="n">cores <span class="o" style="font-weight:bold;">= <span class="n">multiprocessing<span class="o" style="font-weight:bold;">.<span class="n">cpu_count<span class="p">()
<span class="n">pool <span class="o" style="font-weight:bold;">= <span class="n">multiprocessing<span class="o" style="font-weight:bold;">.<span class="n">Pool<span class="p">(<span class="n">processes<span class="o" style="font-weight:bold;">=<span class="n">cores<span class="p">)
<span class="n">xs <span class="o" style="font-weight:bold;">= <span class="nb" style="color:rgb(0,134,179);">range<span class="p">(<span class="mi" style="color:rgb(0,153,153);">5<span class="p">)
<span class="c" style="color:rgb(153,136);font-style:italic;"># method 1: map
<span class="k" style="font-weight:bold;">print <span class="n">pool<span class="o" style="font-weight:bold;">.<span class="nb" style="color:rgb(0,179);">map<span class="p">(<span class="n">f<span class="p">, <span class="n">xs<span class="p">) <span class="c" style="color:rgb(153,136);font-style:italic;"># prints [0,1,4,9,16]
<span class="c" style="color:rgb(153,136);font-style:italic;"># method 2: imap
<span class="k" style="font-weight:bold;">for <span class="n">y <span class="ow" style="font-weight:bold;">in <span class="n">pool<span class="o" style="font-weight:bold;">.<span class="n">imap<span class="p">(<span class="n">f<span class="p">, <span class="n">xs<span class="p">):
<span class="k" style="font-weight:bold;">print <span class="n">y <span class="c" style="color:rgb(153,136);font-style:italic;"># 0,16,respectively
<span class="c" style="color:rgb(153,136);font-style:italic;"># method 3: imap_unordered
<span class="k" style="font-weight:bold;">for <span class="n">y <span class="ow" style="font-weight:bold;">in <span class="n">pool<span class="o" style="font-weight:bold;">.<span class="n">imap_unordered<span class="p">(<span class="n">f<span class="p">, <span class="n">xs<span class="p">):
<span class="k" style="font-weight:bold;">print<span class="p">(<span class="n">y<span class="p">) <span class="c" style="color:rgb(153,136);font-style:italic;"># may be in any order
总结
以上是编程之家为你收集整理的【Python】【并行计算】Python 多核并行计算全部内容,希望文章能够帮你解决【Python】【并行计算】Python 多核并行计算所遇到的程序开发问题。
如果您也喜欢它,动动您的小指点个赞吧