1. Does anyone know the latest update on NumPyPy? PyPy for me is just not a usable proposition because I heavily use Numpy (and Scipy et al). So I am forced to use slow Python + fast Numpy or slow Numpy + fast Python. Very saddening. The C-Extension is just so off the pace, NumPyPy was meant to solve that quandry.
And I know some smart Alec will trot out the usual 'downshift into C' line that everyone (including Guido) use as the final goto solution for performance but that is simply a disgrace in 2017. Even JavaScript is fast. Why can I not choose to write Python and it be fast?? And yet Python 3 is getting slower. Don't agree?
Look at these benchmarks of Python heaps written in Python (not using the C based builtin heapq) https://github.com/MikeMirzayanov/binary-heap-benchmark Python generally is off the pace but Python 3 is about twice as slow as 2 and miles off JavaScript.
But PyPy is proof that Python can be fast. It makes quote/unquote "Pure Python" within striking distance of Go and and when I run that test suit on PyPy, its similar to the Node.js score. Why does this matter?
Because I want to write bloody Python not C.
And it is so tantalisingly close - look at a blog post like: https://dnshane.wordpress.com/2017/02/14/benchmarking-python... The performance of the Fibonacci Heap that someone wrote in quote/unquote "Pure Python", when run in CPython can never compete with HeapQ (the C based builtin lib), but on PyPy it can. Fast code written in Python. So what are the problems holding back PyPy? I think possibly money and number of devs working on stuff. Javascript had Mozilla, Google, Microsoft and Apple in a browser war + loads of open source input.
But is the biggest stumbling block not Guido himself and the core Python devs? Do they just philosophically not agree with PyPy or is it just disinterest?
Well whatever it is, it is heart-breaking to want to write fast code in my favourite language and leverage all its power including Numpy/Scipy etc and not be able to. And yes my use-case is perhaps quite unique, a very CPU intensive service that ideally computes and returns a real-time calculation (that includes 500k function calls) in 10-50ms.
But getting fast Numpy in the PyPy mix (i.e all the speed of the JIT + no worse Numpy) would be a HUGE step forward for me in PyPy adoption. What is the latest? How can I help?
in short - funding. If we can find someone who wants fast numpy AND fast python under the same hood, we can combine the approaches of cpyext and numpypy and make it fast. The project is just too big to do on spare time. I've been trying to find some funding for that for quite a while, but I haven't been able to find any sizable backer just yet.
Newsflash: code like this[1] will never be fast in CPython, and if you write a lot of code like that and are sad when it's slow then you need a different language, especially if you expect it to be as fast as a JIT compiled language like js on v8. Or use something like Cython.
That benchmark is pretty meaningless anyway, IMO. Here are some halfway decent, official and up to date benchmarks comparing python 2 and python 3[2].
Python 3 is slower in some areas, noticilby startup time, but it's not all doom and gloom. It's faster in a lot of places. And productivity is hard to benchmark, but IMO py3 is way better in the area.
Ah the 'smart alec' has appeared. I'm not stupid, I know code like that won't be faster in Python but PyPy shows that it can be a hell of a lot faster than CPython and right up there with Node.js and the travesty is that CPython is so far off the pace and getting slower
Would you please not post uncivilly to HN, regardless of how annoying you find someone's comment? This kind of thing degrades discussion and provokes worse from others. We're hoping to do better than that here.
i suspect "smart alec" was chosen with exactly the intention you're expecting--ie, i'm not a native English speaker but i believe the term when used by adults to refer to other adults is benign--perhaps like the "G"-rated version of "wise-ass" or "pedant." Maybe wrong here, but benefit of the doubt perhaps....
Python 3.6.1 is a lot faster than 3.3.2 which was used back when the heap benchmark was done. In my system, 2.7.10 vs 3.6.1:
C:\Test\PyBench>py -2 test.py
Done in 1188.308127
C:\Test\PyBench>py -3 test.py
Done in 1454.897614
Please bring up to date benchmarks to the discussion, and stop complaining about old problems.
Note: adjusted I the workload to be 1000 less iterations, to get the results fast for this comment, so this numbers aren't comparable to the list in the github repo. But even if I hadn't done that, they wouldn't be because I ran these in my system.
Python, R and other languages are just slower. If that is an issue I again will agree you need to move to another language.
Can I ask what you need the speed for? How long are your reports running? Because a lot of times the reports run in under a few minutes and people just don't sub-set their data to code on. People feel it is "BIG DATA" when it is just annoying data that takes less then a minute to spit out.
The coalescence around Python and R for numerics to the extent it did was kind of premature. They're both great languages but I've given up on implementations of either of them achieving the kinds of speeds that they should be at for the things people are using them for.
I think there'll be more traction expanding the libraries for Julia, Nim, or Kotlin, all of which are much faster than Python, and similar in expressiveness. It's probably easier to create an optimization/ML/linear algebra/RNG/whatever library in Julia, Nim, or Kotlin than trying to get good performance out of R or Python.
I completely understand PyPy and numpy and why Python and R became popular, because there was a need for expressivity in the numerics space and other languages weren't offering it. But if you've been following them for long enough, it's clear that both of them had problems under the hood. I think people just crossed the boundary of appropriate use at some point, because the language syntax is so appealing for these sorts of things.
Maybe I'm wrong about all this and Pypy will deliver but I'm not holding my breath any longer. No offense to the Pypy developers--I'm immensely impressed with their work, and they've produced much more than I ever thought--but I do see some sort of asymptote. I think it would take some serious corporate influx of effort like what happened with javascript, and even then I worry that compatibility issues would rear their head. My guess is the Python 2-3 split would become a Python 2-3-Pypy split--maybe that's fine though.
I'm not a "smart alec" for pointing out that Python is, was and will be bad at heavily numerical, number crunching code. It's not what Python is built for.
I mean... it takes 28 bytes to store a single integer in Python.
this is silly, python is used by some of the biggest number crunchers on earth ... as glue code. If you dont like the dual language paradigm and want fast code go learn fortran.
It's not really a travesty. CPython just isn't designed for that kind of workload, and that's fine, because not many people use it to do that kind of thing.
I think the core devs are doing a fine job with CPython as the reference implementation and developing PyPy takes a different set of expertise; JIT and compilers specifically.
I think sponsorship of PyPy would be welcome -- but it seems non-obvious where that would come from.
Javascript has the fortune of being the language that drives a very important platform -- and Chrome has been a particularly strategic investment for Google to have more control over the web than it ever has before. Java has Android... Python unfortunately doesn't have that sort of standing in any area that I'm aware of.
And that would be fine, and my mind goes to similar examples like Lua where the reference and the JIT versions co-exist but PyPy has not had the impact that LuaJIT has had (for example) on the LUA community.
I agree writing C extensions is not a solution. It is evidently too hard. Even the standard pickle library in Python 3 has a memory corruption bug. http://bugs.python.org/issue23655
We [msft python team] tried to get PyPy some funding, but it didn't go very far. I'll keep trying. We've also started this project to enable jitting for CPython:
I understand that Numpy is fast in CPython because it relies on high speed code being done in C.
What you could do is much simpler --- :
Split your Python application in two parts:
1. Keep your functions that make heavy use of Numpy and Scipy under CPython; expose your algorithms/functions as a web service/REST service/etc running under CPython.
2. The rest of the application, which of course needs to call the functions in (1), can be written in PyPy and call the web service in (1). This is where you would put the general-purpose stuff like web, graphics, database access, and of course all symbolic manipulations that do not require Numpy/Scipy.
> And I know some smart Alec will trot out the usual 'downshift into C' line that everyone (including Guido) use as the final goto solution for performance but that is simply a disgrace in 2017.
Easy gluing of other languages together has long been something I considered a strength...but I suppose to each their own.
> Why can I not choose to write Python and it be fast??
Well there are lots of reasons...including implementation issues and I don't know them all...but I think Python has a very clear productivity niche. Personally, I am ok with Python trading performance for productivity. For the most part, I haven't had Python be so much of a bottleneck that writing a very small part of logic to be performant hasn't solved my use case.
> And yet Python 3 is getting slower. Don't agree?
Yeah I don't agree...that benchmark uses Python 3.3. The corner on Python 3 performance over Python 2 started turning around 3.4. Perhaps a talk from this years PyCon would help illustrate:
Indeed, I would even say that Cython is even more proof that there are frontiers of performance that could be explored. But with PyPy (as with Cython) their are sacrifices you have to make.
Personally, I think the most promising performance improvement that is tantalizingly close for me is Larry Hasting's Gilectomy project:
But at the same time, I am not sure that Python ever needs to be fast running in CPython. With `WASM` perhaps it is better to just compile Python.
I don't know, performance in Python has always been a mixed bag...but personally I think it doesn't get much focus because it doesn't really serve Python's target niche. I don't know if there ever will be (or should) be 1 language to do everything...and as it is Python is a good "productivity" focused language to have in your toolbox so-to-speak.
Then you probably should not use Python, python is more of a glue language which you should strive to make your program looking like a business logic, in real word to solve this problem you would write code such as this:
import time
if __name__ == "__main__":
start = time.clock()
N = 10000000
h = list(range(N))
h.sort()
for i, v in enumerate(h):
assert(i == v)
print("Done in %f" % ((time.clock() - start) * 1000))
$ python3.6 heap.py
Done in 2389.877000
Or if heap needs to be used:
import time
from heapq import heapify, heappop
if __name__ == "__main__":
start = time.clock()
N = 10000000
h = list(range(N))
heapify(h)
for i in range(len(h)):
assert(i == heappop(h))
print("Done in %f" % ((time.clock() - start) * 1000))
$ python3.6 heap.py
Done in 10716.348000
Micro benchmarks are silly because you'll never do those things in real code.
You can read this great article just released about the Python 2017 language summit: "Keeping Python Competitive" [1]. There you can read opinions by many core developers. Pypy is also discussed
Unlike you, Python is _not_ my favorite language, but the matplotlib lock-in is real.
Hopefully a matplotlib-equivalent will materialize on Clojure (where Linear Algebra is plenty fast and the language itself is fast-enough out of the box) so I can be done with Python forever.
I can't answer how numpypy is going, but Numba works pretty well for me to write fast numeric code in python. A bit restricted language and installing llvm is a bit of a hassle, but overall it's great.
What are you currently using to solve that problem? I've ran into that problem too, and had to use C-extensions to make my code faster, which isn't ideal.
PyPy is great -- while I still use CPython for our more complex webapp and associated tools that have heavy dependencies on C-extensions; I increasingly use PyPy for the more mundane cpu/data heavy lifting I do. It's typical to get 2X the performance (comparable to some compiled languages) and still use much of our utility code, configs, etc.
Given that python programs usually run an order of magnitude slower than compiled languages even a 2x performance increase doesn't put it in the "comparable" range from my experience. Not bashing python - I use it regularly - but for computational stuff it's a hog unless you're just passing stuff to C libs - like I have a resource build pipeline that does some blender 3D model transformations - code is written in python and takes forever - equivalent code in C++ would take roughly 1/100 of the time and performance would be irrelevant but atm. we're seriously considering rewriting parts in C++ to reduce build times.
The one time I saw 100x increase in performance in Python-to-C (which was done through Cython) was in code that worked with strings calculating a machine-learning related distance between two strings. The code was doing a lot of accessing particular positions in the strings, which in pure python resulted in slow retrieval of every character (lots of .__getitem__ calls), which were optimized to having 2 predefined empty arrays (in heap, not stack, and their corresponding counters of valid items) and then walking the strings and storing the "hot" values in them.
So it was a very specific case where we could get that 100x speedup at work.
As long as you use the python procedures that are written in C you will not really get much by using pypy. Try it yourself, write some IO and string heavy code and compare. [1]
But as you say: for numeric computations python is slow as molasses.
[1] or just look at something like https://github.com/juditacs/wordcount/blob/master/README.md . The simple py2 version is 2.5 times slower than a java version someone spent a lot of time writing, and less than 2 times slower than a reasonably straightforward C program.
That was the point of the Heap Benchmarks. In CPython you would have to use HeapQ, writing something yourself in Python will be miles off the pace. Whereas in PyPy the Python implementation of a Heap or your own version is comparably fast. As it should be. The 'hunt down the written in C' parts of the standard library is what I am increasingly objecting to.
It all depends on the task -- if your program is calling into optimized C-extensions anyway, converting the whole thing into a compiled language of course offers less of a speedup.
Coming from someone who uses python but doesn't really follow alternative compilers, PyPy sounds great. What are some of the downsides, if any? Are you sacrificing library compatibility for faster core+standard libs?
In addition to being incompatible with (some) third-party libraries, pypy tends to use significantly more memory than cpython. It's also slower than cpython for scripts that don't run long enough to warm up the JIT, so you probably wouldn't want to use it by default. (Disclaimer: I'm basing this on experience with older versions of pypy and haven't verified it recently)
He memory thing is still an issue. I had to go thru. Lot of tuning on max GC size to keep it runnable for long times. Too low and it is slow and too high and it kills the box.
For workloads heavy on JSON operations PyPy has been slower than CPython for me in the past. Because it didn't benefit from the C implementation of simplejson.
In addition to the missing libraries, which has gotten a lot better in the last few years, it crashes a bit more.
I have a syslog proxy that has one huge incoming stream like 50k msgs/second. CPU Could not keep up with CPython but PyPy runs fine and crashes on some low level JIT assertion every so often. I have it setup to use PyPy on the high volume instances and CPython on the low volume instances.
Why are they still comparing to Python 2.7.2? I couldn't find benchmarks against Python 3.5 for their Py3 interpreter.
All the times I tried PyPy I came into a hurdle where one of the libraries I needs doesn't work (or underperforms) in PyPy, the most important ones being Numpy and OpenCV.
So in the end I just gave up with them, and stuck with Python 2/3 and Cython, which solved my speed problems without having to do all the work of C-extensions from the ground up.
The benchmark showing a factor 100x compares a pure python implementation of "pidigits" running in PyPy3 vs one that uses GMP (via gmpy2) in python. I am actually impressed it's only a factor 100. And with progress made by cffi, I'd hopeful that a GMP-using PyPy version could be written rhat matches the speed of the code using gmpy2.
The next benchmark "only" runs 6x slower in PyPy ; still bad, but that paints quite a differen picture.
If it's running pidigits inside PyPy then it shows how much slower the C interface (not PyPys cffi, but it's CPython compatible interface) is compared to CPythons.
For the record, I'm not claiming PyPy3 to be 100x slower, these are benchmarks and they're hairy beasts that we have to shave and try to see what they tell us about performance.
My point is that PyPy3 is still behind in language features (in CPython we have a lot of nice things from 3.6 and 3.7 coming soon, while PyPy still lags behind having the complete 3.5 feature set), and they haven't optimized it as much as their 2.7 branch. But the PyPy people are always showing those "7x faster than CPython" claims (which come from an average of benchmarks, which seems to have been cherry picked to avoid the ones in which they're actually slower than CPython).
On the other hand, with Cython moving over to Py3 was never an issue (it actually helped in some cases), and it always helped to deliver better performance, and in just the right spot where it's needed. True, you have to know about profiling and identifying where to use Cython, but at my workplace it's been a far better tool to solve our performance needs.
Any word on the "single codebase" aspect of supporting both major Python versions? I remember suggesting it years ago at a time when the team wanted to do Mercurial backporting instead. What changed their mind?
That looks like it could fix the lag on CPython releases, so it's a big feature.
Most code that is vanilla Python will work just fine with PyPy. However, in real life, your code most likely also uses (imports) other Python libraries, you just need to make sure they work OK with PyPy.
Many, many, many libraries already work fine with PyPy, so give it a shot.
1. Does anyone know the latest update on NumPyPy? PyPy for me is just not a usable proposition because I heavily use Numpy (and Scipy et al). So I am forced to use slow Python + fast Numpy or slow Numpy + fast Python. Very saddening. The C-Extension is just so off the pace, NumPyPy was meant to solve that quandry.
And I know some smart Alec will trot out the usual 'downshift into C' line that everyone (including Guido) use as the final goto solution for performance but that is simply a disgrace in 2017. Even JavaScript is fast. Why can I not choose to write Python and it be fast?? And yet Python 3 is getting slower. Don't agree? Look at these benchmarks of Python heaps written in Python (not using the C based builtin heapq) https://github.com/MikeMirzayanov/binary-heap-benchmark Python generally is off the pace but Python 3 is about twice as slow as 2 and miles off JavaScript.
But PyPy is proof that Python can be fast. It makes quote/unquote "Pure Python" within striking distance of Go and and when I run that test suit on PyPy, its similar to the Node.js score. Why does this matter? Because I want to write bloody Python not C.
And it is so tantalisingly close - look at a blog post like: https://dnshane.wordpress.com/2017/02/14/benchmarking-python... The performance of the Fibonacci Heap that someone wrote in quote/unquote "Pure Python", when run in CPython can never compete with HeapQ (the C based builtin lib), but on PyPy it can. Fast code written in Python. So what are the problems holding back PyPy? I think possibly money and number of devs working on stuff. Javascript had Mozilla, Google, Microsoft and Apple in a browser war + loads of open source input.
But is the biggest stumbling block not Guido himself and the core Python devs? Do they just philosophically not agree with PyPy or is it just disinterest?
Well whatever it is, it is heart-breaking to want to write fast code in my favourite language and leverage all its power including Numpy/Scipy etc and not be able to. And yes my use-case is perhaps quite unique, a very CPU intensive service that ideally computes and returns a real-time calculation (that includes 500k function calls) in 10-50ms.
But getting fast Numpy in the PyPy mix (i.e all the speed of the JIT + no worse Numpy) would be a HUGE step forward for me in PyPy adoption. What is the latest? How can I help?