Because Python is a dynamic language, making it faster has been a challenge. But over the last couple of years, developers in the core Python team have focused on various ways to do it.
At PyCon 2023, held in Salt Lake City, Utah, several talks highlighted Python's future as a faster and more efficient language. Python 3.12 will showcase many of those improvements. Some are new in that latest version, others are already in Python but have been further refined.
Mark Shannon, a longtime core Python contributor now at Microsoft, summarized many of the initiatives to speed up and streamline Python. Most of the work he described in his presentation centered on reducing Python's memory use, making the interpreter faster, and optimizing the compiler to yield more efficient code.
Other projects, still under wraps but already showing promise, offer ways to expand Python's concurrency model. This will allow Python to better use multiple cores with fewer of the tradeoffs imposed by threads, async, or multiprocessing.
[...] One long-dreamed way to solve this problem is to remove Python's GIL, or Global Interpreter Lock. The GIL synchronizes operations between threads to ensure objects are accessed by only one thread at a time. In theory, removing the GIL would allow true multithreading. In practice—and it's been tried many times—it slows down non-threaded use cases, so it's not a net win.
Core python developer Eric Snow, in his talk, unveiled a possible future solution for all this: subinterpreters, and a per-interpreter GIL. In short: the GIL wouldn't be removed, just sidestepped.
Subinterpreters is a mechanism where the Python runtime can have multiple interpreters running together inside a single process, as opposed to each interpreter being isolated in its own process (the current multiprocessing mechanism). Each subinterpreter gets its own GIL, but all subinterpreters can share state more readily.
[...] With Python 3.12, Snow and his cohort cleaned up Python's internals enough to make subinterpreters useful, and they are adding a minimal module to the Python standard library called interpreters. This gives programmers a rudimentary way to launch subinterpreters and execute code on them.
[...] Another major set of performance improvements Shannon mentioned, Python's new adaptive specializing interpreter, was discussed in detail in a separate session by core Python developer Brandt Bucher.
Python 3.11 introduced new bytecodes to the interpreter, called adaptive instructions. These instructions can be replaced automatically at runtime with versions specialized for a given Python type, a process called quickening. This saves the interpreter the step of having to look up what types the objects are, speeding up the whole process enormously. For instance, if a given addition operation regularly takes in two integers, that instruction can be replaced with one that assumes the operands are both integers.
[...] Python objects have historically used a lot of memory. A Python 3 object header, even without the data for the object, occupied 208 bytes.
Over the last several versions of Python, though, various efforts took place to streamline the way Python objects were designed, finding ways to share memory or represent things more compactly. Shannon outlined how as of Python 3.12, the object header's now a mere 96 bytes—slightly less than half of what it was before.
These changes don't just allow more Python objects to be kept in memory, they also improve cache locality for Python objects. While that by itself may not speed things up as significantly as other efforts, it's still a boon.
[...] The default Python implementation, CPython, has three decades of development behind it. That also means three decades of cruft, legacy APIs, and design decisions that can be hard to transcend—all of which make it hard to improve Python in key ways.
[...] One key issue is the proliferation of C APIs found in CPython, the reference runtime for the language. As of Python 3.8, there are a few different sets of APIs, each with different maintenance requirements. Over the last five years, Stinner worked to make many public APIs private, so programmers don't need to deal as directly with sensitive CPython internals. The long-term goal is to make components that use the C APIs, like Python extension modules, less dependent on things that might change with each version.
(Score: 4, Insightful) by turgid on Wednesday May 03, @12:04PM (21 children)
It sounds like Python has had its day. It has become baroque, complicated and is being used beyond its capabilities. Is it time to start again with a new language?
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 2, Insightful) by Anonymous Coward on Wednesday May 03, @12:34PM (2 children)
I'm fine with perl. Perl has a better track record on backward compatibility than Python, so if you want to talk about future proof, perl is more likely to be future proof than python.
If you're doing multithreaded perl this might be useful: https://perldoc.perl.org/threads::shared [perl.org]
If you're doing multiprocess perl perhaps this: https://metacpan.org/pod/IPC::Shareable [metacpan.org]
If perl is really too slow for the task in most cases you shouldn't be using python either.
(Score: 1, Troll) by jga on Wednesday May 03, @01:27PM (1 child)
I my eyes, being future proof is much more than having a code that just runs. What if you need to change/adapt it? It is not straight forward to read someone else's Perl code, or even your own code, after months/years.
There is a reason why Python has succeeded, and it is not because it technically superior to other languages (in general, it is not), but because it is easy to read/adapt from other sources.
(Score: 2, Insightful) by Anonymous Coward on Wednesday May 03, @05:30PM
I've had no problems with reading my old perl code from more than 15 years ago, which I haven't looked at for years. And I've had no big problems with understanding and modifying perl code in CPAN written by others.
Maybe you're talking about obfuscated or similar perl code or non-perl programmers trying to read and modify perl code?
I agree that perl code is hard for non-perl programmers to adapt/change. But that's same for other popular programming languages too especially C and C++ if you include "without introducing security bugs" 😉.
I've had more problems trying to understand and _safely_ modifying C and C++ stuff written by others.
e.g. the stuff here: https://github.com/isc-projects/dhcp/blob/master/server/ [github.com]
Looks a lot more difficult to understand and change successfully to me than my perl dhcpd. My dhcpd is 3500 lines of code; optionally supports talking/reporting to an IRC server (don't ask me why); stores rules, leases, groups etc in a DB. So VIPs can be prioritized to be given IPs in certain address ranges (e.g. public IPs) if available. You could set up rules so that clients that attach to a particular VLAN are automatically added to a VIP MAC group and from then on their mac addresses are treated as VIPs on the rest of the network/VLANs.
No need for anyone to manually configure stuff to "bless" them - if they've managed to connect to the ports/APs connected to the blessed VLANs they'll be VIPs from then on.
So how hard would it be to add this feature to ISC dhcpd within a few days without adding any security bugs in the process?
Could I write the same thing in python? Considering python 3 was introduced only in Dec 2008, it'd probably mean I'd have had to write it in Python 2 and today my code would no longer work on a supported python interpreter. So much for future proofing.
(Score: 2) by GloomMower on Wednesday May 03, @01:00PM (14 children)
What has become complicated?
(Score: 2) by turgid on Wednesday May 03, @01:21PM (7 children)
Everything to do with Python. It needs to be retired.
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 2) by GloomMower on Wednesday May 03, @01:47PM (6 children)
I hadn't noticed it go so complex. I've been using python since 2004 or so I think. I was hoping you would mention something that I forgot or couldn't remember. But you told me nothing. It is okay if you just don't like python, but was curious what got too complicated.
(Score: 3, Interesting) by turgid on Wednesday May 03, @02:36PM (5 children)
Different versions of the language are incompatible, for starters. Package dependencies are a rats' nest. Then you have the internal problems described above regarding the Global Interpreter Lock. It seems like the design of the interpreter is very much showing its age, and it is somewhat telling that it hasn't been updated yet. That implies that the code isn't in a very good state, that it has become very complicated internally so difficult to change. I have eight 64-bit multi-core/multi-CPU machines in this room, at home. Some are 20 years old. It's surprising that such a popular and important language such as Python still doesn't scale very well on multi-core machines (but then again, neither do most languages).
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 4, Insightful) by GloomMower on Wednesday May 03, @03:02PM
Ahh okay. I hadn't had much experience with version incompatibility besides the 2->3 jump.
GIL has not been a major issue for me also, because most waiting for stuff I do is in IO.
Package management isn't the greatest yes. But it was easier before because there wasn't many packages :), so it is kind of good problem. Using third party package management has helped me a lot there, like pipenv, or poetry.
There are lots of other languages I guess to pick from now.
(Score: 1, Interesting) by Anonymous Coward on Wednesday May 03, @04:04PM (1 child)
> Different versions of the language are incompatible,
I know nothing about Python, but sometimes I work in a mechanical testing lab. Some of their test data post processing is written in Python. They complain that a routine written last year only runs on some of their PCs -- they blame it on different versions of Windows.
(Score: 0) by Anonymous Coward on Wednesday May 03, @05:47PM
Sounds more like a Python problem than a Windows problem. Plenty of non-python stuff written last year or even 2003 still works on Windows 7, 8 and 10 today... 😉
Perhaps this is because the Python bunch ( with their usual lack of commitment to backward compatibility) decided to stop supporting Windows 7 with the latest python stuff and for various reasons ($$$$$?) you still need to support Windows 7.
https://discuss.python.org/t/windows-7-support-for-python-3-9-or-3-10-without-a-fork/13842 [python.org]
My recommendation is to not use python if you want your stuff to keep working for years without rewrites or jumping through hoops.
(Score: 2) by digitalaudiorock on Wednesday May 03, @05:21PM (1 child)
This x 1000. Pretty near EVERY time I've had any interest in using something written in python under Linux, it quickly became clear that it needed versions of stuff that would require going around the distribution's package manager with pip. That's NEVER a good idea...in fact in Gentoo for example you run the risk of breaking the package manager itself (portage is written in python) and being truly screwed. The only viable option in many cases would be to run it in a python virtualenv.
(Score: 2) by GloomMower on Wednesday May 03, @05:30PM
There are good utilities to handle all that for you. pipenv and poetry. Would be good if one of those came with python, or was integrated with pip though.
https://pipenv.pypa.io/en/latest/ [pipenv.pypa.io]
https://python-poetry.org/ [python-poetry.org]
(Score: 3, Informative) by PiMuNu on Wednesday May 03, @02:28PM (5 children)
> What has become complicated?
As a random example:- manipulating strings has become a terrible chore in the last 2-3 years. I have been using python for maybe 15 years and it used to "just work".
(Score: 2) by GloomMower on Wednesday May 03, @02:54PM (3 children)
> As a random example:- manipulating strings has become a terrible chore in the last 2-3 years.
Because of utf8?
The new handling of utf8 has made my life a lot easier, however if I am needing to manipulate raw bytes it is more complicated than how it was handled in python2.
(Score: 2) by PiMuNu on Wednesday May 03, @04:01PM (1 child)
Yes, because of utf8.
May just be I am not used to things, but to me it is very unclear what is a string, what is a bytes object, how utf8 and other encodings interact with each other. I don't understand what a bytes object is. Stuff which used to be "just a string" is now obfuscated in some strange way. I have to invoke a mysterious "decode" to some of the strings I encounter for no reason that I can parse, in order to manipulate them.
In general I struggle to see how python3 is an improvement over python2. I can't think of any changes that are an improvement.
(Score: 2) by GloomMower on Wednesday May 03, @04:54PM
Ahh okay. Yes if all you were ever doing was ASCII or byte level work, python3 is more cumbersome for that. But UTF8 string handling in python3 is much better IMO.
I can see why some people might not like that change, but on the other hand I recognize there are languages other than English. I was resistant at first to everything moving to utf8 but it is starting to be a good thing I think.
(Score: 3, Interesting) by janrinok on Wednesday May 03, @05:07PM
I agree. Those who never use anything other than ascii don't even think of other languages. But even the people on this site have forgotten how they clamoured for utf in our own Perl code. TMB did a sterling job on that, and now any language in the world can be written. Ah, but some ask who cares? Well, Joe Merchant has his signature, others use different languages. I quoted something on my May Day post in French (and English), complete with all the appropriate special characters that the language uses. Look in the journals. there is Korean, Arabic, Russian and many other languages.
And in Python, it is now easily doable.
(Score: 1) by modest on Wednesday May 03, @02:58PM
I'm sorry to ask but can you elaborate on what you mean by this? I think all the same features are there plus the introduction of f-strings was a game changer for readability / simplicity. If I had to guess, you're referring to the changes from Python2 to Python3 around encoding into byte-strings which trips me up too sometimes too but, well, explicit is better than implicit and I feel like those change made lower-level communication & bit twiddling a little more clear.
(Score: 2) by istartedi on Wednesday May 03, @06:00PM
It's gotten a huge boost from the current crop of AI chatbots. It seems like they all produce Python as their default output when asked to generate code. A whole new crop of new programmers are being introduced to it, and this makes sense because IIRC it was originally designed to replace BASIC as an introductory programming language.
Love it or hate it, I don't think it's going away any time soon.
Appended to the end of comments you post. Max: 120 chars.
(Score: 2) by sjames on Wednesday May 03, @06:03PM (1 child)
The language is fine. One implementation may be getting baroque (or just maturing), but there are others.
Compare to C++ where the spec itself is getting baroque and seems to be in danger of swallowing it's own tail.
(Score: 3, Funny) by DannyB on Wednesday May 03, @08:05PM
If it is broke, then fix it.
If it ain't broke, then fix it until it is.
How often should I have my memory checked? I used to know but...
(Score: 3, Interesting) by squeedles on Wednesday May 03, @02:28PM (1 child)
I've recently written a Python native interface to a big C++ package. Having previously done similar native interfaces to Node, .NET, and even back in the day Java and TCL, I'm pretty satisfied with the internals of Python and don't find it particularly complex. There documentation for the internals is a bit sparse, but reasonable given how few people work at that level. It can take a while to figure out how to do a particular thing for the first time, but a little source diving usually does the trick.
As a developer, I really need the limited api (sort of a stable ABI) so that I don't have to chase versions shipping on different platforms or when they drop a new 3.x like this. However, it seems like the limited api is being developed in a real haphazard fashion, and found some "can't get there from here" situations until they finally got to 3.10. I'd really like to see them spend a little more time covering the bases on that.
(Score: 1, Funny) by Anonymous Coward on Wednesday May 03, @03:07PM
How is one supposed to "move fast and break things" with that attitude???
(Score: 2) by istartedi on Wednesday May 03, @05:56PM (3 children)
Probably a naive question, but if your algorithm requires a lot of communication between threads, won't it be difficult to parallelize anyway?
Everything I've ever worked with that could be easily parallelized also had really quick locks and very little communication between threads, or you just fork()'d.
Appended to the end of comments you post. Max: 120 chars.
(Score: 2) by DannyB on Wednesday May 03, @08:03PM (2 children)
The easiest problems to parallelize are called "embarrassingly parallel" problems.
Such as something that you need to do to each and every pixel -- independently from adjacent pixels. So, NOT Conway's Game of Life which requires information about adjacent pixels.
A good example is every pixel in some particular view of the Mandelbrot set, or other related fractal. Each pixel's center has a floating point coordinate, possibly hundreds of digits long, and the calculation of whether it is in the set or not, and how many iterations before it escapes, is determined independently from adjacent pixels. So highly parallelizeable.
How often should I have my memory checked? I used to know but...
(Score: 3, Insightful) by Snotnose on Thursday May 04, @05:04AM (1 child)
Say what now? Life is the perfect thing to parallelize. You ping pong between 2 buffers, A and B. While A is displayed you traverse it to build B. Then display B and build the next generation in A. Lather, rinse, repeat.
If you try using only 1 buffer not only will it look like crap onscreen, it will be wrong. Say 2,2 is dead, but it's live for the next generation. Now when you process 2,3 the cell in 2,2, that should be dead, is now live.
I just passed a drug test. My dealer has some explaining to do.
(Score: 2) by DannyB on Thursday May 04, @02:12PM
If I were to attempt to parallelize Life, I would first implement the Hashlife [wikipedia.org] algorithm.
This already makes Life much faster.
Then I would parallelize that.
How often should I have my memory checked? I used to know but...
(Score: 3, Funny) by DannyB on Wednesday May 03, @07:58PM (3 children)
I cannot seem to find Python 3:12 ?
How often should I have my memory checked? I used to know but...
(Score: 3, Funny) by DannyB on Wednesday May 03, @08:08PM (1 child)
Nevermind, Chat GPT is hear to save the day . . .
How often should I have my memory checked? I used to know but...
(Score: 1, Interesting) by Anonymous Coward on Wednesday May 03, @08:46PM
Python and clean and efficient do not belong in the same sentense.
(Score: 3, Touché) by istartedi on Thursday May 04, @04:35PM
In the Biblical subversion repository, it's Genesis 3:1 [biblehub.com]
Appended to the end of comments you post. Max: 120 chars.