PyPy (Posts by mattip)

PyPy v7.3.23 release

mattip — Wed, 27 May 2026 10:00:00 GMT

PyPy v7.3.23: release of python 2.7, 3.11

The PyPy team is proud to release version 7.3.23 of PyPy after the previous release on April 26, 2026. This is a bug-fix release that fixes an overeager warning about unused coroutines, and some problems around multiple inheritance in c-extensions.

This version includes a change to the bytecode interpreter to use exception tables instead of dedicated opcodes. Now the PyPy disassembly will be closer to CPython format. So far it does not impact performance.

The release includes two different interpreters:

PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
PyPy3.11, which is an interpreter supporting the syntax and the features of Python 3.11, including the stdlib for CPython 3.11.15.

The interpreters are based on much the same codebase, thus the double release. This is a micro release, all APIs are compatible with the other 7.3 releases.

We recommend updating. You can find links to download the releases here:

https://pypy.org/download.html

We would like to thank our donors for the continued support of the PyPy project. If PyPy is not quite good enough for your needs, we are available for direct consulting work. If PyPy is helping you out, we would love to hear about it and encourage submissions to our blog via a pull request to https://github.com/pypy/pypy.org

We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: bug fixes, PyPy and RPython documentation improvements, or general help with making RPython's JIT even better.

If you are a python library maintainer and use C-extensions, please consider making a HPy / CFFI / cppyy version of your library that would be performant on PyPy. In any case, cibuildwheel supports building wheels for PyPy.

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython. It's fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

We provide binary builds for:

x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)
64-bit ARM machines running Linux (aarch64) and macos (macos_arm64).

PyPy supports Windows 32-bit, Linux PPC64 big- and little-endian, Linux ARM 32 bit, RISC-V RV64IMAFD Linux, and s390x Linux but does not release binaries. Please reach out to us if you wish to sponsor binary releases for those platforms. Downstream packagers provide binary builds for debian, Fedora, conda, OpenBSD, FreeBSD, Gentoo, and more.

What else is new?

For more information about the 7.3.23 release, see the full changelog.

Please update, and continue to help us make pypy better.

Cheers, The PyPy Team

PyPy v7.3.22 release

mattip — Tue, 28 Apr 2026 10:00:00 GMT

PyPy v7.3.22: release of python 2.7, 3.11

The PyPy team is proud to release version 7.3.22 of PyPy after the previous release on March 13, 2026. This is a bug-fix release that fixes several issues in the JIT. Among them, a long-standing JIT bug that started appearing when some instance optimizations exposed it. We also cleaned up many of the remaining stdlib test suite failures, which improves CPython compatibility around line numbers in dis.dis, signatures and objclass attributes for builtins, and other quality of life features.

There is now an RPython _pickle module that mirrors the CPython one, greatly speeding up pickling operations. Where before PyPy was 5.7x slower than CPython on the pickle benchmark from the pyperformance benchmark suite, now it is only 1.6x slower [0]. We also added pypy pickler extensions to dump and load lists using list strategies, and enabled them in the ForkingPickler used by multiprocessing, speeding up cases where such objects are passed between PyPy multiprocessing instances.

We also added an RPython json encoder, speeding up json_bench from being 2.6x slower than CPython to being 0.7x (meaning faster).

The release includes two different interpreters:

PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
PyPy3.11, which is an interpreter supporting the syntax and the features of Python 3.11, including the stdlib for CPython 3.11.15.

The interpreters are based on much the same codebase, thus the double release. This is a micro release, all APIs are compatible with the other 7.3 releases.

We recommend updating. You can find links to download the releases here:

https://pypy.org/download.html

Footnotes

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython It's fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

We provide binary builds for:

x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)
64-bit ARM machines running Linux (aarch64) and macos (macos_arm64).

What else is new?

For more information about the 7.3.22 release, see the full changelog.

Please update, and continue to help us make pypy better.

Cheers, The PyPy Team

Using Claude to fix PyPy3.11 test failures securely

mattip — Mon, 23 Mar 2026 10:27:55 GMT

I got access to Claude Max for 6 months, as a promotional move Anthropic made to Open Source Software contributors. My main OSS impact is as a maintainer for NumPy, but I decided to see what claude-code could to for PyPy's failing 3.11 tests. Most of these failures are edge cases: error messages that differ from CPython, or debugging tools that fail in certain cases. I was worried about letting an AI agent loose on my development machine. I noticed a post by Patrick McCanna (thanks Patrick!) that pointed to using bubblewrap to sandbox the agent. So I set it all up and (hopefully securely) pointed claude-code at some tests.

Setting up

There were a few steps to make sure I didn't open myself up to obvious gotchas. There are stories about agents wiping out data bases, or deleting mail boxes.

Bubblewrap

First I needed to see what bubblewrap does. I followed the instructions in the blog post to set things up with some minor variations:

sudo apt install bubblewrap

I couldn't run bwrap. After digging around a bit, I found I needed to add an exception for appamor on Ubuntu 24.04:

sudo bash -c 'cat > /etc/apparmor.d/bwrap << EOF
abi <abi/4.0>,
include <tunables/global>

profile bwrap /usr/bin/bwrap flags=(unconfined) {
  userns,
}
EOF'
sudo apparmor_parser -r /etc/apparmor.d/bwrap

Then bwrap would run. It is all locked down by default, so I opened up some exceptions. The arguments are pretty self-explanatory. Ubuntu spreads the executables around the operating system, so I needed access to various directories. I wanted a /tmp for running pytest. I also wanted the prompt to reflect the use of bubblewrap, so changed the hostname:

cat << 'EOL' >> ./run_bwrap.sh
  function call_bwrap() {
    bwrap \
      --ro-bind /usr /usr \
      --ro-bind /etc /etc \
      --ro-bind /run /run \
      --symlink usr/lib /lib \
      --symlink usr/lib64 /lib64 \
      --symlink usr/bin /bin \
      --proc /proc \
      --dev /dev \
      --bind $(pwd) $(pwd) \
      --chdir $(pwd) \
      --unshare-user --unshare-pid --unshare-ipc --unshare-uts --unshare-cgroup \
      --die-with-parent \
      --hostname bwrap \
      --tmpfs /tmp \
      /bin/bash "$@"
  }
EOL

source ./run_bwrap.sh
call_bwrap
# now I am in a sandboxed bash shell
# play around, try seeing other directories, getting sudo, or writing outside
# the sandbox
exit

I did not do --unshare-network since, after all, I want to use claude and that needs network access. I did add rw access to $(pwd) since I want it to edit code in the current directory, that is the whole point.

Basic claude

After trying out bubblewrap and convincing myself it does actually work, I installed claude code

curl -fsSL https://claude.ai/install.sh | bash

Really Anthropic, this is the best way to install claude? No dpkg?

I ran claude once (unsafely) to get logged in. It opened a webpage, and saved the login to the oathAccount field in ~/.claude.json. Now I changed my bash script to this to get claude to run inside the bubblewrap sandbox:

cat << 'EOL' >> ./run_claude.sh
  claude-safe() {
    bwrap \
      --ro-bind /usr /usr \
      --ro-bind /etc /etc \
      --ro-bind /run /run \
      --ro-bind "$HOME/.local/share/claude" "$HOME/.local/share/claude" \
      --symlink usr/lib /lib \
      --symlink usr/lib64 /lib64 \
      --symlink usr/bin /bin \
      --symlink "$HOME/.local/share/claude/versions/2.1.81" "$HOME/.local/bin/claude" \
      --proc /proc \
      --dev /dev \
      --bind $(pwd) $(pwd) \
      --bind "$HOME/.claude" "$HOME/.claude" \
      --bind "$HOME/.claude.json" "$HOME/.claude.json" \
      --chdir $(pwd) \
      --unshare-user --unshare-pid --unshare-ipc --unshare-uts --unshare-cgroup \
      --die-with-parent \
      --hostname bwrap \
      --tmpfs /tmp \
      --setenv PATH "$HOME/.local/bin:$PATH" \
      claude "$@"
  }
EOL

source ./run_claude.sh
claude-safe

Now I can use claude. Note it needs some more directories in order to run. This script hard-codes the version, in the future YMMV. I want it to be able to look at github, and also my local checkout of cpython so it can examine differences. I created a read-only token by clicking on my avatar in the upper right corner of a github we page, then going to Settings → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token. Since pypy is in the pypy org, I used "Repository owner: pypy", "Repository access: pypy (only)" and "Permissions: Contents". Then I made doubly sure the token permissions were read-only. And checked again. Then I copied the token to the bash script. I also added a ro-bind to the cpython checkout, so I could tell claude code where to look for CPython implementations of missing PyPy functionality.

--ro-bind "$HOME/oss/cpython" "$HOME/oss/cpython" \
--setenv GH_TOKEN "hah, sharing my token would not have been smart" \

Claude /sandbox

Claude comes with its own sandbox, configured by using the /sandbox command. I chose the defaults, which prevents malicious code in the repo from accessing the file system and the network. I was missing some packages to get this to work. Claude would hang until I installed them, and I needed to kill it with kill.

sudo apt install socat
sudo npm install -g @anthropic-ai/sandbox-runtime

Final touches

One last thing that I discovered later: I needed to give claude access to some grepping and git tools. While git should be locked down externally so it cannot push to the repo, I do want claude to look at other issues and pull requests in read-only mode. So I added a local .claude/settings.json file inside the repo (see below for which directory to do this):

{
  "permissions": {
    "allow": [
      "Bash(sed*)",
      "Bash(grep*)",
      "Bash(cat*)",
      "Bash(find*)",
      "Bash(rg*)",
      "Bash(python*)",
      "Bash(pytest*)"
    ]
  }
}

Then I made git ignore it, even when doing a git clean, in a local (not part of the repo) configuration

echo -n .claude >> ~/.config/git/ignore

What about `git push`?

I don't want claude messing around with the upstream repo, only read access. But I did not actively prevent git push. So instead of using my actual pypy repo, I cloned it to a separate directory and did not add a remote pointing to github.com.

Fixing tests - easy

Now that everything is set up (I hope I remembered everything), I could start asking questions. The technique I chose was to feed claude the whole test failure from the buildbot. So starting from the buildbot py3.11 summary, click on one of the F links and copy-paste all that into the claude prompt. It didn't take long for claude to come up with solutions for the long-standing ctype error missing exception which turned out to be due to an missing error trap when already handling an error.

Also a CTYPES_MAX_ARGCOUNT check was missing. At first, claude wanted to change the ctypes code from CPython's stdlib, and so I had to make it clear that claude was not to touch the files in lib-python. They are copied verbatim from CPython and should not be modified without really good reasons.

The fix to raise TypeError rather than Attribute Error for deleting ctype object's value was maybe a little trickier: claude needed to create its own property class and use it in assignments.

The fix for a failing test for a correct repr of a ctypes array was a little more involved. Claude needed to figure out that newmemoryview was raising an exception, dive into the RPython implementation and fix the problem, and then also fix a pure-python __buffer__ shape edge case error.

There were more, but you get the idea. With a little bit of coaching, and by showing claude where the CPython implementation was, more tests are now passing.

Fixing tests - harder

PyPy has a HPy backend. There were some test failures that were easy to fix (a handle not being closed, an annotation warning). But the big one was a problem with the context tracking before and after ffi function calls. In debug mode there is a check that the ffi call is done using the correct HPy context. It turns out to be tricky to hang on to a reference to a context in RPython since the context RPython object is pre-built. The solution, which took quite a few tokens and translation cycles to work out, was to assign the context on the C level, and have a getter to fish it out in RPython.

Conclusion

I started this journey not more than 24 hours ago, after some successful sessions using claude to refactor some web sites off hosting platforms and make them static pages. I was impressed enough to try coding with it from the terminal. It helps that I was given a generous budget to use Anthropic's tool.

Claude seems capable of understanding the layers of PyPy: from the pure python stdlib to RPython and into the small amount of C code. I even asked it to examine a segfault in the recently released PyPy7.3.21, and it seems to have found the general area where there was a latent bug in the JIT.

Like any tool, agentic programming must be used carefully to make sure it cannot do damage. I hope I closed the most obvious foot-guns, if you have other ideas of things I should do to protect myself while using an agent like this, I would love to hear about them.

PyPy v7.3.21 release

mattip — Fri, 13 Mar 2026 10:00:00 GMT

PyPy v7.3.21: release of python 2.7, 3.11

The PyPy team is proud to release version 7.3.21 of PyPy after the previous release on July 4, 2025. This is a bug-fix release that also updates to Python 3.11.15.

The release includes two different interpreters:

PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
PyPy3.11, which is an interpreter supporting the syntax and the features of Python 3.11, including the stdlib for CPython 3.11.15.

The interpreters are based on much the same codebase, thus the double release. This is a micro release, all APIs are compatible with the other 7.3 releases.

We recommend updating. You can find links to download the releases here:

https://pypy.org/download.html

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython It's fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

We provide binary builds for:

x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)
64-bit ARM machines running Linux (aarch64) and macos (macos_arm64).

What else is new?

For more information about the 7.3.21 release, see the full changelog.

Please update, and continue to help us make pypy better.

Cheers, The PyPy Team

PyPy v7.3.20 release

mattip — Fri, 04 Jul 2025 12:00:00 GMT

PyPy v7.3.20: release of python 2.7, 3.11

The PyPy team is proud to release version 7.3.20 of PyPy after the previous release on Feb 26, 2025. The release fixes some subtle bugs in ctypes and OrderedDict and makes PyPy3.11 compatible with an upcoming release of Cython.

The release includes two different interpreters:

PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
PyPy3.11, which is an interpreter supporting the syntax and the features of Python 3.11, including the stdlib for CPython 3.11.13.

The interpreters are based on much the same codebase, thus the double release. This is a micro release, all APIs are compatible with the other 7.3 releases.

We recommend updating. You can find links to download the releases here:

https://pypy.org/download.html

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython It's fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

We provide binary builds for:

x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)
64-bit ARM machines running Linux (aarch64) and macos (macos_arm64).

What else is new?

For more information about the 7.3.20 release, see the full changelog.

Please update, and continue to help us make pypy better.

Cheers, The PyPy Team

PyPy v7.3.19 release

mattip — Wed, 26 Feb 2025 12:00:00 GMT

PyPy v7.3.19: release of python 2.7, 3.10 and 3.11 beta

The PyPy team is proud to release version 7.3.19 of PyPy. This is primarily a bug-fix release fixing JIT-related problems and follows quickly on the heels of the previous release on Feb 6, 2025.

This release includes a python 3.11 interpreter. There were bugs in the first beta that could prevent its wider use, so we are continuing to call this release "beta". In the next release we will drop 3.10 and remove the "beta" label.

The release includes three different interpreters:

PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
PyPy3.10, which is an interpreter supporting the syntax and the features of Python 3.10, including the stdlib for CPython 3.10.16.
PyPy3.11, which is an interpreter supporting the syntax and the features of Python 3.11, including the stdlib for CPython 3.11.11.

The interpreters are based on much the same codebase, thus the triple release. This is a micro release, all APIs are compatible with the other 7.3 releases. It follows after 7.3.17 release on August 28, 2024.

We recommend updating. You can find links to download the releases here:

https://pypy.org/download.html

If you are a python library maintainer and use C-extensions, please consider making a HPy / CFFI / cppyy version of your library that would be performant on PyPy. In any case, both cibuildwheel and the multibuild system support building wheels for PyPy.

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython It's fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

We provide binary builds for:

x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)
64-bit ARM machines running Linux (aarch64) and macos (macos_arm64).

What else is new?

For more information about the 7.3.19 release, see the full changelog.

Please update, and continue to help us make pypy better.

Cheers, The PyPy Team

PyPy v7.3.18 release

mattip — Thu, 06 Feb 2025 12:00:00 GMT

PyPy v7.3.18: release of python 2.7, 3.10 and 3.11 beta

The PyPy team is proud to release version 7.3.18 of PyPy.

This release includes a python 3.11 interpreter. We are labelling it "beta" because it is the first one. In the next release we will drop 3.10 and remove the "beta" label. There are a particularly large set of bugfixes in this release thanks to @devdanzin using fusil on the 3.10 builds, originally written by Victor Stinner. Other significant changes:

We have updated libffi shipped in our portable builds. We also now statically link to libffi where possible which reduces the number of shared object dependencies.
We have added code to be able to show the native function names when profiling with VMProf. So far only Linux supports this feature.
We have added a PEP 768-inspired remote debugging facility.
The HPy backend has been updated to latest HPy HEAD

The release includes three different interpreters:

PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
PyPy3.10, which is an interpreter supporting the syntax and the features of Python 3.10, including the stdlib for CPython 3.10.16.
PyPy3.11, which is an interpreter supporting the syntax and the features of Python 3.11, including the stdlib for CPython 3.11.11.

We recommend updating. You can find links to download the releases here:

https://pypy.org/download.html

VMProf Native Symbol Names

When running VMProf profiling with native profiling enabled, PyPy did so far not produce function names for C functions. The output looked like this:

pypy -m vmprof ~/projects/gitpypy/lib-python/2.7/test/pystone.py
Pystone(1.1) time for 50000 passes = 0.0109887
This machine benchmarks at 4.55011e+06 pystones/second
 vmprof output:
 %:      name:                location:
 100.0%  entry_point          <builtin>/app_main.py:874
 100.0%  run_command_line     <builtin>/app_main.py:601
 100.0%  run_toplevel         <builtin>/app_main.py:93
 100.0%  _run_module_as_main  /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:150
 100.0%  _run_code            /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:62
 100.0%  <module>             /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/site-packages/vmprof/__main__.py:1
 100.0%  main                 /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/site-packages/vmprof/__main__.py:30
 100.0%  run_path             /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:238
 100.0%  _run_module_code     /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:75
 100.0%  <module>             /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:3
 100.0%  main                 /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:60
 100.0%  pystones             /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:67
 100.0%  Proc0                /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:79
 76.9%   <unknown code>
 69.2%   <unknown code>
 53.8%   <unknown code>
 53.8%   <unknown code>
 46.2%   <unknown code>
 46.2%   <unknown code>
 38.5%   <unknown code>
 38.5%   Proc8                /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:212
 30.8%   <unknown code>
 ...

We can now symbolify these C functions and give function names and which shared library they come from, at least on Linux:

Pystone(1.1) time for 50000 passes = 0.218967
This machine benchmarks at 228345 pystones/second
 vmprof output:
 %:      name:                                           location:
 100.0%  entry_point                                     <builtin>/app_main.py:889
 100.0%  run_command_line                                <builtin>/app_main.py:616
 100.0%  run_toplevel                                    <builtin>/app_main.py:95
 100.0%  _run_module_as_main                             /home/user/projects/gitpypy/lib-python/2.7/runpy.py:150
 100.0%  _run_code                                       /home/user/projects/gitpypy/lib-python/2.7/runpy.py:62
 100.0%  <module>                                        /home/user/projects/gitpypy/site-packages/vmprof/__main__.py:1
 100.0%  main                                            /home/user/projects/gitpypy/site-packages/vmprof/__main__.py:30
 100.0%  run_module                                      /home/user/projects/gitpypy/lib-python/2.7/runpy.py:179
 100.0%  _run_module_code                                /home/user/projects/gitpypy/lib-python/2.7/runpy.py:75
 100.0%  <module>                                        /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:3
 100.0%  main                                            /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:60
 100.0%  pystones                                        /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:67
 100.0%  Proc0                                           /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:79
 95.5%   n:pypy_g_execute_frame:0:pypy-c
 91.4%   n:pypy_g_PyFrame_dispatch:0:pypy-c
 63.8%   n:pypy_g_PyFrame_dispatch_bytecode:0:pypy-c
 49.8%   Proc1                                           /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:137
 17.6%   copy                                            /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:53
 13.6%   n:pypy_g_PyFrame_CALL_FUNCTION:0:pypy-c
 10.4%   Proc8                                           /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:212
 8.6%    n:pypy_g_STORE_ATTR_slowpath:0:pypy-c

This becomes even more useful when using the VMProf Firefox converter, which uses the Firefox Profiler Web UI to visualize profiling output:

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython It's fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

We provide binary builds for:

x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)
64-bit ARM machines running Linux (aarch64) and macos (macos_arm64).

What else is new?

For more information about the 7.3.18 release, see the full changelog.

Please update, and continue to help us make pypy better.

Cheers, The PyPy Team

Towards PyPy3.11 - an update

mattip — Sat, 04 Jan 2025 13:29:11 GMT

We¹ are steadily working towards a Python 3.11 interpreter, which will be part of the upcoming PyPy 7.3.18 release. Along with that, we also recently updated speed.pypy.org to compare PyPy's performance to CPython 3.11 (it used to be CPython 3.7).

Why is there no PyPy for Python 3.11?

TL;DR: we are working on it and hopefully will have a beta version soon

We started by merging the exception groups work by Nico Rittinghaus, merging the CPython stdlib for Python 3.11.9, and updating the regex engine to handle atomic groupings. I think these were the largest changes needed to support Python3.11, maybe I missed something else?

We then created a milestone with many of the changes that might not be caught in testing. As of today that milestone stands at 24/40 issues closed, 16 open. This is in addition to the v7.3.18 milestone, which has 11/23 issues closed, 12 open.

We also updated our infrastructure to run nightly buildbot test of the py3.11 branch, including adding py3.11 to our benchmarking site speed.pypy.org.

Then the real work started: fixing these milestone issues and failing stdlib tests. Some of the changes were cosmetic changes to error messages, some were more involved changes to the interpreter to behave more like CPython. For instance, hex(x) where x is an int calls long.format(x, "#x") on CPython where in PyPy we used x.__format__("#x"). This subtle difference caused a failure in the repr of IntEnum. Tracking down problems like these takes time. We are now down to about 250 failing stdlib tests, with thousands passing. For comparison, PyPy3.10, first released in June 2023, still has around 100 failing stdlib tests.

C-extension support

PyPy supports the Python C-API via a cpyext compatibility layer. We "mangle" the CPython symbols to add an extra Py to prevent loading CPython c-extension modules into PyPy, since the ABI is different. So a function like PyLong_FromLong will be exported from the c shared object as PyPyLong_FromLong. One of my long-standing goals is to remove this mangling, but it then requires that our c declarations, inlined functions, and macros in the headers match 1:1 the CPython headers. We can get by with not declaring and implementing parts of the interfaces, but what is declared must be identical. This is a long-term project, with each release the headers get closer to the CPython versions. I hope to concentrate on the PyUnicode interfaces for this release.

Note that the time to do this is before a new version release. Once the version is released, we cannot change the headers significantly.

So what is left?

Summarizing the milestones and other things to do:

Update to the recently released 3.11.11 stdlib
Make sure vmprof works
Update the time module to use more MONOTONIC_CLOCK, implement time.sleep differently, and clean up the many duplicate time like interfaces we have across the codebase. We have the time module, some time routines in the _threading moduleand RPython threading support in therpython/rlib` code. We should also make sure we are using 64-bit time interfaces.
Decide whether zero-cost exceptions gain us in performance and whether we should implement them even if they do not improve performance.
Update our hpy backend to the latest HEAD, which would allow running the hpy numpy fork
Reintegrate the pure-python pyrepl libbrary from CPython 3.13.

What else did I forget?

Why did the benchmark results get worse on speed.pypy.org?

TL;DR: running a benchmark site is hard. Something changed in the benchmark runner, and suddenly benchmarks got 10-15% slower.

PyPy run an instance of codepseed with a very old benchmarking suite that can still run Python2 (remember: that is the language of RPython underlying PyPy). The site runs on PSA infrastructure and the benchmarking machine is generously sponsored by Baroque Software. On Nov 9, there was a sudden jump in benchmarking times. For instance here is the result for the float benchmark. This happened across various benchmark runs: both PyPy2.7 and PyPy3.11alpha, with and without the JIT. After spending some time rerunning various benchmarks, I could only conclude the machine itself had gotten slower, maybe due to some security update in the linux kernel, maybe some change in the hosting platform. This "broke" the front-page comparsison: suddenly "latest" is much slower than the historic benchmarks run previous to the changes in the machine.

That page also recently (as of last week) uses CPython 3.11 as a baseline for comparison, where previously it used CPython3.7. It is common knowledge that the newer CPython versions are faster, and we see this now quite clearly. Diving into individual benchmarks, we can see that ones where PyPy-with-a-jit was comparable to CPython3.7 seem to be the ones that CPython3.11 improved greatly. Looking at a comparison of the runs this can be seen in benchmarks like deltablue and the sqlalchemy family. So the new graph has more lines that extend past 1.5 than the old graph.

New graph with PyPy3.11

Older graph with PyPy3.10

These days most of the work is done by CF Bolz-Tereick and me on a volunteer basis. Want to get involved? Reach out, we would love to expand the team. Have an idea for funding the work? Fantastic, let's talk. ↩

PyPy v7.3.17 release

mattip — Wed, 28 Aug 2024 12:22:08 GMT

PyPy v7.3.17: release of python 2.7 and 3.10

The PyPy team is proud to release version 7.3.17 of PyPy.

This release includes a new RISC-V JIT backend, an improved REPL based on work by the CPython team, and better JIT optimizations of integer operations. Special shout-outs to Logan Chien for the RISC-V backend work, to Nico Rittinghaus for better integer optimization in the JIT, and the CPython team that has worked on the repl.

The release includes two different interpreters:

PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
PyPy3.10, which is an interpreter supporting the syntax and the features of Python 3.10, including the stdlib for CPython 3.10.14.

The interpreters are based on much the same codebase, thus the dual release. This is a micro release, all APIs are compatible with the other 7.3 releases. It follows after 7.3.16 release on April 23, 2024.

We recommend updating. You can find links to download the releases here:

https://pypy.org/download.html

RISC-V backend for the JIT

PyPy's JIT has added support for generating 64-bit RISC-V machine code at runtime (RV64-IMAD, specifically). So far we are not releasing binaries for any RISC-V platforms, but there are instructions on how to cross-compile binaries.

REPL Improvements

The biggest user-visible change of the release is new features in the repl of PyPy3.10. CPython 3.13 has adopted and extended PyPy's pure-Python repl, adding a number of features and fixing a number or bugs in the process. We have backported and added the following features:

Prompts and tracebacks use terminal colors, as well as terminal hyperlinks for file names.
Bracketed paste enable pasting several lines of input into the terminal without auto-indentation getting in the way.
A special interactive help browser (F1), history browser (F2), explicit paste mode (F3).
Support for Ctrl-<left/right> to jump over whole words at a time.

See the CPython documentation for further details. Thanks to Łukasz Langa, Pablo Galindo Salgado and the other CPython devs involved in this work.

Better JIT optimizations of integer operations

The optimizers of PyPy's JIT have become much better at reasoning about and optimizing integer operations. This is done with a new "knownbits" abstract domain. In many programs that do bit-manipulation of integers, some of the bits of the integer variables of the program can be statically known. Here's a simple example:

x = a | 1
...
if x & 1:
    ...
else:
    ...

With the new abstract domain, the JIT can optimize the if-condition to True, because it already knows that the lowest bit of x must be set. This optimization applies to all Python-integers that fit into a machine word (PyPy optimistically picks between two different representations for int, depending on the size of the value). Unfortunately there is very little impact of this change on almost all Python code, because intensive bit-manipulation is rare in Python. However, the change leads to significant performance improvements in Pydrofoil (the RPython-based RISC-V/ARM emulators that are automatically generated from high-level Sail specifications of the respective ISAs, and that use the RPython JIT to improve performance).

PyPy versions and speed.pypy.org

The keen-eyed will have noticed no mention of Python version 3.9 in the releases above. Typically we will maintain only one version of Python3, but due to PyPy3.9 support on conda-forge we maintained multiple versions from the first release of PyPy3.10 in PyPy v7.3.12 (Dec 2022). Conda-forge is sunsetting its PyPy support, which means we can drop PyPy3.9. Since that was the major driver of benchmarks at https://speed.pypy.org, we revamped the site to showcase PyPy3.9, PyPy3.10, and various versions of cpython on the home page. For historical reasons, the "baseline" for comparison is still cpython 3.7.19.

We will keep the buildbots building PyPY3.9 until the end of August, these builds will still be available on the nightly builds tab of the buildbot.

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython It's fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

We provide binary builds for:

x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)
64-bit ARM machines running Linux (aarch64) and macos (macos_arm64).

What else is new?

For more information about the 7.3.17 release, see the full changelog.

Please update, and continue to help us make pypy better.

Cheers, The PyPy Team

Conda-forge proposes sunsetting support for PyPy

mattip — Fri, 09 Aug 2024 06:27:41 GMT

Conda-forge has kindly been providing support for PyPy since 2019. The conda-forge team has been very patient and generous with resources, but it seems the uptake of PyPy has not justified the effort. Major packages still are not available on PyPy, others find it hard to update versions. We don't get much feedback at all about people using PyPy, and even less about PyPy on conda-forge. The conda-forge team has proposed sunsetting PyPy going forward, which means current packages would remain but no new packages would be built. If you have an opinion, you can comment on that PR, or on this blog post.

Since conda-forge supports PyPy3.9 but not PyPy3.10, we have continued releasing PyPy3.9 even though we typically support only one version of PyPy3. With the sunsetting proposal, we will not release any more updates to PyPy3.9. I opened a poll about the intention to drop PyPy3.9. If you have an opinion, please chime in.

PyPy (Posts by mattip)

PyPy v7.3.23 release

PyPy v7.3.23: release of python 2.7, 3.11

What is PyPy?

What else is new?

PyPy v7.3.22 release

PyPy v7.3.22: release of python 2.7, 3.11

What is PyPy?

What else is new?

Using Claude to fix PyPy3.11 test failures securely

Setting up

Bubblewrap

Basic claude

Claude /sandbox

Final touches

What about git push?

Fixing tests - easy

Fixing tests - harder

Conclusion

PyPy v7.3.21 release

PyPy v7.3.21: release of python 2.7, 3.11

What is PyPy?

What else is new?

PyPy v7.3.20 release

PyPy v7.3.20: release of python 2.7, 3.11

What is PyPy?

What else is new?

PyPy v7.3.19 release

PyPy v7.3.19: release of python 2.7, 3.10 and 3.11 beta

What is PyPy?

What else is new?

PyPy v7.3.18 release

PyPy v7.3.18: release of python 2.7, 3.10 and 3.11 beta

VMProf Native Symbol Names

What is PyPy?

What else is new?

Towards PyPy3.11 - an update

Why is there no PyPy for Python 3.11?

C-extension support

So what is left?

Why did the benchmark results get worse on speed.pypy.org?

PyPy v7.3.17 release

PyPy v7.3.17: release of python 2.7 and 3.10

RISC-V backend for the JIT

REPL Improvements

Better JIT optimizations of integer operations

PyPy versions and speed.pypy.org

What is PyPy?

What else is new?

Conda-forge proposes sunsetting support for PyPy

What about `git push`?