Call for donations - PyPy to support Python3!
UPDATE (February 2014):
Thanks to our donors, we have raised 45% of the total so far. Work on this topic has been happening, and continues to happen, within the budget – even if not within the timeline described below. We have simply not found enough time to work on it as much as we wanted, and thus did not consume the money as quickly as predicted. The ratio “progress / $ used” so far corresponds roughly to what we expected. The document below is the original call for proposal, and we still accept donations for this topic.
The release of Python 3 has been a major undertaking for the Python community, both technically and socially. So far the PyPy interpreter implements only version 2 of the Python language and is increasingly used in production systems. It thus contributes to the general risk of a long lasting Python community split where a lot of people continue using Python 2 while others work with Python 3, making it harder for everyone.
The PyPy project is in a unique position in that it could support Python 3 without having to discontinue supporting Python 2, with the possibility of reusing a large part of the code base and fully reusing its unique translation and JIT-Compiler technologies. However, it requires a lot of work, and it will take a long time before we can complete a Python 3 port if we only wait for volunteer work. Thus, we are asking the community to help with funding the necessary work, to make it happen faster. Here is a more detailed view on how our proposed work benefits the Python community and the general public.
Below you'll find the planned stages of work and the associated fundraising targets we need to make things happen. Once we reach the necessary target for each stage, we will start contracting developers. Contracts and money are managed by the non-profit Software Freedom Conservancy of which the PyPy project is a member. The current elected representatives are Carl Friedrich Bolz, Holger Krekel and Jacob Hallen and they will - in close collaboration with Conservancy and the core developers - select the best developers for the Python 3 porting job among well known PyPy contributors.
If you want to see PyPy support Python 3 and Python 2, donate using buttons on the side.
Should we not receive enough donations to complete all stages by 1st March 2012 at the latest, we will try our best to make PyPy support Python 3 anyway. We however reserve the right to shift any unused funds to other PyPy activities when that date is reached. Of course, since the Conservancy is a 501(c)(3) charitable organization incorporated in NY, USA, all funds will, regardless of their use, be spent in a way that benefits the general public, the advancement of Open Source and Free Software, and in particular the PyPy community and the PyPy codebase.
Note For donations higher than $1,000, we can arrange for an invoice and a different payment method to avoid the high Paypal fees. Please contact pypy at sfconservancy.org if you want to know details on how to donate via other means.
Planned stages of work
The goal of this project is to write an interpreter that interprets version 3 of the Python language. To be precise we would aim at having a Python 3.2 interpreter together in the same codebase as the python 2.7 one.
At the end of the project, it will be possible to decide at translation time whether to build an interpreter which supports Python 2.7 or Python 3.2 and both versions will be nightly tested and available from nightly builds.
The focus of this project is on compatibility, not performance. In particular, it might be possible that the resulting Python 3 interpreter will be slower than the Python 2 one. If needed, optimizing and making it more JIT friendly will be the scope of a separate project. Our existing JIT generation technology should apply out of the box; this disclaimer is only about the extra performance we could obtain by tweaking the Python 3 interpreter or writing specific interpreter-guided optimizations in the JIT.
About estimates and costs
For each step, we estimated the time that it would take to complete for an experienced developer who is already familiar with the PyPy codebase. From this number, the money is calculated considering an hourly rate of $60, and a 5% general donation which goes to the Software Freedom Conservancy itself, the non-profit association of which the PyPy project is a member and which manages all the issues related to donations, taxes and payments.
The estimated time to complete the whole project is about 10.5 person-months.
For comparison, the python-3000 mailing list was created in March 2006; Python 3.0 was released in December 2008 and Python 3.1, the first release genuinely suitable for production use (due to the abysmal I/O performance of 3.0) was released in June 2009. During these 3.5 years, a lot of people contributed to the development of Python 3, and while it is hard to turn these numbers into precise person-years, it sounds reasonable to think that in total it took several person-years.
We have the advantage of targeting something that already exists without having to define the destination as they go along, and also the internal architecture of PyPy makes it easier to do the porting.
Step 1: core language
In this step, we implement all the changes to the core language, i.e. everything which is not in the extension modules. This includes, but it is not necessarily limited to the following items, which are split into two big areas:
Sub-step 1.1: string vs unicode and I/O:
- adapt the existing testing infrastructure to support running Python 3 code
- string vs bytes: the interpreter uses unicode strings everywhere.
- the print function
- open is now an alias for io.open, removal of the old file type.
- string formatting (for the part which is not already implemented in Python 2.7)
- the _io module (for the part which is not already implemented in Python 2.7)
- syntactic changes to make io.py importable (in particular: metaclass=... in class declarations)
- Estimate cost: $35,000
Sub-step 1.2: other syntactic changes, builtin types and functions, exceptions:
- views and iterators instead of lists (e.g., dict.items(), map, range & co.)
- new rules for ordering comparisons
- removal of old-style classes
- int/long unification
- function annotations
- smaller syntax changes, such as keyword-only arguments, nonlocal, extended iterable unpacking, set literals, dict and set comprehension, etc.
- changes to exceptions: __traceback__ attribute, chained exceptions, del e at the end of the except block, etc.
- changes to builtins: super, input, next(), etc.
- improved with statement
- Estimate cost: $28,000
Note that the distinction between sub-steps 1.1 and 1.2 is blurry, and it might be possible that during the development we will decide to move items between the two sub-steps, as needed.
For more information, look at the various “What's new” documents:
Total estimate cost: $63,000
Step 2: extension modules
In this step, we implement all the changes to the extension modules which are written in C in CPython. This includes, but it is not necessarily limited to:
- collections, gzip, bz2, decimal, itertools, re, functools, pickle, _elementtree, math, etc.
Estimate cost: this is hard to do at this point, we will be able to give a more precise estimate as soon as Step 1 is completed. As a reference, it should be possible to complete it with $37,000
Step 3: cpyext
The cpyext module allows to load CPython C extensions in PyPy. Since the C API changed a lot between Python 2.7 and Python 3.2, cpyext will not work out of the box in the Python 3 PyPy interpreter. In this step, we will adapt it to work with Python 3 as well.
Note that, even for Python 2, cpyext is still in a beta state. In particular, not all extension modules compile and load correctly. As a consequence, the same will be true for Python 3 as well. As a general rule, we expect that if a Python 2 module works with cpyext, the corresponding Python 3 version will also work when this step is completed, although the details might vary depending on the exact C extension module.
Estimate cost: $5,000
Benefits of This Work to the Python Community and the General Public
Python has become one of the most popular dynamic programming languages in the world. Web developers, educators, and scientific programmers alike all value Python because Python code is often more readable and because Python often increases programmer productivity.
Traditionally, languages like Python ran more slowly than static, compiled languages; Python developers chose to sacrifice execution speed for ease of programming. The PyPy project created a substantially improved Python language implementation, including a fast Just-in-time (JIT) compiler. The increased execution speed that PyPy provides has attracted many users, who now find their Python code runs up to four times faster under PyPy than under the reference implementation written in C. Some programs that are particularly JIT-friendly see even greater speedups. See our speed website.
Meanwhile, the Python community is undergoing significant change, with the introduction of a new ‘version 3’ of the Python language (Python 3). Python 3 breaks some backwards compatibility with 2.x series, so programmers who seek to use Python 3 must port old code. For example, programmers who needed old libraries might want to use Python 2 only and programmers wanting to use new language features would like to use Python 3, but would not be able to use libraries that written for Python 2.x without substantial rewrites. This issue could lead to a dangerous community split: programmers who needed old libraries might only use Python 2 and programmers seeking new language features would use Python 3, but would not be able to use libraries written for Python 2.x.
PyPy currently supports only Python 2.7. While PyPy supports only version 2 of the Python language, PyPy users have an incentive to avoid Python 3. Python programmers must chose between the language features in Python 3, and the substantial performance benefits offered by PyPy.
To address this issue, the PyPy team proposes to implement Python 3 on PyPy. With such improvements to PyPy, PyPy can support the entire Python-using community and hopefully help to prevent any community split. PyPy support of Python 3 would also bring the excellent code execution performance of PyPy to those who wish to migrate to Python 3.
Moreover, by ensuring the latest version of the Python language specification (Py3k) works properly and fully on PyPy, the maximal benefit of collaboration, learning, and software improvement can happen in the entire Python language community.
A broad community of developers support and develop the PyPy project, many of whom work as volunteers. The Py3k grant should help with turning some attention towards implementing Python 3. This will not hinder other directions in which PyPy is going like improving performance. The goal of the PyPy community is to support both Python 2 and Python 3 for the foreseeable future.
PyPy's developers make all PyPy software available to the public without charge, under PyPy's Open Source copyright license, the permissive MIT License. PyPy's license assures that PyPy is equally available to everyone freely on terms that allow both non-commercial and commercial activity. This license allows for academics, for-profit software developers, volunteers and enthusiasts alike to collaborate together to make a better Python implementation for everyone.
Finally, tracing JITs and other programming language execution technology used in PyPy are of current and particular interest in computer science research. PyPy helps cross-pollinate knowledge between academic computer science and industrial use of Python, since PyPy can function well both as a research tool and real-world Python programming language implementation. Continued support and evolution of PyPy in any direction, such as support for Py3k, increases the features available from PyPy and such improvements are expected to spark even more general interest in PyPy itself and the Python programming language generally.