Based on your timeline, a four year gap from "not yet ready to migrate" to "we won't fix security vulnerabilities" is very short.
Python is an open source project I've used and contributed nothing to, so I don't have the right to be a back seat driver. Were it a commercial project and I was a customer, I would be quite upset.
Python 2's EOL was first announced in 2008, it was extended in 2014. That's more than a decade of forewarning that this was coming down the line. A decade of Python2 receiving security fixes.
Were I a commercial customer, and had been told to switch to the new version ten years ago, then it's on me if I still have started the migration yet.
It's interesting seeing who the holdouts are, though. Less than a year ago, the CEO of Sentry (the crash reporter) locked a Github thread with essentially "we'll get there when we get there, sometime in or after 2020." But then, as late as 2016 their stance was pretty much that they would never migrate, so I suppose it's not surprising that they aren't super prepared for it:
Calibre is another well known case, and they don't plan to migrate.
Actually, I can't wait to remove it from my hard drive on Jan 1, since it's one of the most poorly designed apps I've ever used. It's really not surprising at all that they're not able to migrate it, assuming the quality of the app is indicative of the quality of the code. I wouldn't be surprised if the same was true of a lot of applications that are refusing to migrate. (I'll probably have to find an alternative to ebook-convert, the command line tool that's the only part of Calibre I use, or maybe I'll rewrite it in Python 3 myself.)
The only good part of calibre is the `ebook-convert` commandline tool that has a stupid-simple interface (polar opposite of the gui) and handles all kinds of formats that pandoc doesn't.
The gui, Calibre proper, is a total nightmare. Functional, but my god is it esoteric and ugly. I've heard people actually use that gui as their primary ebook reading software and I just can't fathom how they find that tolerable.
I dislike Calibre main GUI as well. It looks a bit dated and all, but really convenient for downloading books from Userfiction forums like Spacebattles via the Fanficfare plugin. Also good for simple editing of ebook metadata. Built-in ebook-viewer is highly configurable with custom keybindings and themable with custom CSS.This is my setup
> No, it doesn't [need to convert to Python 3]. I am perfectly capable of maintaining python 2 myself. Far less work than migrating the entire calibre codebase.
Reality? You can maintain an absurd, unintuitive, dated UI by yourself fairly easily, but even someone who pushes out two minor updates per week is going to be taking on a little too much in my opinion if they believe they can also support a fork of a major language.
His responses leave an incredibly sour taste in my mouth. Not only is he obviously short sighted with his "Python 2 is not the past and Python 3 is not the future" but all his responses just sound like someone who got up on the wrong side of the bed. Every morning.
You couldn't switch to Python 3 until the middle part of this decade unless the stars aligned with your dependencies, the library support wasn't there. And the first releases of Python 3 were glorified betas, the first "usable" version of Python 3 is often considered to be version 3.3 released in 2012.
But you could easily write Python 3 compatible code so that upgrading once your dependacies were ready was easy. You also could have contributed to your dependancies to help them become Python 3 compatible.
I often hear this, but I just don't see this working in practice. A very similar example I have experience with, I've often worked with people who tried to write code that would be "compatible with Windows" (I primary work in a Linux environment). It makes the code more difficult to read and as soon as you try and run it in Windows it fails spectacularly.
You can write code that's compatible with both Python 2 and 3, but I don't think you can expect to be using Python 2 exclusively and write code that works well in Python 3 unless you're constantly running it with Python 3. Especially, if you don't have much firsthand Python 3 experience. This is much easier if your code is isolated from those dependencies and you write tests that run in both Python 2 and 3, but that's quite a bit more of a commitment than you seem to be implying.
There are a ton of open source projects that have done python 2/3 simultaneously in practice.
I get your point though about not being able to run full integration tests due to dependencies. Still, if you were making a good effort to keep the code Python 3 compliant, it would make upgrading much easier.
Also, my other suggestion- contributing to Python 3 compatibility for modules you depend on! That will give you Python 3 experience :)
Quick: run a shell command, split up the reply by line, run a regex on each line, and save the results to a yaml file— now, does what you wrote work correctly on both Python 2.7 and Python 3.3+? Yeah that's what I thought.
It's still hard to get bilingual Python correct today, and it was considerably harder before 3.3.
import subprocess
import re
import yaml
output = subprocess.check_output(['ls', '-l'], universal_newlines=True)
matching_lines = []
for line in output.splitlines():
if re.search(r'total', line):
matching_lines.append(line)
with open('matching_lines.yaml', 'w') as f:
f.write(yaml.dump(matching_lines))
Nice. Yeah my life got a lot easier when I learned about `universal_newlines` changing the subprocess return type under Python 3.
The much worse case of this that I dealt with a few years ago was reading in and modifying XML files, and then saving the modified XML contents as strings in a yaml file (a keyed cache). The input XML files (which I didn't control) were not consistent about having the encoding marked, and that really made things a mess for ElementTree.
This program would crash with a UnicodeEncodeError upon encountering any file with a non-latin character in its name on Windows with Python 3.5 or earlier.
I know that's not the version of Python you were targeting, and I know you didn't write this program to be cross-platform, but I wrote a similar program and I was repeatedly surprised by those sorts of problems. I don't entirely agree with mikepurvis, but I do feel his pain.
No one is arguing that upgrading would be trivial; the argument is that it's necessary work that could have been started 10 years ago. Also, if you run your unit tests on Python 2 and 3, you'll get pretty good pretty fast at writing 2-3 compatible code.
The comment I was specifically replying to asserted that "you could easily write Python 3 compatible code", and could have been doing so 10 years ago.
This is simply untrue— there are a dozen small stumbling blocks that mean that writing bilingual code has a real cost. And it's not as easy as just having the right CI setup, especially if it's ten years ago and most of your dependencies haven't migrated.
Ah, I see the confusion. In this case, "easily" is relative. The assertion "you could easily write Python 3 compatible code" was in response to the (implied?) assertion "you can't do anything to migrate unless your dependencies are migrated". Perhaps "simply" would have been a better word choice than "easily". To your point, writing 2-3 compatible code has a real cost, but to the GP's point, it's the obvious solution to the "can't do anything until deps are migrated" problem.
> To your point, writing 2-3 compatible code has a real cost, but to the GP's point, it's the obvious solution to the "can't do anything until deps are migrated" problem
Q: Ten years ago how would you have explained to your manager that you needed your team to start writing 2-3 compatible code, which obviously takes considerably more time and effort than sticking with 2 compatible code?
At this point, all the code I write is 2/3 compatible, and the things that annoy me are all py3 only features. Backwards compatibility stuff is pretty easy. Most of the easy problems are easy, and six has solved all the hard ones for like 7 years.
And I've been writing some of my code 2/3 compatible since 3.4.
> But you could easily write Python 3 compatible code so that upgrading once your dependacies were ready was easy.
This sounds a bit like telling someone, oh, you could've easily started driving an EV, so that once charging stations were actually out there, you'd have been ready...
> This sounds a bit like telling someone, oh, you could've easily started driving an EV, so that once charging stations were actually out there, you'd have been ready...
While car analogies are notoriously bad, it's much more like "I'm going to buy a plug-in hybrid and use gasoline until charging stations are widely available". You can write code that works in Python 2 that takes zero effort to run in Python 3. I (and many others) have been doing this for years. Just look at the number of packages on PyPI that run, unmodified, on 2 and 3.
I try to write compatible code myself too, and the only reason it's "zero effort" to run in Python 3 is that I've already invested said effort when making it compatible. Getting strings and paths and stdio to be cross-compatible and working correctly in both 2 and 3 can be quite painful in my experience. Which was kind of my point with the vehicle analogy. It requires a significant investment that you're ignoring, and lack of enough underlying support to justify it during that time.
Driving an EV without charging stations is (nearly) useless. You can still run Python 2-3 code with a Python 2 interpreter with no significant consequence.
EV drivers don't need charging stations at all if they have an open plug at home or at work. The metaphor still sort of works that Python developers had plenty of time where maybe they didn't have enough charging stations on the wide open highway roads of Production to use Python 3 there, but certainly could have been happy to use an EV to commute from home (hobby projects) to the work parking lot (side projects, automation projects, new projects) and back while they waited for those "charging stations" to be built.
I don't want to debate the amount of shoehorning necessary to validate the metaphor. We all understand concretely the issue at hand, the metaphor is only obfuscating at this point.
The thing is, a four year gap is very large from the perspective of "should we drop everything else and prioritize porting our library to python3 right now".
If the deadline wasn't 2020 but 2024, then you wouldn't get more time, simply we'd be at "not yet ready to migrate" state right now, as major libraries would not have switched yet.
If the only way your organization can prioritize this is either “drop everything and update” or “completely ignore and do no work to prepare,” then it sounds like Python versioning is not your biggest problem.
I get this, though: porting code affects all parts of the code base, so it generates git conflicts all the time. It's way more of a pain to do than routine feature development or refactors. It gets difficult to have part of the team work on features while another group works on the port to a new version, because that generates a ton of conflicts. It's easier to get the entire team working on the port.
I had to manage a mature codebase upgrading Ruby versions, and it was painful. And that is fairly mild by comparison; not the big jump from Python 2 to 3.
Unfortunately, I think this type of thing actually is the case for many organizations. Trying to explain the business case for paying off technical debt to a non-technical person can be very difficult when "right now everything works."
I can really understand that… but I believe that such organizations will experience significantly more problems than just Python upgrades. (I work for one.)
Library, as is most open source libraries out there, sure.
For a big company-internal Python codebase that also has to wait for all the major libraries to migrate first, that time frame can quickly shrink to 1-2 years, which is very little.
You're waiting for other people to do stuff, and for many of them 1-2 years is the timeframe when they are likely to start working on it. From that perspective 1-2 years is not "too short", it's the "we intentionally chose this as the optimal time". That doesn't really depend on when the deadline is; if you postpone the deadline, then they postpone the migration - as pretty much every library did back in 2014 when the deadline was postponed by 5(!) years.
Beggars can't be choosers; if your company wants or needs major libraries to have migrated 4-5 years before the deadline, then your company has to participate in making that migration happen. Or accept that it's going to be done later than you'd like, because the needs and motivation of these library maintainers are quite different from the motivation of Python core maintainers.
This is dumb. You're blaming people who probably don't get to call the shots. The managers who do have the ability to make different decisions are, themselves, often under the gun to meet other commitments.
Sure, they will have to cope. One of the ways people cope is by accepting security risks and continuing to run with old versions. Another is to decide that Python isn't for them, and move resources toward better platforms with commercial support for older code.
The fact is, that until a few years ago there were too many libraries that were not compatible for most organizations to make the switch. So, effectively the window for those organizations has only been a few years.
There is also the third choice - you don't migrate and keep doing your job (developing features) on the legacy code base. Things like certificates will have ad-hoc solutions pushed whether you like it or not, and the language will keep trundling along.
If there wind up being major security issues, there will be little choice but for someone to take up the mantle and fix them. Because companies aren't going to switch, and when you've got a botnet ravaging the internet they can't flip a switch and do a big-bang rewrite, someone will have to push out a patch for them.
This isn't the end of Python 2, it's just the Python team washing their hands of it. Ask PHP or Cobol users how their attempts to kill legacy codebases worked out for them.
This whole endeavor was complete folly from the Python foundation to begin with. A big-bang rewrite because they didn't like the syntax, and they didn't even fix the GIL while they were at it.
I'm supposed to go and tell all the various projects that I depend on that they have to change their upgrade timescales to suit mine? Even if I had enough time to help them all meet that kind of timescale, why would they agree to fit their changes into my required deadline?
Maintainers are under a lot of pressure. Having users come along and say "we'll add 50 developers to your project if you agree to ship v3 compliance by next month" is not reducing that pressure, it's adding to it.
More programmers != better results or faster delivery. Your solution is just going to create more problems.
You fork the library. You fix the incompatibility issues. You issue PRs back to the mainline to help them adopt the same fixes. At some point in the future you can switch back to the main repo if they've migrated, or you can consider that library abandoned and continue to maintain your own or find an alternate.
At no point do you have to wait on anyone else, you choose to.
You can do that, but then they won't accept your solution, so you just spend time developing a parallel system that you need to support yourself until the end of time. Been there, done that.
If there is significant demand for solving the problem your fork solves, others will migrate to it. This is made harder when the main solution gets to squeak along for 10 extra years on a version of the language that should no longer get first-party support. Of course there is going to be less demand for that Py3 solution when there is no immediate consequence for continuing on Py2. Killing security update support for 2 will boost the demand for libraries using 3, and make it more likely that your fork gets merged or becomes the defacto standard.
The Python 2->3 window is an example of where being too nice to too many users harms the project. Once Python 2 is dead, people will choose between being the sole maintainer of a Python 3 fork of some Python 2 library versus a being a sole maintainer of their entire Python 2 stack from the language layer on up.
How are they supposed to migrate if I can't openly host my solution because it was developed on company time? Merging back into the original project is one thing, but making your solution available some other way is a whole other story.
This is the reality of having lots of open source dependencies. Important changes/bugs may exist, and the maintainers are under no obligation to fix them to suit your business needs. If you want them fixed, you contribute to the code base (or fork it) and fix it.
Still better than my last programming job where the vendor of a vital piece of HW had provided a C++ API (also vital), but refused to provide dlls that would work in anything newer than VS2010. At least with open source you can do something about it.
You shouldn't take "Do you have programmers?" too literally.
Programmers ~= Resources ~= Money
=> If you want your dependencies to be upgraded in a reasonable time frame, try to contribute to financing the maintainers (or support them in a manner _they_ find suitable).
A lot of libraries refused to migrate before Python 3.5(?) came out which was ~4 years ago. IIRC earlier Python 3 versions had performance regressions compared to Python 2.
> Based on your timeline, a four year gap from "not yet ready to migrate" to "we won't fix security vulnerabilities" is very short.
Is it though? Especially since as late as 2014, everyone was under the assumption that in 2015 security support would end. We were already trying to migrate long before that. The five year extension seems pretty reasonable to me.
That's a pretty low bar. I still think that move by Apple was pretty terrible. Especially given that it's 100% impossible to downgrade iOS (your last point).
Python is an open source project I've used and contributed nothing to, so I don't have the right to be a back seat driver. Were it a commercial project and I was a customer, I would be quite upset.