So I’m no expert, but I have been a hobbyist C and Rust dev for a while now, and I’ve installed tons of programs from GitHub and whatnot that required manual compilation or other hoops to jump through, but I am constantly befuddled installing python apps. They seem to always need a very specific (often outdated) version of python, require a bunch of venv nonsense, googling gives tons of outdated info that no longer works, and generally seem incredibly not portable. As someone who doesn’t work in python, it seems more obtuse than any other language’s ecosystem. Why is it like this?
Python is the only programming language that has forced me to question what the difference is between an egg and a wheel.
No, it’s not just you, Python’s tooling is a mess. It’s not necessarily anyone’s fault, but there are a ton of options and a lot of very similarly named things that accomplish different (but sometimes similar) tasks. (pyenv, venv, and virtualenv come to mind.) As someone who considers themselves between beginner and intermediate proficiency in Python, this is my biggest hurdle right now.
Python’s tooling is a mess.
Not only that. It’s a historic mess. Over the years, growing a better and better toolset left a lot of projects in a very messy state. So many answers on Stack Overflow that mention
easy_install- I still don’t know what it is, but I guess it was some kind of protouv.Every time I’m doing anything with Python I ask myself if Java’s tooling is this complicated or I’m just used to it by now. I think a big part of the weirdness is that a lot of Python tooling is tied to the Python installation whereas in Java things like Maven and Gradle are separate. In addition, I think dependencies you install get tied to that Python installation, while in Java they just are in a cache for Maven/Gradle. And in the horrible scenario where you need to use different versions of Maven/Gradle (one place I was at specifically needed Maven 3.0.3 for one project and a different for a different, don’t ask, it’s dumb and their own fault for setting it up that way) at least they still have one common cache for everything.
I guess it also helps that with Java you (often) don’t need platform specific jar files. But Python is often used as an easy and dynamic scripting interface over more performant, native code. So you don’t really run into things like “this artifact doesn’t have a 64 bit arm version for python 2” often with Java. But that’s not a fault of Python’s tooling, it’s just the reality of how it’s used.
You re not stupid, python’s packaging & versionning is PITA. as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem
as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem
A perfect summary of the history of computer code!
Python’s packaging is not great. Pip and venvs help but, it’s lightyears behind anything you’re used to. My go-to is using a venv for everything.
Yes it’s terrible. The only hope on the horizon is
uv. It’s significantly better than all the other tooling (Poetry, pip, pipenv, etc.) so I think it has a good chance of reducing the options to just Pip oruvat least.But I fully expect the Python Devs to ignore it, and maybe even make life deliberately difficult for it like they did for static analysers. They have some strange priorities sometimes.
uv is good but it needs a little more time in the oven.
For the moment I would definitely recommend poetry if you are not a library developer. Poetry’s biggest sin is it’s atrocious performance but it has most of the features you need to work with Python apps today.
Why do you say it needs more time in the oven? I’ve had zero issues with it as a drop-in replacement for Pip in a large commercial project, which is an extremely impressive achievement. (And it was 10x faster.)
I tried Poetry once and it failed to resolve dependencies on the first thing I tried it on. If anything Poetry needs more time in the oven. It also wasn’t 10x faster.
No it’s not. E.g. nobody who starts a new project uses setup.py anymore
Are you sure? I’m not very active in that ecosystem, but if that was prevalent in the past, surely there’s still tutorials and stuff out there that people would follow and create such projects even today?
More than that, it seems to me that the official python docs for packaging [still] talks about setup.py. Why would people not use that?
Sure, there was some hyperbole. Some people need some specific setuptools plugin or something. Almost nobody.
when the official docs are telling you to use it, then it’s used. You can have no expectation of people to think the tooling isn’t shit when it’s literally the official recommendation.
It doesn’t. read the first words behind the link you posted:
Page Status: Outdated
Here is the actual one: https://packaging.python.org/en/latest/tutorials/packaging-projects/
OP seems to be trying to install older projects, rather than creating a new project.
Yeah the tooling sucks. The only tooling I’ve liked is Poetry, I never have trouble installing or packaging the apps that use it.
I’m no Python expert either and yeah, from an outsider’s perspective it seems needlessly confusing.
easy_installthat’s never been easy,pipthat should absolutely be put on a Performance Improvement Plan, and now thisvenvnonsense.You can criticize javascript’s ridiculous dependencies all you want (left-pad?), but one thing that they absolutely got right is how to manage them. Everything’s in
node_modulesand that’s it. Yeah, you might get eleven copies of left-pad on your system, but you know what you NEVER get? Version conflicts between projects you’re working on.It’s something of a “14 competing standards” situation, but uv seems to be the nerd favourite these days.
I still do the python3 -m venv venv && source venv/bin/activate
How can uv help me be a better person?
- let
pyproject.tomltrack the dependencies and dev-dependencies you actually care about
- dependencies are what you need to run your application
- dev-dependencies are not necessary to run your app, but to develop it (formatting, linting, utilities, etc)
- it can track exactly what’s needed ot run the application via the
uv.lockfile that contains each and every lib that’s needed. - uv will install the needed Python version for you, completely separate from what your system is running.
uv syncanduv run <application>is pretty much all you need to get going- it’s blazingly fast in everything
Thank you for explaining so clearly. Point 3 is indeed something I’ve ran into before!
- let
And pip install -r requirements.txt
Fuck it, I just use sudo and live with the consequences.
You’ll see when you start your second project why this doesn’t work.
This! Haven’t used that one personally, but seeing how good ruff is I bet it’s darn amazing, next best thing that I used has been PDM and Poetry, because Python’s first party tooling has always been lackluster, no cohesive way to define a project and actually work it until relatively recently
I moved all our projects (and devs) from poetry to uv. Reasons were poetry’s non standard pyproject.toml syntax and speed, plus some weird quirks, e. g. if poetry asks for input and is not run with the verbose flag, devs often don’t notice and believe it is stuck (even though it’s in the default project README).
Personally, I update uv on my local machine as soon as a new release is available so I can track any breaking changes. Couple of months in, I can say there were some hiccups in the beginning, but currently, it’s smooth sailing, and the speed gain really affects productivity as well, mostly due to being able to not break away from a mental “flow” state while staring at updates, becoming suspicious something might be wrong. Don’t get me wrong, apart from the custom syntax (poetry partially predates the pyproject PEP), poetry worked great for us for years, but uv feels nicer.
Recently, “uv build” was introduced, which simplified things. I wish there was an command to update the lock file while also updating the dependency specs in the project file. I ran some command today and by accident discovered that custom dependency groups (apart from e. g. “dev”) have made it to uv, too.
“uv pip” does some things differently, in particular when resolving packages (it’s possible to switch to pip’s behavior now), but I do agree with the decisions, in particular the changes to prevent “dependency confusion” attacks.
As for the original question: Python really has a bit of a history of project management and build tools, I do feel however that the community and maintainers are finally getting somewhere.
cargo is a bit of an “unfair” comparison since its development happened much more aligned with Rust and its whole ecosystem and not as an afterthought by third party developers, but I agree: cargo is definitely a great benchmark how project and dependency management plus building should look like, along with rustup, it really makes the developer experience quite pleasant.
The need for virtual environments exists so that different projects can use different versions of dependencies and those dependencies can be installed in a project specific location vs a global, system location. Since Python is interpreted, these dependencies need to stick around for the lifetime of the program so they can be imported at runtime. poetry managed those in a separate folder in e. g. the user’s cache directory, whereas uv for example stores the virtual environment in the project folder, which I strongly prefer.
cargo will download the matching dependencies (along with doing some caching) and link the correct version to the project, so a conceptual virtual environment doesn’t need to exist for Rust. By default, rust links everything apart from the C runtime statically, so the dependencies are no longer neesed after the build - except you probably want to rebuild the project later, so there is some caching.
Finally, I’d also recommend to go and try setting up a project using astral’s uv. It handles sane pyproject.toml files, will create/initialize new projects from a template, manages virtual environments and has CLI to build e. g. wheels or source distribution (you will need to specify which build backend to use. I use hatchling), but thats just a decision you make and express as one line in the project file. Note: hatchling is the build backend, hatch is pypa’s project management, pretty much an alternative to poetry or uv.
uv will also install complete Python distributions (e. g. Python 3.12) if you need a different interpreter version for compatibility reasons
If you use workspaces in cargo, uv also does those.
uv init, uv add, uv lock --upgrade, uv sync, uv build and how uv handles tools you might want to install and run should really go a long way and probably provide an experience somewhat similar to cargo.
I think you responded to the wrong comment, I didn’t question the need for uv or other tools like that
I did that on purpose, i. e. I wanted to confirm your thoughts about uv, drifted off into a general rant, remembered OP’s original question and later realized it would have been better framed as a top level comment. In my defense, I was in an altered state of mind at the time.
venv nonsense
I mean, the fact that it isn’t more end-user invisible to me is annoying, and I wish that it could also include a version of Python, but I think that venv is pretty reasonable. It handles non-systemwide library versioning in what I’d call a reasonably straightforward way. Once you know how to do it, works the same way for each Python program.
Honestly, if there were just a frontend on venv that set up any missing environment and activated the venv, I’d be fine with it.
And I don’t do much Python development, so this isn’t from a “Python awesome” standpoint.
pyenv and uv let you install and switch between multiple Python versions.
As for uv, those come from the Python build standalone project, if I remember correctly, pyenv also installs from there, but don’t quote me on that.
I’m not sure this can be really fixed with Python 3, maybe we just have to hope for Python 4
It’s fixed, and the python version had nothing to do with it. Just use hatch
Ah yes, the 15th standard we’ve been waiting for!
It’s not a standard, it’s built on standards.
You can also use Poetry (which recently grew standard metadata support) or plain
uv venvif you want to do things manually but fast.Just use this one… or any of this 4 others.
This is the issue for us, python outsiders. Each time we try we get a different answer with new tools. We are outside of the comtunity, we don’t know the trend, old and new, pro and cons.
Your first recommandation is hatch… first time I’ve heard of it. Uv seems trendy in this thread, but before that it was unknown to me too.
As I understands it, it should be pip’s job. When it detect I’m in a project it install packages in it and python use them. It can use any tool under the hood, but the default package manager shoud be able to do it on its own.
Uv and pip do the same thing, uv is just faster.
Hatch has the same role as Poetry or tox: managing environments for you.
Applications should be packaged properly, in a self contained installer for exactly this demographic. It’s not Python’s fault that this isn’t common practice.
The difficulty with python tooling is that you have to learn which tools you can and should completely ignore.
Unless you are a 100x engineer managing 500 projects with conflicting versions, build systems, docker, websites, and AAAH…
- you don’t really need venvs
- you should not use more than on package manager (I recommend pip) and you should cling to it with all your might and never switch. Mixing e.g. conda, on linux system installers like apt, is the problem. Just using one is fine.
- You don’t “need” need any other tools. They are bonuses that you should use and learn how to use, exactly when you need them and not before. (type hinting checker, linting, testing, etc…)
Why is it like this?
Isolation for reliability, because it costs the businesses real $$$ when stuff goes down.
venvs exists to prevent the case that “project 1” and “project 2” use the same library “foobar”. Except, “project 1” is old, the maintainer is held up and can’t update as fast and “project 2” is a cutting edge start up that always uses the newest tech.
When python imports a library it would use “the libary” that is installed. If project 2 uses foobar version 15.9 which changed functionality, and project 1 uses foobar uses version 1.0, you get a bug, always, in either project 1 or project 2. Venvs solve this by providing project specific sets of libraries and interpreters.
In practice for many if not most users, this is meaningless, because if you’re making e.g. a plot with matplotlib, that won’t change. But people have “best practices” so they just do stuff even if they don’t need it.
It is a tradeoff between being fine with breakage and fixing it when it occurs and not being fine with breakage. The two approaches won’t mix.
very specific (often outdated) version of python,
They are giving you the version that they know worked. Often you can just remove the specific version pinning and it will work fine, because again, it doesn’t actually change that much. But still, the project that’s online was the working state.
Coming at this from the JS world… Why the heck would 2 projects share the same library? Seems like a pretty stupid idea that opens you up to a ton of issues, so what, you can save 200kb on you hard drive?
everyone focuses on the tooling, not many are focusing on the reason: python is extremely dynamic. like, magic dynamic you can modify a module halfway through an import, you can replace class attributes and automatically propagate to instances, you can decompile the bytecode while it’s running.
combine this with the fact that it’s installed by default and used basically everywhere and you get an environment that needs to be carefully managed for the sake of the system.
js has this packaging system down pat, but it has the advantage that it got mainstream in a sandboxed isolated environment before it started leaking out into the system. python was in there from the beginning, and every change breaks someone’s workflow.
the closest language to look at for packaging is probably lua, which has similar issues. however since lua is usually not a standalone application platform it’s not a big deal there.
and yet that all works fine in Ruby, which came out around the same time as Python and yet has had Bundler for 15 years now.
Python - 15+ package managers and build tools Ruby - 1
the closest language to look at for packaging is probably lua, which has similar issues. however since lua is usually not a standalone application platform it’s not a big deal there.
no the closest language is literally Ruby, it’s almost the exact same language, except the tooling isn’t insane and it came out only a few years after python.
good point, ruby is a good comparison. although, ruby is very different under the hood. it’s magically dynamic in a completely different way, and it also never really got the penetration on the system level that python did.
none of this is to take away from the fact that python packaging is bad. i know how to work it because i’ve been programming in python for 14 years, but trying to teach people makes the problem obvious. and yet.
With all the hype surrounding Python it’s easy to forget that it’s a really old language. And, in my opinion, the leadership is a bit of a mess so there hasn’t been any concerted effort on standardizing tooling.
Some unsolicited advice from somebody who is used more refined build environments but is doing a lot of Python these days:
The whole
venvthing isn’t too bad once you get the hang of it. But be prepared for people to tell you that you’re using the wrong venv for reasons you’ll never quit understand or likely need to care about. Just use the bundled “python -m venv venv” and you’ll be fine despite other “better” alternatives. It’s bundled so it’s always available to you. And feel free to just drop/recreate your venv whenever you like or need. They’re ephemeral and pretty large once you’ve installed a lot of things.Use “pipx” to install python applications you want to use as programs rather than libraries. It creates and manages venvs for them so you don’t get library conflicts. Something like “pip-tools” for example (pipx install pip-tools).
Use “pyenv” to manage installed python versions - it’s a bit like “sdkman” for the JVM ecosystem and makes it easy to deal with the “specific versions of python” stuff.
For dependencies for an app - I just create a requirements.txt and “pip install -r requirements.txt” for the most part… Though I should use one of the 80 better ways to do it because they can help with updating versions automatically. Those tools mostly also just spit out a requirements.txt in the end so it’s pretty easy to migrate to them. pip-tools is what my team is moving towards and it seems a reasonable option. YMMV.
This.
venv
pip-toolsSpecify your primary dependencies in pyproject.toml and use pip-compile to keep stuff locked in requirements.txt to exact versions (or even hashes).
Though after working with cargo a bit, I would love to have all of this in a first-class program, hope uv can get there.
Tried to install Automatic1111 for Stable Diffusion in an Arch distrobox, and despite editing the .sh file to point to the older tarballed Python version as advised on Github, it still tells me it uses the most up to date one that’s installed system wide and thus can’t install pytorch. And that’s pretty much where my personal knowledge ends, and apparently that of those (i.e. that one person) on Github. ¯\_(ツ)_/¯
Always funny when people urge you to ask for help but no one ends up actually helping.
Lol this is exactly why I made this post. I ended up using ComfyUI instead which has other, different python issues, but I got it working (kinda, no GPU but it’s fine it works)
