SVI/ProxmoxVE
1
0
Fork 0
mirror of https://github.com/community-scripts/ProxmoxVE.git synced 2026-03-31 06:34:17 -04:00
Code Issues 2.6k Packages Projects Releases 10 Wiki Activity

Use virtual python environments in scripts/use apt where applicable #864

New Issue
Closed
opened 2025-11-20 04:55:23 -05:00 by saavagebueno · 6 comments
saavagebueno commented 2025-11-20 04:55:23 -05:00
Owner
Copy Link

Originally created by @LaurenceBarnes on GitHub (Apr 6, 2025).

Originally assigned to: @MickLesk on GitHub.

🌟 Briefly describe the feature

Use pipx or pyenv to install packages that interfere with the system-wide installation, or check the software's requirements.txt whether it can be installed through apt.

📝 Detailed description

As discussed in https://github.com/community-scripts/ProxmoxVE/issues/3654, it is not recommended to modify the python installation using pip install because it is managed by the system's package manager. Currently, the scripts such as https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/misc/update-lxcs.sh or https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/install/sabnzbd-install.sh use rm -rf /usr/lib/python3.*/EXTERNALLY-MANAGED to circumvent this, but that is not the method intended by Linux or Python.

As soon as this is given attention, please drop me a ping - I'd happily look into affected scripts and check what commands need to be run to do it properly, that should make it easier for you to utilize those steps inside of the scripts.

💡 Why is this useful?

On this, it is best to quote this from stackoverflow:

The proper way to install Python libraries and applications is to install them in a Python virtual environment whenever possible (the exceptions to this rule are quite rare).

The error message describes two common ways to accomplish this: either by creating a virtual environment yourself, or for applications, by using pipx—a tool which will create a virtual environment for you and install the application in that virtual environment.

&

If you have considered your options carefully and are still sure that you want to install packages "system-wide" and risk breaking your system (for example, by overwriting libraries that were part of tools written in Python that came with your system), Pip needs to be given permission to do so.

&

Theoretically, removing or renaming the "marker" file (/usr/lib/python3.x/EXTERNALLY-MANAGED) would also disable the block, but this is a bad idea. The file was put there for a reason, and it's at least as easy to use the intended mechanisms instead.

This also gives a bit more context:

Long story: Your distribution is trying to protect you against mixing apt provided packages and pip provided packages. Mixing two package managers (apt and pip here) is always a bad idea and the source of many issues.

It also says

Using apt install python3-xxx. It does not cleanly apply for you as you're having a requirements.txt, not a single dependency. It would work if you have only a few requirements in the file and can do it manually for each, like apt install python3-xxx python3-yyy python3-zzz. In this case there's no weird mixing of package managers: you installed python3 using apt, you're installing your dependencies using apt: no surprises.

so in the case of SABnzbd, the requirements could also be installed through apt if applicable https://github.com/sabnzbd/sabnzbd/blob/develop/builder/requirements.txt. This would need to be checked on a case-by-case basis.

Originally created by @LaurenceBarnes on GitHub (Apr 6, 2025). Originally assigned to: @MickLesk on GitHub. ### 🌟 Briefly describe the feature Use pipx or pyenv to install packages that interfere with the system-wide installation, or check the software's `requirements.txt` whether it can be installed through apt. ### 📝 Detailed description As discussed in https://github.com/community-scripts/ProxmoxVE/issues/3654, it is not recommended to modify the python installation using `pip install` because it is managed by the system's package manager. Currently, the scripts such as https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/misc/update-lxcs.sh or https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/install/sabnzbd-install.sh use `rm -rf /usr/lib/python3.*/EXTERNALLY-MANAGED` to circumvent this, but that is not the method intended by Linux or Python. As soon as this is given attention, please drop me a ping - I'd happily look into affected scripts and check what commands need to be run to do it properly, that should make it easier for you to utilize those steps inside of the scripts. ### 💡 Why is this useful? On this, it is best to quote [this from stackoverflow](https://stackoverflow.com/questions/75608323/how-do-i-solve-error-externally-managed-environment-every-time-i-use-pip-3/75722775#75722775): > The proper way to install Python libraries and applications is to install them in a Python virtual environment whenever possible (the exceptions to this rule are quite rare). > > The error message describes two common ways to accomplish this: either by creating a virtual environment yourself, or for applications, by using [pipx](https://pipx.pypa.io/)—a tool which will create a virtual environment for you and install the application in that virtual environment. & > If you have considered your options carefully and are still sure that you want to install packages "system-wide" and **risk breaking your system** (for example, by overwriting libraries that were part of tools written in Python that came with your system), Pip needs to be given permission to do so. & > Theoretically, removing or renaming the "marker" file (`/usr/lib/python3.x/EXTERNALLY-MANAGED`) would also disable the block, but this is a bad idea. The file was put there for a reason, and it's at least as easy to use the intended mechanisms instead. [This](https://stackoverflow.com/questions/75602063/pip-install-r-requirements-txt-is-failing-this-environment-is-externally-mana/75696359#75696359) also gives a bit more context: > Long story: Your distribution is trying to protect you against mixing `apt` provided packages and `pip` provided packages. Mixing two package managers (`apt` and `pip` here) is always a bad idea and the source of many issues. It also says > Using `apt install python3-xxx.` It does not cleanly apply for you as you're having a `requirements.txt`, not a single dependency. It would work if you have only a few requirements in the file and can do it manually for each, like `apt install python3-xxx python3-yyy python3-zzz`. In this case there's no weird mixing of package managers: you installed `python3` using `apt`, you're installing your dependencies using `apt`: no surprises. so in the case of SABnzbd, the requirements could also be installed through `apt` if applicable https://github.com/sabnzbd/sabnzbd/blob/develop/builder/requirements.txt. This would need to be checked on a case-by-case basis.
saavagebueno added the enhancement label 2025-11-20 04:55:23 -05:00
saavagebueno commented 2025-11-20 04:55:24 -05:00
Author
Owner
Copy Link

@michelroegl-brunner commented on GitHub (Apr 10, 2025):

@LaurenceBarnes As stated by Mick in the last issue you opened, and you offering to look into this i kindly ask you to do so. Please prepare a PR with the changes that would be needed to the affectet scripts so we as Maintainer can look at it.
Currently we focus on other things in the codebase and this is a really low priority, as it works for most people the way we do it.

If you prepare the changes with a PR we can check/test it acordingly.
Thank you.

@michelroegl-brunner commented on GitHub (Apr 10, 2025): @LaurenceBarnes As stated by Mick in the last issue you opened, and you offering to look into this i kindly ask you to do so. Please prepare a PR with the changes that would be needed to the affectet scripts so we as Maintainer can look at it. Currently we focus on other things in the codebase and this is a really low priority, as it works for most people the way we do it. If you prepare the changes with a PR we can check/test it acordingly. Thank you.
saavagebueno commented 2025-11-20 04:55:24 -05:00
Author
Owner
Copy Link

@LaurenceBarnes commented on GitHub (Apr 10, 2025):

@LaurenceBarnes As stated by Mick in the last issue you opened, and you offering to look into this i kindly ask you to do so. Please prepare a PR with the changes that would be needed to the affectet scripts so we as Maintainer can look at it.
Currently we focus on other things in the codebase and this is a really low priority, as it works for most people the way we do it.

If you prepare the changes with a PR we can check/test it acordingly.
Thank you.

Hi, thanks for the reply. It was also mentioned that this was rather a feature request, so I raised this as one in order to track this.

If you don't mind it, I'd be happy if you leave it open until I'll get around to providing input.

@LaurenceBarnes commented on GitHub (Apr 10, 2025): > @LaurenceBarnes As stated by Mick in the last issue you opened, and you offering to look into this i kindly ask you to do so. Please prepare a PR with the changes that would be needed to the affectet scripts so we as Maintainer can look at it. > Currently we focus on other things in the codebase and this is a really low priority, as it works for most people the way we do it. > > If you prepare the changes with a PR we can check/test it acordingly. > Thank you. Hi, thanks for the reply. It was also mentioned that this was rather a feature request, so I raised this as one in order to track this. If you don't mind it, I'd be happy if you leave it open until I'll get around to providing input.
saavagebueno commented 2025-11-20 04:55:24 -05:00
Author
Owner
Copy Link

@dreulavelle commented on GitHub (Apr 14, 2025):

Better off switching to uv honestly.

@dreulavelle commented on GitHub (Apr 14, 2025): Better off switching to `uv` honestly.
saavagebueno commented 2025-11-20 04:55:25 -05:00
Author
Owner
Copy Link

@logictoad commented on GitHub (May 23, 2025):

Better off switching to uv honestly.

Agreed - I added this in place of the upgrade script as a simple brute force method to upgrade in the container

  rm pyproject.toml
  uv init
  uv venv
  uv pip install certbot-dns-cloudflare
  #$STD python3 -m pip install --no-cache-dir certbot-dns-cloudflare
@logictoad commented on GitHub (May 23, 2025): > Better off switching to `uv` honestly. Agreed - I added this in place of the upgrade script as a simple brute force method to upgrade in the container ``` rm pyproject.toml uv init uv venv uv pip install certbot-dns-cloudflare #$STD python3 -m pip install --no-cache-dir certbot-dns-cloudflare ```
saavagebueno commented 2025-11-20 04:55:25 -05:00
Author
Owner
Copy Link

@MickLesk commented on GitHub (May 24, 2025):

Its WIP to switch all to uv

@MickLesk commented on GitHub (May 24, 2025): Its WIP to switch all to uv
saavagebueno commented 2025-11-20 04:55:25 -05:00
Author
Owner
Copy Link

@logictoad commented on GitHub (May 24, 2025):

Its WIP to switch all to uv

Yeah all good just thought I'd post a solution for the interim

@logictoad commented on GitHub (May 24, 2025): > Its WIP to switch all to uv Yeah all good just thought I'd post a solution for the interim
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
🛑 Failure to comply with the guidelines
breaking change
bug
bug
bugfix
deferred
delete script
dependencies
enhancement
external
feature
github
help wanted
Implemented in VED waiting push to Main
in project pipeline
invalid
investigation
json
maintenance
new script
new script
nice to have
not a script issue
not planned
organization
pull-request
Mirrored from GitHub Pull Request
 question
refactor
rename script
security
update script
website
wontdo
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/ProxmoxVE#864
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?