The are many possibilities to package a Python app, and I have to admit
I am pretty confused.

Here is what I have:

A Python command line app which requires some packages which are not in
the standard library.

I am on Linux and like to have an executable (perhaps a zip file with a
shebang; whatever) which runs on different Linux systems.

Different mean
- perhaps different glibc versions
- perhaps different Python versions

In my specific case this is:
- RedHat 8.4 with Python 3.6.8
- Ubuntu 20.04 LTS with Python 3.8.10
- and finally Fedora 33 with Python 3.9.9

Is this possible to do? If yes which tool would I use for this?

--
Manfred

Hi Manfred,

Manfred Lotz <ml_news@posteo.de> writes:

> The are many possibilities to package a Python app, and I have to admit
> I am pretty confused.
>
> Here is what I have:
>
> A Python command line app which requires some packages which are not in
> the standard library.
>
> I am on Linux and like to have an executable (perhaps a zip file with a
> shebang; whatever) which runs on different Linux systems.
>
> Different mean
> - perhaps different glibc versions
> - perhaps different Python versions
>
> In my specific case this is:
> - RedHat 8.4 with Python 3.6.8
> - Ubuntu 20.04 LTS with Python 3.8.10
> - and finally Fedora 33 with Python 3.9.9
>
>
> Is this possible to do? If yes which tool would I use for this?

I use poetry[1] on CentOS 7 to handle all the dependencies and create a
wheel which I then install to a custom directory with pip3.

You would checkout the repository with your code on the target system,
start a poetry shell using the Python version required, and then build
the wheel. From outside the poetry shell you can set PYTHONUSERBASE and
then install with pip3. You then just need to set PYTHONPATH
appropriately where ever you want to use your software.

Different Python versions shouldn't be a problem. If some module
depends on a specific glibc version, then you might end up in standard
dependency-hell territory, but you can pin module versions of
dependencies in poetry, and you could also possibly use different
branches within your repository to handle that.

HTH

Loris

Footnotes:
[1] https://python-poetry.org

--
This signature is currently under construction.

Hi Loris,

On Wed, 08 Dec 2021 15:38:48 +0100
"Loris Bennett" <loris.bennett@fu-berlin.de> wrote:

> Hi Manfred,
>
> Manfred Lotz <ml_news@posteo.de> writes:
>
> > The are many possibilities to package a Python app, and I have to
> > admit I am pretty confused.
> >
> > Here is what I have:
> >
> > A Python command line app which requires some packages which are
> > not in the standard library.
> >
> > I am on Linux and like to have an executable (perhaps a zip file
> > with a shebang; whatever) which runs on different Linux systems.
> >
> > Different mean
> > - perhaps different glibc versions
> > - perhaps different Python versions
> >
> > In my specific case this is:
> > - RedHat 8.4 with Python 3.6.8
> > - Ubuntu 20.04 LTS with Python 3.8.10
> > - and finally Fedora 33 with Python 3.9.9
> >
> >
> > Is this possible to do? If yes which tool would I use for this?
>
> I use poetry[1] on CentOS 7 to handle all the dependencies and create
> a wheel which I then install to a custom directory with pip3.
>
> You would checkout the repository with your code on the target system,
> start a poetry shell using the Python version required, and then build
> the wheel. From outside the poetry shell you can set PYTHONUSERBASE
> and then install with pip3. You then just need to set PYTHONPATH
> appropriately where ever you want to use your software.
>

In my case it could happen that I do not have access to the target
system but wants to handover the Python app to somebody else. This
person wants just to run it.

> Different Python versions shouldn't be a problem. If some module
> depends on a specific glibc version, then you might end up in standard
> dependency-hell territory, but you can pin module versions of
> dependencies in poetry, and you could also possibly use different
> branches within your repository to handle that.
>

I try to avoid using modules which depeng on specific glibc.

Although, it seems that it doesn't really help for my use case I will
play with poetry to get a better understanding of its capabilities.

--
Thanks a lot,
Manfred

Hi Manfred,

Manfred Lotz <ml_news@posteo.de> writes:

> Hi Loris,
>
> On Wed, 08 Dec 2021 15:38:48 +0100
> "Loris Bennett" <loris.bennett@fu-berlin.de> wrote:
>
>> Hi Manfred,
>>
>> Manfred Lotz <ml_news@posteo.de> writes:
>>
>> > The are many possibilities to package a Python app, and I have to
>> > admit I am pretty confused.
>> >
>> > Here is what I have:
>> >
>> > A Python command line app which requires some packages which are
>> > not in the standard library.
>> >
>> > I am on Linux and like to have an executable (perhaps a zip file
>> > with a shebang; whatever) which runs on different Linux systems.
>> >
>> > Different mean
>> > - perhaps different glibc versions
>> > - perhaps different Python versions
>> >
>> > In my specific case this is:
>> > - RedHat 8.4 with Python 3.6.8
>> > - Ubuntu 20.04 LTS with Python 3.8.10
>> > - and finally Fedora 33 with Python 3.9.9
>> >
>> >
>> > Is this possible to do? If yes which tool would I use for this?
>>
>> I use poetry[1] on CentOS 7 to handle all the dependencies and create
>> a wheel which I then install to a custom directory with pip3.
>>
>> You would checkout the repository with your code on the target system,
>> start a poetry shell using the Python version required, and then build
>> the wheel. From outside the poetry shell you can set PYTHONUSERBASE
>> and then install with pip3. You then just need to set PYTHONPATH
>> appropriately where ever you want to use your software.
>>
>
> In my case it could happen that I do not have access to the target
> system but wants to handover the Python app to somebody else. This
> person wants just to run it.

For what ever reasons, there does not seem to be much focus on this kind
of deployment for Python. Similar to the way things are with Perl, the
assumption seems to be that you have a working environment and install
any dependencies needed to get the program you have been given working.

I have never used it, but you might want to look at something like
pyinstaller

https://pyinstaller.readthedocs.io

However, it looks as if it is aimed towards bundling a single script. I
don't know how it would work if you have a more complex program
consisting of a number of modules.

>> Different Python versions shouldn't be a problem. If some module
>> depends on a specific glibc version, then you might end up in standard
>> dependency-hell territory, but you can pin module versions of
>> dependencies in poetry, and you could also possibly use different
>> branches within your repository to handle that.
>>
>
> I try to avoid using modules which depeng on specific glibc.

So would I, but you mentioned it above in your definition of
'different'.

> Although, it seems that it doesn't really help for my use case I will
> play with poetry to get a better understanding of its capabilities.

You're right, poetry doesn't seem to address your main problem.
Nevertheless, it might be useful for developing your program before you
get to the question of how to distribute it

Cheers,

Loris

--
This signature is currently under construction.

Manfred Lotz wrote at 2021-12-8 12:31 +0100:
>The are many possibilities to package a Python app, and I have to admit
>I am pretty confused.
>
>Here is what I have:
>
>A Python command line app which requires some packages which are not in
>the standard library.

Are they on PyPI or can they be downloaded to PyPI?
In this case, you could install it via `pip` and
a "requirements" file (describing what is necessary).

You could also build an archive (e.g. `tar`, `zip`)
and create small installer script which unpacks and installs as necessary.
You could put the archive at the end of the script (such
that you have a single file which gets executed to do everything).

> On 8 Dec 2021, at 18:27, Manfred Lotz <ml_news@posteo.de> wrote:
>
> The are many possibilities to package a Python app, and I have to admit
> I am pretty confused.
>
> Here is what I have:
>
> A Python command line app which requires some packages which are not in
> the standard library.
>
> I am on Linux and like to have an executable (perhaps a zip file with a
> shebang; whatever) which runs on different Linux systems.
>
> Different mean
> - perhaps different glibc versions
> - perhaps different Python versions
>
> In my specific case this is:
> - RedHat 8.4 with Python 3.6.8
> - Ubuntu 20.04 LTS with Python 3.8.10
> - and finally Fedora 33 with Python 3.9.9
>
>
> Is this possible to do? If yes which tool would I use for this?

Have a look at pyinstaller it knows how to collect up all your dependancies and build a folder of files you can distribute. It’s also got a single file mode that you might find useful. The end user does not need to install python.

I am looking at using it to build gui apps a macOs.

You will have to experiment to find out if it solves you glib case concerns.

Barry
>
>
> --
> Manfred
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

On Thu, 9 Dec 2021 17:34:03 +0000
Barry <barry@barrys-emacs.org> wrote:

> > On 8 Dec 2021, at 18:27, Manfred Lotz <ml_news@posteo.de> wrote:
> >
> > The are many possibilities to package a Python app, and I have to
> > admit I am pretty confused.
> >
> > Here is what I have:
> >
> > A Python command line app which requires some packages which are
> > not in the standard library.
> >
> > I am on Linux and like to have an executable (perhaps a zip file
> > with a shebang; whatever) which runs on different Linux systems.
> >
> > Different mean
> > - perhaps different glibc versions
> > - perhaps different Python versions
> >
> > In my specific case this is:
> > - RedHat 8.4 with Python 3.6.8
> > - Ubuntu 20.04 LTS with Python 3.8.10
> > - and finally Fedora 33 with Python 3.9.9
> >
> >
> > Is this possible to do? If yes which tool would I use for this?
>
> Have a look at pyinstaller it knows how to collect up all your
> dependancies and build a folder of files you can distribute. It’s
> also got a single file mode that you might find useful. The end user
> does not need to install python.
>
> I am looking at using it to build gui apps a macOs.
>
> You will have to experiment to find out if it solves you glib case
> concerns.
>

I played with pyinstaller which worked fine. However, it builds a
dynamic executable and thus it is glibc version dependent. Means, I
have to build different executables for differen glibc versions.

So, it seems I will have to check how executable zip archives are
supposed to work.

--
Manfred

On Thu, 9 Dec 2021 18:18:26 +0100
"Dieter Maurer" <dieter@handshake.de> wrote:

> Manfred Lotz wrote at 2021-12-8 12:31 +0100:
> >The are many possibilities to package a Python app, and I have to
> >admit I am pretty confused.
> >
> >Here is what I have:
> >
> >A Python command line app which requires some packages which are not
> >in the standard library.
>
> Are they on PyPI or can they be downloaded to PyPI?
> In this case, you could install it via `pip` and
> a "requirements" file (describing what is necessary).
>

Acutally, those packages are on pypi.

I created a minimal example where I have two files

hello.py
========

#!/usr/bin/python3

from message import hello
import typer

def main():
hello()

if __name__ == "__main__":
typer.run(main)

message.py
==========

def hello():
print("hello world")

pyinstaller worked fine taking care of message.py and typer module. But
as said in my other reply it is glibc version dependent.

> You could also build an archive (e.g. `tar`, `zip`)
> and create small installer script which unpacks and installs as
> necessary. You could put the archive at the end of the script (such
> that you have a single file which gets executed to do everything).

This I will try next.

--
Manfred

On 2021-12-09 19:35:37 +0100, Manfred Lotz wrote:
> I played with pyinstaller which worked fine. However, it builds a
> dynamic executable and thus it is glibc version dependent. Means, I
> have to build different executables for differen glibc versions.

Just build it for the oldest version. Even if you don't, it may not
matter. I just compiled a hello world program on Ubuntu 20 and ran it on
Debian 6 (which is 10 years old). Obviusly a real program has more
opportunities to run into compatibility problems, but glibc doesn't
change all that much.

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

On 12/9/21 11:35, Manfred Lotz wrote:

> I played with pyinstaller which worked fine. However, it builds a
> dynamic executable and thus it is glibc version dependent. Means, I
> have to build different executables for differen glibc versions.
>
> So, it seems I will have to check how executable zip archives are
> supposed to work.
>

For me at least, I'm still not sure what you are trying to accomplish.

"A Python command line app which requires some packages which are not in
the standard library."

If your code and its dependencies are "pure python" then you only need
to build a wheel package and you should be good to go. If you think you
have Python version issues, most of the good checkers will take a
version argument, so if you think you need to be compatible with
3.6-3.10 you give it a version of 3.6 to check against, and it should
spot things that don't work all the way back to that base version, and
then you can find ways to code around it.

If the things you call on are binary, it gets a little more complicated.
Wheels that contain compiled bits need to match the version of Python
that's going to run them (unless they use the stable ABI, but that's not
terribly common yet).

You shouldn't run into glibc versioning problems. Most of glibc has
been extraordinarily stable for nearly two decades, and within the range
of distributions you've mentioned there should not be problems unless
something is reaching into very esoteric areas. Other system libraries
- maybe not so much. If you really think you're going to have this
level of binary-compatibility problem, the flavor-of-the-month technique
is to build a self-contained bundle, such as a Snap or Flatpak, or...

Quote:

> New packaging formats like Snap, Flatpak and AppImage are providing
distribution agnostic packages that work on most Linux distributions.

But that's no longer a Python-specific question at that point...

As I said, I can't really deduce the details of what you're trying to
accomplish, just hoping you don't buy yourself more trouble than you
actually need to - this might be easier than you think.

On Thu, 9 Dec 2021 21:23:58 +0100
"Peter J. Holzer" <hjp-python@hjp.at> wrote:

> On 2021-12-09 19:35:37 +0100, Manfred Lotz wrote:
> > I played with pyinstaller which worked fine. However, it builds a
> > dynamic executable and thus it is glibc version dependent. Means, I
> > have to build different executables for differen glibc versions.
>
> Just build it for the oldest version.

Wasn't aware of that.

> Even if you don't, it may not
> matter. I just compiled a hello world program on Ubuntu 20 and ran it
> on Debian 6 (which is 10 years old). Obviusly a real program has more
> opportunities to run into compatibility problems, but glibc doesn't
> change all that much.
>
> hp
>

I have build an executable with pyinstaller on Ubuntu 20.04 and it
didn't run on Redhat 8.4.

Doing it the other way round helps indeed, i.e. building on Redhat 8.4
and then it runs on Ubuntu 20.04.

Thank you.

--
Manfred

On Thu, 9 Dec 2021 13:51:00 -0700
Mats Wichmann <mats@wichmann.us> wrote:

> On 12/9/21 11:35, Manfred Lotz wrote:
>
> > I played with pyinstaller which worked fine. However, it builds a
> > dynamic executable and thus it is glibc version dependent. Means, I
> > have to build different executables for differen glibc versions.
> >
> > So, it seems I will have to check how executable zip archives are
> > supposed to work.
> >
>
> For me at least, I'm still not sure what you are trying to accomplish.
>

I like to offer my command line app to some people who are not really
command line geeks. Means, I don't want to have to tell them to install
packages via pip and stuff like that. Simply take a file, make it
executable and run it.

--
Manfred

Manfred Lotz <ml_news@posteo.de> writes:

> pyinstaller worked fine taking care of message.py and typer module. But
> as said in my other reply it is glibc version dependent.

Perhaps the included freeze.py script (included in the CPython source
that is, in Tools/freeze) is worth considering as well. Although it also
seems to create a dynamic executable by default (I tried with your hello
example), it seems to me it's possible to edit the generated makefile
and replace -shared with -static in a few places. Didn't try that
though.