Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

Sex, Drugs & Linux Rules -- MaDsen Wikholm, mwikholm@at8.abo.fi

devel / comp.lang.python / Re: Call julia from Python: which package?

SubjectAuthor
o Re: Call julia from Python: which package?Oscar Benjamin

1
Re: Call julia from Python: which package?

<mailman.17.1639782095.1567.python-list@python.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=16439&group=comp.lang.python#16439

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: oscar.j....@gmail.com (Oscar Benjamin)
Newsgroups: comp.lang.python
Subject: Re: Call julia from Python: which package?
Date: Fri, 17 Dec 2021 23:01:21 +0000
Lines: 55
Message-ID: <mailman.17.1639782095.1567.python-list@python.org>
References: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
<CAHVvXxScU6ONFGq66Uy37Bk0k+HOnKiKF51y_9ZUbT0S+-N6yA@mail.gmail.com>
<CAPTjJmormmRVfRj-HWYcT1yLvGifA=MV7KjO_c2HG5NauufA0g@mail.gmail.com>
<CAHVvXxRQmwaPdNgXPBb31jn48k7gHP3LT-c-eyYsV42WYxfybg@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
X-Trace: news.uni-berlin.de bLeGu+PKGNVcf0G3Vub5wg+4vBXht8Bb0MrYIR3Z/tJg==
Return-Path: <oscar.j.benjamin@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=PxVUnW46;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.004
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'library.': 0.05;
"python's": 0.05; 'usage': 0.05; '"python': 0.07; 'underlying':
0.07; 'python.': 0.08; '18,': 0.09; 'angelico': 0.09; 'cc:addr
:python-list': 0.09; 'int': 0.09; 'likes': 0.09; 'objects,': 0.09;
'overhead': 0.09; 'timing': 0.09; 'subject:Python': 0.12;
'problem.': 0.15; '(it': 0.16; '(there': 0.16; 'approach,': 0.16;
'cc:name:python': 0.16; 'cpython': 0.16; 'date,': 0.16; 'heavily':
0.16; 'integer': 0.16; 'purely': 0.16; 'received:209.85.208.42':
0.16; 'received:mail-ed1-f42.google.com': 0.16; 'slow': 0.16;
'subject:package': 0.16; 'subject:which': 0.16; 'wrapper': 0.16;
'wrote:': 0.16; 'python': 0.16; 'says': 0.17; 'values': 0.17;
'code.': 0.17; 'uses': 0.19; 'implement': 0.19;
'cc:addr:python.org': 0.20; 'machine': 0.22; 'languages': 0.22;
'fri,': 0.22; 'sat,': 0.22; 'code': 0.23; 'idea': 0.24; '(and':
0.25; 'cc:2**0': 0.25; 'library': 0.26; 'subject:Call': 0.26;
"wasn't": 0.26; 'done': 0.28; 'chris': 0.28; 'asked': 0.29;
'approach': 0.31; 'dec': 0.31; 'think': 0.32; 'message-
id:@mail.gmail.com': 0.32; 'but': 0.32; "i'm": 0.33; 'particular':
0.33; 'same': 0.34; 'work.': 0.34; 'header:In-Reply-To:1': 0.34;
'received:google.com': 0.34; 'handling': 0.35;
'from:addr:gmail.com': 0.35; 'possibly': 0.36; 'currently': 0.37;
'subject:from': 0.37; 'using': 0.37; "it's": 0.37;
'received:209.85': 0.37; 'means': 0.38; 'received:209': 0.39;
'received:209.85.208': 0.39; 'still': 0.40; 'advantage': 0.40;
'case.': 0.40; 'should': 0.40; 'inline': 0.61; 'skip:o 10': 0.61;
"there's": 0.61; 'here': 0.62; 'true': 0.63; 'simply': 0.63;
'between': 0.63; 'range': 0.64; 'clear': 0.64; 'operations.':
0.64; 'shows': 0.67; 'types': 0.67; 'url:%3': 0.67; 'operations':
0.68; 'assembly': 0.69; 'creator': 0.69; 'hybrid': 0.69; 'within':
0.69; '2016': 0.70; '2021': 0.71; 'compared': 0.71; 'note:': 0.71;
'url:%20': 0.71; 'performance': 0.71; '(that': 0.84; 'optimise':
0.84; 'oscar': 0.84; 'referenced': 0.84; 'represented': 0.84;
'said:': 0.84; 'type.': 0.84; 'turned': 0.95; 'colleagues': 0.97
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc; bh=uTTObptHYwM2FC0GQvbRudHhp+W67APjs2A/W1dI7Zo=;
b=PxVUnW46eQFVsQkMxNDl61muVkctZ0qp/GXWTipl1Z8d2kP9PTPaTIuZ0DydusNkMx
icEAJ8LWGy7VPTlvnyybyjfoprxIE0nyyXYJZmeqKBN44/9Bc8BIF8SXgsX+XU2cTgtg
kcWp38THa6GPeDSEgWeh9AP5m/4Ctmyz7MYSyxwfYmu1VkenrGwuFyc3J2DQ7PLvfPhy
i/E2Lj4IkLow7A8d1l7vBRQskX9ZfR2KpeMjMEk0Y04zRP7H3A/Bu8k6i7AKktMOXPII
vhXKB6DqXNXzX0YAB5uzCthsTH1Is1X1ZuGB1eLxPlijnd3s0rouYoEfQbkuEBW1h/oP
EohQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc;
bh=uTTObptHYwM2FC0GQvbRudHhp+W67APjs2A/W1dI7Zo=;
b=zHFqTydtnO0mSRcOw0BRhoq3SbY2L7jvyukUmVyTkhxVFO3Wg3GLQ+SIYLGRpmYD1K
pVR82D8GJpzLjovgnp8E8HucYzZzc78glGuo/KSHghmfJmcCUSA4hniipWlJgFZbj3lp
Bc0svkSpEJ8uPsi7oj86DGWNa8k69linLc14q9kMDRgZgzmKH/1WEDeZAlePtttFTTuM
52Ag/OE5Czmw38wJ0hUbtJg05NGdUNWswS21uNHc2W3LvgcfxzyBsqd8hEcUDmwkJGMD
zoMmhNy5gt10SHTiJ1mMpSt9z6vZz4kULhaMHGoBQeFTfs11wix1E35FloaK5d1UyPyq
R9Zw==
X-Gm-Message-State: AOAM5338EW7WeIhAamy5lETe8MU6vmc1tO5XLLIUOV4zdRsbgEp+llZa
CmlwT5W53q+e1ZWTLZrspXivA8Uxi4pQf6o+2JU=
X-Google-Smtp-Source: ABdhPJyGZ/ureNSnLBC2i1dNQkcY2ICKZPi58Hn6YF3GxEFgDOj/jRCNGUcSI7Fphy+st9NUmPB4Wb0OUi/pTaktfI0=
X-Received: by 2002:a17:906:300e:: with SMTP id
14mr4105996ejz.732.1639782092994;
Fri, 17 Dec 2021 15:01:32 -0800 (PST)
In-Reply-To: <CAPTjJmormmRVfRj-HWYcT1yLvGifA=MV7KjO_c2HG5NauufA0g@mail.gmail.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAHVvXxRQmwaPdNgXPBb31jn48k7gHP3LT-c-eyYsV42WYxfybg@mail.gmail.com>
X-Mailman-Original-References: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
<CAHVvXxScU6ONFGq66Uy37Bk0k+HOnKiKF51y_9ZUbT0S+-N6yA@mail.gmail.com>
<CAPTjJmormmRVfRj-HWYcT1yLvGifA=MV7KjO_c2HG5NauufA0g@mail.gmail.com>
 by: Oscar Benjamin - Fri, 17 Dec 2021 23:01 UTC

On Fri, 17 Dec 2021 at 22:40, Chris Angelico <rosuav@gmail.com> wrote:
>
> On Sat, Dec 18, 2021 at 9:24 AM Oscar Benjamin
> <oscar.j.benjamin@gmail.com> wrote:
> > When I timed the result in Julia and in Python I found that the Julia
> > code was slower than the Python code. Of course I don't know how to
> > optimise Julia code so I asked one of my colleagues who does (and who
> > likes to proselytise about Julia). He pointed me to here where the
> > creator of Julia says "BigInts are currently pretty slow in Julia":
> > https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast.
> > I should make clear here that I used the gmpy2 library in Python for
> > the basic integer operations which is a wrapper around the same gmp
> > library that is used by Julia. That means that the basic integer
> > operations were being done by the same gmp library (a C library) in
> > each case. The timing differences between Python and Julia are purely
> > about overhead around usage of the same underlying C library.
>
> Point of note: "Python actually uses a hybrid approach where small
> integer values are represented inline and only when values get too
> large are they represented as BigInts" might be sorta-kinda true for
> Python 2, but it's not true for Python 3. In all versions of CPython
> to date, all integers are objects, they're not "represented inline"
> (there *are* some languages that do this intrinsically, and I think
> that PyPy can sometimes unbox integers, but CPython never will); and
> the performance advantage of Py2's machine-sized integers clearly
> wasn't worth recreating in Py3. (It would be possible to implement it
> in Py3 as an optimization, while still having the same 'int' type for
> both, but nobody's done it.)
>
> So if Python's integers are faster than Julia's, it's not because
> there's any sort of hybrid approach, it's simply because CPython is
> more heavily optimized for that sort of work.
>
> (That said: I have no idea whether a StackOverflow answer from 2016 is
> still applicable.)

To be clear: I wasn't using Python's int type. I used the gmpy2.mpz
type which is precisely the same mpz from the same gmp library that
Julia uses. I'm told by my colleague that Julia has a lot of overhead
when using "heap types" which is possibly the cause of the problem.

The other library I referenced (flint) does have a super-optimised
approach for handling small integers natively as machine types at the
same time as supporting large integers and even includes hand-written
assembly for the most basic operations. That library is significantly
faster than Python's int or gmpy2/gmp.

In any case what it shows is that Julia does not have reduced overhead
*in general* (i.e. for all situations) as compared to Python. I
expected that but I also thought that this particular example would be
within the range of things where Julia out-performed Python and that
turned out not to be the case.

--
Oscar

1
server_pubkey.txt

rocksolid light 0.9.81
clearnet tor