Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

The clothes have no emperor. -- C. A. R. Hoare, commenting on ADA.

devel / comp.lang.python / Re: Pandas or Numpy

SubjectAuthor
* Pandas or NumpyTobiah
+- Re: Pandas or NumpyChris Angelico
+- Re: Pandas or NumpyJulius Hamilton
+- Re: Pandas or NumpyAvi Gross
`- Re: Pandas or NumpyMarco Sulla

1
Pandas or Numpy

<ssjsi3$3h0$1@gioia.aioe.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=16791&group=comp.lang.python#16791

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!aioe.org!6uZihruI5Omezy/vf6DSxw.user.46.165.242.75.POSTED!not-for-mail
From: tob...@tobiah.org (Tobiah)
Newsgroups: comp.lang.python
Subject: Pandas or Numpy
Date: Sun, 23 Jan 2022 07:34:26 -0800
Organization: Aioe.org NNTP Server
Message-ID: <ssjsi3$3h0$1@gioia.aioe.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Info: gioia.aioe.org; logging-data="3616"; posting-host="6uZihruI5Omezy/vf6DSxw.user.gioia.aioe.org"; mail-complaints-to="abuse@aioe.org";
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
Thunderbird/91.5.0
Content-Language: en-US
X-Notice: Filtered by postfilter v. 0.9.2
 by: Tobiah - Sun, 23 Jan 2022 15:34 UTC

I know very little about either. I need to handle score input files
for Csound. Each line is a list of floating point values where each
column has a particular meaning to the program.

I need to compose large (hundreds, thousands, maybe millions) lists
and be able to do math on, or possibly sort by various columns, among other
operations. A common requirement would be to do the same math operation
on each value in a column, or redistribute the values according to an
exponential curve, etc.

One wrinkle is that the first column of a Csound score is actually a
single character. I was thinking if the data types all had to be the
same, then I'd make a translation table or just use the ascii value
of the character, but if I could mix types that might be a smidge better.

It seems like both libraries are possible choices. Would one
be the obvious choice for me?

Thanks!

Re: Pandas or Numpy

<mailman.242.1642958716.3079.python-list@python.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=16792&group=comp.lang.python#16792

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: ros...@gmail.com (Chris Angelico)
Newsgroups: comp.lang.python
Subject: Re: Pandas or Numpy
Date: Mon, 24 Jan 2022 04:25:04 +1100
Lines: 30
Message-ID: <mailman.242.1642958716.3079.python-list@python.org>
References: <ssjsi3$3h0$1@gioia.aioe.org>
<CAPTjJmq3Ky22JGq1a3Ndz5NL+4f=QNLbA173G-8dExgJB1rPnQ@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
X-Trace: news.uni-berlin.de HAtXaSR5vxKC6AwLr0J01AGh7WgkAwPI/xEj3ln8xmuw==
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=bP2gtV2v;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.033
X-Spam-Evidence: '*H*': 0.93; '*S*': 0.00; '2022': 0.05; 'math': 0.05;
'floating': 0.09; 'pandas': 0.09; 'types.': 0.09; 'choices.':
0.16; 'chrisa': 0.16; 'column': 0.16; 'exponential': 0.16;
'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'me?':
0.16; 'numpy,': 0.16; 'sounds': 0.16; 'wrinkle': 0.16; 'wrote:':
0.16; 'values': 0.17; 'libraries': 0.19; 'to:addr:python-list':
0.20; 'input': 0.21; 'maybe': 0.22; "i'd": 0.24; 'seems': 0.26;
'else': 0.27; 'thinking': 0.28; 'same,': 0.32; 'message-
id:@mail.gmail.com': 0.32; 'but': 0.32; "i'm": 0.33; 'particular':
0.33; 'able': 0.34; 'same': 0.34; 'header:In-Reply-To:1': 0.34;
'received:google.com': 0.34; 'fine': 0.35; 'handling': 0.35;
'meaning': 0.35; 'yes,': 0.35; 'from:addr:gmail.com': 0.35;
'files': 0.36; 'built': 0.36; 'mon,': 0.36; 'possibly': 0.36;
'lists': 0.37; "it's": 0.37; 'received:209.85': 0.37; 'could':
0.38; 'received:209': 0.39; 'single': 0.39; 'handle': 0.39;
'list': 0.39; 'use': 0.39; 'table': 0.39; 'program.': 0.40;
'both': 0.40; 'should': 0.40; 'me.': 0.62; 'data,': 0.63;
'everything': 0.63; 'operations.': 0.64; 'requirement': 0.64;
'top': 0.65; 'among': 0.65; 'types': 0.67; 'mix': 0.69; 'obvious':
0.69; 'little': 0.73; 'choice': 0.76; 'expert,': 0.84;
'redistribute': 0.84; 'score': 0.91
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to;
bh=71SJpZLJVUgDaj8aP6hZyWLLzMdU+2MWBZziyfnwkBs=;
b=bP2gtV2vGzeWxp46rLiUrEfiY0Op5IZzAGgVap+fjkXvRez87oeUGrpu7BaJZ4Y2Np
3GmX8Itp/0GJiOHn4zAo5QBfaiO3pWXqA1JTZN5oXSoZx+lA4J8e35re9/PIdbcTI4w6
RtJ7AP1sKF2Zto9B/kNM7rlA6KqzXe9gzN+9OsyVRimU7DmXfvOxrmjZLJgPhTysSZcA
fOHAhCxggIPxjkDqUV89Jv3Qstl3wBysa5DweIiHrRkIUkk7w+/BsizypsRlLsULO5hK
r65+OJEqyQ7jlfqE24tzE/ic3i1GOt5P2cET8n1R/AVL7tf1qwKdebf1dJmr9HeMNE6+
52eQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to;
bh=71SJpZLJVUgDaj8aP6hZyWLLzMdU+2MWBZziyfnwkBs=;
b=Y81vAFp+IxXwliVUznxtyYuwfPqEN0s55Btsn3xkj0Xje2xSRwRFJajf+ZrX+nwEgO
5xEm3xGv+WsllFqAdDyJw61DRh4xkDp+DpY+TuifyavP5nxKWVUVpzZIxZrzu/HvDwbw
o5rianV0NDqX8f90O8Az2zFMwgrtQuGDtyOQloMrA8fy1Ywv4ppxWeaFFSoqdhNudC4h
IPf79sue/NZdIDK1++M9X1EuKoLnCPJNDVIbmgutpDeIFyqzVhiLm1FZr37RHifGo1dH
MXj5FDl0OVtmH0ke0o5CRIbF57VEBm6N/dQRH1HCHrYrDjtHlDsI/Tpu9BE6eNsogbLz
0/3Q==
X-Gm-Message-State: AOAM533m62SYbvKM21JfxsvUelc8uX9Bakk2GCGbNL7QWH3w/sILCQRX
0ZDlp4rp9v6CtztS4vbNJ9vbkNk3RetflHbnMDh5MYzX
X-Google-Smtp-Source: ABdhPJz6EO2Y8gi7i/ocAgt9awpXliyOlcXdKABNTKdRa7NC1cniCgdfm4bXgsg+D6slKNeEV4BI+2IoLujuZaVEGtE=
X-Received: by 2002:a1c:f316:: with SMTP id q22mr8689343wmq.33.1642958714948;
Sun, 23 Jan 2022 09:25:14 -0800 (PST)
In-Reply-To: <ssjsi3$3h0$1@gioia.aioe.org>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAPTjJmq3Ky22JGq1a3Ndz5NL+4f=QNLbA173G-8dExgJB1rPnQ@mail.gmail.com>
X-Mailman-Original-References: <ssjsi3$3h0$1@gioia.aioe.org>
 by: Chris Angelico - Sun, 23 Jan 2022 17:25 UTC

On Mon, 24 Jan 2022 at 04:10, Tobiah <toby@tobiah.org> wrote:
>
> I know very little about either. I need to handle score input files
> for Csound. Each line is a list of floating point values where each
> column has a particular meaning to the program.
>
> I need to compose large (hundreds, thousands, maybe millions) lists
> and be able to do math on, or possibly sort by various columns, among other
> operations. A common requirement would be to do the same math operation
> on each value in a column, or redistribute the values according to an
> exponential curve, etc.
>
> One wrinkle is that the first column of a Csound score is actually a
> single character. I was thinking if the data types all had to be the
> same, then I'd make a translation table or just use the ascii value
> of the character, but if I could mix types that might be a smidge better.
>
> It seems like both libraries are possible choices. Would one
> be the obvious choice for me?
>

I'm not an expert, but that sounds like a job for Pandas to me. It's
excellent at handling tabular data, and yes, it's fine with a mixture
of types. Everything else you've described should work fine (not sure
how to redistribute on an exponential curve, but I'm sure it's not
hard).

BTW, Pandas is built on top of Numpy, so it's kinda "both".

ChrisA

Re: Pandas or Numpy

<mailman.245.1642961119.3079.python-list@python.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=16795&group=comp.lang.python#16795

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: juliusha...@gmail.com (Julius Hamilton)
Newsgroups: comp.lang.python
Subject: Re: Pandas or Numpy
Date: Sun, 23 Jan 2022 19:05:06 +0100
Lines: 52
Message-ID: <mailman.245.1642961119.3079.python-list@python.org>
References: <ssjsi3$3h0$1@gioia.aioe.org>
<CAPTjJmq3Ky22JGq1a3Ndz5NL+4f=QNLbA173G-8dExgJB1rPnQ@mail.gmail.com>
<CAEsMKX3qeN4GbSXzfFkYS+7RFWCyAjoypyrSJKKLis0u7b494A@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de nHoU9ipgYc8OWiOlGSVsDA/BiCX/lI/+tGy3wo7r359Q==
Return-Path: <juliushamilton100@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=YfBDG9pX;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.004
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; '2022': 0.05; 'math': 0.05;
'angelico': 0.09; 'cc:addr:python-list': 0.09; 'floating': 0.09;
'numpy': 0.09; 'pandas': 0.09; 'received:209.85.219': 0.09;
'spreadsheet': 0.09; 'types.': 0.09; '&gt;': 0.14; 'cc:no real
name:2**0': 0.14; 'url:mailman': 0.15; 'both.': 0.16; 'choices.':
0.16; 'chrisa': 0.16; 'column': 0.16; 'exponential': 0.16;
'mathematical': 0.16; 'me?': 0.16; 'numpy,': 0.16; 'sounds': 0.16;
'wrinkle': 0.16; 'wrote:': 0.16; 'values': 0.17; 'libraries':
0.19; 'cc:addr:python.org': 0.20; 'input': 0.21; 'maybe': 0.22;
"i'd": 0.24; 'url-ip:188.166.95.178/32': 0.25; 'url-
ip:188.166.95/24': 0.25; 'url:listinfo': 0.25; 'cc:2**0': 0.25;
'url-ip:188.166/16': 0.25; 'seems': 0.26; 'library': 0.26; 'else':
0.27; 'chris': 0.28; 'thinking': 0.28; 'url-ip:188/8': 0.31;
'same,': 0.32; 'message-id:@mail.gmail.com': 0.32; 'but': 0.32;
"i'm": 0.33; 'particular': 0.33; 'able': 0.34; 'same': 0.34;
'header:In-Reply-To:1': 0.34; 'received:google.com': 0.34;
'definitely': 0.35; 'fine': 0.35; 'handling': 0.35; 'meaning':
0.35; 'yes,': 0.35; 'from:addr:gmail.com': 0.35; 'files': 0.36;
'built': 0.36; 'mon,': 0.36; 'possibly': 0.36; 'lists': 0.37;
"it's": 0.37; 'received:209.85': 0.37; 'could': 0.38;
'received:209': 0.39; 'single': 0.39; 'handle': 0.39; 'list':
0.39; 'use': 0.39; 'table': 0.39; 'data.': 0.40; 'program.': 0.40;
'both': 0.40; 'should': 0.40; 'skip:o 10': 0.61; 'me.': 0.62;
'data,': 0.63; 'everything': 0.63; 'operations.': 0.64;
'requirement': 0.64; 'top': 0.65; 'among': 0.65; 'types': 0.67;
'mix': 0.69; 'obvious': 0.69; 'little': 0.73; 'choice': 0.76;
'expert,': 0.84; 'redistribute': 0.84; 'score': 0.91
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc; bh=F1BzjeiDLciSCuzRspc0P8UlgNotY9liXZnWASZ270Q=;
b=YfBDG9pXHOybPb58PQ6OPaKre2TmHxjTKMHvOPuiRzOf756f+nB02KfH9vkRi1hJz/
gdTvxCQor8ryjwZVKZTFzJbHZmFFvidbUkNLIOaGYPS58xeQWcq6ksv3HnNPyEL6F50R
+InZleAZIHZ3QizvUt2Q99U6l5ZrhR3uDLwEQdeNPzLIt8e/gu2CWS36TjOFfzXipWkl
T3u16zyoCgDzpeJlz2+pQby+jsDev3w73Awqua35Lh9x7Vx9THx3kAxy4rljSEkyXHww
9I7tNanK/HreA4OS7UmDudhj0gAgzPp1puh3cSBaW0dKerXU5qaqJFYLZ0gMImCaz5E4
vcnQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc;
bh=F1BzjeiDLciSCuzRspc0P8UlgNotY9liXZnWASZ270Q=;
b=2Do8xeM8qz5JgRzzaI7NnSeWzdSIMyRxul0WPiAIENfLTAIAjvJNMRzPS6y4CLmJAT
rWK9Uq717hpZhHrmH3jbGyw1YbeXyfSQONPhHVWN51h8DDiZt676Q65unzzOx7wlmfSc
yJyNlq3hd66VXbYxCRVoswvnaOgPD87S5nx6oB+8F2um6Q1z4ZUrooX6yEar8bJXn2aB
MoJQjZ3WYYCmYrySIN3ja8QMdwECSxG+YftwrdWv8yM9xBp+4rv4+5jPTm54RPQiOY8C
YXjJLKNlqwf3itea3k0uWH1duKtTYJJKoE+lSYIGUxGO9JT6WBuEcIk506hufgc9Zf22
/rCw==
X-Gm-Message-State: AOAM530H81XUqaNo+Y1eJ1TTSVVuWinLru9jZyAR3P8+X6LYw+1t9g9/
ZX2IZpgZ71Mzdcj59ZTOg+niyoTPn1FmXKc7zoB/r+bUvvo=
X-Google-Smtp-Source: ABdhPJwRQZLl1nvAlBeuuLIjkJzTL8bby8ALy4okAehRbpAJOEN51atZC/4vjUC9ZIj0rxjxnVjBqAMClxN37q3Hlxc=
X-Received: by 2002:a5b:4c2:: with SMTP id u2mr18405309ybp.668.1642961116869;
Sun, 23 Jan 2022 10:05:16 -0800 (PST)
In-Reply-To: <CAPTjJmq3Ky22JGq1a3Ndz5NL+4f=QNLbA173G-8dExgJB1rPnQ@mail.gmail.com>
X-Content-Filtered-By: Mailman/MimeDel 2.1.39
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAEsMKX3qeN4GbSXzfFkYS+7RFWCyAjoypyrSJKKLis0u7b494A@mail.gmail.com>
X-Mailman-Original-References: <ssjsi3$3h0$1@gioia.aioe.org>
<CAPTjJmq3Ky22JGq1a3Ndz5NL+4f=QNLbA173G-8dExgJB1rPnQ@mail.gmail.com>
 by: Julius Hamilton - Sun, 23 Jan 2022 18:05 UTC

Hey,

I don’t know but in case you don’t get other good answers, I’m pretty sure
Numpy is more of a mathematical library and Pandas is definitely for
handling spreadsheet data.

So maybe both.

Julius

On Sun 23. Jan 2022 at 18:28, Chris Angelico <rosuav@gmail.com> wrote:

> On Mon, 24 Jan 2022 at 04:10, Tobiah <toby@tobiah.org> wrote:
> >
> > I know very little about either. I need to handle score input files
> > for Csound. Each line is a list of floating point values where each
> > column has a particular meaning to the program.
> >
> > I need to compose large (hundreds, thousands, maybe millions) lists
> > and be able to do math on, or possibly sort by various columns, among
> other
> > operations. A common requirement would be to do the same math operation
> > on each value in a column, or redistribute the values according to an
> > exponential curve, etc.
> >
> > One wrinkle is that the first column of a Csound score is actually a
> > single character. I was thinking if the data types all had to be the
> > same, then I'd make a translation table or just use the ascii value
> > of the character, but if I could mix types that might be a smidge better.
> >
> > It seems like both libraries are possible choices. Would one
> > be the obvious choice for me?
> >
>
> I'm not an expert, but that sounds like a job for Pandas to me. It's
> excellent at handling tabular data, and yes, it's fine with a mixture
> of types. Everything else you've described should work fine (not sure
> how to redistribute on an exponential curve, but I'm sure it's not
> hard).
>
> BTW, Pandas is built on top of Numpy, so it's kinda "both".
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Re: Pandas or Numpy

<mailman.247.1642962553.3079.python-list@python.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=16797&group=comp.lang.python#16797

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: avigr...@verizon.net (Avi Gross)
Newsgroups: comp.lang.python
Subject: Re: Pandas or Numpy
Date: Sun, 23 Jan 2022 18:29:09 +0000 (UTC)
Lines: 95
Message-ID: <mailman.247.1642962553.3079.python-list@python.org>
References: <ssjsi3$3h0$1@gioia.aioe.org>
<CAPTjJmq3Ky22JGq1a3Ndz5NL+4f=QNLbA173G-8dExgJB1rPnQ@mail.gmail.com>
<CAEsMKX3qeN4GbSXzfFkYS+7RFWCyAjoypyrSJKKLis0u7b494A@mail.gmail.com>
<983499973.718802.1642962549481@mail.yahoo.com>
Reply-To: Avi Gross <avigross@verizon.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de /H3PeFWLEdG7DMc1QBt1ewy374+B2f8sBqBGePasFrlQ==
Return-Path: <avigross@verizon.net>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=verizon.net header.i=@verizon.net header.b=NKY3GIse;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.003
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; '2022': 0.05; 'math': 0.05;
'&gt;&gt;&gt;': 0.07; 'modules': 0.07; 'sun,': 0.07; 'angelico':
0.09; 'cc:addr:python-list': 0.09; 'dataframe': 0.09; 'fair,':
0.09; 'floating': 0.09; 'general,': 0.09; 'hey,i': 0.09;
'language,': 0.09; 'numpy': 0.09; 'pandas': 0.09; 'spreadsheet':
0.09; 'tricks': 0.09; 'types.': 0.09; '&gt;': 0.14; 'url:mailman':
0.15; 'both.': 0.16; 'choices.': 0.16; 'chrisa': 0.16;
'collection,': 0.16; 'column': 0.16; 'compiled': 0.16;
'dataframe.': 0.16; 'exponential': 0.16; 'integer': 0.16;
'large.': 0.16; 'mathematical': 0.16; 'me?': 0.16; 'numpy,': 0.16;
'packages.': 0.16; 'sounds': 0.16; 'structures': 0.16; 'subset':
0.16; 'unsigned': 0.16; 'usage,': 0.16; 'values,': 0.16;
'wrinkle': 0.16; 'wrote:': 0.16; 'python': 0.16; 'values': 0.17;
'applications': 0.17; 'libraries': 0.19; 'cc:addr:python.org':
0.20; 'input': 0.21; 'languages': 0.22; 'maybe': 0.22; 'code':
0.23; "i'd": 0.24; 'skip:- 10': 0.25; 'url-ip:188.166.95.178/32':
0.25; 'url-ip:188.166.95/24': 0.25; 'url:listinfo': 0.25;
'cc:2**0': 0.25; 'programming': 0.25; 'url-ip:188.166/16': 0.25;
'seems': 0.26; 'space': 0.26; 'library': 0.26; 'creating': 0.27;
'else': 0.27; 'bit': 0.27; 'done': 0.28; 'chris': 0.28; 'mostly':
0.28; 'series': 0.28; 'thinking': 0.28; 'ideas': 0.28; 'takes':
0.31; 'url-ip:188/8': 0.31; 'it.\xc2\xa0': 0.32; 'objects': 0.32;
'same,': 0.32; 'structure': 0.32; 'but': 0.32; "i'm": 0.33;
'there': 0.33; 'particular': 0.33; 'able': 0.34; 'same': 0.34;
'header:In-Reply-To:1': 0.34; 'definitely': 0.35; 'fine': 0.35;
'handling': 0.35; 'meaning': 0.35; 'yes,': 0.35; 'files': 0.36;
'built': 0.36; 'mon,': 0.36; 'possibly': 0.36; 'people': 0.36;
'lists': 0.37; 'missing': 0.37; 'using': 0.37; "it's": 0.37;
'could': 0.38; 'means': 0.38; 'two': 0.39; 'quite': 0.39; 'added':
0.39; 'copies': 0.39; 'single': 0.39; 'skip:o 10': 0.61; 'skip:h
10': 0.61; 'from:': 0.62; 'me.': 0.62; 'to:': 0.62; 'data,': 0.63;
'true': 0.63; 'everything': 0.63; 'full': 0.64; 'operations.':
0.64; 're:': 0.64; 'requirement': 0.64; 'your': 0.64; 'top': 0.65;
'among': 0.65; 'parts': 0.65; 'look': 0.65; 'types': 0.67;
'generally': 0.67; 'operations': 0.68; 'and,': 0.69; 'cc:': 0.69;
'mix': 0.69; 'obvious': 0.69; 'ignore': 0.71; 'little': 0.73;
'costs': 0.74; 'recognized': 0.75; 'features': 0.75; 'choice':
0.76; 'life': 0.77; 'sent:': 0.78; 'header:Reply-To:1': 0.79;
'spent': 0.81; 'admit': 0.84; 'expert,': 0.84; 'garbage': 0.84;
'hamilton': 0.84; 'indexes': 0.84; 'luck.': 0.84; 'redistribute':
0.84; 'flexible': 0.91; 'largely': 0.91; 'received:74.6.132':
0.91; 'score': 0.91
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=verizon.net; s=a2048;
t=1642962550; bh=ojlJC+9CwtIpVFsAwe4IYJL8Iy8wXA7hVHiwyae38uY=;
h=Date:From:Reply-To:Cc:In-Reply-To:References:Subject:From:Subject:Reply-To;
b=NKY3GIseCsvt1VqfbNGDlVaJMpt4Fy7spm0lXO88eAoUWmWQ1bSjBrf5CTeGM+iK2C6SgcagR8dNqZbP9BrCXBwl0nERz8iv1cS75y42h7mAXXlLA0Iw3WgJz5VbP2CEQ81aujC+1FeKUNlxhopohnRCVAQ0Ro7pwLSBLlstPjYRB+xpFPflGQUfrpsGYbERXS768oowHWelzrDKwlDi+M176RZiV3wxABvMfqbpGAu2R4IKFnQfKv/YfADBTSLKgzKzVvGl/Z1edc4OrC1t7X2LKIPZYgcIaA7afy0achyfL7/pP+y21JXRsIKQOeqppCJ7weVkJccFbYBNtzjFxg==
X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
t=1642962550; bh=rjrAsQZmYh9UtlRrIuTdzSJc8zNke1f/LlUbE/O+45R=;
h=X-Sonic-MF:Date:From:Subject:From:Subject;
b=qBZaoTRrzfIUt3/8WYZ7vnkDVBibjzOzDLt4q7lmyv3b2RrqJaOMVWv+ZFcUHuvCne16JQl0mlC92sDYhtZSAAQctXwoTrSe6nIc5Hj1lC82qqZG3ozmu/RpomO++o0PC3+DOO5R5Xh4+nR5JkTFtOueQoVVnHTTckEWe0AOcxaP9jzNss2MC6XMWPz/3ocI9M0IRkEDw0skM5I/KOgCxUU0pq4VAUq0Pj2rE6iGE7fRLV6p4pEH3mycPIbRIqcH3qqUidpoZ8vEJagG9luNp/xuO2+1xTFq/MWQHkJSgfoxG/06CI/OtWVc2ZuM9ljEqmMCrjbC8mtbpzXwVykWhA==
X-YMail-OSG: gB6amAcVM1nI_e2Ob5.S0Gb1aqGPR9YxDd7DLyK2OFN_XTn2J_wUZw_DfheX2oK
zhaIoNggInBl65XdUn64Jk7ouVktlU08D.cgsaPb4fED1nKHwBRhYChDdGR5mSYwU8MB8fOSb4om
muhzzDeUevj1XyLfnw4d42Mv1YtEwd400DW47Du6MR3cI8NTz9fmYkRQxR1ecGlPbFvDEp5X4aJj
bThlEXYXp30pukukWl86PgUDR6gSiDwaZYHfuATQa7AFZcY4xTspFHuLqdHenEfLRMU7TWkn2yWJ
97mMQa7irpdTZGjgjMl4qvltVr6Ph7KJSz_Sxxo0KEerBnqMh2OmIkAVRVSPU_ZFTUQQUaKWf1RM
v1SKLnvw36.ZD4nFsPrnBVukNkDvTSkAdq15QitsDoRvLS34AhV9eQtfMCHWKpV_twhkc.4IT64u
pA8cOLWjai5OWJvDQOVSntlC5MGEXVPfaWvN5TJ.j9QoR07.hRFh4BX5ecS3RkCvlTeYnDOFZ85y
gQadcnB1RJ6h5Fhl8PaFZuSrzQqVVIvePS.hwrYaIOLX4N2kwXDJ7g_sPr6tXO8lRhCm9wc3G0Dk
Qt2igpoywm4LMMvlAnwDmyT0eFCicl0EOVC0FXVFV87vCJ9Bp5Qa0DO2JlUrViMkfVAI1Sz7Kgs.
l6l87gpvzOKJazbkKEiH4kH_Rx0SSdDm1HlZY5Qizcu9SFNRQn1qw3Ytg58xTW6xoNaK93kfkEp3
uiZ.21sb.dqHtF92O_QEQ.W3ISoP1eSmhk2k4hZzRLuLOqyax1BYjmi4E1_bmRHrfotdIjCKtV8j
NC2eu773nOWyLGgaceEMHYswhep7nAsN62AwPUo5qK2NJSg_tJo_PJz7WRqNOD8JDJl34t5GwyXi
7_JyYcbkGUT0xr6P.VVw_vBO7f2YyjaKL4fJW1_sFp.nYY9y6nSHE0lRNs6CYh00TguQh1r_VP_0
Hw10anRbQMexarUw_jwrHJM6aq8KoQlabx3FSvlcQHQv0sRirWXD_PtEDeWDfrPWBJKuSTyAw_7w
xi2SqrRY8ZyCMkr7MnXOPwG2Bj_bhYJWpwRcWdBwn8tuOCSNQQAGErYZCxK7fTh8ba68TQ1dXdbb
n4QsiupGc0C4UHXktKZRh4bOpxxhtwxrzcSkYJ1dbM5LUSWrGQY_wyqYi9sZ9qbDjfkJ11HsRn49
ThesMRmGR4gMskQ4XzRzChp0JzGMHEkCQTHLdQhEmv4jyMxk9f04vVXpe9Qh4UpB3DFdmjY11yXd
0jKTSTIWao3qqjO2oivgt2.GQYAN3lT.AXMPCCUTTHOHoGY2RE9VnMVxoHUqMSi3D7yDPQNBjAAM
BEKAN1pc6Q9GJBsHqPyuO7l2QoBmbfoifeQdFnYv9j1jbys6QVR62fXaZdK3S1c0OyYhs6eDaPBs
xFB8xE8XpycVDufyCAUtOSsr9XJ.PHj3VdIKBACbv3QBKVdMcMj9K5G4z4wJBblQL3Q7.lEB2msA
VvdMkOGn9pWGnyVeTH8Vc8uFmDIkL3vUMMZJLvkx2hwN6ENq4cJXvJBodmE3n4HWMEEe32yiCaOx
T_9ZPXCwOINy7010aKebZ6wvL1oHbxl4Cb7AR3zg57ev00U0WiTx.BDoNRopXyEnJ3X5m8TKX3lj
AaWtpSQuIFGq9wjzLmwfj8EEyflWT_foDLiRCW64iVPUBtugvXoOD9AI_VdmXYxPIXeMHByAsfsW
5RqdQObW0YEeSE8dRcwpXaCT5YOicHN3kdzQrBXgNr0ljcE.fBGrhW_rFvmInjRzQ3rMIR21_p6g
lQrUfemFsk8dCyAh407N9bu4eYbfCxTh0FcrACrI0DLeMMc8rZwqA1GJtfv8wjIfhWht2XC_.BxQ
_mnR_j.d0BoBcwYsoEl3zku57q5ePsEBEhfRnfNdIT7OOymyXoZlNrWweAXoJyHCW_Jh23OhbM1J
8Zi718DuVQmhd45Q0cJaZ7YtRblgGKTv86U38kRqpj.hRWJYchHvBb.3A4JLXhvnn.B0xqc8yVLy
lzJ2p.7sWYGowpuWUsgnKI8VGgGfgZKDqrZ5.NuR0EnCgBbYyEO7GWgqc7SVMIP.SxbMXOkAN5aq
JkT6RTAXTKUXRrgJ8ppWf3MDhABc68gu_26waFngfXWV7nGZW72OosdbY4JRsPcgxtqBoHWXGS_J
a3526ZMyt8V.9hOaiCA69kI0z
X-Sonic-MF: <avigross@verizon.net>
In-Reply-To: <CAEsMKX3qeN4GbSXzfFkYS+7RFWCyAjoypyrSJKKLis0u7b494A@mail.gmail.com>
X-Mailer: WebService/1.1.19615 aolwebmail
X-Content-Filtered-By: Mailman/MimeDel 2.1.39
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <983499973.718802.1642962549481@mail.yahoo.com>
X-Mailman-Original-References: <ssjsi3$3h0$1@gioia.aioe.org>
<CAPTjJmq3Ky22JGq1a3Ndz5NL+4f=QNLbA173G-8dExgJB1rPnQ@mail.gmail.com>
<CAEsMKX3qeN4GbSXzfFkYS+7RFWCyAjoypyrSJKKLis0u7b494A@mail.gmail.com>
 by: Avi Gross - Sun, 23 Jan 2022 18:29 UTC

Definitely it sounds like you may use both. Quite a bit of what people do using DataFrame objects includes working on copies of individual columns, which often are numpy Series or the like and in the other direction, can be used to create or amend a pandas DataFrame. Plus, many operations used to select some subset of rows will use things like  a data structure holding the integer indexes you want or a boolean where True means take it and False means ignore it. 
Many real life applications just incorporate both numpy as np and pandas as pd and sometimes also use other Python functionality such as lists or matrices which unfortunately generally are just a list of lists. 
Python was built with a different philosophy than some languages like R in which much of what is in numpy and pandas is built in, in  other ways, and has been extended by packages. Python built mostly on flexible lists so the modules you are asking about do make faster and perhaps better versions. And, to be fair, python has lots of nifty features that R is largely missing and had to be added externally. 
Both used properly can do a nice job on the kind of things you want but with a warning. Your description suggests some of the data you will be using or making can get quite large. So make sure you look into the dtypes for parts of your data so you do not store small integers in full sized integers but in signed or unsigned bytes. Data with only two possible values, might be stored as boolean. And note many operations can be done in place, rather than creating a new object. If you are worried about space usage, or time spent on garbage collection, as in any programming language, there are recognized ideas about how you might tighten up your code using existing paradigms. I do admit some tricks have costs and it takes real testing to see if it even matters to try. But in general, much of numpy and pandas are already optimized in lots of compiled code  so using these rather than python lists and other data structures can already be a big plus.
Good luck.

-----Original Message-----
From: Julius Hamilton <juliushamilton100@gmail.com>
To: Chris Angelico <rosuav@gmail.com>
Cc: python-list@python.org
Sent: Sun, Jan 23, 2022 1:05 pm
Subject: Re: Pandas or Numpy

Hey,

I don’t know but in case you don’t get other good answers, I’m pretty sure
Numpy is more of a mathematical library and Pandas is definitely for
handling spreadsheet data.

So maybe both.

Julius

On Sun 23. Jan 2022 at 18:28, Chris Angelico <rosuav@gmail.com> wrote:

> On Mon, 24 Jan 2022 at 04:10, Tobiah <toby@tobiah.org> wrote:
> >
> > I know very little about either.  I need to handle score input files
> > for Csound.  Each line is a list of floating point values where each
> > column has a particular meaning to the program.
> >
> > I need to compose large (hundreds, thousands, maybe millions) lists
> > and be able to do math on, or possibly sort by various columns, among
> other
> > operations.  A common requirement would be to do the same math operation
> > on each value in a column, or redistribute the values according to an
> > exponential curve, etc.
> >
> > One wrinkle is that the first column of a Csound score is actually a
> > single character.  I was thinking if the data types all had to be the
> > same, then I'd make a translation table or just use the ascii value
> > of the character, but if I could mix types that might be a smidge better.
> >
> > It seems like both libraries are possible choices.  Would one
> > be the obvious choice for me?
> >
>
> I'm not an expert, but that sounds like a job for Pandas to me. It's
> excellent at handling tabular data, and yes, it's fine with a mixture
> of types. Everything else you've described should work fine (not sure
> how to redistribute on an exponential curve, but I'm sure it's not
> hard).
>
> BTW, Pandas is built on top of Numpy, so it's kinda "both".
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list

Re: Pandas or Numpy

<mailman.268.1643239011.3079.python-list@python.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=16820&group=comp.lang.python#16820

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: Marco.Su...@gmail.com (Marco Sulla)
Newsgroups: comp.lang.python
Subject: Re: Pandas or Numpy
Date: Thu, 27 Jan 2022 00:16:12 +0100
Lines: 14
Message-ID: <mailman.268.1643239011.3079.python-list@python.org>
References: <ssjsi3$3h0$1@gioia.aioe.org>
<ga2rugp1e4ibur3sa8b724fo1h78ttvv45@4ax.com>
<CABbU2U96LkqUsX5-T6MjDFtRAnz5aN+UA7aWb9hM5dvxCodiFA@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
X-Trace: news.uni-berlin.de MuCnXkQiv2AclgvUSaVcJQ2ac3YXbxX3MyrA6WPmYeUw==
Return-Path: <elbarbun@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=jlYawcf0;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'csv': 0.03; '2022': 0.05;
'row': 0.05; 'okay': 0.07; 'cc:addr:python-list': 0.09;
'comparison': 0.09; 'enough.': 0.09; 'numpy': 0.09; 'pandas':
0.09; 'received:209.85.219': 0.09; 'cc:no real name:2**0': 0.14;
'"apply': 0.16; 'builtin': 0.16; 'creation,': 0.16;
'from:name:marco sulla': 0.16; 'furthermore,': 0.16; 'pandas.':
0.16; 'wrote:': 0.16; 'cc:addr:python.org': 0.20; 'all,': 0.20;
'code': 0.23; 'cc:2**0': 0.25; 'tried': 0.26; 'module': 0.31;
'transform': 0.32; 'message-id:@mail.gmail.com': 0.32; 'but':
0.32; 'header:In-Reply-To:1': 0.34; 'received:google.com': 0.34;
'from:addr:gmail.com': 0.35; 'mon,': 0.36; 'really': 0.37;
'received:209.85': 0.37; 'received:209': 0.39; 'should': 0.40;
'your': 0.64; 'saw': 0.65
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc; bh=kVncG2McfGz3cmWzqIgkQWZjhiSrD+eweli2VEAY9Zs=;
b=jlYawcf07hkmdvzhQAb0Mg7gWoQM4D93nLrg+g9w61cWXhJBMGn3y/PXqzEUipusRE
3ZhQGQ+1jtfHcB7Tu5iylZbYzoemxeV7ypcKR9VKB2VXefLT++RY+tO2vryourwLlW8C
36dP0H7KKDiXR6pw9TeGu0tcwlSCIT9sa3QV4o+6MCl+6Jqpjo9dMYvkPgc/ErspJUxw
oyyVteUsH+XJBXwjzb59/6uLEhh+bVW21WrECEsNmGgWzE/SIJM9vd/sdIqo9lisYSuK
ds+SXDdadOgH4cweuWZeic/ouF/EZRZi+dHhrGUCuFXclh9dGiICNtjs6ISponJSkSaC
J96Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc;
bh=kVncG2McfGz3cmWzqIgkQWZjhiSrD+eweli2VEAY9Zs=;
b=ah0NsznAGbMbyJYzHW4D62yHRykTbXELQ/TEd+SPJ8Sc005h4OeU4lw8EgrThT4S+h
rSpX08Mwo1C9k1Xsi1zy9NSymYOF77mrcOIPf1a276uJsL6PwOEKC/4Vci5vaG7xIXzN
63Ly7145I8Uul/9oQGf0Hy6n4/CRoXSAuJarwGX55KqxtUcESLS4azsdBmaw325VMkDX
uKsSkHxi0PqY4aSRp3fOPetipJ61++wphlheGfgNVNLg9jG36oXBIYMeorGDa2e4tVpM
lHc9Uk0mTVdA8ASJoxn8EN4T7lXvDrH5E5rP8uicHamQ2TL4L2wET9/+93JsOl9gTjPM
o2fQ==
X-Gm-Message-State: AOAM532ET3G0cQl0yTMv68mP1iFF07DOycWV8ueg8+547+de1dWYCoR0
EruKhqEpE2o6FM75uRCAsr9IB3Bx5keXvDMLMVs=
X-Google-Smtp-Source: ABdhPJw/wOJ2lvD3cLh1yq2pa3QUVqdbF7JcPIQZZsyCWNPtXnvK3iJb7iivP2TLDWU0h+82fvj2sxQJPDZERflnpo0=
X-Received: by 2002:a25:40c2:: with SMTP id n185mr1854130yba.536.1643239008583;
Wed, 26 Jan 2022 15:16:48 -0800 (PST)
In-Reply-To: <ga2rugp1e4ibur3sa8b724fo1h78ttvv45@4ax.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CABbU2U96LkqUsX5-T6MjDFtRAnz5aN+UA7aWb9hM5dvxCodiFA@mail.gmail.com>
X-Mailman-Original-References: <ssjsi3$3h0$1@gioia.aioe.org>
<ga2rugp1e4ibur3sa8b724fo1h78ttvv45@4ax.com>
 by: Marco Sulla - Wed, 26 Jan 2022 23:16 UTC

On Mon, 24 Jan 2022 at 05:37, Dennis Lee Bieber <wlfraed@ix.netcom.com> wrote:
> Note that the comparison warns that /indexing/ in pandas can be slow.
> If your manipulation is always "apply operationX to columnY" it should be
> okay -- but "apply operationX to the nth row of columnY", and repeat for
> other rows, is going to be slow.

In my small way, I can confirm. In one of my previous works, we used
numpy and Pandas. Writing the code in Pandas is quick, but they just
realised that was really slow, and they tried to transform as much
Panda code to numpy code as possible.

Furthermore, I saw that they were so accustomed with Pandas that they
used it for all, even for a simple csv creation, when the csv builtin
module is enough.

1
server_pubkey.txt

rocksolid light 0.9.81
clearnet tor