I'd like get a statistic of how often each loop is used in practice.

I was trying to take a look at the Python's standard libraries --- those
included in a standard installation of Python 3.9.6, say --- to see
which loops are more often used among while and for loops. Of course,
since English use the preposition ``for'' a lot, that makes my life
harder. Removing comments is easy, but removing strings is harder. So
I don't know yet what I'll do.

Have you guys ever measured something like that in a casual or serious
way? I'd love to know. Thank you!

Hope Rouselle <hrouselle@jevedi.com> writes:
>Have you guys ever measured something like that in a casual or serious

import ast
import pathlib
rootname=r'''<ABSOLUTE PATH TO A DIRECTORY WITH SOURCE CODE>'''
rootpath=pathlib.Path(rootname)
rootiterable=rootpath.glob('**/*.py')
first = True
WhileCount = 0
ForCount = 0
for filepath in rootiterable:
try:
with filepath.open(encoding="utf-8") as file:
source=file.read()
parse=ast.parse(source, filename=str(file))
for entry in ast.walk(parse):
if isinstance(entry, ast.While):
WhileCount+=1
print( f"{ForCount=}, {WhileCount=}" )
elif isinstance(entry, ast.For):
ForCount+=1
print( f"{ForCount=}, {WhileCount=}" )
except SyntaxError as inst:
if first:
print( f"{sys.exc_info()[ 0 ] =}" )
print( f"{type( inst ) =}" )
print( f"{inst.args =}" )
print( f"{inst =}" )
print( f"skipping {filepath}." )
first=False

ram@zedat.fu-berlin.de (Stefan Ram) writes:

> Hope Rouselle <hrouselle@jevedi.com> writes:
>>Have you guys ever measured something like that in a casual or serious
>
> import ast
> import pathlib
> rootname=r'''<ABSOLUTE PATH TO A DIRECTORY WITH SOURCE CODE>'''
> rootpath=pathlib.Path(rootname)
> rootiterable=rootpath.glob('**/*.py')
> first = True
> WhileCount = 0
> ForCount = 0
> for filepath in rootiterable:
> try:
> with filepath.open(encoding="utf-8") as file:
> source=file.read()
> parse=ast.parse(source, filename=str(file))
> for entry in ast.walk(parse):
> if isinstance(entry, ast.While):
> WhileCount+=1
> print( f"{ForCount=}, {WhileCount=}" )
> elif isinstance(entry, ast.For):
> ForCount+=1
> print( f"{ForCount=}, {WhileCount=}" )
> except SyntaxError as inst:
> if first:
> print( f"{sys.exc_info()[ 0 ] =}" )
> print( f"{type( inst ) =}" )
> print( f"{inst.args =}" )
> print( f"{inst =}" )
> print( f"skipping {filepath}." )
> first=False

You are so wonderful! Thanks quite a lot. Here's what I got:

ForCount=18703, WhileCount=2505

I have pretty much just the standard libraries and a couple more ---
sympy and some xlsxwriter library.

On Sun, Aug 29, 2021 at 7:40 AM Hope Rouselle <hrouselle@jevedi.com> wrote:
>
> I'd like get a statistic of how often each loop is used in practice.
>
> I was trying to take a look at the Python's standard libraries --- those
> included in a standard installation of Python 3.9.6, say --- to see
> which loops are more often used among while and for loops. Of course,
> since English use the preposition ``for'' a lot, that makes my life
> harder. Removing comments is easy, but removing strings is harder. So
> I don't know yet what I'll do.
>
> Have you guys ever measured something like that in a casual or serious
> way? I'd love to know. Thank you!

For analysis like this, I recommend using the Abstract Syntax Tree:

https://docs.python.org/3/library/ast.html

You can take a Python source file, parse it to the AST, and then walk
that tree to see what it's using. That will avoid any false positives
from the word "for" coming up in the wrong places.

ChrisA

If you're doing this analysis, I'd be pretty interested in how many of
those while loops where 'while True:'

I'd wager 75%. But may be completely off

Steve

On Sat, 28 Aug 2021 at 23:03, Hope Rouselle <hrouselle@jevedi.com> wrote:

> ram@zedat.fu-berlin.de (Stefan Ram) writes:
>
> > Hope Rouselle <hrouselle@jevedi.com> writes:
> >>Have you guys ever measured something like that in a casual or serious
> >
> > import ast
> > import pathlib
> > rootname=r'''<ABSOLUTE PATH TO A DIRECTORY WITH SOURCE CODE>'''
> > rootpath=pathlib.Path(rootname)
> > rootiterable=rootpath.glob('**/*.py')
> > first = True
> > WhileCount = 0
> > ForCount = 0
> > for filepath in rootiterable:
> > try:
> > with filepath.open(encoding="utf-8") as file:
> > source=file.read()
> > parse=ast.parse(source, filename=str(file))
> > for entry in ast.walk(parse):
> > if isinstance(entry, ast.While):
> > WhileCount+=1
> > print( f"{ForCount=}, {WhileCount=}" )
> > elif isinstance(entry, ast.For):
> > ForCount+=1
> > print( f"{ForCount=}, {WhileCount=}" )
> > except SyntaxError as inst:
> > if first:
> > print( f"{sys.exc_info()[ 0 ] =}" )
> > print( f"{type( inst ) =}" )
> > print( f"{inst.args =}" )
> > print( f"{inst =}" )
> > print( f"skipping {filepath}." )
> > first=False
>
> You are so wonderful! Thanks quite a lot. Here's what I got:
>
> ForCount=18703, WhileCount=2505
>
> I have pretty much just the standard libraries and a couple more ---
> sympy and some xlsxwriter library.
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Stestagg <stestagg@gmail.com> writes:
>If you're doing this analysis, I'd be pretty interested in how many of
>those while loops where 'while True:'

Here, about 40 %.

Below is a new version that is supposed to also count "if True:"
and "if 1:" and also removes a bug with the exception handling
(missing "import sys" and missing "UnicodeDecodeError"). I'm not
experienced in using the AST module, so there might still be bugs
in the program!

import ast
import pathlib
import sys
rootname=r'''PATH TO SOURCE DIR (WITHOUT A BACKQUOTE AT THE END)'''
rootpath=pathlib.Path(rootname)
rootiterable=rootpath.glob('**/*.py')
WhileCount = 0
TrueCount = 0
ForCount = 0
for filepath in rootiterable:
try:
with filepath.open(encoding="utf-8") as file:
source=file.read()
parse=ast.parse(source, filename=str(file))
for entry in ast.walk(parse):
if isinstance(entry, ast.While):
WhileCount+=1
if ( isinstance( entry.test, ast.Constant ) and
hasattr( entry.test, "value") and
( isinstance( entry.test.value, bool )
and entry.test.value == True or
isinstance( entry.test.value, int )
and entry.test.value != 0 )):
TrueCount+=1
print( f"{ForCount=}, {WhileCount=}, {TrueCount=}" )
elif isinstance(entry, ast.For):
ForCount+=1
print( f"{ForCount=}, {WhileCount=}" )
except ( SyntaxError, UnicodeDecodeError )as inst:
print( f"skipping {filepath}." )
print( f"{sys.exc_info()[ 0 ] =}" )
print( f"{type( inst ) =}" )
print( f"{inst.args =}" )
print( f"{inst =}\n" )
print( f"{ForCount=}, {WhileCount=}, {TrueCount=}" )

On 8/28/2021 9:31 AM, Hope Rouselle wrote:
> I'd like get a statistic of how often each loop is used in practice.

My guess is that for loops are at least twice as common as while loops.

> I was trying to take a look at the Python's standard libraries --- those
> included in a standard installation of Python 3.9.6, say --- to see
> which loops are more often used among while and for loops. Of course,
> since English use the preposition ``for'' a lot, that makes my life
> harder. Removing comments is easy, but removing strings is harder. So
> I don't know yet what I'll do.

Try something like

fors = 0
for file in files:
for line in file:
if re.match(r"/s*for .* in ", line):
fors += 1

This excludes comprehensions, which have replaced some for loops.

--
Terry Jan Reedy

On Sun, 29 Aug 2021 at 00:04, Stefan Ram <ram@zedat.fu-berlin.de> wrote:

> Stestagg <stestagg@gmail.com> writes:
> >If you're doing this analysis, I'd be pretty interested in how many of
> >those while loops where 'while True:'
>
> Here, about 40 %.

Thanks! That's an interesting stat. I might have to look more into how
while loops are actually used in other people's code.

>
> Below is a new version that is supposed to also count "if True:"
> and "if 1:" and also removes a bug with the exception handling
> (missing "import sys" and missing "UnicodeDecodeError"). I'm not
> experienced in using the AST module, so there might still be bugs
> in the program!
>
> import ast
> import pathlib
> import sys
> rootname=r'''PATH TO SOURCE DIR (WITHOUT A BACKQUOTE AT THE END)'''
> rootpath=pathlib.Path(rootname)
> rootiterable=rootpath.glob('**/*.py')
> WhileCount = 0
> TrueCount = 0
> ForCount = 0
> for filepath in rootiterable:
> try:
> with filepath.open(encoding="utf-8") as file:
> source=file.read()
> parse=ast.parse(source, filename=str(file))
> for entry in ast.walk(parse):
> if isinstance(entry, ast.While):
> WhileCount+=1
> if ( isinstance( entry.test, ast.Constant ) and
> hasattr( entry.test, "value") and
> ( isinstance( entry.test.value, bool )
> and entry.test.value == True or
> isinstance( entry.test.value, int )
> and entry.test.value != 0 )):
> TrueCount+=1
> print( f"{ForCount=}, {WhileCount=}, {TrueCount=}" )
> elif isinstance(entry, ast.For):
> ForCount+=1
> print( f"{ForCount=}, {WhileCount=}" )
> except ( SyntaxError, UnicodeDecodeError )as inst:
> print( f"skipping {filepath}." )
> print( f"{sys.exc_info()[ 0 ] =}" )
> print( f"{type( inst ) =}" )
> print( f"{inst.args =}" )
> print( f"{inst =}\n" )
> print( f"{ForCount=}, {WhileCount=}, {TrueCount=}" )
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Chris Angelico <rosuav@gmail.com> writes:

> On Sun, Aug 29, 2021 at 7:40 AM Hope Rouselle <hrouselle@jevedi.com> wrote:
>>
>> I'd like get a statistic of how often each loop is used in practice.
>>
>> I was trying to take a look at the Python's standard libraries --- those
>> included in a standard installation of Python 3.9.6, say --- to see
>> which loops are more often used among while and for loops. Of course,
>> since English use the preposition ``for'' a lot, that makes my life
>> harder. Removing comments is easy, but removing strings is harder. So
>> I don't know yet what I'll do.
>>
>> Have you guys ever measured something like that in a casual or serious
>> way? I'd love to know. Thank you!
>
> For analysis like this, I recommend using the Abstract Syntax Tree:
>
> https://docs.python.org/3/library/ast.html
>
> You can take a Python source file, parse it to the AST, and then walk
> that tree to see what it's using. That will avoid any false positives
> from the word "for" coming up in the wrong places.

Wonderful. Someone here actually posted a nice ``routine'' that does
the job marvelously well. I had the impression the poster posted only
to solve my request --- impressive and appreciated.

Terry Reedy <tjreedy@udel.edu> writes:

> On 8/28/2021 9:31 AM, Hope Rouselle wrote:
>> I'd like get a statistic of how often each loop is used in practice.
>
> My guess is that for loops are at least twice as common as while loops.

Scanning just the Python 3.9.6's Lib/ directory --- using the ast module
and the program someone kindly posted here ---, I found for loop
appearing 598% more times than while loops. The ratio was something
like that 6.98..., excluding files that due to encoding errors (or
whatever) could not be parsed. (I reported these numbers elsewhere in
this thread.)

>> I was trying to take a look at the Python's standard libraries --- those
>> included in a standard installation of Python 3.9.6, say --- to see
>> which loops are more often used among while and for loops. Of course,
>> since English use the preposition ``for'' a lot, that makes my life
>> harder. Removing comments is easy, but removing strings is harder. So
>> I don't know yet what I'll do.
>
> Try something like
>
> fors = 0
> for file in files:
> for line in file:
> if re.match(r"/s*for .* in ", line):
> fors += 1
>
> This excludes comprehensions, which have replaced some for loops.

That's nice too. Thanks for the alternative.