First, I do this:

proc p0.StartUp.SetUpIcons {} {
set ::BaseVars_WindowIconFile "$::BaseVars_iconsdir/eye5.png"
set ::BaseVars_WindowIconFile [image create photo -file $::BaseVars_WindowIconFile]
wm iconphoto . -default $::BaseVars_WindowIconFile

load "$::BaseVars_iconsdir/libtksvg0.12.so"
set ::CustomVars_IconFile_Dir_Close "$::BaseVars_iconsdir/folder_closed3.svg"
set ::CustomVars_IconFile_Dir_Open "$::BaseVars_iconsdir/folder_open3.svg"
set ::CustomVars_IconFile_CommonFile "$::BaseVars_iconsdir/file_black.svg"
set ::CustomVars_IconFile_Skinny "$::BaseVars_iconsdir/skinny.svg"
set iconsize $::GUIoptions_IconSize
set ::CustomVars_Image_IconFile_Dir_Close \
[image create photo ::CustomVars_IconFile_Dir_Close -file "$::CustomVars_IconFile_Dir_Close" -format "svg -scaletoheight $iconsize"]
set ::CustomVars_Image_IconFile_Dir_Open \
[image create photo ::CustomVars_IconFile_Dir_Open -file "$::CustomVars_IconFile_Dir_Open" -format "svg -scaletoheight $iconsize"]
set ::CustomVars_Image_IconFile_CommonFile \
[image create photo ::CustomVars_IconFile_CommonFile -file $::CustomVars_IconFile_CommonFile -format "svg -scaletoheight $iconsize"]
set ::CustomVars_Image_IconFile_Skinny \
[image create photo ::CustomVars_IconFile_Skinny -file $::CustomVars_IconFile_Skinny -format "svg -scaletoheight $iconsize"]
}

There is also this relevant proc:

p.GUI.InsertCorrectIcon {widget type target}

It's long and tedious, let's see just one part of it, it should
be enough:

if {$type == "typedir"} {
if {[file exists $::CustomVars_IconFile_Dir_Close]} {
$widget image create end -image $::CustomVars_Image_IconFile_Dir_Close
$widget insert end { }
} else {$widget insert end {D}}
return
}

OK. Then later on I build boxes with a file listing:

foreach line $organizedlist {
set insw $::Widgets_unipane.fileListPane.filenameframe.box
p.GUI.InsertCorrectIcon $insw [lindex $line 0] [lindex $line 1]
$insw insert end "[lindex $line 1]\n"
set insw $::Widgets_unipane.fileListPane.sizeframe.box
p.GUI.InsertCorrectIcon $insw [lindex $line 0] "___!SKINNY_ICON!___"
$insw insert end "[p.Utils.FormatUnits [lindex $line 2]]\n"
set insw $::Widgets_unipane.fileListPane.dateframe.box
p.GUI.InsertCorrectIcon $insw [lindex $line 0] "___!SKINNY_ICON!___"
$insw insert end "[clock format [lindex $line 3] -format "%Y-%m-%d %H:%M:%S"]\n"
set insw $::Widgets_unipane.fileListPane.permissionframe.box
p.GUI.InsertCorrectIcon $insw [lindex $line 0] "___!SKINNY_ICON!___"
$insw insert end "[p.Utils.Deoctalize [lindex $line 4]]\n"
set insw $::Widgets_unipane.fileListPane.ownerframe.box
p.GUI.InsertCorrectIcon $insw [lindex $line 0] "___!SKINNY_ICON!___"
$insw insert end "[lindex $line 5]\n"
}

And it all seemed to work so well until I "opened" a large directory.
Then it all gets pretty ugly.

For example, I have this directory with 3,000 items. My application
takes 5 seconds to "load" that directory. The delay is quite noticeable.

And it gets worse. I have another directory inside ~/Mail, one of my
oldest email accounts, with 12,500 items. My application takes a full
minute to load that directory. For comparison, SpaceFM takes 8 seconds
and PCManFM takes 3 seconds.

My application can load that huge directory almost instantaneously if I
disable the icons. It's clear the icons slow everything down. Badly.
But they are important aesthetically.

Is there some way around this problem? Can I make it faster?

Much, much faster?

--
Luc
>>

Hi Luc,
>My application can load that huge directory almost instantaneously if I
>disable the icons. It's clear the icons slow everything down. Badly.
>But they are important aesthetically.
>
>Is there some way around this problem? Can I make it faster?
>
>Much, much faster?
one way is to load just as many icons as are visible at one moment
(maybe plus a few more so that the user can scroll already while you
load the next 'page'). And I would create a 'dict' holding one key for
every entry in your directory to remember any icon already loaded.
Good luck
Helmut

Luc <luc@sep.invalid> wrote:
>
> And it gets worse. I have another directory inside ~/Mail, one of my
> oldest email accounts, with 12,500 items. My application takes a
> full minute to load that directory. For comparison, SpaceFM takes 8
> seconds and PCManFM takes 3 seconds.
>
> My application can load that huge directory almost instantaneously if I
> disable the icons. It's clear the icons slow everything down. Badly.
> But they are important aesthetically.

You didn't need all that complicated code here. You needed to digest
it down to an example case, like so:

#!/usr/bin/wish

image create photo IMAGE -file /usr/share/icons/hicolor/32x32/apps/kicad.png

text .t
pack .t

set begin [clock milliseconds]
for {set i 0} {$i < 12500} {incr i} {
.t insert end "$i: "
.t image create end -image IMAGE
.t insert end ":$i\n"
}

#make sure work to actually display is complete
update

set end [clock milliseconds]
puts "[expr {$end - $begin}] ms elapsed"

The above is what I mean when I tell you to post "working code" -- and
additionally: "easy to understand code". Compare the understandability
of the above to what you posted. Note, if you don't have a "kicad.png"
then substitute another png icon in /usr/share/icons/hicolor/32x32/
that you *do* have.

Running the above on my machine, I get:

11149 ms elapsed

If I comment out the "image create" call, I get:

123 ms elapsed

> Is there some way around this problem?

Nothing that will be simple, as this looks to be a slowness within the
text widget itself. So short of posting a patch to the C code of the
text widget to remove what looks like an O(n^2) complexity factor when
inserting images there is not a whole lot you can do.

> Can I make it faster?

Possibly, but you'd have to setup a system where you fill out the
widget with text, without images, then insert the images just before
lines scroll into view, avoiding the need to "create" everything up
front.

> Much, much faster?

Avoid images or use Unicode characters that are the same, or close
enough, to your current images as substitutes for the images you
currently use.

On Sat, 30 Dec 2023 05:39:30 -0000 (UTC), Rich wrote:

>You didn't need all that complicated code here.

I thought I should because you always ask for code and because I had
done it in a different way that was even slower, then I found the
current way which is a lot faster and maybe I could still be doing
something wrong or stupid.

>Avoid images or use Unicode characters that are the same, or close
>enough, to your current images as substitutes for the images you
>currently use.

I thought about using Unicode and I think that would be great although
too many users would disagree and demand custom iconsets. But alas,
the only Unicode characters I could find that would satisfactorily
represent files and folders are out of Tcl/Tk's reach, such as

1F5C0 🗀 1F5C1 🗁 1F4BC 💼 1F5B9 🖹 1F4C4 📄

TK can't display those.

I wonder if they are going to make it through intact here. Let's see.

--
Luc
>>

Luc <luc@sep.invalid> wrote:
> On Sat, 30 Dec 2023 05:39:30 -0000 (UTC), Rich wrote:
>
>>You didn't need all that complicated code here.
>
> I thought I should because you always ask for code and because I had
> done it in a different way that was even slower, then I found the
> current way which is a lot faster and maybe I could still be doing
> something wrong or stupid.

Code that demonstrates the issue. Not "a chunk of your project's
code". The difference is significant, and important.

>>Avoid images or use Unicode characters that are the same, or close
>>enough, to your current images as substitutes for the images you
>>currently use.
>
> I thought about using Unicode and I think that would be great although
> too many users would disagree and demand custom iconsets.

If you are going to allow for user custom icon sets then that means
some trickery to work around the issue. One option might be to build a
"virtual scrolling" setup where your text widget only has the number of
lines visible on screen, plus some small amount, and you "scroll" up
and down by replacing lines in the widget rather than by scrolling a
widget with 12,500 lines.

> But alas, the only Unicode characters I could find that would
> satisfactorily represent files and folders are out of Tcl/Tk's reach,
> such as
>
> 1F5C0 🗀 1F5C1 🗁 1F4BC 💼 1F5B9 🖹 1F4C4 📄
>
> TK can't display those.

Tcl 9 should correct that (as far as I understand). But that does not
help you now.

> I wonder if they are going to make it through intact here. Let's see.

No, I got blank boxes here as well.

On Sun, 31 Dec 2023 00:23:33 -0000 (UTC), Rich wrote:

>If you are going to allow for user custom icon sets then that means
>some trickery to work around the issue.

I think the problem is in 'image create'. That already sounds like
heavy work. Done thousands of times, it can't be fast.

I wish there was some way we could create an image once and "cache" it
then use it like a rubberstamp without having to "create" it every
single time. I've been trying to figure out some way to do that or to
turn everything into one single huge string to be inserted in one fell
swoop, but there doesn't seem to be a way. :-(

>> 1F5C0 🗀 1F5C1 🗁 1F4BC 💼 1F5B9 🖹 1F4C4 📄
>> TK can't display those.

>Tcl 9 should correct that (as far as I understand). But that does not
>help you now.
>> I wonder if they are going to make it through intact here. Let's see.
>
>No, I got blank boxes here as well.

"As well"? Must be your email/news reader. They all got through intact
for me. And the version of email reader I'm using was released seven
years ago.

--
Luc
>>

On Sun, 31 Dec 2023 00:23:33 -0000 (UTC), Rich wrote:

>If you are going to allow for user custom icon sets then that means
>some trickery to work around the issue. One option might be to build a
>"virtual scrolling" setup where your text widget only has the number of
>lines visible on screen, plus some small amount, and you "scroll" up
>and down by replacing lines in the widget rather than by scrolling a
>widget with 12,500 lines.

I've thought about that, but it seems too hard. "Scrolling" means a lot
of things. Maybe the user will use arrow keys. Maybe PageUp/PageDown.
Maybe the mouse wheel. Maybe Home/End to jump to the beginning and end.
And there is selection. Maybe it's a long selection that extends beyond
the limits of the fake view.

Imagine managing all that. That alone will cost me more work than the
entire application. I don't think I want to afford that.

I will sooner scrap the text widget model for buttons or labels or
something else. I don't know. This is one heck of a show-stopper.

--
Luc
>>

Luc <luc@sep.invalid> wrote:
> On Sun, 31 Dec 2023 00:23:33 -0000 (UTC), Rich wrote:
>
>>If you are going to allow for user custom icon sets then that means
>>some trickery to work around the issue.
>
> I think the problem is in 'image create'. That already sounds like
> heavy work. Done thousands of times, it can't be fast.
>
> I wish there was some way we could create an image once and "cache" it

You already do that (and my short example does as well). The "image"
is only created once, in this line:

image create photo IMAGE -file /usr/share/icons/hicolor/32x32/apps/kicad.png

The text widget is reusing that same image over and over. But
somewhere in the code for the text widget is some code that expends
ever more time for each new image created within the text (for the text
widget, 'image create' is not creating a new image, it is creating a
new image annotation within the text).

Given the slowdown, it feels like an algorithm that is doing something
like measuring the height of every prior line already in the widget for
each new image that is inserted. Doing something like that would lead
to the amount of slowdown seen. Note that I don't know what is causing
the excessive work, just guessing at one possibility.

> then use it like a rubberstamp without having to "create" it every
> single time. I've been trying to figure out some way to do that or to
> turn everything into one single huge string to be inserted in one fell
> swoop, but there doesn't seem to be a way. :-(

Right now, given the slowdown seen, working out a way to not have to
insert so many images, almost all of which won't be visible on screen
unless the user scrolls, will probably be the way to go.

>>> 1F5C0 🗀 1F5C1 🗁 1F4BC 💼 1F5B9 🖹 1F4C4 📄
>>
>>No, I got blank boxes here as well.
>
> "As well"? Must be your email/news reader. They all got through intact
> for me. And the version of email reader I'm using was released seven
> years ago.

It is also possible that the font I'm using simply does not have glyphs
for those codepoints.

Just for the record, I can fill up a text widget with 20,000 buttons
with both text and image in under 500 ms.

How can buttons be so fast, even with images?

I also tried filling it up with labels and with other one-line text
widgets. Those were much, much too slow. And I didn't even try them
with images.

I don't think I can really use buttons though. I suspect they will be
too clumsy for my purposes.

Back to the drawing board...

package require Tk
wm withdraw .
set w [toplevel .top]

set wf [frame $w.frame]
pack $wf -fill both -expand 1

set sb [scrollbar $wf.sb -orient vertical -command "$w.tx yview"]
set tx [text $wf.tx -font {Arial 14} -yscrollcommand [list $sb set]]
$sb configure -command [list $tx yview]]
pack $sb -fill y -side right
pack $tx -fill both -expand 1
focus $tx
bind .top <Escape> {exit 0}

set begin [clock milliseconds]
image create photo IMAGE -file /usr/share/icons/hicolor/32x32/apps/elk.png
for {set x 1} {$x <= 20000} {incr x} {
button $tx.b$x -text $x -width 80 -cursor arrow -image IMAGE -compound left
pack $tx.b$x
$tx window create end -window $tx.b$x
$tx insert end "\n"
} set end [clock milliseconds]
puts "[expr {$end - $begin}] ms elapsed"

--
Luc
>>

Luc <luc@sep.invalid> wrote:
> Just for the record, I can fill up a text widget with 20,000 buttons
> with both text and image in under 500 ms.

I get 502ms for my last run, which had two mods:

button $tx.b$x -text $x -width 80 -cursor arrow -image IMAGE -compound l
#pack $tx.b$x
$tx insert end "$x "
$tx window create end -window $tx.b$x
$tx insert end "$x\n"

Widgets that you plan to "insert" into a text widget do not need to be
packed (packing is for display within a frame or toplevel, not for
inside another widget).

And I added text in front and behind (just in case that made any
difference). It did not seem to.

> How can buttons be so fast, even with images?

Simple. The code path used for "window create" is different from the
code path used for "image create" and the "window create" code path
does not have the O(n^2) algorithm within it.

> I also tried filling it up with labels and with other one-line text
> widgets. Those were much, much too slow. And I didn't even try them
> with images.

Ok, so a way around the "slow images" is to use buttons containing
images.

> I don't think I can really use buttons though. I suspect they will be
> too clumsy for my purposes.

Why? You can make them into "nothing windows" (remove all bindings,
turn off all decorations) such that they don't change or push or do
anything. Beyond a bit of "manual memory management" (you'll want to
either reuse existing buttons for destroy/recreate them as needed) they
do solve your performance problem with regular image create in a text.

> Back to the drawing board...

Why, it seems you've found an excellent workaround. The complexity
here (tracking your "image buttons" for memory management purposes) is
significantly lower than building up a "virtual scrolling" system.

Meanwhile you might also consider filing a bug report on the text
widget performance with image create (and now that you've discovered
the "image in a button" workaround, you have further evidence that the
issue is not directly related to "images" themselves, but something
different between "image create" and "window create").

On Sun, 31 Dec 2023 16:53:28 -0000 (UTC), Rich wrote:

>Ok, so a way around the "slow images" is to use buttons containing
>images.
>
>Why, it seems you've found an excellent workaround.

Meh. Not so fast.

The code I posted never changes the font. As soon as I specify a font,
the whole thing becomes slower by a factor of 70x!

Back to the drawing board...

--
Luc
>>

Greetings from the drawing board.

My threaded version of the thing I'm working on is actually slower than
the previous design.

I still struggle to understand threads so maybe I did something wrong.

Or maybe threads won't help in this case. What do you think?

--------------------
package require Tk
package require Thread

image create photo IMAGE -file /usr/share/icons/hicolor/32x32/apps/elk.png

text .t -font {Freesans 14}
pack .t
focus .t
bind .t <Escape> {exit 0}

set begin [clock milliseconds]

set linecount 4500
set chunksize 1500
set chunks [dict create]
for {set x 1} {$x <= $linecount} {incr x} {
set y [expr {$x - 1}]
if {![info exists chunkstart$x] && ![info exists chunkstart1]} {
set chunkstart$x 1
} else {set chunkstart$x [expr {[set chunkend$y] + 1}]}
set chunkend$x [expr {[set chunkstart$x] + [expr {$chunksize - 1}]}]
if {[set chunkend$x] >= $linecount} {set chunkend$x $linecount}
dict set chunks chunk$x "start [set chunkstart$x] end [set chunkend$x]"
if {[set chunkend$x] >= $linecount} {break}
}

# confirming that the chunks are alright
#foreach ch [dict keys $chunks] {puts "$ch starts at [dict get $chunks $ch "start"]\n$ch ends at [dict get $chunks $ch "end"]\n"}

# create all the empty lines first
for {set i 1} {$i <= $linecount} {incr i} {
.t insert end " line$i\n"
} proc populate {} {
foreach ch [dict keys $::chunks] {
for {set x [dict get $::chunks $ch "start"]} {$x <= [dict get $::chunks $ch "end"]} {incr x} {
.t image create $x.0 -image IMAGE
}
}
} foreach chunk [dict keys $chunks] {
set tid_$chunk [thread::create {thread::wait}]
unset -nocomplain resultof_$chunk
thread::send -async [set tid_$chunk] [populate] resultof_$chunk
if {![info exist resultof_$chunk]} {vwait resultof_$chunk}
}

update
set end [clock milliseconds]

focus .t
..t see end
puts "[expr {$end - $begin}] ms elapsed"
--------------------

--
Luc
>>

Luc <luc@sep.invalid> wrote:
> On Sun, 31 Dec 2023 16:53:28 -0000 (UTC), Rich wrote:
>
>>Ok, so a way around the "slow images" is to use buttons containing
>>images.
>>
>>Why, it seems you've found an excellent workaround.
>
>
> Meh. Not so fast.
>
> The code I posted never changes the font. As soon as I specify a font,
> the whole thing becomes slower by a factor of 70x!

How did you specify the font?

Luc <luc@sep.invalid> wrote:
> Greetings from the drawing board.
>
> My threaded version of the thing I'm working on is actually slower than
> the previous design.
>
> I still struggle to understand threads so maybe I did something wrong.
>
> Or maybe threads won't help in this case. What do you think?

Threads won't help. The reason why is the slowness is from the code of
the single text widget that is doing the display of all the lines.

No matter how many threads you launch, they all end up serialized on
that one single text widget, that is doing all the work of formatting
the display data. And the slowdown is somewhere inside the code that
does the "formatting for display".

On Sun, 31 Dec 2023 20:07:00 -0000 (UTC), Rich wrote:

>Luc <luc@sep.invalid> wrote:
>> On Sun, 31 Dec 2023 16:53:28 -0000 (UTC), Rich wrote:
>>
>>>Ok, so a way around the "slow images" is to use buttons containing
>>>images.
>>>
>>>Why, it seems you've found an excellent workaround.
>>
>>
>> Meh. Not so fast.
>>
>> The code I posted never changes the font. As soon as I specify a font,
>> the whole thing becomes slower by a factor of 70x!
>
>How did you specify the font?
>
**************************

The same code I posted yesterday. It specifies a font for the text
widget that contains everything.

438 ms

I realized the text widget is just a container and doesn't need any
specific font. So I removed it.

440 ms

Then I added the font specification to the line that creates the button:

button $tx.b$x -font {Arial 14} -text $x -width 80 -cursor arrow -image IMAGE -compound left

76111 ms

Yikes.

--
Luc
>>

On Sun, 31 Dec 2023 20:08:57 -0000 (UTC), Rich wrote:

>Threads won't help. The reason why is the slowness is from the code of
>the single text widget that is doing the display of all the lines.
>
>No matter how many threads you launch, they all end up serialized on
>that one single text widget, that is doing all the work of formatting
>the display data. And the slowdown is somewhere inside the code that
>does the "formatting for display".
**************************

Threads will help if they give me a responsive GUI while the application
does something else behind the curtains. The way I did it, it is not
responsive until all the threads have finished their tasks.

--
Luc
>>

On 12/31/2023 11:14 AM, Luc wrote:
> Greetings from the drawing board.
>
> My threaded version of the thing I'm working on is actually slower than
> the previous design.

> foreach chunk [dict keys $chunks] {
> set tid_$chunk [thread::create {thread::wait}]
> unset -nocomplain resultof_$chunk
> thread::send -async [set tid_$chunk] [populate] resultof_$chunk
> if {![info exist resultof_$chunk]} {vwait resultof_$chunk}
> }

In the above, [populate] is not done in the second thread, but is instead processed before thread::send even sees it. So, whatever populate returns, is sent into a thread and when that's done, it sets your return variable.

Looking at populate, the return value is likely the last [.t image create], which returns a unique identifier. I suspect the thread sees that and actually throws an error, since it's not a valid script to execute. I forget whether that ends up killing the thread or just goes silently into the night.

Also, each thread runs in it's own interpreter. So, that .t created in the main thread is not visible to the second thread. In fact, tk in a second thread even has its own event loop and everything else that a gui has. It's almost like being a separate process. And if you do a package require Tk in that second thread, you'll end up with another widget tree beginning at .

Luc <luc@sep.invalid> wrote:
> On Sun, 31 Dec 2023 20:07:00 -0000 (UTC), Rich wrote:
>>How did you specify the font?
>>
>
> Then I added the font specification to the line that creates the button:
>
> button $tx.b$x -font {Arial 14} -text $x -width 80 -cursor arrow -image IMAGE -compound left
>
> 76111 ms

You specified the font in a much less than optimal way.

You want to create a named font using "font create", then use that
named font for the "-font" option.

Your way probably had Tk redoing all the font rasterizing anew for each
line. Using a named Tk font you only pay the font lookup and
rasterizing once. An added advantage is that if you then "font
configure" that named font later, the updates propagate automatically
to every widget that uses the named font.

Rich et al,

In case you're interested, my bug report has received comments:

https://core.tcl-lang.org/tk/tktview/737abf45230d38c450c031d1c89ea84811d54e3e

I appreciate it very very much.

When that fix hits the streets I will be able to rewrite the largest
proc in my application in a much simpler way. But I still like to think
that the more complicated version will always be there for users of old
Tcl/Tk versions to fall back on. And I learned how to use threads.
No effort is lost in life until life itself is lost.

Thank you all.

--
Luc
>>