novaBBS - comp.misc - Re: State of the Terminal

State of the Terminal
=====================
March 12, 2024

This is a companion article to my talk at Neovimconf 2023.

I have been using Vim/Neovim as my full time text editor for close to
10 years. I've spent a lot of time in the terminal and have become
very aware of the many flaws and idiosyncrasies of this bizarre
platform. But I also think it gets a lot of things right! And I'm not
alone in this belief: terminal based tools are still widely popular
even in the presence of many alternatives (the StackOverflow
developer survey shows that Neovim is the "most loved" editor 3 years
in a row).

It's only been in the last couple of years that I've begun to dig
deep into the inner workings of how terminal emulators, and the
applications that run inside of them, really work. I've learned that
there is a lot of innovation and creative problem solving happening
in this space, even though the underlying technology is over half a
century old.

I've also found that many people who use terminal based tools
(including shells like Bash and editors like Vim) know very little
about terminals themselves, or some of the modern features and
capabilities they can support.

In this article, we'll discuss some of the problems that terminal
based applications have historically had to deal with (and what the
modern solutions are) as well as some features that modern terminal
emulators support that you may not be aware of.

But first, some (very) brief history.

Background & History
====================
Most terminal emulators today can directly trace their roots back to
the DEC VT100. The VT100 was not the first video terminal, nor was it
the last, but it was the most popular (at the time). And as we've
learned from history many times since, what becomes popular creates
the de facto standard for everything that comes after.

DEC VT100 Jason Scott, CC BY 2.0 via Wikimedia Commons
<https://gpanders.com/img/DEC_VT100_terminal.jpg>

Video terminals were an improvement on the teletype machines that
preceded them. They could move the cursor around the screen to create
interactive interfaces. They could use color, and clear and redraw
their displays quickly without feeding out reams of paper.

Different video terminals had their own unique way of doing things
using unique, proprietary escape codes (a sequence of bytes beginning
with the escape 0x1b character). This made life difficult for
applications because they had to know which of these sequences to
use. Libraries and helper programs (e.g. termcap) were created to
help ameliorate these issues (we still live with the descendant of
these early libraries, terminfo).

Eventually, formal standards were created, such as ECMA-48 and ANSI
X3.64 (from which the term "ANSI escape codes" derives), which
defined a set of standard escape sequences. The DEC VT100 was the
first video terminal to support these new standards. Its popularity,
combined with the new standards, meant that programs now had a set of
known good escape sequences they could reliably use. Its popularity
spawned many clones, which in turn supported the same sequences for
compatibility with applications.

<https://en.wikipedia.org/wiki/ANSI_escape_code>

Graphical window systems eventually replaced hardware video
terminals, but users still wanted to use the terminal based programs
they were accustomed to (you know how those vi people are). In 1984,
work began on a software terminal emulator at MIT. This emulator
became part of the X project and was named Xterm. Xterm implemented
its own features which did not exist on the video terminals it
emulated, such as mouse tracking and a configurable color palette.
These features were in turn copied by Xterm clones, until eventually
Xterm itself became the new de facto standard.

<https://en.wikipedia.org/wiki/Xterm>

Terminal Emulator Basics
========================
Terminal based applications write two kinds of data to the terminal
emulator: printable text that is displayed to the user, and control
codes, which modify the terminal emulator's state. Control codes are
either single bytes in the C0 character set (bytes 0x00 through 0x1f)
or sequences of bytes that begin with the escape character (0x1b).
These sequences are most commonly referred to as "escape sequences",
and it is these sequences that do the bulk of the heavy lifting in
terminal applications.

Most control codes from the C0 character set are not used today, but
regardless of experience with terminals or terminal applications,
most developers are likely familiar with control codes such as \r
(carriage return), which moves the cursor to the beginning of the
current line, and \n (line feed), which moves the cursor to the next
line.

Escape sequences are varied and numerous, but the vast majority used
in practice fall into one of three categories: Control Sequence
Introducer (CSI), Device Control String (DCS), and Operating System
Command (OSC).

CSI sequences are those which begin with the prefix ESC [ (0x1b
0x5b). Escape sequences in this category are those which reposition
the cursor, change the cursor style, clear the screen, set foreground
and background colors, and more.

OSC sequences are those which begin with the prefix ESC ] and are
typically used for things that modify or interact with the user's
environment outside of the terminal emulator itself (hence the name
"Operating System Command"). Examples are reading from or writing to
the system clipboard, changing the title of the terminal emulator's
window, or sending desktop notifications.

Xterm maintains a list of all of the control sequences it supports on
its website, which, along with vt100.net, forms an informal
pseudo-specification for VT100 emulators. Note that this list may not
contain some control sequences used by other, modern terminal
emulators for features which Xterm does not support (e.g. the Kitty
keyboard protocol, which we'll discuss later).

<https://invisible-island.net/xterm/ctlseqs/ctlseqs.html>

<https://vt100.net/>

Escape sequences are actually quite easy to use, and you can even do
it straight from your shell. Try running the following command from
any shell:

printf '\e[1;32mHello \e[0;4;31mworld!\n\e[0m'

This command will print the text "Hello world!", with "Hello" in
green, bold text and "world!" in red, underlined text.

The escape sequences used here are of the form CSI <parameters> m,
which is so common it has its own name: Select Graphic Rendition
(SGR). The SGR escape sequence sets foreground and background colors
for all printed text. The first escape sequence in the example
\e[1;32m enables the bold attribute (1) and sets the foreground color
to green (32). The second escape sequence \e[0;4;31m first clears any
existing styles (0), then enables the underline attribute (4), and
finally sets the foreground text color to red (31). Finally, the last
escape sequence \e[0m resets all styles back to their defaults.

Another use case for simple CSI sequences is redrawing text on the
screen on an already existing line (e.g. for a progress bar or text
that updates itself over time). Hint: look at \r, CSI A, and CSI K.

Most escape sequences are sent from the application to the terminal
emulator, but occasionally the terminal emulator sends escape
sequences to the application. Usually this is done in response to a
query from the application (for instance, to determine if a certain
mode is set).

Problems & Solutions
====================
Terminal emulators are descended from old, legacy technologies, which
brings with it its fair share of problems. Many of these problems
have been (mostly) solved, or at least ameliorated, while others are
still active areas of innovation and research.

Key Encoding
------------
Terminal emulators and terminal applications communicate through a
stream of bytes. When a user presses a key the terminal sends the
byte representation of the character associated with that key. The
old video terminals only supported ASCII so this was, generally,
fairly straightforward.

Modifier keys like Ctrl and Alt complicate this situation. Alt
modified keys are encoded by prefixing the character with an Esc. But
this has a problem: including an extra Esc byte for the Alt modifier
introduces ambiguity between Alt modified key presses and two
separate key presses. When an application sees Esc C, should it
interpret it as Alt-C or did the user press Esc and then press C?
Applications usually solve this by measuring the amount of time
between Esc and the next character. If the time is less than some
defined interval, it is considered an Alt modified key press (Vim
uses the ttimeoutlen option, tmux uses the escape-time option).

Ctrl modified keys are an even bigger problem. When Ctrl is used as a
modifier, the shifted2 version of the key has the 7th bit masked off
(for example, C is 0x43 and after masking the 7th bit the byte
becomes 0x03). This means that not only can the Shift modifier not be
used in conjunction with Ctrl, but that certain Ctrl modified keys
are completely indistinguishable from other control codes.

Click here to read the complete article

Subject	Author
State of the Terminal	Ben Collver
Re: State of the Terminal	Retrograde
Re: State of the Terminal	D
Re: State of the Terminal	candycanearter07
Re: State of the Terminal	Johanne Fairchild
Re: State of the Terminal	Johanne Fairchild
Re: State of the Terminal	Johanne Fairchild
Re: State of the Terminal	Theo
Re: State of the Terminal	Anton Shepelev
Re: State of the Terminal	Lawrence D'Oliveiro

Brain fried -- Core dumped

computers / comp.misc / Re: State of the Terminal