Curses! – A Post That Ends Up Being About Capturing Ncurses Output And Converting It To HTML

So I had an idea the other day! I started to implement it, but it turned out it was already done way better elsewhere, so I dropped it. But let’s talk about it anyway, because it’s not the worst idea ever and I had a fun experience as a result.

For some reason, I was under the impression that a really good web-based Interactive Fiction interpreter wasn’t available. I have no idea why I held this idea, because I had been using Parchment to play Z-Machine games online for the past couple of days. Maybe I figured there wasn’t a good online Glulx interpreter, but I was wrong about that too. In any case, the idea was simple: Provide a reasonable method of playing Interactive Fiction online with near-perfect emulation of how it would feel when played on a native interpreter.

The method I chose was chosen because I am lazy. It was literally to be a bunch of different things I found online all cobbled together. The fundamental plan was to run a Linux Z-Machine interpreter for each user, take ‘screenshots’ of the terminal, convert this screenshot to HTML and serve it to the user via a web interface. Input would be sent down to the interpreter from the browser, ‘screenshots’ get sent back up. This would have meant the user experienced ‘lag’ when typing, but I was just pleased that I thought I struck upon a novel method of delivering an authentic native Z-Machine experience in the browser.

Rather than talk about why that was all a terrible, terrible idea, let’s focus on the journey and not the aborted destination. While the idea didn’t pan out, I did learn a thing or two about the terminal and ANSI codes. In fact, I even found and fixed a bug in tmux! The fix hasn’t been accepted as a patch yet, and it might not be because it’s not the cleanest solution to the problem… But I did discover a bug, so yay!

So how did I end up doing that? Well, it turns out that the very first step of the journey was a doozy. Capturing output from the terminal is not usually very difficult. I was using a Z-Machine interpreter called Frotz to do the heavy-lifting of running the IF games. Naturally, my first attempt was redirecting the output of Frotz to a file and finding a tool to convert the ANSI to something more web-friendly. That’s when I came across ansi2html.sh, which didn’t work at the time for a very particular reason. Frotz uses something called ncurses to draw its screen. Rather than scroll the terminal when new text has to be printed, it prints over the existing space by using special terminal control codes called cursor movement instructions. These instructions are used by ‘printing’ them to the output. When one of these characters is ‘printed’ by the terminal, the cursor is moved according to the instruction rather than a new character being printed. After being moved, new text is drawn over the text that was previously present in that spot.

This means that Frotz doesn’t output lines delimited by newline characters. Instead, it prints out one very long line that uses these cursor instructions to move the cursor around and redraw over already printed characters. This confuses ansi2html, which tries very hard but eventually can’t keep up with the complicated sequence of cursor movements. This explanation doesn’t do the process justice, so here’s a few pictures. First up, here’s a picture of what Frotz looks like when it’s running (or if the output file is printed using ‘cat’):
Frotz

That looks pretty standard, right? Here’s what the underlying ANSI looks like:
ANSI Horror

Wow. Interestingly, you can ‘cat’ this file and the output looks exactly the same as the first screenshot (Technically speaking. You would have to strip out a screen-clear instruction first). So ‘cat’ can handle it, but ansi2html.sh produces this:
Nice try, ansi2html

It’s a good effort, but it’s clear that ansi2html can’t do this. Other tools purported to be able to convert ncurses ANSI output to something more usable, but I tried them to no avail. I needed some sort of pre-processing step: To convert this mess of ANSI characters into a more easily manageable mess of ANSI characters. Enter tmux.

Tmux is a terminal multiplexer, just like ‘screen’. It has one advantage over ‘screen’ that I was interested in: It can capture coloured ‘screenshots’ of the terminal and write them to a hardcopy. A hardcopy is basically just a screen capture, like the output file from earlier but stripped of cursor control characters and the like. Tmux generates this by examining each individual character in its viewing pane and deducing which ANSI flags would have to be set to produce that character. In effect, tmux produces a perfect facsimile of terminal output, minus any crazy terminal control characters. Generating the hardcopy isn’t hard. I opened a terminal and booted up tmux. Once within tmux (which again, functions very similarly to ‘screen’), I started Frotz and used another terminal to send the commands to the tmux process, causing it to capture its current viewing pane to a buffer and then saving that buffer to a file called tmux.hardcopy:
tmux capture-pane -eJ ; tmux save-buffer tmux.hardcopy

So did it work? We should be able to ‘cat’ the hardcopy and get back the same image as the first one.
So close

Okay, but where did the colour go?! I can see it in the header, but it suddenly disappears. Well, ignoring that, how does ansi2html perform with the hardcopy?
Nearly there

That’s so much better! But the colour is still missing. What gives? Well, it turns out that tmux has a bug which incorrectly sets certain flags. If certain attributes change between characters (in this case, the ‘reverse colours’ attribute), a special character is printed which resets all attributes. The remaining attributes are then re-added via their own special character instructions. However, the colour instructions are not refreshed like the other attributes, so suddenly we’re missing all the colour from the output from that point on because tmux believes it only has to print the colour changing code once at the start of the hardcopy, rather than after every attribute reset. So I wrote a patch for tmux! It’s one of my first experiences with contributing to open source.

As I was writing this, one of the tmux maintainers got back to me regarding my patch: It’s good, but it’s not quite there. It has its own problems, but we’re working through it to get a proper fix together. Hopefully no one will ever actually need the patch above!

Anyway, shortly after finding/fixing the tmux bug, I realised the project as it was envisaged was fairly pointless. However, I did get to find a real-life bug in something open source! And I got a blog post out of the whole experience too. And of course, this little post wouldn’t be complete without a final screenshot of ansi2html finally taking on an ncurses program and winning:
Victory!

Strictly speaking, I had to modify ansi2html to get the reverse colours working correctly. You can find my modified version at this gist here. But yeah! There you go: How to capture output from programs that use the ncurses library and convert them to html! Looks like something good came out of the journey after all. Maybe I’ll talk about how I tracked down the tmux bug next time?

Interactive Fiction

>look

Castle Darbenghast

You stand in a massive courtyard, surrounded on all sides by stone walls that seem to touch the sky. In front of you, there is a marble pedestal carved with ornate shapes of pugs.

There is a book on the pedestal.

>take book
Taken.

>read book
The book is a collection of something called 'blog posts'. You turn to the latest one and begin to read..


Interactive Fiction is a bit like a book that you control, in the same way that your average AAA blockbuster from Ubisoft or EA is a movie that you take the lead role in. The primary difference, naturally, is the medium. Far Cry or Halo may be worth 60,000 words per second according to the old adage, but with a piece of interactive fiction the pictures you paint are your own. Interactive Fiction (or IF) games are also known as text adventures. What follows is a ton of waffle: If you get bored, just skip to the end of the post. I have some links that you must click.

In a piece of interactive fiction the reader is given a chunk of text. After reading the text, the player must now react to it. By providing an action to the game, such as ‘look’ or ‘punch mugger’, the game decides the outcome of the action and prints another block of text to respond to. This event-response interaction continues until the story reaches an ending, whether that be a happy one (becoming king of the land), or one that’s slightly less optimal (becoming the court jester).

While the parsing and variety of actions in these games are impressive, they are limited by the imagination of the game’s author. Pretty much everything you type will be of the form ‘verb noun’, with a smallish number of verbs. Don’t be tricked into believing you’re writing a story with the computer: You’re an actor in someone else’s world, and they’ve already decided how the story should end.

In this regard, I would compare Interactive Fiction to a computerised version of a highly railroaded Dungeons and Dragons game. The dungeon master already knows how you the players will advance, and it’s up to the group to stay on the track defined by the dungeon master or face the consequences.. Or even simply be flat-out told they can’t perform certain actions for fear of derailing the story. However, much like a well-run railroad, a great piece of interactive fiction won’t ever leave you wondering how to move forward, or trying to get into the head of the creator to attempt to divine the next move. Instead, you feel like a person in a living breathing world, doing what would come naturally. Even if you can’t break the flow of the game, with some good direction and writing you’ll actually want to stay in the flow.

Another example may be in order. Below, I’ve played through a tiny amount of the legendary IF known as ‘Zork‘. My inputs are prefaced with a >, the rest of the text is provided by the game.


Copyright (c) 1981, 1982, 1983 Infocom, Inc. All rights reserved.
ZORK is a registered trademark of Infocom, Inc.
Revision 88 / Serial number 840726

West of House
You are standing in an open field west of a white house, with a boarded front
door.
There is a small mailbox here.

>open mailbox
Opening the small mailbox reveals a leaflet.

>get leaflet
Taken.

>read leaflet
"WELCOME TO ZORK!

ZORK is a game of adventure, danger, and low cunning. In it you will explore
some of the most amazing territory ever seen by mortals. No computer should be
without one!"

>go southeast
South of House
You are facing the south side of a white house. There is no door here, and all
the windows are boarded.

>go northeast
Behind House
You are behind the white house. A path leads into the forest to the east. In
one corner of the house there is a small window which is slightly ajar.

>open window
With great effort, you open the window far enough to allow entry.

>enter house
Kitchen
You are in the kitchen of the white house. A table seems to have been used
recently for the preparation of food. A passage leads to the west and a dark
staircase can be seen leading upward. A dark chimney leads down and to the east
is a small window which is open.
On the table is an elongated brown sack, smelling of hot peppers.
A bottle is sitting on the table.
The glass bottle contains:
A quantity of water

>


And so on. The example has hopefully made the idea behind IF clear: Interactive Fiction is a story that you have a degree of control over. In particular, you control which paragraph gets printed when. You do this by navigating and interacting with a world described entirely by text (in the majority of games. Some feature graphics, but those are far and few between).

Why is this so amazing? Well, truth be told, in this age of AAA blockbusters and superb indie studios, maybe text adventures are a bit archaic. However, interactive fiction still has its merits. For one thing, it’s still fun, just like how reading hasn’t aged a day since the press was invented. For another, people are still writing text adventures! You might have noticed in the example above that Zork was first published in 1981. People are still making these games, over twenty years on. They’re still wonderful, ageless experiences. From a new reader’s perspective, there is a vast library of IF out there and, because of the near universality of how they’re played, once you’ve played one you’ll know how to play them all. For potential story-tellers, IF is a very interesting medium to work with. Outside of the classic Choose-Your-Own-Adventure books, there are very few text-based ways for authors to provide an interactive experience for their readers. What better way than to let them discover with their own actions the world you’ve crafted?

Basically, if you like books and you like pretending you’re a character in a book, you’re going to get along fine with Interactive Fiction.

Before I set you loose on some of my favourite introductory examples of the genre, let me give you a quick primer on a few common text adventure commands:

  • >go <direction> – There is seldom a more often used command than the go command. Given a direction, this moves the player from one location to another in the game’s world. So common is this command, most parsers allow the user to drop the word go entirely, understand implicitly that a command like >north means >go north. The directions themselves can be shortened: North becomes ‘n’, south becomes ‘s’, etc. The classic directions in IF are the compass cardinals and intermediates, along with ‘up’ and ‘down’. Sometimes games will prefer you use terms like ‘enter’ or ‘move’.
  • >inventory – This command lists the players inventory. Most parsers will understand ‘i’ to be shorthand for ‘inventory’. Example output:
    >i
    You are carrying:
    A leaflet
  • >get <noun> – This command moves an item from the world into the player’s inventory. There is the opposite command as well: drop. Example output:
    >get knife
    Taken.
  • >look – This command displays the description of the current area the player is in. Usually, you are shown this description upon entering a room via ‘go’, but you might want to see it again to check if anything has changed (new items, a slightly different description of that brick wall..). Most parsers will understand ‘l’ to be shorthand for ‘look’. If you provide a noun to the command, you can examine an item in your inventory or in the room more closely. Example output:
    >l
    Attic
    This is the attic. The only exit is a stairway leading down.
    A large coil of rope is lying in the corner.
    On a table is a nasty-looking knife.

These are just a selection of the seriously huge number of potential commands. More avant-garde games may not even implement some of these standard commands, or others simply might have no use for things like an inventory. If you’re wondering how to perform a certain action, try typing it to see if it works. For instance, if you have a closed jar and you want it to be open, try typing >open jar! You might be pleasantly surprised. If the author is doing a good job, most of the items you come across will have obvious uses, or contain a hint as to the proper verb in its description. Parsers also give you hints if you seem to mean one thing but have typed another; for instance, if you type unlock door, the parser is likely to prompt you: With what?. That’s your cue to say door key or some such. One last thing about commands: You can usually type ‘save’ and ‘load’ to.. Well, you can probably figure that one out.

Okay, it’s time. You’re in for a treat. To play interactive fiction and most text adventures, you’ll need an interpreter and game files. The interpreter will run the game files, which are typically in .z5 or .z8 (known as Inform) format. There’s also the option of playing games line using online interpreters. I’ll close out this article with links to WinFrotz, a Windows Inform Interpreter, the IF Database, where you can find many, MANY games, and links to some great introductory or otherwise noteworthy IF games:

  • Zork – Where would we be without Zork? Explore the dungeon, get treasures, solve puzzles. Very pure, and very hard. This game is many people’s first experience with IF, but I’d say it’s probably too unforgiving. If you’re a glutton for punishment, go for it!
  • Photopia – A story by a modern Interactive Fiction master named Adam Cadre, Photopia is a perfect introduction to Interactive Fiction. Don’t read the reviews. Either play it online, or if you want the real experience, grab a Glulx interpreter and the game file proper and play it like that. Just don’t read the reviews, okay? If I remember right, the DOS version also works a treat.
  • Spider and Web – You’re a tourist in a big city, and have somehow ended up in an empty back-alley. How strange. Safely playable online, and fairly forgiving in terms of mechanics and difficulty. Not 100% introductory, but definitely a shining example of a well-written piece of fiction.
  • 9:05 – Another game by Adam Cadre. You wake up at 9:05. Time for another day. Very, very short game. I’d say it’s a brilliant introduction to IF. Play online, be amazed.
  • Ad Verbum – Something a little more unconventional. If you’re a fan of puns and wordplay, this is a delight. The parser is a little different from your average game. Maybe cut your teeth on one of the other games in this list before coming to this one. The objective is simple: Remove all the items from a condemned house. Of course, it’s not that simple. Here’s the online link.

Good luck, and have fun!


You finish reading the blog post.

>quit
See you next time!