Markdown emoji: shortcodes and the pandoc emoji extension

The test file held one line: :heart:, :smile: and a made-up :notanemoji: between plain words. With the emoji extension off, pandoc gave back the colons and words exactly as written.

Output here comes from a single macOS test pass plus a few added runs, and the pandoc release stayed at 3.8.3 for every one. The markdown emoji input went through the markdown and commonmark readers with the emoji extension off and on. This post stays ASCII, so each emoji character pandoc wrote is given here as a labeled code point.

Emoji shortcodes in plain CommonMark

The spec, as searched here, holds no shortcode rule. A case-blind search, grep -ci 'emoji', matched 0 lines. Searches for :smile:, for :heart: and for shortcode came back 0 each time. A control search, grep -ci 'numeric character reference', matched 10 lines in that file.

The spec lists the colon among its ASCII punctuation characters (lines 332 to 337). With no emoji rule in the file, a word between two colons is ordinary text to the spec. Pandoc adds a shortcode route through an extension called emoji.

Pandoc's emoji extension listing

Pandoc prints a reader's extensions when given the --list-extensions option. Run for this post, pandoc --list-extensions=markdown | grep emoji printed -emoji. Swapping in commonmark printed -emoji as well.

In this test, that minus sign matched the output: neither reader made an emoji span until +emoji went on the reader name. Turning it on takes -f markdown+emoji or -f commonmark+emoji for that run. The command is otherwise unchanged.

Shortcodes with the emoji extension on

The test file emoji.md:

I :heart: it :smile: and :notanemoji: too.

Extension left off:

• pandoc -f markdown -t html emoji.md returned <p>I :heart: it :smile: and :notanemoji: too.</p>.
• pandoc -f commonmark -t html emoji.md gave an identical line.

Extension turned on:

• pandoc -f markdown+emoji -t html emoji.md built two span elements. The first one opened with <span class="emoji" data-emoji="heart">. Pandoc wrapped its HTML partway through the second opening tag, so the next output line began with class="emoji" data-emoji="smile">.
• pandoc -f commonmark+emoji -t html emoji.md matched it exactly. An od dump of each run printed matching hex, byte for byte.

This post can't quote the span contents directly, since they hold bytes outside ASCII. The od -An -tx1 dump showed these runs inside the two spans:

• In the heart span: e2 9d a4 ef b8 8f. Those six bytes are two code points, U+2764 and then U+FE0F (shown here as code points; the post itself stays ASCII).
• In the smile span: f0 9f 98 84. Those four bytes are one code point, U+1F604 (also shown as a code point).

The data-emoji attribute holds the shortcode name in plain ASCII. A grep for class="emoji" on the markdown+emoji output printed 2, and the same grep on output from plain markdown printed 0.

The heart came out with a second code point, U+FE0F, worth three more bytes in the file. If a byte count on converted output looks too high, the guide to characters you can't see covers scanning a file for them.

Plain text output kept the characters too. The od dump of pandoc -f markdown+emoji -t plain emoji.md held both byte runs, with no span around either one.

Unknown names, code spans and escapes

Look at the made-up name first. :notanemoji: came through both +emoji runs as typed, colons included. So the two names pandoc knew became spans, and the invented one stayed text in this test.

Code came next. The added test file emoji-x-code.md:

In code `:smile:` stays.

The commonmark+emoji run and pandoc -f markdown+emoji -t html emoji-x-code.md both returned <p>In code <code>:smile:</code> stays.</p>. The shortcode inside the code span stayed text in both.

Then a backslash. The added file emoji-x-escape.md held I \:smile: it. on one line. With markdown+emoji and with commonmark+emoji, pandoc returned <p>I :smile: it.</p>. The backslash was dropped and no span appeared.

The spec backs this up. Per line 481, every ASCII punctuation mark accepts a backslash before it, and the colon appears in that set (lines 332 to 335). For the full set of escapes, see the escaping walkthrough.

Shortcodes inside a heading weren't part of this test. For heading syntax and anchor links, see the heading reference.

A numeric reference instead of a shortcode

Say you want the character in the output while the source file stays ASCII. The spec has its own route for that: a hexadecimal numeric character reference, written as &# then x or X, 1 to 6 hex digits and a semicolon (lines 653 to 655). It is read as the matching Unicode character (lines 656 to 657).

The added file emoji-x-ref.md held I 😄 it. on one line. A bare -f commonmark, with no extension added, handled it. Its od dump, joined onto one line with single spaces, read:

3c 70 3e 49 20 f0 9f 98 84 20 69 74 2e 3c 2f 70 3e 0a

That is <p>I , then f0 9f 98 84 for U+1F604 (given as a code point), then it.</p> and a newline. Those four bytes match the smile span's bytes. This output had no span and no class around them.

The trade is readability: the source shows a hex code where a shortcode shows a name. The spec also says references aren't recognized in code spans or code blocks (lines 610 to 611).

Test a shortcode line in pandoc

Shortcode came out as text? Put that single line through pandoc in isolation.

1. Save the line in a new file by itself, as emoji.md did.
2. Check the extension: pandoc --list-extensions=markdown | grep emoji. It printed -emoji here.
3. Convert with -f markdown+emoji -t html, or -f commonmark+emoji for the CommonMark reader.
4. Count spans: pandoc -f markdown+emoji -t html emoji.md | grep -c 'class="emoji"' printed 2. The grep counts lines, and each class="emoji" sat on its own output line here.
5. Count lines holding non-ASCII bytes: pandoc -f markdown+emoji -t html emoji.md | LC_ALL=C grep -c '[^ -~]' printed 2. With -f markdown and no +emoji, the same count printed 0.
6. Read the code points: pandoc -f markdown+emoji -t html emoji.md | od -An -tx1. Match the byte runs against the list under Shortcodes with the emoji extension on.

Where Sashmark fits

Sashmark's planned design calls for typing edits directly onto the rendered page, paired with what the plan names a clean Round-Trip, under which lines nobody touched would keep their exact original bytes. The same plan holds a Changes View as well, meant to mark whatever shifted after your prior read, and the design would have it run with no git involved. It is in development and has not been released. The app itself has no AI features. See the planned features, the help pages and the roadmap for where each part stands.

More at sashmark.com.