Markdown smart quotes: pandoc's smart extension tested

You typed a plain double quote in a Markdown file. The HTML output showed a curved one instead, and a pair of hyphens turned into a single dash. Or the reverse happened: you wanted curved quotes and got the straight marks you typed.

In pandoc, this is the smart extension. This post runs a small set of test files through four pandoc reader settings and shows the output, so you can tell which setting gave which result.

The test setup

The tests ran once, on one Mac, using pandoc 3.8.3. The six main test files each hold one line made with printf, using plain ASCII quotes, hyphens and dots. Each of the six went through four reader settings:

• -f commonmark
• -f commonmark+smart
• -f markdown
• -f markdown-smart

Each run used -t html --ascii. With --ascii, pandoc wrote the curved quotes, dashes and dots in these tests as numeric references, such as “. That keeps the output readable here, since this page holds ASCII only.

What the CommonMark spec says about quotes

Searching the text of the CommonMark spec, version 0.31.2, for the words "smart", "curly" and "typographic" turned up none of them. The spec does list both quote marks, " and ', as ASCII punctuation characters, and its backslash escape rule lets you escape any character in that group. The escape characters guide covers that rule in full.

So in this test, any change from straight to curved came from pandoc's smart extension, set by the reader name and the + or - after it.

Turning smart on and off in pandoc

Pandoc can list a reader's extensions with a plus or minus sign in front of the name:

pandoc --list-extensions=commonmark | grep smart
pandoc --list-extensions=markdown | grep smart

Pandoc 3.8.3 printed -smart for commonmark and +smart for markdown. The minus means off by default and the plus means on by default. Adding +smart to a reader name turns it on, and -smart turns it off.

Here is the test file sq-double.md:

Jane said "Doe Holdings" won't wait.

The four runs gave two different results:

Reader setting	Output
commonmark	<p>Jane said "Doe Holdings" won't wait.</p>
commonmark+smart	<p>Jane said &#x201C;Doe Holdings&#x201D; won&#x2019;t wait.</p>
markdown	<p>Jane said &#x201C;Doe Holdings&#x201D; won&#x2019;t wait.</p>
markdown-smart	<p>Jane said "Doe Holdings" won't wait.</p>

Both settings with smart on curved the quotes. Both settings with it off left them as typed. Plain markdown curved them with no flag at all, since smart is on by default in that reader.

Double quotes, single quotes and apostrophes

In the sq-double.md output above, “ sits where the quote opens and ” where it closes. The apostrophe in won't became ’.

The file sq-single.md holds Jane's 'draft' plan. With smart on, both commonmark+smart and markdown gave this:

<p>Jane&#x2019;s &#x2018;draft&#x2019; plan</p>

The single quote that opens draft became ‘. The one that closes it and the apostrophe in Jane's both became ’, the same reference. With smart off, the line came back exactly as typed.

In this test, without --ascii, pandoc wrote the curved marks as real characters. An od -c dump of the commonmark+smart output for sq-double.md showed each one as a multi-byte character.

Decades, dashes and three dots

The smart extension touches more than quotes. The file sq-decade.md holds The '90s plan. With smart on, pandoc gave <p>The &#x2019;90s plan</p>: the quote before the digits became ’, the same reference it used for the apostrophe in won't.

The file sq-dashes.md holds Pages 3--5 of the plan---Jane Doe... and gave this with smart on:

<p>Pages 3&#x2013;5 of the plan&#x2014;Jane Doe&#x2026;</p>

Two hyphens became –, three hyphens became —, and three dots became …. With smart off, under commonmark or markdown-smart, the hyphens and dots stayed as typed.

Code spans and backslashes

Two kinds of input kept straight quotes even with smart on.

The first is a code span. The file sq-code.md holds this line:

Run `echo "Doe"` and keep "Jane".

With smart on, pandoc gave <code>echo "Doe"</code> with straight quotes, while the quotes around Jane outside the code became “ and ”. For more on spans and fenced blocks, see the code blocks guide.

A backslash is the other kind. The file sq-escaped.md holds Jane typed \"Doe\" and Doe\'s. and all four reader settings gave the same line:

<p>Jane typed "Doe" and Doe's.</p>

The backslashes are gone and the quotes and the apostrophe stayed straight, with smart on or off. Per the CommonMark spec, a backslash has no escape effect inside a code span, so this route is for quotes in plain text.

Curly quotes typed into the source

In this test, turning smart off didn't straighten curly quotes that were already in the file. An extra test file, sq-extra-curly.md, holds Jane said, then the word Doe between real curved double quotes, written with printf as UTF-8 bytes. Both commonmark and markdown-smart gave the same output:

<p>Jane said &#x201C;Doe&#x201D;.</p>

Checking a file for smart quotes

These steps use the test files from this post. Use your own file name in their place.

1. Check the reader's default. Run pandoc --list-extensions=commonmark | grep smart, or the same with markdown. A leading - means off by default and a + means on.
2. Render with the setting you plan to use. Run pandoc -f commonmark+smart -t html --ascii sq-double.md, or swap in markdown-smart or another setting from the table.
3. Read the output for references. Look for “, ”, ‘, ’, –, — or … where you typed straight marks, hyphens or dots.
4. Check the source bytes. Count the lines that hold a non-ASCII byte with grep -c '[^ -~]' sq-double.md, run under the C locale (LC_ALL=C). It printed 0 for that file. For sq-extra-curly.md, which holds two curved quotes on one line, it printed 1, since grep -c counts lines with a match. The invisible characters guide has more byte checks.
5. Look at the bytes. Run od -c sq-extra-curly.md to see where the multi-byte characters sit.
6. Keep one mark straight. Put a backslash just before it in plain text, or move it into a code span, then render again.

Where Sashmark fits

In the Sashmark v1 plan, you would edit a Markdown file in place on its rendered page. The goal set for that is a clean Round-Trip, where every line you did not touch should keep the bytes it had before. The same v1 plan lists a Changes View to flag edits made to the file since your previous read, with no git needed. It is in development and has not been released. The app itself has no AI features. See the planned features, the help pages and the roadmap for where each part stands.

Follow progress at sashmark.com.