Markdown LaTeX macros: newcommand in pandoc tests

Markdown LaTeX macros are short commands you define once with \newcommand and then call inside math, such as \R standing in for \mathbb{R}. Pandoc swapped each call inside math for its definition in this test whenever the latex_macros extension was on. A single small file was fed to pandoc under three reader settings, with two target formats, and the results are quoted below.

The conversions ran on macOS, under a pandoc whose version string reads 3.8.3. Nothing here reaches past the output this one test produced. For dollar-sign delimiters, see math in Markdown; this post skips them.

Two macros in one test file

The input is mac.md. It holds only ASCII bytes, so here is cat mac.md exactly as printed:

\newcommand{\R}{\mathbb{R}}
\newcommand{\norm}[1]{\lVert #1 \rVert}

Let $x \in \R$ and take $\norm{x}$.

$$\norm{x} \le 1$$

Plain text with \R in it.

Line 1 defines \R, which takes no argument. Line 2 defines \norm, which takes one argument, written #1 inside the definition. Line 4 calls both macros in inline math. Line 6 calls \norm in display math. Line 8 types \R in a plain sentence with no math around it, to see what a macro outside math turns into.

These five commands were run from the folder holding the file:

pandoc -f markdown -t html mac.md
pandoc -f markdown -t latex mac.md
pandoc -f markdown-latex_macros -t html mac.md
pandoc -f markdown-latex_macros -t latex mac.md
pandoc -f commonmark+tex_math_dollars -t html mac.md

All five returned exit code 0. The extra test for this post ran them a second time and saved the output, so the quotes below can be checked against two logs.

Macro calls in the HTML output

The pandoc extension list prints +latex_macros when asked about markdown, which marks the extension as on by default. The first command used that default:

pandoc -f markdown -t html mac.md

Five code points in pandoc's HTML fall outside ASCII. U+2208 (element of) is shown as in. U+211D (double-struck capital R) is shown as R. U+2016 (double vertical line) is shown as ||. U+2264 (less-than or equal to) is shown as <=. U+2004 (three-per-em space), which pandoc placed before and after U+2208 and U+2264, is shown as a plain space. With those five swaps made, here is the HTML, ASCII-normalized:

<p>Let <span class="math inline"><em>x</em> in R</span> and take <span
class="math inline">||<em>x</em>||</span>.</p>
<p><span class="math display">||<em>x</em>|| <= 1</span></p>
<p>Plain text with in it.</p>

Each line of the real output went through od -An -tx1 in the extra test. On line 1 it found e2 88 88, the UTF-8 bytes for U+2208, and then e2 84 9d, the bytes for U+211D, with e2 80 84 (U+2004) in front of each. Line 2 shows e2 80 96, which is U+2016, on both sides of <em>x</em>. Line 3 has U+2016 twice more, then U+2004, then e2 89 a4, which is U+2264, then U+2004 again before the 1.

Read together, those bytes say both macros were expanded inside math. The call \R turned into the double-struck letter, and \norm{x} turned into <em>x</em> between two double bars. Neither \newcommand line appears anywhere in the HTML.

The plain sentence is a different story. Pandoc wrote Plain text with in it. for that line, so the \R is gone in this test, with one space left between with and in. That fourth line dumped as plain ASCII bytes, so nothing was hiding there. This run gives no evidence that a macro typed outside math reaches HTML output.

Macro calls in the TeX output

The second command kept the same reader and asked for TeX:

pandoc -f markdown -t latex mac.md

Since this output is ASCII, here it is exactly:

\newcommand{\R}{\mathbb{R}}
\newcommand{\norm}[1]{\lVert #1 \rVert}

Let \(x \in \mathbb{R}\) and take \(\lVert x \rVert\).

\[\lVert x \rVert \le 1\]

Plain text with \mathbb{R} in it.

Both \newcommand lines stayed at the top, unchanged. Inside the math, pandoc still swapped in the definitions: \R became \mathbb{R}, and \norm{x} became \lVert x \rVert. Inline math was wrapped in \( and \), while the display line went between \[ and \].

This time the plain-text \R on the last line came out expanded to \mathbb{R}. So the two writers parted ways on that one line: the HTML run dropped it and the TeX run expanded it. A grep -c 'mathbb' on the saved TeX file counted 3 lines, which are the definition on line 1, the inline math on line 4 and the plain sentence on line 8.

The latex_macros extension switched off

Appending -latex_macros to the reader name took the extension away for this run:

pandoc -f markdown-latex_macros -t html mac.md

Pandoc printed three warnings to standard error. The log holds the first one like this:

[WARNING] Could not convert TeX math x \in \R, rendering as TeX:
  x \in \R
          ^
  unexpected control sequence \R
  expecting "%", "\\label", "\\tag", "\\nonumber", whitespace or "\\allowbreak"

The other two warnings named \norm, once for the inline call and once for the display call. The HTML that followed is pure ASCII, so it appears as pandoc wrote it:

<p>Let <span class="math inline">$x \in \R$</span> and take <span
class="math inline">$\norm{x}$</span>.</p>
<p><span class="math display">$$\norm{x} \le 1$$</span></p>
<p>Plain text with in it.</p>

The math spans now carry the raw TeX, delimiters included, and neither macro name was replaced. The two \newcommand lines are missing from this HTML as well. The plain sentence lost its \R once more.

The TeX writer with the same reader setting was run next:

pandoc -f markdown-latex_macros -t latex mac.md

\newcommand{\R}{\mathbb{R}}
\newcommand{\norm}[1]{\lVert #1 \rVert}

Let \(x \in \R\) and take \(\norm{x}\).

\[\norm{x} \le 1\]

Plain text with \R in it.

Here the definitions were written at the top again, and every call passed through untouched, in the math and in the plain sentence. Set beside the TeX from the default reader, the two files differ on lines 4, 6 and 8, where the calls sit.

commonmark with tex_math_dollars added

Since -tex_math_dollars appears in the commonmark extension list, the fifth command switched that pandoc extension on with a plus sign:

pandoc -f commonmark+tex_math_dollars -t html mac.md

The same three warnings came out, naming \R once and \norm twice. This HTML is ASCII too:

<p>\newcommand{\R}{\mathbb{R}} \newcommand{\norm}[1]{\lVert #1
\rVert}</p>
<p>Let <span class="math inline">$x \in \R$</span> and take <span
class="math inline">$\norm{x}$</span>.</p>
<p><span class="math display">$$\norm{x} \le 1$$</span></p>
<p>Plain text with \R in it.</p>

The two definition lines ended up as a single paragraph of visible text, rewrapped so the break falls after #1. The math spans kept their raw TeX, as in the run with latex_macros removed. The plain sentence kept \R with its backslash. That fits CommonMark 0.31.2 at lines 489-490, where a backslash placed before a character other than ASCII punctuation stays a literal backslash. More on that rule sits in escaping characters in Markdown.

As for the latex_macros name itself, it does not show up in the commonmark extension list at all. A grep for it in the saved list counted 0 lines, and so did the same grep on a fresh pandoc --list-extensions=commonmark run. As a control, a grep for tex_math_dollars in that list counted 1.

Steps to check your own macros

The extra test ran each step below on mac.md. Swap mac.md for whatever file you want to check.

1. Get TeX with the calls expanded: pandoc -f markdown -t latex mac.md -o mac-check.tex.
2. Count the definitions that came through with grep -c 'newcommand' mac-check.tex. For mac.md this printed 2.
3. Look at a plain sentence that calls a macro. Here grep -n 'Plain text' mac-check.tex printed 8:Plain text with \mathbb{R} in it. so search for words from your own sentence.
4. Produce HTML without the extension, sending the warnings to a separate file: pandoc -f markdown-latex_macros -t html mac.md -o mac-off.html 2> mac-off.err.
5. Count the warnings with grep -c 'unexpected control sequence' mac-off.err. It printed 3 for this file, matching the three math spans in mac.md that call a macro.

Converting several files in one run brings its own path questions, and relative paths across multiple files tests the rebase_relative_paths pandoc extension. Fence syntax for blocks like the ones above is in code blocks in Markdown.

Where Sashmark fits

If Sashmark reaches its first version as intended, you could correct a typo right where the text is drawn. The aim of a clean Round-Trip there would be that no line you leave alone would change, down to the byte. Changes View, another item on that version's list, would mark which passages moved after your previous visit, no git required.

It is in development and has not been released. The app itself has no AI features. See the planned features, the help pages and the roadmap for where each part stands.

More at sashmark.com.