Invisible characters in Markdown: find and test them

A heading won't render. A list item turns into a plain line that starts with a dash. You open the file, and it looks fine.

The cause can be a byte you can't see. This post builds small test files holding real invisible characters in markdown, shows the shell commands that reveal each one, and reports what pandoc made of them. Why tools add these bytes is linked under When git and editors are involved.

How the tests were run

Each test file was made with printf, so it holds the real bytes. An invisible character can't be shown in a code block, so this post writes it two ways: as the escape you'd type in printf, such as \xC2\xA0, or as the text that cat -vet and od -c print for it.

Every result below comes from one run of pandoc 3.8.3 on macOS, called with the command pandoc -f commonmark -t html file.md. Results from other tools aren't covered here. The commands set LC_ALL=C, as the tests did. In the cat -vet output, a $ marks the end of each line, so anything between the last visible character and the $ is something you couldn't see.

Carriage returns (CRLF endings)

When a file uses CRLF endings, a carriage return sits before each line feed. Under CommonMark 0.31.2, a line ending can be a line feed, a lone carriage return, or a carriage return plus a line feed.

The test file crlf.md held a name, a company, an empty line and two list items, all with CRLF endings. Running LC_ALL=C cat -vet crlf.md printed this:

Jane Doe^M$
Doe Holdings^M$
^M$
- one^M$
- two^M$

Each ^M is a carriage return. To count the lines holding one, run grep -c $'\r' crlf.md. It printed 5 for crlf.md and 0 for lf.md, which has the same text with plain line feeds. The file command agreed, calling crlf.md ASCII text, with CRLF line terminators.

Pandoc gave the same HTML for both files: one paragraph, then a list with two items.

Trailing spaces

Trailing spaces sit in cat -vet output as blank room between the text and the $, and grep -n ' $' file.md lists every line whose final character is a space, with its number. In this test, two trailing spaces gave <br /> in pandoc. One gave no break, so the two lines joined: pandoc returned a single paragraph holding both lines, with no <br />. The rule behind that is in the line breaks guide.

Byte order mark (BOM)

The test files for this section began with the three bytes \xEF\xBB\xBF, a UTF-8 byte order mark. Running LC_ALL=C cat -vet bom-heading.md printed its first line like this:

M-oM-;M-?# Doe Holdings$

The M-oM-;M-? part is the mark. To check for it, LC_ALL=C grep -c $'^\xEF\xBB\xBF' file.md counts lines that open with those bytes. It gave 1 for both test files that had the mark and 0 for lf.md. The file command described bom-heading.md as Unicode text, UTF-8 (with BOM) text.

The spec text (version 0.31.2) never mentions a byte order mark. In this test, pandoc still read the heading, and bom-heading.md came out as <h1>Doe Holdings</h1>. Front matter came through too. Run through pandoc -f commonmark+yaml_metadata_block -t html -s --metadata pagetitle=x bom-frontmatter.md, the file gave Doe Holdings as the document title, matching the result for the same file without the mark.

Non-breaking spaces

A non-breaking space is the character U+00A0, typed in printf as \xC2\xA0. In cat -vet output it shows as M-BM- followed by a space, so the heading test printed as #M-BM- Doe Holdings$.

CommonMark requires one or more spaces or tabs after the opening # run of a heading, before its text. A list marker needs the same before the item's text. A space, in the spec's terms, is U+0020, so a non-breaking space doesn't count.

The tests matched. With a non-breaking space after #, pandoc returned a <p> paragraph, not an <h1>. With one after -, it returned a <p> paragraph, not a list. Placed between two words, as in Jane\xC2\xA0Doe, the character stayed in pandoc's HTML.

To find them, run LC_ALL=C grep -n $'\xC2\xA0' file.md. It listed line 1 of each of the three test files that held one and printed nothing for lf.md.

Zero-width spaces

A zero-width space is U+200B, typed in printf as \xE2\x80\x8B. In cat -vet output it shows as M-bM-^@M-^K, and od -c prints its three bytes as 342 200 213.

This character is neither U+0020 nor a tab, so the heading rule from Non-breaking spaces applies. With a zero-width space between # and the text, pandoc returned <p> instead of a heading.

Bold text was the second test. The file zwsp-emphasis.md was built from the escape **Jane\xE2\x80\x8B**Doe, which puts a zero-width space just before the closing **. Pandoc left both pairs of asterisks in the output as literal characters. The control file, **Jane**Doe with nothing hidden, gave <strong>Jane</strong>Doe. This post doesn't trace that result to a line in the spec; it's what pandoc gave in this test.

Between two words, as in Jane\xE2\x80\x8BDoe, pandoc kept the character, and od -c of the HTML showed the same three bytes. Search for it with LC_ALL=C grep -n $'\xE2\x80\x8B' file.md.

Missing final newline

The last byte tells you. Running tail -c 1 file.md | od -c printed s for nofinal.md and \n for lf.md. In cat -vet output, the last line of nofinal.md had no closing $.

The spec allows the last line to stop at end of file without a line ending, and pandoc rendered nofinal.md as a single paragraph holding both lines. For saves that should leave a missing final newline untouched, read the round-trip save explainer.

How to scan a file for invisible characters

Run these on the file that misbehaves. Swap file.md for its name.

1. Run LC_ALL=C cat -vet file.md and read the line ends. A ^M before the $ means a carriage return, and a gap before the $ means trailing spaces.
2. Run LC_ALL=C grep -n '[^[:print:][:space:]]' file.md to list lines holding bytes that the C locale counts as neither printable nor space characters. In the tests it caught the non-breaking space, the zero-width space and the byte order mark. It also flags real text: it listed accent.md, whose only unusual character is an e with an acute accent. A hit means "look here."
3. Count the lines holding a carriage return with grep -c $'\r' file.md and list trailing spaces with grep -n ' $' file.md. In a separate test, the grep from step 2 printed nothing for crlf.md or for trail2.md, the test file with two trailing spaces, so these two need their own check.
4. Pin down a hit from step 2 with a targeted search: LC_ALL=C grep -n $'\xC2\xA0' file.md for non-breaking spaces, LC_ALL=C grep -n $'\xE2\x80\x8B' file.md for zero-width spaces, and LC_ALL=C grep -c $'^\xEF\xBB\xBF' file.md for a byte order mark.
5. Check the final byte with tail -c 1 file.md | od -c. A \n means the file ends with a newline.
6. For a line that renders wrong, run pandoc -f commonmark -t html file.md and set its HTML next to the cat -vet view of the same line.

If a line still puzzles you, od -c file.md prints every byte of the file.

When git and editors are involved

A diff can mark a line as changed when only its line ending moved; the git guide handles that in its section on line-ending noise. For why some editors and formatters add or strip these bytes when they save, read this explainer on reformatting at save time.

Where Sashmark fits

Still being built, Sashmark has a v1 plan that includes editing in place on the page as rendered. Each save in that plan is meant as a clean Round-Trip: lines you didn't edit keep the exact bytes they had. It is in development and has not been released. There's also a planned Changes View, marking what differs from the copy you last read, no git needed. The app itself has no AI features. See the planned features, the help pages and the roadmap for where each part stands.

Keep up with the project at sashmark.com.