--- title: "Invisible characters in Markdown: find and test them" slug: markdown-invisible-characters excerpt: "Numbered shell steps for checking a Markdown file sit under the heading How to scan a file for invisible characters." quickAnswer: "Separate sections named Carriage returns (CRLF endings), Trailing spaces, Byte order mark (BOM), Non-breaking spaces, Zero-width spaces and Missing final newline each show relevant checking commands and sample output. How the tests were run says each test file was made with printf and names the pandoc 3.8.3 command used on macOS. How to scan a file for invisible characters lists numbered steps you can follow on a file of your own." keyTakeaways: - "The section Carriage returns (CRLF endings) shows the cat -vet view of them plus a grep that counts the lines holding one." - "Non-breaking spaces and Zero-width spaces each have a section with test files and the HTML pandoc gave for them." - "Under Byte order mark (BOM) you'll find the mark as cat -vet prints it, a grep that counts it, and a front matter test." - "For your own file, How to scan a file for invisible characters lists the commands as numbered steps." - "Trailing spaces and Missing final newline are short sections, and each links to a related post." publishedDate: 2026-10-20 dateModified: 2026-10-20 author: Bobby Huang cluster: D role: post status: published faq: - q: "Why doesn't my line starting with # turn into a heading?" a: "Both the Non-breaking spaces section and the Zero-width spaces section include a heading test file and the HTML pandoc gave for it." - q: Where can I see carriage returns in a Markdown file? a: "Carriage returns (CRLF endings) names the commands that reveal them, with sample output." - q: Does a byte order mark break YAML front matter? a: "Byte order mark (BOM) reports a front matter test run through pandoc, with and without the mark." - q: Will the scan flag accented letters too? a: "How to scan a file for invisible characters has a note on this, backed by a test file that holds an accented letter." related: - markdown-escape-characters - markdown-headings-anchor-links - markdown-bold-italic --- A heading won't render. A list item turns into a plain line that starts with a dash. You open the file, and it looks fine. The cause can be a byte you can't see. This post builds small test files holding real invisible characters in markdown, shows the shell commands that reveal each one, and reports what pandoc made of them. Why tools add these bytes is linked under When git and editors are involved. ## How the tests were run Each test file was made with `printf`, so it holds the real bytes. An invisible character can't be shown in a code block, so this post writes it two ways: as the escape you'd type in `printf`, such as `\xC2\xA0`, or as the text that `cat -vet` and `od -c` print for it. Every result below comes from one run of pandoc 3.8.3 on macOS, called with the command `pandoc -f commonmark -t html file.md`. Results from other tools aren't covered here. The commands set `LC_ALL=C`, as the tests did. In the `cat -vet` output, a `$` marks the end of each line, so anything between the last visible character and the `$` is something you couldn't see. ## Carriage returns (CRLF endings) When a file uses CRLF endings, a carriage return sits before each line feed. Under CommonMark 0.31.2, a line ending can be a line feed, a lone carriage return, or a carriage return plus a line feed. The test file `crlf.md` held a name, a company, an empty line and two list items, all with CRLF endings. Running `LC_ALL=C cat -vet crlf.md` printed this: ```text Jane Doe^M$ Doe Holdings^M$ ^M$ - one^M$ - two^M$ ``` Each `^M` is a carriage return. To count the lines holding one, run `grep -c $'\r' crlf.md`. It printed 5 for `crlf.md` and 0 for `lf.md`, which has the same text with plain line feeds. The `file` command agreed, calling `crlf.md` `ASCII text, with CRLF line terminators`. Pandoc gave the same HTML for both files: one paragraph, then a list with two items. ## Trailing spaces Trailing spaces sit in `cat -vet` output as blank room between the text and the `$`, and `grep -n ' $' file.md` lists every line whose final character is a space, with its number. In this test, two trailing spaces gave `
` in pandoc. One gave no break, so the two lines joined: pandoc returned a single paragraph holding both lines, with no `
`. The rule behind that is in the [line breaks guide](markdown-line-breaks.md). ## Byte order mark (BOM) The test files for this section began with the three bytes `\xEF\xBB\xBF`, a UTF-8 byte order mark. Running `LC_ALL=C cat -vet bom-heading.md` printed its first line like this: ```text M-oM-;M-?# Doe Holdings$ ``` The `M-oM-;M-?` part is the mark. To check for it, `LC_ALL=C grep -c $'^\xEF\xBB\xBF' file.md` counts lines that open with those bytes. It gave 1 for both test files that had the mark and 0 for `lf.md`. The `file` command described `bom-heading.md` as `Unicode text, UTF-8 (with BOM) text`. The spec text (version 0.31.2) never mentions a byte order mark. In this test, pandoc still read the heading, and `bom-heading.md` came out as `

Doe Holdings

`. Front matter came through too. Run through `pandoc -f commonmark+yaml_metadata_block -t html -s --metadata pagetitle=x bom-frontmatter.md`, the file gave `Doe Holdings` as the document title, matching the result for the same file without the mark. ## Non-breaking spaces A non-breaking space is the character U+00A0, typed in `printf` as `\xC2\xA0`. In `cat -vet` output it shows as `M-BM-` followed by a space, so the heading test printed as `#M-BM- Doe Holdings$`. CommonMark requires one or more spaces or tabs after the opening `#` run of a heading, before its text. A list marker needs the same before the item's text. A space, in the spec's terms, is U+0020, so a non-breaking space doesn't count. The tests matched. With a non-breaking space after `#`, pandoc returned a `

` paragraph, not an `

`. With one after `-`, it returned a `

` paragraph, not a list. Placed between two words, as in `Jane\xC2\xA0Doe`, the character stayed in pandoc's HTML. To find them, run `LC_ALL=C grep -n $'\xC2\xA0' file.md`. It listed line 1 of each of the three test files that held one and printed nothing for `lf.md`. ## Zero-width spaces A zero-width space is U+200B, typed in `printf` as `\xE2\x80\x8B`. In `cat -vet` output it shows as `M-bM-^@M-^K`, and `od -c` prints its three bytes as `342 200 213`. This character is neither U+0020 nor a tab, so the heading rule from Non-breaking spaces applies. With a zero-width space between `#` and the text, pandoc returned `

` instead of a heading. Bold text was the second test. The file `zwsp-emphasis.md` was built from the escape `**Jane\xE2\x80\x8B**Doe`, which puts a zero-width space just before the closing `**`. Pandoc left both pairs of asterisks in the output as literal characters. The control file, `**Jane**Doe` with nothing hidden, gave `JaneDoe`. This post doesn't trace that result to a line in the spec; it's what pandoc gave in this test. Between two words, as in `Jane\xE2\x80\x8BDoe`, pandoc kept the character, and `od -c` of the HTML showed the same three bytes. Search for it with `LC_ALL=C grep -n $'\xE2\x80\x8B' file.md`. ## Missing final newline The last byte tells you. Running `tail -c 1 file.md | od -c` printed `s` for `nofinal.md` and `\n` for `lf.md`. In `cat -vet` output, the last line of `nofinal.md` had no closing `$`. The spec allows the last line to stop at end of file without a line ending, and pandoc rendered `nofinal.md` as a single paragraph holding both lines. For saves that should leave a missing final newline untouched, read [the round-trip save explainer](what-is-a-byte-for-byte-round-trip-save.md). ## How to scan a file for invisible characters Run these on the file that misbehaves. Swap `file.md` for its name. 1. Run `LC_ALL=C cat -vet file.md` and read the line ends. A `^M` before the `$` means a carriage return, and a gap before the `$` means trailing spaces. 2. Run `LC_ALL=C grep -n '[^[:print:][:space:]]' file.md` to list lines holding bytes that the C locale counts as neither printable nor space characters. In the tests it caught the non-breaking space, the zero-width space and the byte order mark. It also flags real text: it listed `accent.md`, whose only unusual character is an e with an acute accent. A hit means "look here." 3. Count the lines holding a carriage return with `grep -c $'\r' file.md` and list trailing spaces with `grep -n ' $' file.md`. In a separate test, the grep from step 2 printed nothing for `crlf.md` or for `trail2.md`, the test file with two trailing spaces, so these two need their own check. 4. Pin down a hit from step 2 with a targeted search: `LC_ALL=C grep -n $'\xC2\xA0' file.md` for non-breaking spaces, `LC_ALL=C grep -n $'\xE2\x80\x8B' file.md` for zero-width spaces, and `LC_ALL=C grep -c $'^\xEF\xBB\xBF' file.md` for a byte order mark. 5. Check the final byte with `tail -c 1 file.md | od -c`. A `\n` means the file ends with a newline. 6. For a line that renders wrong, run `pandoc -f commonmark -t html file.md` and set its HTML next to the `cat -vet` view of the same line. If a line still puzzles you, `od -c file.md` prints every byte of the file. ## When git and editors are involved A diff can mark a line as changed when only its line ending moved; the git guide handles that in [its section on line-ending noise](how-to-diff-markdown-files-with-git.md). For why some editors and formatters add or strip these bytes when they save, read [this explainer on reformatting at save time](why-editors-reformat-markdown-on-save.md). ## Where Sashmark fits Still being built, Sashmark has a v1 plan that includes editing in place on the page as rendered. Each save in that plan is meant as a clean Round-Trip: lines you didn't edit keep the exact bytes they had. It is in development and has not been released. There's also a planned Changes View, marking what differs from the copy you last read, no git needed. The app itself has no AI features. See the [planned features](/features), the [help pages](/help) and the [roadmap](/roadmap) for where each part stands. Keep up with the project at [sashmark.com](https://www.sashmark.com/).