How I Built a Fast HTML Parser Using Regex and TypeScript

By Elson Correia
Photo by Jay Zhang on Unsplash

Photo by Jay Zhang on Unsplash

While working on a side project, the need to parse HTML came up, and to save time, I tried the fastest HTML parsers I could find. After fighting and trying to hack them, I realized I needed a custom or super customizable one to fit all project needs. Unfortunately, I had no luck. So, I created one.

I thought it was a simple enough thing to do…

The Motivation

For my specific project, I needed something fast, which was easy to find, but I needed to be customizable enough. However, everything I found mainly failed in two areas:

  • They offered no way to tap into nodes while they were being parsed — That’s something I desperately needed.
  • They offered no ability to specify custom API for the parsed result, forcing me to learn something new they came up with or remain stuck with really non-performant APIs. — This ability would allow me to adapt the parser to the project, not vice versa.

Some offer customizations that often come with performance loss — I wanted both performance and customization. Additionally, I needed it to work in any JavaScript runtime environment, and because I was going to use it in a client library, it needed to be light.

Here is a list of best parsers I tried: html-parser, htmljs-parser (good callback options), html-dom-parser, html5parser, cheerio (really good offering more than just a parsing solution), parse5 (as good as cheerio), htmlparser2, htmlparser, node-html-parser (really fast)

Is my parser better? (disclaimer)

I am not claiming my parser is better than any of the above or that everyone should use it instead. I created a parser specifically to solve a problem I was having. I don’t believe in single solutions, and one should always try to find the best tool for the job— or create one if necessary.

The Result

  • It is a ~40kB package, ~4kB minified, and <2kB CDN size when used in the browser.
  • Using the htmlparser-benchmark package, it benchmarks around 1.68957 ms/file ± 1.11577 by default, and using third-party API like jsDOM, it benchmarks around 26.3847 ms/file ± 18.8658 .
  • Simple to use. It's just a function with two arguments, and one of them is optional. If you know DOM API, you already understand the result. parse(SOME_HTML_STRING)
  • Highly customizable: You decide which and how the API looks for the parsed result. It can adapt to any project.

This cocktail of features is all I needed. It is small enough to include as a dependency for something I’ll use in the browser yet powerful, so I can build anything on top, not to mention how easy it is to learn and use.

parse(SOME_HTML_STRING, (node) => {
  // handle node here
})

This makes it easy to perform things on the node, like collecting nodes or performing changes, but the best part is that I could also provide a DOM Document object to be used, even a custom one if I want to.

parse(SOME_HTML_STRING, document) // pass the document from env

parse(SOME_HTML_STRING, MyCustomDOMDocument) // or a custom one

By default, it uses a custom lite DOM Document-like that I created because the DOM API was a little slow, affecting my parser performance from 1.8 to 27.3 milliseconds average parse duration per file.

As a matter of fact, for my project, I created a new Document API specific to my project, which I will happily tell you about in a different post.

That’s all the API…Nothing else!

Just a parse function you can call with an HTML string and optional callback, Document, or custom Document object of your choosing. It does not get simpler than that.

There is no need to provide options for how individual things get parsed. The parse is optimal for anything, including comments, script, and SVG tags.

The Algorithm

I used a Regex and a while loop for this, and if you ask any pro about Regex and HTML, they will tell you that it is insane to attempt this because HTML has so many edge cases.

My early pattern quickly taught me that the goal is not to try to catch all the edge cases but to identify the known ones and let the rest be a “wild guess.” So what I did was create a pattern that would look for:

  • : an empty open tag
  • : an empty self-closing tag
  • : a closing tag
  • : an open tag with attributes where ATTRIBUTES is just any text in that place.
  • : a self-closing tag with attributes
  • : a comment

I ended up with this pattern:

/<!--([^]*?(?=-->))-->|<(\/|!)?([a-z][a-z0-9-]*)\s*([^>]*?)(\/?)>/gi

As you can see, I dont match the content of those tags or specific patterns for attributes. Everything in between is treated as text, and those are grabbed in between matches.

I keep a stack to track the parent element, so I know where to push Nodes and then pop them from a stack when I match the closing tag. Let me break it down:

  • Whenever I match an open or a self-closing tag, I create an element and process attributes if there are any. I push the element to a stack as long as it is not a self-closing tag.
  • If I match a closing tag, I check the last element in the stack, and if the tag name is the same, I pop it from the stack, but first, I collect all the text between the closing tag match index and the end of the open tag index. Any text is added as a text node.
  • Match all comments and append them to the last element in the stack (the parent).
  • If an open script tag is matched, start a search for the closing tag by creating a separate stack and a loop to check possible labels inside the script tag (totally possible) and discard them as just part of the text body of the script tag. Once the closing tag is found, collect everything in between as the script tag body.

Now, let me show you a snippet of what it looks like. First, I created a parse a function that takes a string, and inside, I have my pattern and variables to track the matches and the last index of the matches with a stack array to track parent-child relationships.

const parse = (markup: string) => {
    const pattern =
        /<!--([^]*?(?=-->))-->|<(\/|!)?([a-z][a-z0-9-]*)\s*([^>]*?)(\/?)>/gi
    const stack: Array<Node | DocumentFragment> = [document.createDocumentFragment()]
    let match: RegExpExecArray | null = null
    let lastIndex = 0

    // all following code here
}

Notice that the stack starts with a document fragment, and if I close all tags correctly (depending on whether they all got closed in the HTML string), it should be the only one in the stack in the end.

Next, I added my while loop, where I match and assign the match variable with the result looping while there is a result.

...

while ((match = pattern.exec(markup)) !== null) {
    const [
        ,
        comment,
        bangOrClosingSlash,
        tagName,
        attributes,
        selfClosingSlash,
    ] = match;
}

Inside the circle, I decompose my match and cleverly name the parts of the pattern in which the match happened. With this, the first thing I do is ignore all <!doctype> tag, but track the lastIndex

// ignore !doctype
if (bangOrClosingSlash === '!') {
    lastIndex = pattern.lastIndex
    continue
}

I ignored it because this marks the document itself, and if you can’t create such a node.

I continue collecting the last item in the stack…, the parent node.

const stackLastItem = stack.at(-1)

…and grabbing any possible text before the tag and updating the lastIndex

// collect any text before tag
if (match.index >= lastIndex + 1) {
    const text = markup.slice(lastIndex, match.index)
    const node = document.createTextNode(text)
    stackLastItem.appendChild(node)
}

lastIndex = pattern.lastIndex

I then check if the match happened on a comment and collect it as well, but break to the next game if thats is the case.

 if (comment) {
      const node = document.createComment(comment)
      stackLastItem.appendChild(node)
      continue
  }

I then check if the matched tag is a closing tag and remove the element from the stack. This signals that the node and its descendants are all collected.

if (bangOrClosingSlash) {
    // tag was closed
    if (new RegExp(tagName, 'i').test(stackLastItem?.tagName)) {
        stack.pop()
    }
    continue
}

Otherwise, if it is a self-closing tag, I create it and append it to its parent (the last item in the stack).

if (selfClosingTag) {
    const node = document.createElement(tagName.toLowerCase())

    setAttributes(node, attributes) // handle all attributes
    stackLastItem.appendChild(node)
    continue
}

The setAttributes function here is a simple function that takes all the strings in the tag from the tag name to the > symbol and creates attributes. It looks something like this:

const setAttributes = (node: Element, attributes: string) => {
    attributes = attributes?.trim()
    if (attributes) {
        const attrPattern =
            /([a-z][\w-.:]*)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|(\S+)))?/gi
        let match: RegExpExecArray | null = null

        while ((match = attrPattern.exec(attributes))) {
            const name = match[1]
            const value =
                match[2] ||
                match[3] ||
                match[4] ||
                (new RegExp(`^${match[1]}\\s*=`).test(match[0]) ? '' : null)

            node.setAttribute(name, value ?? '')
        }
    }
}

Here, you can see my second pattern for key-value attributes and another while loop.

I finish the loop body by appending the element to its parent and pushing it to the stack.

stackLastItem.appendChild(node)
stack.push(node)

But after the loop, I need to collect any text that could come after all the tags and append it to the document fragment so I can return it.

if (lastIndex < markup.length) {
    const text = markup.slice(lastIndex)
    const node = doc.createTextNode(text)
    stack[0]appendChild(node)
}

return stack[0]

Additional edge cases

HTML has a lot of edge cases, and it is a complex markup language. It is easy for developers to mess up syntax, resulting in undesirable results. There were some additional cases I came across which needed to be addressed:

  • Script tags: whenever an open-close script tag is matched, there should be an internal loop to find the closing script tag. This can be done by creating a small stack and matching possible tags until the stack is empty. This is because any HTML string inside the script tag is not to be rendered, and my pattern matches those.
<script>
  const html = `<p>some text</p><script src="pHandler"/>`;
</script>
  • Native self-closing tags: HTML already has certain tags that are considered self-closing tags. When you come across these, there is no need to try to find content. I keep a pattern for those, and I change the selfClosingTag check to see if it's a known native self-closing tag.
const selfClosingTag = selfClosingTags.test(tagName) ||
  selfClosingSlash === '/';
  • SVG and ELEMENT namespaces: We cannot use document.createElement method to create SVG tags. We must specify the SVG namespace to make the correct element. Therefore, use the document.createElementNS where we can specify the namespace; this happens whenever an SVG tag is matched. Everything after should have an SVG namespace until the SVG closing tag is found.
const ns = /svg/i.test(tagName)
  ? NSURI.SVG
  : /html/i.test(tagName)
  ? NSURI.HTML
  : stackLastItem.namespaceURI;

const node = doc.createElementNS(ns, tagName)
  • Boolean attributes: HTML has native boolean attributes. Initially, I had to handle them separately, but then I realized that I could expect them as no value attributes and adjust my attribute regex pattern.
<input type="radio" checked/>
  • DOCTYPE: Creating a DOCTYPE tag has to do with the document, and because the root is a Document fragment, ignoring this tag was the best way. It is easy to grab the result and add it to a document, therefore creating such a doctype.
document.appendChild(parse(SOME_HTML, document));
  • Conditional comments: Comments can get tricky, especially when you want to cover conditional comments or commented-out HTML. Luckily, I could find a pattern I could rely on, and no extra logic was needed.

Parsed result API

I created a Document-like API with only these APIs:

interface DocumentLike {
    createTextNode: (nodeValue: string) => TextLike;
    createComment: (nodeValue: string) => CommentLike;
    createDocumentFragment: () => DocumentFragmentLike;
    createElementNS: (ns: string, nodeName: string) => ElementLike;
}

It looks and behaves like DOM nodes to an extent, but it only has the necessary, and it's built for performance. That’s how I was able to pull some impressive benchmark numbers. Here is what the rest of the API looks like:

interface NodeLike {
    readonly nodeType: number;
    readonly nodeName: string;
    nodeValue: string;
}

interface CommentLike extends NodeLike {}

interface TextLike extends NodeLike {}

interface ElementLike extends NodeLike {
    readonly tagName: string;
    outerHTML: string; // not needed during parsing
    textContent: string;
    readonly childNodes: Array<NodeLike>;
    readonly children: Array<ElementLike>;
    readonly attributes: NamedNodeMapLike;
    setAttribute: (name: string, value?: string) => void;
    appendChild: (node: NodeLike | ElementLike | DocumentFragmentLike) => void;
}

interface DocumentFragmentLike extends Omit<ElementLike, 'outerHTML', 'setAttribute', 'attributes'> {
}

It only contains what the parser needs, making it easily interchangeable with the Document object from the browser or package like jsdom. With this, you can create your version to control the result to fit your needs.

…and this is the type of control I needed for my project.

Next Steps

As I said, I built this for a different project and will soon write about those projects as well. A project like this is never done, and I always look for ways to improve it.

You can try it for yourself and tell me about your experience and findings. Any improvements and suggestions are welcomed.

@beforesemicolon/html-parser *HTML parser for NodeJs and Browsers. Latest version: 0.2.3, last published: 2 days ago. Start using…*www.npmjs.com

Take away

As a front-end engineer, a project like this gives me a different perspective on how things work, allowing me to understand and optimize what I do even more. I learned much and even more with all the projects built on this.

Follow and subscribe to learn more about my projects…

Illustration for “How I Built a Fast HTML Parser Using Regex and TypeScript”

YouTube Channel: Before Semicolon Website: beforesemicolon.com

Share this article