Project DANSO \U0001FA88: Document Abstract Notation for Semantic Operations

DANSO is a specialized markdown parser designed to enhance the performance of Language Models (LLMs) when processing markdown documents. It addresses the limitations of naive document processing approaches and provides a targeted solution to improve processing quality, coherence, and fidelity to the original content.

Key Features

• Shallow Parsing -- DANSO performs a shallow parsing of markdown elements, categorizing them into frontmatter, MDX/HTML tags, code blocks, and paragraphs. This allows LLMs to better understand the structure and context of the document.
• Targeted Logic -- Each type of parsed entity undergoes corresponding processing logic, ensuring that only the necessary parts are processed while preserving the structure and functionality of the document.
• Frontmatter Handling -- DANSO identifies and processes only the string values in the frontmatter, such as title and description, keeping the frontmatter structure intact.
• MDX/HTML Tag Preservation -- The parser recognizes MDX and HTML tags and processes only the inner strings, preserving the tags and their functionality.
• Code Block Extraction -- Code blocks are extracted as a whole, along with basic metadata. DANSO can optionally process code comments while keeping the code itself unmodified.
• Paragraph Context -- The raw text of each paragraph is provided as a whole, allowing LLMs to understand the context and generate more coherent outputs.

Why DANSO?

Traditional document processing approaches often face challenges when dealing with markdown documents, such as:

• Unintentional processing of MDX tags and modification of code blocks, leading to syntax errors and altered functionality.
• Limited context due to parsing at the HTML tag level, resulting in suboptimal outputs and difficulty in reordering sentences.