Add support for dir=rtl in MathML and TeX, with automatic detection of rtl languages - #1572

This PR implements right-to-left support in MathJax by adding support for dir="rtl" attributes, and automatic detection of RTL languages in token nodes. Its handling of RTL text is not the complete Unicode bidirectional algorithm (which would require much more character-by-character data), but should suffice for most situations. Ones where it does not can be handled via mtextInheritFont: true, where the system text renderer will be used.

The main idea is to use transform: scalex(-1) to flip expressions with dir="rtl", but to apply scalex(-1) again for token nodes so that their glyphs appear in the correct orientation. This requires some extra g container nodes in SVG output in order to have an appropriate reference for the transform-origin CSS property.

Within a token node, the text is broken into separate runs of RTL and LTR text, and the RTL sections are reversed before rendering. Some characters that have reflected versions (like parentheses, brackets, and braces) have their reflected versions substituted, and those that require mirroring in the output are handled by using scalex(-1) again. This doesn't implement the complete unicode bidi algorithm, and it doesn't always handle numbers intermixed with RTL languages as that algorithm requires, for example. But is gets a wide range of situations correct.

In TeX input, \text{} and related text-mode macros can include internal mathematics, which interrupts the processing or RTL content. For example, the MathML

<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
<mtext>&#x5D0; (12, 34) &#x5D1;&#x5D2;</mtext>
</math>

produces

in Firefox's native MathML support, while

<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
  <mrow>
    <mtext>&#x5D0;&#xA0;</mtext>
    <mrow data-mjx-texclass="ORD">
      <mo stretchy="false">(</mo>
      <mn>12</mn>
      <mo>,</mo>
      <mn>34</mn>
      <mo stretchy="false">)</mo>
    </mrow>
    <mtext>&#xA0;&#x5D1;&#x5D2;</mtext>
  </mrow>
</math>

(the result of \text{\U{5D0} $(12, 34)$ \U{5D1}\U{5D2}}, where I use \U{...} in order to prevent the browser from reordering the Hebrew letters when it displays this post) is rendered as

In order to handle this better, TeX's internal math processing will mark the containing mrow with dir="rtl" when the first letter character in the text is from an RTL language. In addition, new \rtl and \ltr macros are defined to TeX to allow additional control over the rendering direction.

Input Details

In order to handle the RTL attributes, the output jax needs to walk the MathML tree to identify RTL sections for special handling. Most of the time, however, this will not be necessary, as most math is LTR, so in order to avoid the overhead, the MathML input jax keeps track of whether a dir attribute appears int he expression and marks the root element with a bidi property that the output jax uses to decide wether it needs to do the special bidi pre-processing. This is done in the MathMLCompile.ts file.

The TeX input has a similar mechanism. Any macro that sets the dir attribute also sets the inputData.bidi value on the MathItem being processed, and the TeX input jax transfers that to the root node's bidi property when it finishes the compilation.

The internalMath function in ParseUtil.ts is modified to check if the first letter is an RTL one, and keeps track of that letter in the dir variable (it does so when \U{...} is processed, or if the character is not one of the ones handled by the big if-then structure). In the end, if a letter has been found (dir is not empty), then if there are multiple MML nodes in the text (e.g., there was internal math notation in the text), and if the direction letter is an RTL one, we set dir to rtl, otherwise set it to ''). Finally, after the MathML has been turned into an mrow or mstyle, we add the dir="rtl", if needed.

In order to make the \rtl and \ltr easier to produce, I moved the GetArgumentMml() function from the html package to the TexParser.ts file, then add the new macros to the base package. The method used adds the needed dir attribute and marks the inputData.bidi value if the direction is rtl (no need to pre-process if everything is ltr).

We add the new macros to the textmacros list, so that they can be used in text mode. Because the textmacros package replaces the internalMath function, we need to handle the addition of the dir attribute to the mrow for the text that it produces, just like for the original internalMath method.

First, because the TextParser is a separate parser from the current TexParser, we copy the mathItem from the TexParser so that the macros that need to set the bidi property have access to it. Then in the saveText() method, we check for the first letter character, and set the dir property appropriately. Finally, in mml(), we add the dir attribute, if needed.

Output Details

First, look at the common.ts file. We add a new pre-filter that checks if the bidi property is set on the root element, and if so, it checks if the default for dir has been changed (by the user configuration) to rtl, and if so, makes the attribute explicit. Then we recursively look for dir attributes and mark the nodes that change the direction. For token elements, if the direction is currently reversed, we reverse the token contents to get it back to normal. We also mark RTL tokens as needing their text mixed runs reversed (e.g., Latin text before Hebrew will be reversed).

The common/Wrapper.ts is where the processing of RTL text is handled. The remapChars() function now handles the various mirroring actions that are needed. First, the code points are made into a string, and the string is split by runs of RTL text. If we are not in an RTL mode, then we may have groups of RTL text separated by whitespace or numbers, and these need to be presented in reversed order, so we look for those and reverse the groups by hand.

If we are in RTL mode, the ranges are reversed, then, for each range (LTR then RTL then LTR, etc), if reversing, we strip of non-letter characters at the beginning and ending, otherwise we use the full range if not reversing. We split the main text into unicode code points, and if we are in an RTL section (every other one), we reverse the characters. If we had opening or closing character (which is only when reversing), we move the opening ones to the end of the range, mirroring any that can be. If there were closing characters, we move them to the beginning, mirroring any that can be. Note that special characters interior to a range are not mirrored. Characters that have mirrored equivalents are set to those values, and the ones that must be mirrored by the output jax are marked by being set to the negative of their code point values.

This algorithm works well for most situations, but there are some situations that it doesn't get right. For example, \text{abc \U{5D0}\U{5D1}\U{5D1} 123} in LTR mode should swap the order of the numbers and the Hebrew groups, but this algorithm doesn't do that. For now, if you need that behavior, you should set mtextInheritFont: true so that the browser handles the bidi processing.

The common/Wrapper.ts file also include a large table of the reversible and mirrored characters from the Unicode specification. There is more there than is probably needed for MathJax, so some could be commented out to save space, if that seems appropriate.

The common wrapper for TextNode is modified to handle the negative code points for mirrored characters. The mn and mo wrappers, which override remapChars(), now call super.remapChars() to get the proper handling of the RTL characters. In addition, the mo handling of stretchy characters checks to see if the character needs to be reversed before stretching it, and stores the character in a new stretchC property (so we don't have to compute that again later).

What remains is the CHTML and SVG implementations for the reversing regions and mirrored characters.

In chtml.ts, we add CSS styles that reverse the marked sub-expressions and mirrored characters. We use a separate mjx-mirror DOM element for mirrored characters, since they may appear within a group of characters in a token element, and we don't want to mirror all the others.

In chtml/Wrapper.ts, we had a new handleBidi() method that uses the pre-processed reverse properties to determine when the data-mjx-reverse attributes need to be added.

In chtml/Wrappers/TextNode.ts, when an explicit font is used (and we create an unknown text entry), we add the dir="rtl" property, if needed. Otherwise, when processing individual characters, we check if they are to be mirrored (and convert back to a positive code point value), and create the mjx-c earlier, then wrap it in mjx-mirror if needed when it is added to the parent.

Similarlty, for SVG output, we add CSS to handle the reversing, and add a handleBidi() method. In this case, it adds a new g element with data-mjx-reverse attribute. This is so that the transform-box: content-box CSS is the correct, and transform-origin: center will be the center of that box. Because of that extra element, the place() method needs to take that into account.

The placeChar() function deals with mirroring an individual character.

The SVG TextNode.ts wrapper handles the unknown text. When the direction is set to rtl, that changes which end of the text the starting point is at, so we add a translation to push it back into the proper location (a bit awkward but works; there might be a better way).

Finally, the pattern for RTL character ranges is added to util/string.ts so that it can be used by multiple modules.