Txt2tags writer by farvardin · Pull Request #6551 · jgm/pandoc

farvardin · 2020-07-21T15:04:14Z

hello,

I've made a txt2tags writer, adapted from Dokuwiki writer. It works rather ok but will be improved. Could you integrate this in pandoc please?

thank you

jgm · 2020-07-21T16:45:41Z

    , writeTEI
    , writeTexinfo
    , writeTextile
+	, writeTxt2Tags


This should line up with the other entries

yes, that's strange because I got some compilation warnings into other places, which I fixed, and I believed it was clean when I've submitted the PR. I'll check this soon.

jgm · 2020-07-21T16:48:47Z

Thank you! This seems a good start, but it still needs some work before it's merge-ready.

At the very least, we need automated tests. The recommended (and easiest) approach would be to add to test/Tests/Old.hs (this requires creating a golden file test/writer.t2t).

Another thing you might try is doing round-trip tests with the t2t reader. This will help identify bugs and missing things.

tarleb · 2020-12-28T19:13:20Z

Thanks @farvardin, I think it would be great if we could get this moving again. I see you already added an "old-style" golden file test, but no code to use it. Are you still interested in advancing this? Please let us know how we can help.

I had quick read through the writer source, I think it looks good. Would you be interested in an in-depth review, or do you rather just want to get it merged soon?

farvardin · 2020-12-29T23:32:58Z

@tarleb if you could help me how to improve it, it would be great. I don't really understand the test-thing.

tarleb · 2021-01-02T09:44:15Z

Gladly.

You can enable the tests by adding the appropriate commands to test/Tests/Old.hs, lines 162-164. Check the other formats to see what's required.

It would also be great if you could take a look at the doclayout package. It is used in plain text writers to avoid overlong lines. If I understand correctly, then txt2tags supports breaking long paragraphs into shorter lines by inserting newline characters. This is what doclayout is designed to do, see also pandoc's --columns option.

LMK if you run into problems.

farvardin · 2021-01-03T20:01:07Z

@tarleb Thank you, I understand better the test process now. I've added the txt2tags writer to the file and it seems to work now. About the Lint error the missing file is in my pull request (I've never used lint or Haskell before and I'm not really a developper) and therefore I don't know why it's complaining.

I've had a look at doclayout but even with the example I can't figure out what I could do with this, sorry. I've seen all the writers are importing it, but I don't even find where they really use it in the code.

tarleb · 2021-01-06T11:30:23Z

The linting issue is caused by a missing entry in pandoc.cabal. All test files must be listed under extra-source-files; otherwise they won't be included in the source distribution, causing test failure for people building from source. I'm guilty of doing that (more than once 😔), so John added this nice check. See here.

The Doc Text type from doclayout behaves mostlyl just like Text, so most of the work should be updating the function type signatures. If you'd like to discuss this with less latency, feel free to reach out in the fp-chat slack chat.

farvardin · 2021-01-18T19:16:17Z

@tarleb thanks again for the explanation. I should have figured it out myself by looking for the reference to "writer.rtf" for example, since it can only be found in pandoc.cabal.

lint is no longer complaining, so I suppose it could be merged officially!

I've just converted an epub file into txt2tags, using the command "pandoc -f epub -t txt2tags doc.epub > doc.t2t" and it works fine!

tarleb

Those are good news, thanks! I went through the code and it looks good; I made inline comments whenever I was unsure about something.

I'm not sure if @jgm considers it essential, but it would be best if we could switch the result type of blockToTxt2Tags from Text to Doc Text so pandoc's --columns option would work better with this writer. Do you have time to give it a try?

tarleb · 2021-01-18T20:35:06Z

+                       "javascript", "latex", "lisp", "lua", "matlab", "mirc", "mpasm", "mysql", "nsis", "objc",
+                       "ocaml", "ocaml-brief", "oobas", "oracle8", "pascal", "perl", "php", "php-brief", "plsql",
+                       "python", "qbasic", "rails", "reg", "robots", "ruby", "sas", "scheme", "sdlbasic",
+                       "smalltalk", "smarty", "sql", "tcl", "", "thinbasic", "tsql", "vb", "vbnet", "vhdl",


The empty string in this list strikes me as weird. Is it intentional?

No, I'm pretty sure I've just copied this part from the DokuWiki.hs without modification. I'm not sure how this part should work. The new code in DokuWiki is quite different now:

I'll remove the empty string, though I'm sure most of this part could be removed. But if I change it myself, I might break it!

tarleb · 2021-01-18T20:47:18Z

+
+inlineToTxt2Tags _ il@(RawInline f str)
+  | f == Format "Txt2Tags" = return str
+  | f == Format "html"     = return $ "<html>" <> str <> "</html>"


Most other <html> tags are uppercase, but here they are lower case. Is there a difference?

I suppose there is no difference. Again, this part is straight from the DokuWiki.hs file:

Those are good news, thanks! I went through the code and it looks good; I made inline comments whenever I was unsure about something.

I'm not sure if @jgm considers it essential, but it would be best if we could switch the result type of blockToTxt2Tags from Text to Doc Text so pandoc's --columns option would work better with this writer. Do you have time to give it a try?

I don't know how to do this, sorry. I've just hacked the existing dokuwiki code so it's outputting valid txt2tags markup (most of the difference is with the headings)

tarleb · 2021-01-18T21:07:43Z

+  case txt of
+     [Str s] | "mailto:" `T.isPrefixOf` src -> return $ "<" <> s <> ">"
+             | escapeURI s == src -> return src
+     _  -> if isURI src


Is this if statement necessary? I'm not aware of a string with a leading / that would be recognized as valid URI.

it's just copied from Dokuwiki... most txt2tags syntax is similar to dokuwiki, so I suppose it's legit

probably the code should be refined in the future

tarleb · 2021-01-18T21:34:27Z

+                          , stUseTags = False
+                          , stBackSlashLB = False }
+
+type Txt2Tags m = ReaderT WriterEnvironment (StateT WriterState m)


Looks like we don't need the WriterState, so we could shorten this to ReaderT WriterEnvironment m. The function runTxt2Tags would then become runTxt2Tags = flip runReaderT def.

I don't know anything about haskell, so I don't understand the difference, sorry :(

This is copy-pasted together without you knowing Haskell?

I think the main stumbling block for us will be maintenance. We have to ensure that this will be maintained after it's merged. Not sure how to proceed there, I'm not too keen to up to take up an additional responsibility. @jgm, what do you think?

the txt2tags language is not evolving, for 2 reasons: it's stable for many years, and also it's possible to add custom markup with some preproc and postprocessors, so the core doesn't need changing. So the only maintenance would be to adapt the writer code in the case pandoc change something in its internal code. But it would require the same maintenance as for the dokuwiki writer.

tarleb · 2021-01-18T21:52:40Z

Oh, I forgot: I think we should also add a test/tables.t2t file, as otherwise table output would remain untested.

farvardin · 2021-01-18T22:27:05Z

Oh, I forgot: I think we should also add a test/tables.t2t file, as otherwise table output would remain untested.

done

tarleb · 2021-01-19T08:45:33Z

I think the main stumbling block for us will be maintenance. We have to ensure that this will be maintained after it's merged. Not sure how to proceed there, I'm not too keen to up to take up an additional responsibility. @jgm, what do you think?

jgm · 2021-01-19T17:57:26Z

Have you considered putting this through its paces with round trip tests?
pandoc -f native -t txt2tags -s | pandoc -f txt2tags -t native -s should ideally give you what you started from. (You could try it with testsuite.native and txt2tags.native in the test directory.)

Pandoc doesn't generally offer a round-tripping guarantee, so we usually don't put things like this in the test suite, but doing this is a good way to identify bugs.

I'd also echo @tarleb's comment that doclayout should be used. It's not used in some of the older writers, and in formats where newlines cause paragraph breaks, but I'd rather have it supported in new writers. This is a matter of

importing Text.DocLayout
changing the writer function types to produce Doc rather than Text
using doclayout's functions to create the Doc values. e.g. where t is a Text, literal t is a Doc. For breakable spaces, use space. There are also functions for nesting, hanging indents, etc. -- see doclayout documentation on hackage.

Anyway, it's good to know that you don't need to know Haskell to contribute to pandoc.

farvardin · 2021-01-20T19:56:35Z

I've done it, and it's rather faithful (I've written in the header of the writer which parts are not correctly working, but it's not very important, like definition lists)

while testing your command, I've noticed the writer is using the "txt2tags" tag, while the reader is using "t2t". I don't know which one is better.

pandoc -f html sample.html -t txt2tags -s | pandoc -f t2t -t html -s > test.html

about doclayout it's totally beyond my skills, I'm sorry.

jgm · 2021-01-21T00:13:08Z

What are you using for sample.html?

jgm · 2021-01-21T02:23:51Z

Can you try with test/testsuite.native? That exercises most of the standard pandoc features, and you wouldn't get irrelevant differences due to HTML formatting.

jgm · 2021-01-21T02:25:12Z

As for the name, we'd better stick with t2t just for consistency's sake.

farvardin · 2021-01-21T10:01:10Z

What are you using for sample.html?

it was a sample document from the txt2tags website (https://txt2tags.org/sample.html)

Can you try with test/testsuite.native? That exercises most of the standard pandoc features, and you wouldn't get irrelevant differences due to HTML formatting.

I still get several differences because pandoc creates id and adds some < p > markup in the original export (or maybe I've missed an option to avoid this) but yet it's still acceptable as result.

As for the name, we'd better stick with t2t just for consistency's sake.

I suppose it would be more logical to rename the reader part to txt2tags, because markdown and dokuwiki are written respectively "markdown" and "dokuwiki", but I understand if you prefer not breaking the previous behavior. So I've just changed all referencies of txt2tags to "t2t" in the writer part option, which will be a new addition, so it wouldn't confuse people.

jgm · 2021-01-21T16:39:21Z

@@ -20,8 +20,7 @@ Txt2Tags:  <https://www.Txt2Tags.org/>
   - Definition lists are broken
   - Tables could be improved
   - Some formats make better results than others (html is ok, md is somehow broken)


Can you explain this comment?

I don't remember, it was 6 months ago. I suppose when writing from markdown, some elements were not correctly rendered

jgm · 2021-01-21T16:39:38Z

   - Definition lists are broken
   - Tables could be improved
   - Some formats make better results than others (html is ok, md is somehow broken)
-   - code related to dokuwiki only could be removed


Is there still dokuwiki-specific code? IF so, let's remove it.

I suppose there are still some code, yes

jgm · 2021-01-21T16:41:52Z

Yes, I agree, txt2tags would make more sense, but I think it's more important not to break existing behavior, and to remain consistent.

jgm · 2021-01-21T16:47:23Z

+$pagetitle$
+$author-meta$
+$date-meta$
+
+
+


This is why you're getting all that blank space at the beginning. Do you want the 3 blank lines if there's a title? If there's no title block, would it be better to omit them? You can use conditionals like

$if(pagetitle)$ $pagetitle$ $author-meta$ $date-meta$ $endif$

jgm · 2021-01-21T16:52:29Z

I ran writer.t2t through txt2tags, and it appears that some of the syntax is invalid. (e.g. the <HTML> bits).
Have you tried this experiment? (Also with tables.t2t.)
I think this writer still needs some work before it's ready.

farvardin · 2021-01-21T18:26:14Z

I ran writer.t2t through txt2tags, and it appears that some of the syntax is invalid. (e.g. the <HTML> bits).
Have you tried this experiment? (Also with tables.t2t.)
I think this writer still needs some work before it's ready.

I've removed the html part, and now the test no longer works. There are some parts I'll never be able to fix (def lists), so I suppose I'll give up then, I'll use the generated binary for myself. There are fewer and fewer people who are still using txt2tags today, so it's not a big deal :(

jgm · 2021-01-22T18:16:41Z

You can update the tests, no?
Or by "no longer works" do you mean that the new output is invalid t2t?

jgm · 2021-01-22T18:18:08Z

Note: dokuwiki falls back to HTML for tables and lists whose contents are too complex to represent using the native syntax. I don't know if t2t offers any fallback possibility like that, so there's a question what to do if some of the things pandoc can represent can't be renedred in t2t. We could simply skip them and issue a warning, I suppose.

farvardin · 2021-01-22T21:20:29Z

You can update the tests, no?
Or by "no longer works" do you mean that the new output is invalid t2t?

I don't know, I've updated the code, and thought I've also updated the test, but I still get:

      ------------------------------------------------------------------------
      --- writer.t2t
      +++ /temp/github/pandoc/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.0.1.0/build/pandoc/pandoc --data-dir=../data --quiet testsuite.native -r native -w t2t --columns=78 --variable pandoc-version= -s
      +   1 
      +   2 
      +   3 
      +   4 
      +   5 
      +   6 
      ------------------------------------------------------------------------

so I don't know what is wrong.

farvardin · 2021-01-22T21:24:28Z

Note: dokuwiki falls back to HTML for tables and lists whose contents are too complex to represent using the native syntax. I don't know if t2t offers any fallback possibility like that, so there's a question what to do if some of the things pandoc can represent can't be renedred in t2t. We could simply skip them and issue a warning, I suppose.

it's still possible to start a line with % which means it's a comment. It won't be interpreted, but user could see it and check and correct it.

jgm · 2021-01-23T02:23:41Z

make TESTARGS=--accept will regenerate all the tests with the current output of the writer.
(Then it's very important to inspect all the changes before committing.)

…i version

- Fix inline code to use `` `` `` `` instead of `''%%...%%''` - Fix raw block/inline format name to lowercase `txt2tags` - Fix HTML raw passthrough syntax (`"""\n...\n"""` for blocks, `""..."""` for inline) - Remove unused `stBlockQuoteLevel` field; blockquotes render as plain content - Fix `escapeString` to also escape `--` and ` `` ` - Fix definition list: use `: **term**` with indented content instead of DokuWiki `* **term**` - Remove leftover DokuWiki-specific `languageNames` map and `isSimpleList` helpers - Add `Tests.Writers.Txt2Tags` with unit tests covering inline formatting, escaping, headers, code blocks, lists, definition lists, blockquotes, horizontal rules, tables, links, images, and raw blocks Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Txt2Tags writer: fix syntax, add tests

- Fix inline code to use `` `` `` `` instead of `''%%...%%''` - Fix raw block/inline format name to lowercase `txt2tags` - Fix HTML raw passthrough syntax: `"""\n...\n"""` for blocks, `""..."""` for inline - Remove unused `stBlockQuoteLevel` field; blockquotes render as plain content - Fix `escapeString` to also escape `--` and ` `` ` - Fix definition list: use `: **term**` with indented content instead of DokuWiki `* **term**` - Remove leftover DokuWiki-specific `languageNames` map and `isSimpleList` helpers - Add `Tests.Writers.Txt2Tags` with unit tests covering inline formatting, escaping, headers, code blocks, lists, definition lists, blockquotes, horizontal rules, tables, links, images, and raw blocks Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix previous commit?

Add `writeTxt2Tags` to the export list so it is accessible via the `Text.Pandoc` module, fixing a compilation error in tests.

Txt2Tags writer: export writeTxt2Tags from Text.Pandoc.Writers

fixed tests

farvardin · 2026-04-21T17:38:37Z

@jgm is the test ok now? it looks like there is no longer the previous issue with this writer.

farvardin · 2026-06-11T15:58:51Z

Closing in favor of #11699 - rebased onto current main as a single clean commit to fix the check-commit-msg-length CI failure (the history here had gotten tangled across multiple forks).

jgm reviewed Jul 21, 2020

View reviewed changes

tarleb added the new:writer label Jan 2, 2021

tarleb reviewed Jan 18, 2021

View reviewed changes

jgm reviewed Jan 21, 2021

View reviewed changes

jgm force-pushed the main branch from 3ff001b to 3da4d41 Compare December 18, 2024 23:02

jgm force-pushed the main branch from 60c147d to bfcff3e Compare May 12, 2025 00:38

jgm force-pushed the main branch from 4bb4f7f to 74351e4 Compare December 24, 2025 22:44

farvardin closed this Feb 15, 2026

farvardin force-pushed the master branch from 6879a5e to 1e4fc72 Compare February 15, 2026 20:29

farvardin added 2 commits February 15, 2026 21:50

adding again txt2tags writer after sync from upstream

31129fe

adding t2t files

d27f051

farvardin reopened this Feb 15, 2026

farvardin added 3 commits February 15, 2026 23:49

updating src/Text/Pandoc/Writers/Txt2Tags.hs to follow latest dokuwik…

4a00e28

…i version

disable ci

3d611d6

adding basic instructions

0d7b179

jgm force-pushed the main branch from ac068c2 to 54453a3 Compare March 22, 2026 09:17

farvardin and others added 13 commits April 10, 2026 00:08

Merge branch 'jgm:main' into master

0277ff6

Merge pull request #2 from luginf/master

c78a046

Txt2Tags writer: fix syntax, add tests

Merge branch 'jgm:main' into master

ddcba71

restore github ci stuff

5e8313c

remove personal note

8cc2681

Merge branch 'master' of github.com:luginf/pandoc

dc354e8

Merge pull request #3 from luginf/master

f475403

fix previous commit?

Txt2Tags writer: export writeTxt2Tags from Text.Pandoc.Writers

af4d48a

Add `writeTxt2Tags` to the export list so it is accessible via the `Text.Pandoc` module, fixing a compilation error in tests.

Merge pull request #4 from luginf/txt2tags-clean

24dcb21

Txt2Tags writer: export writeTxt2Tags from Text.Pandoc.Writers

fixed tests

9780b6b

Merge pull request #5 from luginf/txt2tags-clean

fea6755

fixed tests

Merge branch 'jgm:main' into master

9214ca2

luginf mentioned this pull request Jun 11, 2026

Add Txt2Tags writer #11699

Open

farvardin closed this Jun 11, 2026

Uh oh!

Conversation

farvardin commented Jul 21, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgm commented Jul 21, 2020

Uh oh!

tarleb commented Dec 28, 2020

Uh oh!

farvardin commented Dec 29, 2020

Uh oh!

tarleb commented Jan 2, 2021

Uh oh!

farvardin commented Jan 3, 2021

Uh oh!

tarleb commented Jan 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

farvardin commented Jan 18, 2021

Uh oh!

tarleb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tarleb Jan 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

farvardin Jan 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tarleb commented Jan 18, 2021

Uh oh!

farvardin commented Jan 18, 2021

Uh oh!

tarleb commented Jan 19, 2021

Uh oh!

jgm commented Jan 19, 2021

Uh oh!

farvardin commented Jan 20, 2021

Uh oh!

jgm commented Jan 21, 2021

Uh oh!

jgm commented Jan 21, 2021

Uh oh!

jgm commented Jan 21, 2021

Uh oh!

farvardin commented Jan 21, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgm commented Jan 21, 2021

Uh oh!

Choose a reason for hiding this comment

tarleb commented Jan 6, 2021 •

edited

Loading

tarleb left a comment •

edited

Loading

tarleb Jan 18, 2021 •

edited

Loading

farvardin Jan 19, 2021 •

edited

Loading