pandoc 3.6

[jgm]

Click to expand changelog

• Add mdoc as input format (Evan Silberman). This change introduces a reader for mdoc, a roff-derived semantic markup language for manual pages. This reader has been developed almost exclusively against mandoc’s documentation and implementation of mdoc as a reference, and the real-world manual pages tested against are those from the OpenBSD base system. Of ~3500 manuals in mdoc format shipped with a fresh OpenBSD install, 17 cause the mdoc reader to exit with a parse error. Any further chasing of edge cases is deferred to future work.

• New module: Text.Pandoc.Readers.Mdoc, exporting readMdoc [API change].

• Issue warnings for duplicate YAML metadata keys (Warn on duplicate or conflicting YAML metadata key-values within a metadata block #10312).

• Ensure that --sandbox affects --embed-resources. Previously it did not (contrary to what was implied by the manual), which means that an image with URL /etc/passwd would leak an encoded version of that file to HTML output with --self-contained or --embed-resources, even if --sandbox was used. Thanks to Samuel Mortenson for pointing out the issue.

• Text.Pandoc.App.OutputSettings: add sandbox' function. This computes the sandboxed files from Opt and avoids code repetition.

• Docx reader:

  • Parse index references as empty spans with attributes (Word Index Entry Render as the HTML to WORD #10171). Attributes included are entry, and optionally bold, italic, yomi, see.
  • Don’t create multiple paragraphs for title or subtitle (DOCX Reader: Is it correct behaviour / best solution to use the last title element present in file? #10359). If there are multiple paragraphs with Title or Subtitle style, use only the first for metadata.
  • Handle case where Zotero itemData has different id from the citationItem id. In this case we use the citationItemId in the bibliography as well, overriding the referenceId in the itemData (docx+citations from zotero: different ids used for in-text and CSL-YAML #10366).

• LaTeX reader:

  • Put parsed minipage in specially marked Div (Minipages ignored when converting from LaTeX #10266).

• HTML reader:

  • Parse footnotes defined by dpub-aria roles (HTML reader: parse footnotes using dpub-aria roles #5294).

• MediaWiki reader:

  • Fix indented tables with caption (mediawiki reader fails on tables when the line following a table caption starts with spaces #10390).
  • Fix parsing of col/rowspan (Handle col/rowspans in mediawiki tables #6992).

• Typst reader:

  • Avoid generating empty paragraphs.
  • Support underparen, overparen.
  • Fix #quote attribution. If attribution is not present, don’t print the -- (Typst: different typst output when writing to .typ file or to pdf with the typst engine #10320).
  • Fix typo in unicode code point for em dash (see Typst: different typst output when writing to .typ file or to pdf with the typst engine #10320).

• Commonmark reader:

  • implicit_figures should check for empty caption and not produce an implicit figure in this case (commonmark_x reader with implicit_figures isn't checking whether the caption is empty? #10429).

• RST reader:

  • Use a new one-pass parsing strategy. Instead of having an initial pass where we collect reference definitions, we create links with target ##SUBST##something or ##REF##something or ##NOTE##something, and resolve these in a pass over the parsed AST. This allows us to handle link references that are not at the top level (Error parsing reStructuredText hyperlink inside directive #10281).
  • Ignore newlines in URL in explicit link (Error parsing reStructuredText hyperlink split into two lines #10279).
  • Handle block level substitutions.
  • Support :file: on raw directive (Fix RST raw directive to allow file includes? #8584).
  • Implement option lists (RST: Option Lists not supported #10318).
  • Avoid putting metadata in Para (reStructuredText metadata is wrapped in <p> #7766). Create MetaInlines when possible, just as with markdown input. MetaBlocks is still used when there are multiple paragraphs or non-paragraph content. This change also affects field lists.
  • Fix linked substitutions (reStructuredText Replacement Text in References Doesn't Work #6588). E.g. |Python|_.
  • Support inline anchors (reStructuredText inline internal targets not supported #9196).
  • Explicit links define references (rst references with one underscore g <www>_ not remembered #5081). For example, Go to `g`_ `g <www.example.com>`_. should produce two links to www.example.com.

• EPUB writer:

  • Use standardized filename for cover image instead of the original name (EPUB cover image paths are not URL-encoded (causing spaces in filename to break cover images) #10404). This avoids problems with e.g. filenames containing spaces.

• Markdown writer:

  • Issue INFO warning when not rendering table, e.g., when raw_html is disabled and the table can’t be fit into a supported markdown table format (commonmark/gfm: -raw_html turns tables with <h3> tag into [TABLE]. #10407).
  • Respect empty LineBlock lines in plain output (Evan Silberman). The plain writer behaved as a markdown variant with Ext_line_blocks turned off, and so empty lines in a line block would get eliminated.

• LaTeX writer:

  Ensure that beamer footnotes go on frame, not column (Beamer+columns+footnotes can be improved #5769).

• HTML writer:

  • Unwrap empty incremental divs (Incremental / Non Incremental Divs around list leave an empty <div> in HTML slides #10328, Albert Krewinkel). Divs are unwrapped if the only purpose of the div seems to be to control whether lists are presented incrementally on slides.

• Typst writer:

  • Make template sensitive to a page-numbering variable (Add page numbering (to remove it) to typst template #10370). This can be set to an empty string (or, in metadata, to false) for no page numbers.
  • Make smart extension work (Typst Writer should not escape single and double quotes #10271). If smart is not enabled, a command in the default template will disable smartquote substitutions. When smart is enabled, render curly apostrophes as straight and escape straight apostrophes. When smart is disabled, render curly apostrophes as curly and don’t escape straight apostrophes. Similarly for quotes, em and en dashes. This should give more idiomatic typst output, with fewer unnecessary escapes.

• ANSI writer:

  • Respect empty LineBlock lines (Evan Silberman).

• JATS writer:

  • Correct spelling of suppress attribute (JATS: correct suppress attribute #10350, Andreas Deininger).

• Typst template:

  • Remove definitions.typst partial.
  • Remove unnecessary definition of endnote.
  • Incorporate the one remaining definition into default.typst.
  • Use typst 0.12 code for two column layout (typst 0.12 two column layout changes #10294, Luis Rivera).
  • Note: the new templates presuppose typst 0.12; if you try to use an earlier version of typst, an error will be raised.

• LaTeX/Beamer template:

  • Split fonts.latex partial into two parts: fonts.latex and font-settings.latex.
  • In beamer template, load beamer theme between fonts.latex and font-settings.latex. This allows a theme (such as metropolis) to set its own default font, while still allowing the user to override it. This fixes a regression in pandoc 3.5 (Font change on beamer slides when building with pandoc 3.5 #10297).
  • Note: Users who have custom templates based on pandoc 3.5 templates will need to add font-settings.latex() after fonts.latex() in the latex template. In a beamer template, the beamer theme-setting code needs to be moved between these two partials.

• ConTeXt template: Ensure that font names don’t wrap (unwanted newline #10305).

• epub.css: remove background-color (Update epub.css #10264, Suraj Patil). With this greyish background color, epubs look bad on a Kindle (The background of a epub book on kindle is a grey color that makes it annoying to read the book #10263).

• Text.Pandoc.ImageSize: add WebP support (Evan Silberman, Add WebP support to ImageSize #10397). Add Webp constructor on ImageType [API change].

• Text.Pandoc.Readers.Roff and a new unexported module Text.Pandoc.Readers.Roff.Escape: parameterize Roff escaping (Evan Silberman) [API change]. This allows code to be reused between the mdoc and man readers, despite the differing Token types.

• Text.Pandoc.PDF:

  • PDF via LaTeX: always do max runs if toc is present (Regression: Wrong page numbering of multi-page TOC #10308). The old method (checking to see if toc hash had changed) is not completely reliable.
  • Use .source extension, not .html, in toPdfViaTempFile (Windows: infinite loop creating empty temp files when file deletion prohibited by ACL #10314).

• Text.Pandoc.Logging: add YamlWarning constructor to LogMessage [API change] (Warn on duplicate or conflicting YAML metadata key-values within a metadata block #10312).

• Text.Pandoc.Format: remove duplicate typst entry (Text.Pandoc.Format: remove duplicate typst entry #10388, Caleb Mclennan).

• Fix a typo in the ua.yaml localization for ‘See’ (Jens).

• Lua subsystem (Albert Krewinkel):

  • Remove prefixes from Lua type names (Types reported by pandoc.utils.types are inconsistent #8574). Lua type names were inconsistent with regard to the use of prefixes; all prefixes are removed now, and Lua types now have the same name as the Haskell types. The use of app-specific prefixes is suggested by the Lua manual to avoid collisions. However, this shouldn’t be a problem with pandoc, as it cannot be used as a Lua package.

• doc/libraries.md: Add newly developed Haskell packages. Sort list alphabetically (Albert Krewinkel).

• doc/lua-filters.md: document pandoc.List:iter method (Albert Krewinkel). List objects have a new function iter that returns an iterator function that returns the next list item on each call.

• MANUAL.txt:

  • Clarify what the example of YAML EPUB metadata shows (feature request, web site manual. Is this an example of YAML or epub? #10405).
  • Fix typo in template syntax (MANUAL.txt: fix typo in template syntax #10265, Pascal Wagler).
  • Update manual with information on openxml template (`docx` template documentation #10273).
  • Clarify that --variable can only assign string values (Issue setting a variable of type Dictionary from the command line #10298).

• Fix comments in TEI writer referring to DocBook (Fix comments in TEI writer referring to DocBook #10430, Evan Silberman).

• Fix several typos in documentation (Fix typos #10349, Andreas Deininger).

• Allow Diff 1.0.

• Add font-settings.latex partial to pandoc.cabal (Pandoc nightly build is missing font-settings.latex #10379).

• Bump upper bound for data-default.

• Use latest typst, texmath, pandoc-lua-marshal, commonmark-pandoc, commonmark-extensions, skylighting, skylighting-format-blaze-html.

---

This discussion was created from the release pandoc 3.6.

[jgm]

I'm pleased to announce the release of pandoc 3.6,
available in the usual places:

Binary packages & changelog:
https://github.com/jgm/pandoc/releases/tag/3.6

Source & API documentation:
http://hackage.haskell.org/package/pandoc-3.6

Some changes of particular note (inspect changelog for the full list):

* mdoc is now supported as an output format.
* --sandbox now affects --embed-resources; previously --embed-resources
  or --self-contained could leak data from the file system even if used
  with --sandox.
* Support for index entries in docx reader.
* Many improvements to RST parsing.
* Typst, LaTeX, and ConTeXt template changes.
* Prefixes removed form Lua type names.

API changes:

* New module Text.Pandoc.Readers.Mdoc, exporting readMdoc.
* In Text.Pandoc.ImageSize, new Webp constructor on ImageType.
* Text.Pandoc.Logging: new YamlWarning constructor on LogMessage.
* New module Text.Pandoc.Readers.Roff.Escape exporting escape,
  escapeArg, RoffLikeLexer(..).

Thanks to all who contributed, especially Evan Silberman
for the new mdoc reader, and a special welcome to new contributors
Andreas Deininger, Luis Rivera and Suraj Patil.

[benz0li]

Unofficial and untested Linux/RISC-V 64-bit (linux/riscv64) release assets are available at https://gitlab.b-data.ch/jgm/pandoc/-/releases/3.6.

[anniekatz]

Hi there. Upon installing this version of pandoc for Windows x64 MSI installer via the release channel, my antivirus software flagged an octet stream as a "known Trojan" and removed the installer. After reinstalling, the pandoc executable itself was flagged as "ML/Augur trojan" and removed. I know there have been false positives in the past. Has anyone else had an issue with this?

[jgm]

That's not good! (But it is probably not something we can do anything about.)
What antivirus software are you using?

[jgm]

Reporting this to your antivirus software manufacturer as a false positive might be helpful for others.

[Philamericus]

Hi. Both ESET and Windows Defender blocked it. :(

[jgm]

I wish I knew what to do about this!