Recursive remote includes does not work in the preprocessor

[dabelenda]

I have a documentation using antora, dozens of diagrams. Only a single diagram is not rendered.
In the antora debug logs I only get a single message:

Skipping c4plantuml block. POST https://kroki.io/c4plantuml/svg - server returns an empty response or a 404 status code

Sadly in the current setup I have no more logs. To get what happens in kroki I would need to setup a local kroki instance, but I expect it to be an encoding error when POSTing the diagram source since it works for all smaller diagrams.

The kroki returns an SVG if I POST the diagram source it manually using:

curl https://kroki.io/c4plantuml/svg --data-binary '@diagram.txt'

[dabelenda]

I could provide more logs or information if somebody tells me how to do the troubleshooting to actually get that data.

[ggrossetie]

Could you please share your diagram? Or a diagram that reproduces this error?

[dabelenda]

@Mogztter here is the diagram causing the issue (anonymized but exactly the same size):

@startuml
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/v2.0.1/C4_Container.puml

title LAMP @XXXX High-Level Design

System(elk, "ELK logging stack")
System(mysql, "MySQL hosting database management system.")
System(nfs, "NAS for Website storage")

System_Boundary(k8s, "Kubernetes") {
  System(crossplane, "Crossplane Core with MySQL provider")
  System_Boundary(monitoring, "Monitoring System") {
    System(prometheus, "Prometheus")
    System(s3_exporter, "S3 Exporter")
  }

  System_Boundary(argocd, "ArgoCD deployments configuration", "https://XXXXXXXXXXXXXXXXXX/tkgi/argocd/-/tree/test/tkgi-appofapps/templates") {
    System(website_deployments, "Website Deployments Definitions")
    System(core_deployment, "Hosting Core deployment")
  }

  System_Boundary(hosting_core, "Core of LAMP hosting", "https://XXXXXXXXXXXXXXXXXX/tkgi/lamp/stack-deployment") {
    System(reverse_proxy, "Global Reverse-Proxy (httpd)", "Has access to complete storage, allow .htaccess processing. Has automatic reload of config present in separate ConfigMaps.")
    System(webdav, "WebDAV Access", "Has access to complete storage. Has automatic reload of configuration present in separate ConfigMaps.")
    System(crossplane_provider_conf, "CrossPlane MySQL provider configuration")
    System(backup, "Backup System")
    System(logstash, "Logstash", "Collects logs from all pods of the hosting stack.")
    System(servicemonitor_httpd, "Service Monitor for httpd exporter")

    Rel_D(crossplane_provider_conf, crossplane, "Configuration that will be used by websites to create their websites via Crossplane.", "CRD in k8s API")
  }

  System_Boundary(hosting_website, "Website Definition", "https://XXXXXXXXXXXXXXXXXX/tkgi/argocd/-/tree/test/lamp-site-fpm") {
    System(website_php, "PHP-FPM processing")
    System(website_revproxy_conf, "Reverse Proxy Configuration", "ConfigMap that configures the reverse-proxy.")
    System(website_webdav_conf, "WebDAV Configuration")
    System(db_request, "DataBase request for Crossplane")
    System(servicemonitor_php, "Service Monitor for php metrics")
    System(servicemonitor_backup, "Service Monitor for the backups of this website.")
  }

  System_Ext(ldap, "LDAP", "not in k8s")
  System_Ext(s3, "XXXXXXXXXX", "not in k8s")

  Rel_U(servicemonitor_httpd, prometheus, "Configures the scrapping of the httpd exporter", "CRD in k8s")
  Rel_U(servicemonitor_backup, prometheus, "Configures the scrapping of the s3 exporter", "CRD in k8s")
  Rel_U(servicemonitor_php, prometheus, "Configures the scrappig of the fpm exporter", "CRD in k8s")
  Rel_D(website_deployments, hosting_website, "Defines the deployment with all dependencies")
  Rel_D(core_deployment, hosting_core, "Defines the deployment of the Hosting Core with all dependencies")
  Rel_D(db_request, crossplane_provider_conf, "Requests a DB through the provider configuration, by name.", "CRD in k8s api")
  Rel_D(website_webdav_conf, webdav, "Configures a vhost for that hosting to let an LDAP group members write the data", "ConfigMap in k8s api")
  Rel_D(website_revproxy_conf, reverse_proxy, "Configures a vhost for that hosting to let end-users access the website", "ConfigMap in k8s api")
}

Rel_D(crossplane, mysql, "Configures MySQL databases and rights", "mysql")
Rel_D(webdav, ldap, "Authentication", "ldap")
Rel_D(website_php, nfs, "Access to only this website's data via subdir mount", "nfs")
Rel_D(reverse_proxy, nfs, "Access to all the websites data via a single share mount", "nfs")
Rel_D(webdav, nfs, "Access to all the websites data via a single share mount", "nfs")
Rel_D(backup, s3, "Periodic Push backup of the website data")
Rel_D(logstash, elk, "Pushes the logs to ELK stack", "kafka")

@enduml

[ggrossetie]

It's more portable to use !include <C4/C4_Container> instead of !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/v2.0.1/C4_Container.puml.

but I expect it to be an encoding error when POSTing the diagram source since it works for all smaller diagrams.

Diagram are not encoded when using POST requests, the content is sent in the JSON payload.

I could provide more logs or information if somebody tells me how to do the troubleshooting to actually get that data.

If you have docker installed, you can run Kroki locally using: docker run -p8000:8000 yuzutech/kroki. Then, configure kroki-server-url to point to http://localhost:8000

[dabelenda]

Oh I did not know about this include syntax. And apparently it is related to the include...
I don't know what happens exactly but when using my include from github my local kroki says that System_Boundary does not exist. If I your the <C4/C4_Container.puml> syntax it works locally.

Also I can see that actually all my graphs are sent using POST and only my the one with !include <C4/C4_Context> is using a GET... is there some handling in asciidoctor-kroki that fetches the include and bundles it in the diagram sent to the server?

[ggrossetie]

Kroki does not resolve remote includes for security reasons, that's why !include <C4/C4_Container> is recommended. PlantUML includes the latest release of PlantUML C4 so when you are using !include <C4/C4_Container> it will use this bundled version.

Also I can see that actually all my graphs are sent using POST

The extension will (be default) use a POST request if the URI length is longer than kroki-max-uri-length (default 4000) characters, otherwise it will use a GET request.
In other words, if you diagram exceeds a size threshold, it will use a POST request instead of a GET request.

[dabelenda]

All My graphs are smaller than 4000 bytes (and sent using GET) now that I changed then all to !include <C4/C4_Container> before this change they were all sent using POST. This is what I can tell from my local kroki instance's log.

This is why I asked if asciidoctor-kroki is resolving includes locally before sending them to the kroki server. There is something really weird happening.

[ggrossetie]

Yes, the preprocessor will replace !include with the actual content:

https://github.com/Mogztter/asciidoctor-kroki/blob/059510b500ad9e39733c27ba88b563736fc17ade/src/preprocess.js#L215

That's probably why the size was > 4000 bytes.
However, it does not explain why it wasn't working when using the preprocessor + POST request. Could you please share the log of your local Kroki instance when using !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/v2.0.1/C4_Container.puml?

[dabelenda]

There is nearly nothing in the logs of kroki, only that the function System_Boundary is not defined.

My guess is that readPlantUmlInclude does not recursively manage the includes (did not read the code).
The version 2.0.1 of C4_Context.puml includes the version 2.0.1 of C4.puml which in turn defines System_Boundary. So probably since it is a remote URL it is not handled by kroki and so System_Boundary is not defined.

[dabelenda]

Sorry wrong click...

[ggrossetie]

Recursive local include works but maybe recursive remote include does not 🤔