Rsyslog Omclickhouse Custom Template Bug: A Deep Dive

Let's dive into a puzzling issue encountered while trying to use a custom template with omclickhouse in Rsyslog. It appears the Rsyslog configuration parser is throwing errors, and we need to figure out why. This article aims to dissect the problem, explore potential causes, and offer solutions, ensuring your logs flow smoothly into ClickHouse.

The Problem: Custom Template Woes

The core issue revolves around using a custom template to format messages for omclickhouse. When a custom template is defined and referenced in the Rsyslog configuration, the parser flags an error, specifically complaining about the absence of SQL or stdSQL options, even when stdSQL is explicitly enabled.

Here's the problematic configuration snippet:

template(name="CustomClickHouseFmt" stdSQL="on" type="string" string="
"INSERT INTO rsyslog.SystemEvents (severity, facility, timestamp, hostname, fromhost_ip, tag, message)
VALUES (%syslogseverity%, %syslogfacility%, '%timereported:::date-unixtimestamp%', '%hostname%', '%fromhost-ip%', '%syslogtag:1:32%', '%msg%')"
")

action(
   type="omclickhouse"
   server="127.0.0.1"
   port="8123"
   user="writer"
   pwd="*********"
   usehttps="off"
   bulkmode="on"
   template="CustomClickHouseFmt"
   #stdSQL="on"
)

When running rsyslogd -N1 to validate the configuration, the following error arises:

rsyslogd: version 8.2510.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: Action disabled. To use this action, you have to specify the SQL or stdSQL option in your template!  [v8.2510.0 try https://www.rsyslog.com/e/2008 ]
rsyslogd: error during parsing file /etc/rsyslog.d/99-clickhouse.conf, on or before line 39: errors occurred in file '/etc/rsyslog.d/99-clickhouse.conf' around line 39 [v8.2510.0 try https://www.rsyslog.com/e/2207 ]

Enabling stdSQL="on" within the action parameters doesn't resolve the issue. The error message persists, suggesting a deeper problem. The official omclickhouse documentation doesn't list stdSQL as a valid action parameter, adding to the confusion. This could indicate a bug or a misunderstanding of how custom templates are intended to function with omclickhouse.

Environment Details

To provide a clear picture, here are the environmental specifics:

• Rsyslog Version: 8.2510.0 (aka 2025.10)
• Platform: Ubuntu 24.04.3 LTS
• Deployment Method: adiscon launchpad binaries

Potential Causes and Solutions

Let's explore the potential causes and how we might address them.

1. Bug in Rsyslog Version

• Explanation: It's possible there's a bug in the specific Rsyslog version being used (8.2510.0). Bugs can creep into any software, and this could be one that affects custom template parsing for omclickhouse.
• Solution: The first step is to check if this bug is known. Review the Rsyslog changelogs and issue trackers (check the github issue tracker) for version 8.2510.0 and later versions. If it's a known bug, upgrading to a patched version is the ideal solution. If an upgrade isn't immediately feasible, look for any documented workarounds.

2. Incorrect Template Syntax

• Explanation: While the template looks correct at first glance, subtle syntax errors can sometimes evade immediate detection. Even a minor deviation from the expected format can cause the parser to choke.
• Solution: Double-check the template syntax with extreme scrutiny. Ensure that all quotes are properly escaped, that there are no missing or extra characters, and that the variable names (%syslogseverity%, etc.) are spelled correctly and are valid Rsyslog properties. Try simplifying the template to a minimal working example (e.g., just inserting a fixed string) to see if the basic template structure is being parsed correctly. Then, gradually add complexity back in until the error reappears, helping to pinpoint the exact location of the syntax error.

3. Module Compatibility Issue

• Explanation: There might be an incompatibility between the omclickhouse module and the core Rsyslog version, especially concerning custom template handling. Modules sometimes have dependencies or expectations regarding the core system that, if unmet, can lead to unexpected behavior.
• Solution: Ensure that the omclickhouse module is the correct version for your Rsyslog installation. Check the module's documentation for any compatibility notes or required dependencies. Try reinstalling the omclickhouse module to ensure that all its files are correctly placed and configured.

4. Unexpected Interaction with stdSQL

• Explanation: The error message's insistence on stdSQL might indicate an internal check that's not behaving as expected. Even though the documentation doesn't explicitly mention stdSQL as an action parameter for custom templates, the code might still be evaluating it somehow.
• Solution: Even though it didn't work initially, try explicitly setting `stdSQL=