Hello,
I have been working in log management for a couple of years now. Across all the clients I've met, kiwi syslog had been in use for quite a while.
From a functionality perspectives, amazing things were achieved with it by operational teams.
But I am no expert at configuring kiwi syslog although somewhat familiar with it.
I am often involved in building centralized log management infrastructure and here where I always get stuck with kiwi syslog.
Perhaps there is a hidden config option that I missed ?
Implementing a centralized log management infrastructure often dictates that all logs (syslog) are to be sent to a single destination, the centralized log management.
This destination is always defined with high performance and high resilience in mind e.g. VIP, load balancers, failover systems
For any other systems that requires access to the logs, a live unmodified copy is forwarded to them.
In other words, we just built a syslog relay chain.
And with as much respect I have for your product, making kiwi syslog the first in that relay chain in a central log management system is not an option.
Nor is double-feeding from the source, building a central log management is all about having a single destination for logs where redistribution is performed there.
Whenever I walk into a department that has been running kiwi syslog for a while, they have implemented a lot of automation with it.
Obviously, they (and I agree) want to keep using it.
So the simplest solution would be to forward logs from the centralized syslog server TO the deparment kiwi syslog server.
This ways the enterprise is happy, centralized log management is in place AND that department is happy, the same interface they are using is still there.
Thats where I hit a snag.
To my knowledge, Kiwi syslog ALWAYS take as the source of the message the IP address even if it receives properly RFC3164 or RFC5424 messages containing hostnames.
Therefore, using kiwi syslog in a relay chain where its not the first one in the relay makes all source the previous IP address.
Yes spoofing can be used in the relay chain, but its not elegant, it slows down throughput quite a lot and more often than not, does get blocked by security guidelines.
Almost all advanced syslog server in the field are configurable and allow to use either the hostname contained in properly formatted syslog messages as the source host.
For improperly formatted messages, then the IP of the connected socket is taken.
Also, with some templating, its even possible in the first relay to add in the message an ORIGINATING IP prefix and get the hostname from there.
On output I saw that rsyslog supports adding such prefix.
My questions are:
1. Is there a way to configure kiwi syslog to take the source from inside the syslog message received because it was prefixed with "originating address=4.4.4.4" for example ?
2. Is there a way to configure kiwi syslog to take the source from the hostname syslog header and if it fails to take it from the connected socket ?
Without a way to do any of the above, Kiwi simply doesn't support being on the receiving end of a syslog relay chain and ends up being discarded where it still had lots of value.
Most large enterprises are really looking at central log management, and message brokers like kafka to store the logs and allow for log distribution.
Feeding specific logs from Kafka to kiwi syslog would be a tremendous help for operational teams but e.g. if all the logs have as a source a single IP address, the Kafka cluster instead of the real IP of their firewall, it makes this forwarding useless.
Presuming that I read the doc and havent missed anything, if rsyslog could support on TCP and UDP input a setting that instruct to look for ORIGINATING ADDRESS inserted it the messatge and use this IP address as the source for display, that would be amazing.
Hoping I overlooked some part of the documentation, otherwise is there anyone else who sees this an extremely important feature to support ?