Central Logging
Warning
This part is outdated. It worked fine for Ubuntu but since I added monitoring stack with Grafana + Loki + Promtail, it's not needed anymore. I have included log collection in that section, so go and look here Monitoring after you have your Kubernetes cluster up and running.
One thing that can make your life easier, in case something unexpected happens, is to have logs. I have decided to use my control01
node as a central logging server using rsyslog.
On logging server
Create folder where we store the logs:
# as root
mkdir /var/log/central
Rsyslog will use TCP/UDP port 514, but you need to enable it. Edit /etc/rsyslog.conf
, and make sure these lines look like this (uncommented):
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
Next create config to tell rsyslog to put all logs in previously created folder, create /etc/rsyslog.d/central.conf
$template RemoteLogs,"/var/log/central/%HOSTNAME%.log"
*.* ?RemoteLogs
This will put all logs under /var/log/central/<hostname>.log
Last thing, and this is kind of optional, we need to tell logrotate about this, and have it rotate the logs, so you don't end up with 100+MB text files.
Create file /etc/logrotate.d/central
/var/log/central/*.log
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
invoke-rc.d rsyslog rotate >/dev/null 2>&1 || true
endscript
}
- rotate - How many rotated copies to keep before removing the oldest one.
- weekly - Rotate log every 7 days.
- missingok - If the log file is missing, go on to the next one without issuing an error message.
- notifempty - Do not rotate the log if it is empty.
- compress - Gzip the logs.
- delaycompress - Postpone compression of the previous log file to the next rotation cycle.
- sharedscripts - Because we are going to use wildcard, we need this argument, telling logrotate this setting is for multiple logs.
- postrotate - What to do after rotation is finished, in this case invoke rsyslog rotate.
Some more info about options: https://linux.die.net/man/8/logrotate
Restart rsyslog
systemctl restart rsyslog
That’s it for a server, no need to restart logrotate; that will be run via cron.
On logging clients
Now we set up nodes to send their logs to our server. Our server is called control01
, and all nodes have this entry in their /etc/hosts
file. We have did this here: OS setting
192.168.0.101 control01 control01.local
All you need to do is make sure you put following line *.* @@control01.local:514
(of course with your hostname or the IP of your logging server) at the start of /etc/rsyslog.conf
.
Warning
Do not do this on control01 ( your logging server ), that one is already logging to local files no need to also send the logs to localhost and possibly get logging loop ( Thanks Vincent for pointing this out. )
For me, including the comments, the top of that file looks like this:
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
*.* @@control01.local:514
#################
#### MODULES ####
#################
.
.
.
Nothing else, just restart rsyslog
systemctl restart rsyslog
Next, check the folder on the logging server. New logs should start appearing there in a seconds.
ubuntu@control01:/var/log/central$ ls
control01.log control02.log control03.log cube01.log cube02.log cube03.log cube04.log cube05.log cube06.log
ubuntu@control01:/var/log/central$
lnav
Just a nifty little program to watch your logs in real time, with filters and so on.
sudo apt install lnav
lnav /var/log/central/*.log