Logging of Pgpool-II on Kubernetes

-

logging

Logging is an important topic and particularly useful for troubleshooting, debugging and monitoring. Many applications have their own built-in logging mechanism. Pgpool-II logging mechanism is similar to PostgreSQL logging mechanism. Pgpool-II log management system supports two ways for logging messages (stderr and syslog) and the logging collector process collects the logs sent to stderr and redirects them into log files. However, how can we manage the logging system on Kubernetes? In this blog, I will describe how to manage Pgpool-II container logs on Kubernetes.

Logging on Kubernetes 

Kubernetes supports the functionality to view the container logs by using “kubectl logs” command. 

kubectl logs <pod name>

However, logging on Kubernetes is complicated because the logs will be deleted when containers are terminated or recreated.

Fluentd helps you to collect container logs and send the logs to desired destinations such as Amazon S3, MySQL, MongoDB, etc.

How to use Fluentd on Kubernetes?

Kubernetes logs the stdout and stderr streams of each container in a pod to a file in JSON format. The log file is located in /var/log/containers directory of the node where the pod is running, and the filename contains the pod name, the namespace and the container name.

To read the containerized application log files and expose to other destinations, we need a node-level logging agent that runs on each node.

Fluentd is an open-source log collector that allows you to collect and analyze various application logs and send them to your desired data storage destinations. In this article we use Fluentd as the logging agent. 

Because Fluentd must run on each node, we deploy it as a DaemonSet on Kubernetes. In the next sections, I will describe how to deploy and configure Fluentd on Kubernetes to manage Pgpool-II container logs.

Managing Pgpool-II logs using Fluentd

Pgpool-II container writes its logs to stderr streams.
The pgpool container log files will be stored on each node like:

$ ll /var/log/containers/ | grep pgpool
lrwxrwxrwx 1 root root  94  3月 31 10:27 pgpool-565c85fbd-9sddg_default_pgpool-8e4f9154eac929f8b2fdfa4e59faa934da41374efc28b5c047b42158079b4661.log -> /var/log/pods/default_pgpool-565c85fbd-9sddg_e23dc5a4-7677-4fd8-972c-7df5016f780b/pgpool/0.log
lrwxrwxrwx 1 root root  94  3月 31 10:27 pgpool-565c85fbd-kbpzq_default_pgpool-b6e86e2dcfc6753bf57b306daa559467b89b201e55f4ff18964bed1113dc8f00.log -> /var/log/pods/default_pgpool-565c85fbd-kbpzq_b2af5056-fd0f-4168-bbbc-206c0b27e8ae/pgpool/0.log

Step 1 : Grant permissions

First, we need to grant Fluentd permissions so that Fluentd is able to collect logs from pgpool container. Here we assume that pgpool pod is deployed in the default namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: default

Let’s save the manifests above in fluentd_rbac.yaml and create it.

$ kubectl create -f fluentd_rbac.yaml
serviceaccount/fluentd created
clusterrole.rbac.authorization.k8s.io/fluentd created
clusterrolebinding.rbac.authorization.k8s.io/fluentd created

Step 2 : Configure Fluentd to collect Pgpool-II logs

We can use the manifest provided by fluentd-kubernetes-daemonset to deploy fluentd as a DaemonSet. By default the configuration file fluent.conf under /fluentd/etc will be used to start Fluentd. In this article we use this default Fluentd configuration file and mount Pgpool-II's dedicated configuration file defined in ConfigMap below to pgpool container.

The ConfigMap is:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-pgpool-configmap
  labels:
    app: fluentd-pgpool-config
data:
  pgpool.conf: |-
    <source>
      @type tail
      path /var/log/containers/*_pgpool-*.log
      pos_file /var/log/pgpool.log.pos
      tag pgpool
      read_from_head true
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    <filter pgpool>
      @type record_transformer
      enable_ruby
      <record>
        log ${record["log"].strip}
      </record>
    </filter>

    <filter pgpool>
      @type parser
      key_name log
      <parse>
         @type none
      </parse>
    </filter>

    <match pgpool>
      @type s3
      s3_bucket "#{ENV['S3_BUCKET_NAME']}"
      s3_region "#{ENV['S3_BUCKET_REGION']}"
      path ${tag[0]}
      s3_object_key_format "pgpool-log/%Y-%m-%d/pgpool-log-%{index}.%{file_extension}"
      <inject>
        time_key time
        tag_key tag
        localtime false
      </inject>
      <buffer>
        @type file
        path /var/log/fluentd-buffers/s3-pgpool.buffer
        timekey "#{ENV['S3_TIMEKEY'] || '3600'}"
        timekey_use_utc true
        chunk_limit_size "#{ENV['S3_CHUNK_LIMIT_SIZE'] || '256m'}"
      </buffer>
    </match>

Let’s save the ConfigMap above in fluentd-pgpool-config.yaml and create it.

kubectl create -f fluentd-pgpool-config.yaml

Step 3 : Deploy Fluentd as DaemonSet

Here we use the latest Docker image v1.12 under https://github.com/fluent/fluentd-kubernetes-daemonset/tree/master/docker-image/. Because I will save the logs to Amazon S3, I use debian-s3.

In the step 2 we have defined the Pgpool-II's dedicated ConfigMap fluentd-pgpool-configmap. Next, Let's deploy Fluentd DaemonSet and mount the ConfigMap defined in the step 2 to /fluentd/etc/conf.d.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-daemonset
spec:
  selector:
    matchLabels:
      name: fluentd-pod
  template:
    metadata:
      labels:
        name: fluentd-pod
    spec:
      serviceAccountName: fluentd
      containers:
        - name: fluentd-container
          image: fluent/fluentd-kubernetes-daemonset:v1.12-debian-s3-1
          env:
            - name: S3_BUCKET_NAME
              value: "your s3 bucket name"
            - name: S3_BUCKET_REGION
              value: "your s3 bucket region"
            - name: FLUENTD_SYSTEMD_CONF
              value: "disable"
            - name: FLUENT_UID
              value: "0"
          resources:
            requests:
              cpu: 200m
              memory: 500Mi
            limits:
              memory: 1Gi
          volumeMounts:
            - name: config-volume
              mountPath: /fluentd/etc/conf.d
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config-volume
          configMap:
            name: fluentd-pgpool-configmap
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

Let’s save the DaemonSet manifest above in fluentd.yaml and create it.

kubectl create -f fluentd.yaml

Now, you can find the logs stored in Amazon S3.


 

 


Comments

Post a Comment

Popular posts from this blog

Installing Pgpool-II on Debian/Ubuntu

-
First of all, many thanks to the PostgreSQL Global Development Group (PGDG) for creating Pgpool-II packages for Debian and Ubuntu. This post shows you step by step how to install Pgpool-II using official APT repository provided by PGDG. Prerequisites This blog assumes you have already installed two PostgreSQL 14 servers and setup streaming replication between the PostgreSQL servers.   For testing purpose,we set trust authentication in pg_hba.conf . In a production environment, Please follow the PostgreSQL and Pgpool-II documentation to configure proper authentication settings.  Install Pgpool-II To use the PostgreSQL apt repository, follow the steps below. Create the repository configuration file: # echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list Import the repository signing key: # wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - Update the package lists: #
Read more

Query Load Balancing in Pgpool-II

-
Image
Pgpool-II is a PostgreSQL cluster management tool. The major features of Pgpool-II are: Connection pooling Query load balancing Automated failover Watchdog (High availability of Pgpool-II) Replication In memory query cache In this post, I will describe the query load balancing mechanism in Pgpool-II and the relevant configuration parameters. Why use query load balancing? Query load balancing can distribute database server workloads across multiple PostgreSQL servers. Nowadays, most database systems use multiple replicated database servers to achieve high availability for PostgreSQL. Pgpool-II takes the advantage of the replication feature in order to distribute the workloads across multiple PostgreSQL servers. If there is a PostgreSQL cluster with multiple PostgreSQL servers, Pgpool-II is able to distribute READ queries across those PostgreSQL servers. The benefit of READ queries load balancing is, improve the system's throughput reduce the load on each PostgreSQL server optimize
Read more

Connection Pooling in Pgpool-II

-
Image
Pgpool-II is a cluster management tool for PostgreSQL that can cache connections to PostgreSQL servers. This blog introduces Pgpool-II connection pooling feature and shows how to configure connection pooling in Pgpool-II. What is connection pooling? Establishing and maintaining Database connections are expensive. The reason is that you have to establish a network connection, perform authentication and so on. Database connection pooling is a cache of database connections to keep database connections open so that the connection can be reused when a connection is required by a future request. Reusing an active connection rather than establishing a new connection each time a connection is requested can improve performance and save system resources. How does connection pooling work in Pgpool-II? Pgpool-II caches established connections to the PostgreSQL servers and reuses them whenever a new request with the same properties (i.e. user name, database, protocol version, and other con
Read more