Docker Logging Guide Part 1: Basic Concepts and Importance

Docker logging
Comments 0

Share to social media

Docker has gained popularity as a containerization platform that allows you to develop, deploy, and execute applications faster. It packages applications and their dependencies into standardized entities known as containers. These containers are lightweight, portable, and capable of operating independently. Containers allow developers to build, deploy, and manage applications in different working environments. In addition, containers can be used for both development and production.

Docker logging is an essential process in managing containers within a production setting. It enables the monitoring and troubleshooting of applications. This facilitates the detection and resolution of potential issues. When you implement Docker logging you can easily identify and fix issues that may arise when building Docker containers.

For logging to work properly, it’s important to capture logs from the application itself. Additionally, you can also capture on the host system, and the Docker service. When you implement a combination of logging techniques, it enables efficient logging of Dockerized applications.

In this two-part series, we will discuss Docker logging from the basic concepts to advanced concepts. In this part 1, we will cover what is Docker Logging, Docker logging drivers, and why it is important to perform Docker logging when building containers. By the end of the article, you will be familiar with what is Docker Logging and its importance when building Docker Containers as a DevOps engineer.

So, let’s get started!

What is involved with Docker Logging?

Docker logging involves the collection, storage, and organization of log information that applications generate within Docker containers. As applications execute within Docker containers, they generate diverse log messages. These log messages are both standard output (stdout) and standard error (stderr) streams. Docker’s logging mechanisms facilitate the storage and retention of these logs. The log messages serve various purposes such as monitoring, diagnosing issues, debugging, and ensuring auditability.

Logging Drivers

Docker offers various logging drivers that dictate the storage location and format of log messages. These logging drivers are listed below:

json-file

The json-file logging driver writes container logs in JSON format to local files on the Docker host. Each log entry is formatted as a JSON object, providing structured data for easy parsing and analysis.

It’s suitable for environments requiring local log storage with structured log data. Enables seamless integration with log management tools supporting JSON log formats.

syslog

The syslog logging driver forwards container logs to the syslog daemon on the Docker host, allowing logs to be further processed and forwarded to remote syslog servers or stored locally.

It’s deal for integrating Docker logs with existing syslog infrastructure or centralizing log management using syslog-compatible tools and services. Enables standardized log collection and aggregation.

Journald

The journald logging driver sends container logs to the systemd journal on the Docker host, providing centralized logging capabilities alongside system logs. Logs are stored and managed within the systemd journal.

It’s suitable for environments leveraging systemd journal as the primary logging mechanism. Facilitates centralized log analysis and management alongside system logs.

Fluentd

The fluentd logging driver streams container logs to an instance of Fluentd, a robust log collector and aggregator. Fluentd can process and route logs to various destinations, including Elasticsearch, Kafka, or cloud storage.

It enables flexible log routing and aggregation in Docker environments, supporting centralized log processing and storage. Ideal for organizations with complex log management requirements and diverse log destinations.

Awslogs

The awslogs logging driver sends container logs directly to Amazon CloudWatch Logs, a managed log management service provided by AWS. Logs are stored, analyzed, and monitored using CloudWatch Logs features.

It’s designed for seamless integration with AWS environments, allowing container logs to be centrally managed and monitored alongside other AWS services. Facilitates comprehensive log analysis and monitoring within the AWS ecosystem.

Gelf

The gelf logging driver forwards container logs to a Graylog Extended Log Format (GELF) endpoint, typically a Graylog server. GELF format provides features such as structured logging and log enrichment.

It’s suitable for organizations utilizing Graylog for centralized log management and analysis. Enables structured log processing and visualization, facilitating efficient log search and analysis.

logentries

The logentries logging driver streams container logs to the Logentries service, a cloud-based log management and analytics platform. Logentries provides features for log aggregation, search, and visualization.

It’s ideal for organizations leveraging Logentries for centralized log management and analysis. Facilitates real-time log monitoring, alerting, and troubleshooting in Docker environments.

Splunk

The splunk logging driver sends container logs directly to Splunk Enterprise or Splunk Cloud, enabling centralized log management and analysis. Splunk provides features for log indexing, search, and visualization.

Designed for seamless integration with Splunk for enterprise-grade log management and analysis. Enables organizations to leverage Splunk’s powerful log analytics capabilities for Docker environments.

Default Logging Option

Docker containers use the json-file logging driver as the default option. This option stores logs directly on the Docker host’s local filesystem. Nevertheless, many organizations prefer centralized logging solutions, which consolidate logs from numerous containers and hosts. This approach streamlines the analysis and administration of log data, enhancing operational efficiency.

Why It’s Important to Perform Docker Logging When Building Containers

In this section, we will discuss why it’s important to perform Docker logging when building containers. The following are some of the benefits of Docker logging:

Monitoring Application Health

Docker logging allows developers and administrators to monitor the health and performance of applications during the build process. Developers can ensure that the application initializes correctly. It also ensures dependencies are installed successfully, and configuration settings are applied as intended. 

Monitoring application health during the build process helps identify any potential issues early on. This allows for timely troubleshooting and resolution.

In this example, we’ll use Python’s built-in logging module to log important events in a Flask web application. We’ll implement logging directly into the application code to monitor HTTP requests and application errors.

The first step is creating the Flask application and configuring the logging logic:

Next, let’s create a Docker container for the Flask web application:

Build and run the Docker container using the following commands:

Output:

Docker logging

Output:

To view logs from a running Docker container for the Flask web application use the following command:

Output:

When running a Flask application, the default behavior for logging is to write logs to specific files, such as access.log and error.log, located in the /var/log/python directory. However, to facilitate Docker’s logging management capabilities, the Python configuration within the container has been altered. This modification directs access logs to /dev/stdout and error logs to /dev/stderr instead. Using this configuration, Docker can effectively collect and handle the application logs. Furthermore, the access logs are formatted in JSON to simplify their usage with various log management tools.

Alternatively, the official Python Docker image uses a different strategy to achieve the same outcome. It establishes symbolic links from /var/log/python/access.log to /dev/stdout and from /var/log/nginx/error.log to /dev/stderr. These approaches ensure that the logs the Flask application produces are readily accessible through the Docker CLI. This provides flexibility in managing and analyzing log data.

Diagnosing Issues

Logging can offer valuable insights into encountered errors or warnings while installing dependencies, compiling applications, or performing other tasks. This streamlines the debugging process. It also contributes to ensuring the accurate and dependable construction of containers.

For diagnosing issues during the build process, let’s log the output of npm commands in a Node.js application.

Next, let’s create a Docker container for the Node.js application:

Build and run the Docker container using the following commands:

Output:

Then excute:

And this will output:

To view logs from a running Docker container for the Node.js application use the following command:

Output:

Ensuring Security Compliance

Docker logging during the container build process can ensure security compliance. When logging security scans, you easily can detect potential security vulnerabilities or misconfigurations. You can then rectify them before deploying containers in production environments.

To enable security compliance, we’ll log the output of security scans in a Java application.

Next, let’s create a Docker container for the Java application:

Build and run the Docker container using the following commands:

Output:

To view logs from a running Docker container for the Java application:

Output:

In each example, logging is incorporated directly into the application code to monitor important events, diagnose issues, and ensure security compliance. These logs have been viewed using the docker logs command. The docker logs commands provide valuable insights into application behavior and facilitate debugging. It also monitors efforts throughout the container lifecycle.

Conclusion

In conclusion, Docker logging is a critical aspect of managing containerized applications efficiently. In this article, we discussed the basic concepts of Docker logging such as what is Docker Logging, Docker logging drivers and why it important to perform Docker logging when building containers.

That’s all for this article, see you in the next article. In the next article which is a part 2, we will cover more advanced Docker logging concepts and best practices. See you in the next article.

 

About the author

Bravin Wasike

See Profile

Bravin is a creative DevOps engineer and Technical Writer. He loves writing about software development. He has experience with Docker, Kubernetes, AWS, Jenkins, Terraform, CI/CD, and other DevOps tools. Bravin has written many articles on DevOps tools such as Kubernetes, Docker, Jenkins, Azure DevOps, Aws, CI/CD, Terraform, Ansible, Infrastructure as code, Infrastructure Provisioning, Monitoring and Configuration, Git, Source code management and Azure.

Bravin's contributions