{"id":102329,"date":"2024-06-17T20:17:47","date_gmt":"2024-06-17T20:17:47","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=102329"},"modified":"2024-07-09T18:35:16","modified_gmt":"2024-07-09T18:35:16","slug":"docker-logging-guide-part-1-basic-concepts-and-importance","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/devops\/containers-and-virtualization\/docker-logging-guide-part-1-basic-concepts-and-importance\/","title":{"rendered":"Docker Logging Guide Part 1: Basic\u00a0Concepts and Importance"},"content":{"rendered":"

Docker<\/a>\u00a0has gained popularity as a containerization platform that allows you to develop, deploy, and execute applications faster. It packages applications and their dependencies into standardized entities known as containers. These containers are lightweight, portable, and capable of operating independently. Containers allow developers to build, deploy, and manage applications in different working environments. In addition,\u00a0containers can be used\u00a0for both development and production.<\/p>\n

Docker logging<\/a>\u00a0is an essential process in managing containers within a production setting. It enables the monitoring and troubleshooting of applications.\u00a0This\u00a0facilitates the detection and resolution of potential issues. When you implement Docker logging\u00a0you\u00a0can\u00a0easily\u00a0identify and fix\u00a0issues\u00a0that may arise when building Docker containers.<\/p>\n

For logging to work\u00a0properly,\u00a0it’s\u00a0important\u00a0to capture logs from the application\u00a0itself.\u00a0Additionally,\u00a0you can also capture on the host\u00a0system,\u00a0and the Docker service.\u00a0When you implement a combination of logging techniques, it enables efficient logging of Dockerized applications.<\/p>\n

In this two-part series, we will discuss Docker logging from\u00a0the\u00a0basic\u00a0concepts\u00a0to advanced concepts.\u00a0In this part 1, we will cover\u00a0what is\u00a0Docker Logging, Docker logging drivers, and why it is\u00a0important\u00a0to perform Docker logging when building containers. By the end of the article, you will be familiar with\u00a0what is\u00a0Docker Logging and its importance when building Docker Containers as a DevOps engineer.<\/p>\n

So,\u00a0let\u2019s\u00a0get started!<\/p>\n

What is involved with Docker Logging?<\/h2>\n

Docker logging involves the collection, storage, and organization of log information that applications generate within Docker containers. As applications execute within Docker containers, they generate diverse log messages. These log messages are both standard output (stdout<\/code>) and standard error (stderr<\/code>) streams. Docker’s logging mechanisms facilitate the storage and retention of these logs. The log messages serve various purposes such as monitoring, diagnosing issues, debugging, and ensuring auditability.<\/p>\n

Logging Drivers<\/h2>\n

Docker offers various logging drivers that dictate the storage location and format of log messages. These logging drivers are listed below:<\/p>\n

json-file<\/a><\/strong><\/h3>\n

The json-file<\/code> logging driver writes container logs in JSON format to local files on the Docker host. Each log entry is formatted as a JSON object, providing structured data for easy parsing and analysis.<\/p>\n

It\u2019s suitable for environments requiring local log storage with structured log data. Enables seamless integration with log management tools supporting JSON log formats.<\/p>\n

syslog<\/a><\/strong><\/h3>\n

The syslog<\/code> logging driver forwards container logs to the syslog daemon on the Docker host, allowing logs to be further processed and forwarded to remote syslog servers or stored locally.<\/p>\n

It\u2019s deal for integrating Docker logs with existing syslog infrastructure or centralizing log management using syslog-compatible tools and services. Enables standardized log collection and aggregation.<\/p>\n

Journald<\/a><\/strong><\/h3>\n

The journald<\/code> logging driver sends container logs to the systemd<\/code> journal on the Docker host, providing centralized logging capabilities alongside system logs. Logs are stored and managed within the systemd journal.<\/p>\n

It\u2019s suitable for environments leveraging systemd journal as the primary logging mechanism. Facilitates centralized log analysis and management alongside system logs.<\/p>\n

Fluentd<\/a><\/strong><\/h3>\n

The fluentd<\/code> logging driver streams container logs to an instance of Fluentd, a robust log collector and aggregator. Fluentd can process and route logs to various destinations, including Elasticsearch, Kafka, or cloud storage.<\/p>\n

It enables flexible log routing and aggregation in Docker environments, supporting centralized log processing and storage. Ideal for organizations with complex log management requirements and diverse log destinations.<\/p>\n

Awslogs<\/a><\/strong><\/h3>\n

The awslogs<\/code> logging driver sends container logs directly to Amazon CloudWatch Logs, a managed log management service provided by AWS. Logs are stored, analyzed, and monitored using CloudWatch Logs features.<\/p>\n

It\u2019s designed for seamless integration with AWS environments, allowing container logs to be centrally managed and monitored alongside other AWS services. Facilitates comprehensive log analysis and monitoring within the AWS ecosystem.<\/p>\n

Gelf<\/a><\/strong><\/h3>\n

The gelf <\/code>logging driver forwards container logs to a Graylog Extended Log Format (GELF) endpoint, typically a Graylog server. GELF format provides features such as structured logging and log enrichment.<\/p>\n

It\u2019s suitable for organizations utilizing Graylog for centralized log management and analysis. Enables structured log processing and visualization, facilitating efficient log search and analysis.<\/p>\n

logentries<\/a><\/strong><\/h3>\n

The logentries<\/code> logging driver streams container logs to the Logentries service, a cloud-based log management and analytics platform. Logentries provides features for log aggregation, search, and visualization.<\/p>\n

It\u2019s ideal for organizations leveraging Logentries for centralized log management and analysis. Facilitates real-time log monitoring, alerting, and troubleshooting in Docker environments.<\/p>\n

Splunk<\/a><\/strong><\/h3>\n

The splunk<\/code> logging driver sends container logs directly to Splunk Enterprise or Splunk Cloud, enabling centralized log management and analysis. Splunk provides features for log indexing, search, and visualization.<\/p>\n

Designed for seamless integration with Splunk for enterprise-grade log management and analysis. Enables organizations to leverage Splunk’s powerful log analytics capabilities for Docker environments.<\/p>\n

Default Logging Option<\/h3>\n

Docker containers use the json-file logging driver<\/a> as the default option. This option stores logs directly on the Docker host’s local filesystem. Nevertheless, many organizations prefer centralized logging solutions, which consolidate logs from numerous containers and hosts. This approach streamlines the analysis and administration of log data, enhancing operational efficiency.<\/p>\n

Why It’s Important to Perform Docker Logging When Building Containers<\/h2>\n

In this section, we will discuss why it’s important to perform Docker logging when building containers. The following are some of the benefits of Docker logging:<\/p>\n

Monitoring Application Health<\/h3>\n

Docker logging allows developers and administrators to monitor the health and performance of applications during the build process. Developers can ensure that the application initializes correctly. It also ensures dependencies are installed successfully, and configuration settings are applied as intended.\u00a0<\/p>\n

Monitoring application health during the build process helps identify any potential issues early on. This allows for timely troubleshooting and resolution.<\/p>\n

In this example, we’ll use Python’s built-in logging module to log important events in a Flask web application. We’ll implement logging directly into the application code to monitor HTTP requests and application errors.<\/p>\n

The first step is creating the Flask application and configuring the logging logic:<\/p>\n

# app.py\r\nfrom flask import Flask, request\r\nimport logging\r\napp = Flask(__name__)\r\n\r\n# Configure logging\r\nlogging.basicConfig(level=logging.INFO)\r\n@app.route('\/')\r\ndef index():\r\n\u00a0 \u00a0\u00a0app.logger.info('Handling index request')\r\n\u00a0 \u00a0\u00a0return 'Hello, World!'\r\n@app.route('\/error')\r\ndef error():\r\n\u00a0 \u00a0\u00a0app.logger.error('An error occurred')\r\n\u00a0 \u00a0\u00a0return 'Internal Server Error', 500\r\nif __name__ == '__main__':\r\n\u00a0 \u00a0\u00a0app.run(debug=True)<\/pre>\n
Next, let’s create a Docker container for the Flask web application:<\/p>\n
# Dockerfile\r\nFROM python:3.9\r\n\r\n# Copy application code\r\nCOPY app.py \/app\/\r\n\r\n# Set working directory\r\nWORKDIR \/app\r\n\r\n# Install Flask\r\nRUN pip install Flask\r\n\r\n# Expose port\r\nEXPOSE 5000\r\n\r\n# Command to run the application\r\nCMD [\"python\", \"app.py\"]<\/pre>\n
Build and run the Docker container using the following commands:<\/p>\n
docker build -t <image-name\/tag> .<\/pre>\n
Output:<\/p>\n
\"Docker<\/p>\n
docker run -p 5000:5000 <image-name\/tag><\/pre>\n
Output:<\/p>\n
<\/p>\n
To view logs from a running Docker container for the Flask web application use the following command:<\/p>\n
docker logs [OPTIONS] CONTAINER<\/pre>\n
Output:<\/p>\n
$ docker logs mystifying_germain\r\n\/docker-entrypoint.sh: \/docker-entrypoint.d\/ is not empty, will attempt to perform configuration\r\n\/docker-entrypoint.sh: Looking for shell scripts in \/docker-entrypoint.d\/\r\n\/docker-entrypoint.sh: Launching \/docker-entrypoint.d\/10-listen-on-ipv6-by-default.sh\r\n10-listen-on-ipv6-by-default.sh: info: Getting the checksum of \/etc\/nginx\/conf.d\/default.conf\r\n10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in \/etc\/nginx\/conf.d\/default.conf\r\n\/docker-entrypoint.sh: Sourcing \/docker-entrypoint.d\/15-local-resolvers.envsh\r\n\/docker-entrypoint.sh: Launching \/docker-entrypoint.d\/20-envsubst-on-templates.sh\r\n\/docker-entrypoint.sh: Launching \/docker-entrypoint.d\/30-tune-worker-processes.sh\r\n\/docker-entrypoint.sh: Configuration complete; ready for start up\r\n2024\/04\/15 11:44:22 [notice] 1#1: using the \"epoll\" event method\r\n2024\/04\/15 11:44:22 [notice] 1#1: nginx\/1.25.2\r\n2024\/04\/15 11:44:22 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10)\u00a0\r\n2024\/04\/15 11:44:22 [notice] 1#1: OS: Linux 5.10.16.3-microsoft-standard-WSL2\r\n2024\/04\/15 11:44:22 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576\r\n2024\/04\/15 11:44:22 [notice] 1#1: start worker processes\r\n2024\/04\/15 11:44:22 [notice] 1#1: start worker process 30\r\n2024\/04\/15 11:44:22 [notice] 1#1: start worker process 31\r\n2024\/04\/15 11:44:22 [notice] 1#1: start worker process 32\r\n2024\/04\/15 11:44:22 [notice] 1#1: start worker process 33\r\n2024\/04\/15 11:44:22 [notice] 1#1: start worker process 34\r\n2024\/04\/15 11:44:22 [notice] 1#1: start worker process 35\r\n2024\/04\/15 11:44:22 [notice] 1#1: start worker process 36\r\n2024\/04\/15 11:44:22 [notice] 1#1: start worker process 37<\/pre>\n
When running a Flask application, the default behavior for logging is to write logs to specific files, such as access.log and error.log, located in the \/var\/log\/python<\/code> directory. However, to facilitate Docker’s logging management capabilities, the Python configuration within the container has been altered. This modification directs access logs to \/dev\/stdout<\/code> and error logs to \/dev\/stderr<\/code> instead. Using this configuration, Docker can effectively collect and handle the application logs. Furthermore, the access logs are formatted in JSON to simplify their usage with various log management tools.<\/p>\n
Alternatively, the official Python Docker image uses a different strategy to achieve the same outcome. It establishes symbolic links from \/var\/log\/python\/access.log<\/code> to \/dev\/stdout<\/code> and from \/var\/log\/nginx\/error.log<\/code> to \/dev\/stderr<\/code>. These approaches ensure that the logs the Flask application produces are readily accessible through the Docker CLI. This provides flexibility in managing and analyzing log data.<\/p>\n
Diagnosing Issues<\/h3>\n
Logging can offer valuable insights into encountered errors or warnings while installing dependencies, compiling applications, or performing other tasks. This streamlines the debugging process. It also contributes to ensuring the accurate and dependable construction of containers.<\/p>\n
For diagnosing issues during the build process, let’s log the output of npm<\/code> commands in a Node.js application.<\/p>\n
\/\/ index.js\r\nconst express = require('express');\r\nconst app = express();\r\n\/\/ Logging\r\nconsole.log('Starting application...');\r\napp.get('\/', (req, res) => {\r\n\u00a0\u00a0res.send('Hello, World!');\r\n});\r\napp.get('\/error', (req, res) => {\r\n\u00a0\u00a0console.error('An error occurred');\r\n\u00a0\u00a0res.status(500).send('Internal Server Error');\r\n});\r\nconst PORT = process.env.PORT || 3000;\r\napp.listen(PORT, () => {\r\n\u00a0\u00a0console.log(`Server running on port ${PORT}`);\r\n});<\/pre>\n
Next, let’s create a Docker container for the Node.js application:<\/p>\n
# Dockerfile\r\nFROM node:14\r\n\r\n# Copy application code\r\nCOPY index.js \/app\/\r\n\r\n# Set working directory\r\nWORKDIR \/app\r\n\r\n# Install Express\r\nRUN npm install express\r\n\r\n# Command to start the application\r\nCMD [\"node\", \"index.js\"]<\/pre>\n
Build and run the Docker container using the following commands:<\/p>\n
docker build -t <image-name\/tag> .<\/pre>\n
Output:<\/p>\n
<\/p>\n
Then excute:<\/p>\n
docker run -p 3000:3000 <image-name\/tag><\/pre>\n
And this will output:<\/p>\n
$ docker run -p 3000:3000 js\r\nStarting application...\r\nServer running on port 3000<\/pre>\n
To view logs from a running Docker container for the Node.js application use the following command:<\/p>\n
docker logs [OPTIONS] CONTAINER<\/pre>\n
Output:<\/p>\n
$ docker logs romantic_bhaskara\r\nStarting application...\r\nServer running on port 3000<\/pre>\n
Ensuring Security Compliance<\/h3>\n
Docker logging during the container build process can ensure security compliance. When logging security scans, you easily can detect potential security vulnerabilities or misconfigurations. You can then rectify them before deploying containers in production environments.<\/p>\n
To enable security compliance, we’ll log the output of security scans in a Java application.<\/p>\n
\/\/ Main.java\r\npublic class Main {\r\n\u00a0 \u00a0\u00a0public static void main(String[] args) {\r\n\u00a0 \u00a0 \u00a0 \u00a0\u00a0\/\/ Logging\r\n\u00a0 \u00a0 \u00a0 \u00a0\u00a0System.out.println(\"Starting application...\");\r\n\u00a0 \u00a0 \u00a0 \u00a0\u00a0\/\/ Main application logic\r\n\u00a0 \u00a0 \u00a0 \u00a0\u00a0\/\/ ...\r\n\u00a0 \u00a0\u00a0}\r\n}<\/pre>\n
Next, let’s create a Docker container for the Java application:<\/p>\n
# Dockerfile\r\nFROM openjdk:11-jdk\r\n# Copy application code\r\nCOPY Main.java \/app\/\r\n# Set working directory\r\nWORKDIR \/app\r\n# Compile Java code\r\nRUN javac Main.java\r\n# Command to start the application\r\nCMD [\"java\", \"Main\"]<\/pre>\n
Build and run the Docker container using the following commands:<\/p>\n
docker build -t <image-name\/tag> .\r\ndocker run -p <port>:<port> <image-name\/tag><\/pre>\n
Output:<\/p>\n
Starting application...<\/pre>\n
To view logs from a running Docker container for the Java application:<\/p>\n
docker logs [OPTIONS] CONTAINER<\/pre>\n
Output:<\/p>\n
$ docker logs CONTAINER\r\nStarting application...<\/pre>\n
In each example, logging is incorporated directly into the application code to monitor important events, diagnose issues, and ensure security compliance. These logs have been viewed using the docker logs<\/code> command. The docker logs<\/code> commands provide valuable insights into application behavior and facilitate debugging. It also monitors efforts throughout the container lifecycle.<\/p>\n
Conclusion<\/h3>\n
In conclusion, Docker logging is a critical aspect of managing containerized applications efficiently. In this article, we discussed the basic concepts of Docker logging such as what is Docker Logging, Docker logging drivers and why it important to perform Docker logging when building containers.<\/p>\n
That\u2019s all for this article, see you in the next article. In the next article which is a part 2, we will cover more advanced Docker logging concepts and best practices. See you in the next article.<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
Docker\u00a0has gained popularity as a containerization platform that allows you to develop, deploy, and execute applications faster. It packages applications and their dependencies into standardized entities known as containers. These containers are lightweight, portable, and capable of operating independently. Containers allow developers to build, deploy, and manage applications in different working environments. In addition,\u00a0containers can……<\/p>\n","protected":false},"author":342511,"featured_media":102330,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[143513,53],"tags":[159078],"coauthors":[159023],"class_list":["post-102329","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-containers-and-virtualization","category-featured","tag-docker"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/102329","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/342511"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=102329"}],"version-history":[{"count":5,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/102329\/revisions"}],"predecessor-version":[{"id":102996,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/102329\/revisions\/102996"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media\/102330"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=102329"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=102329"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=102329"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=102329"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}