{"id":90870,"date":"2021-05-10T21:22:38","date_gmt":"2021-05-10T21:22:38","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=90870"},"modified":"2021-05-10T21:22:38","modified_gmt":"2021-05-10T21:22:38","slug":"setting-up-a-kafka-test-environment-with-kafdrop","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/development\/dotnet-development\/setting-up-a-kafka-test-environment-with-kafdrop\/","title":{"rendered":"Setting up a Kafka test environment with Kafdrop"},"content":{"rendered":"

The\u00a0first article<\/a>\u00a0of this series on Apache Kafka explored very introductory concepts around the event streaming platform, a basic installation, and the construction of a fully functional application made with .NET, including the production and consumption of a topic message via command line.<\/p>\n

From a daily life standpoint, it\u2019s challenging to manage Kafka brokers, partitions, topics, producers, and consumers all via command line. An interface would be quite helpful.<\/p>\n

There is a ton of available options for managing your Kafka brokers for web UI applications. Perhaps\u00a0Confluent\u2019s version<\/a>\u00a0is one of the most complete, although it is part of a paid combo for mostly enterprise means.<\/p>\n

Amongst the myriad of open-source options, Kafdrop<\/a> stands out for being simple, fast, and easy to use. It is an open-source web project that allows you to view information from Kafka brokers as existing topics, consumers, and even the content of messages sent.<\/p>\n

This article explores creating a more flexible test environment to work alongside the .NET app built in the previous article. This way, you\u2019ll have more powerful tools to understand what\u2019s happening with your topics.<\/p>\n

Getting Ready<\/h2>\n

The previous article made use of the wurstmeister<\/em> Docker images. Because of the maturity of Confluent Docker images, this article will migrate the docker-compose to make use of its images. They\u2019re called confluentinc<\/a>.<\/p>\n

Take a look at the updated content of the docker-compose.yml<\/em> now:<\/p>\n

Listing 1. Updating the docker-compose.yml file<\/strong><\/p>\n

version: '3'\r\nservices:\r\n  zookeeper:\r\n    image: confluentinc\/cp-zookeeper:latest\r\n    networks: \r\n      - broker-kafka\r\n    environment:\r\n      ZOOKEEPER_CLIENT_PORT: 2181\r\n      ZOOKEEPER_TICK_TIME: 2000\r\n  kafka:\r\n    image: confluentinc\/cp-kafka:latest\r\n    networks: \r\n      - broker-kafka\r\n    depends_on:\r\n      - zookeeper\r\n    ports:\r\n      - 9092:9092\r\n    environment:\r\n      KAFKA_BROKER_ID: 1\r\n      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\r\n      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT:\/\/kafka:29092,PLAINTEXT_HOST:\/\/localhost:9092\r\n      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT\r\n      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT\r\n      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\r\n  kafdrop:\r\n    image: obsidiandynamics\/kafdrop:latest\r\n    networks: \r\n      - broker-kafka\r\n    depends_on:\r\n      - kafka\r\n    ports:\r\n      - 19000:9000\r\n    environment:\r\n      KAFKA_BROKERCONNECT: kafka:29092\r\nnetworks: \r\n  broker-kafka:\r\n    driver: bridge  <\/pre>\n
The first thing is to update its version. That\u2019s a good way to ensure that your Docker files will also keep versioning through the many changes they may suffer.<\/p>\n
You\u2019ll keep the images of Kafka and Zookeeper, although the image sources are different this time. If you\u2019re willing to keep working with the previous ones, that\u2019s ok too. Just make sure to consult the vendor-specific settings that may be required for a broker as such.<\/p>\n
The new network called broker-kafka<\/code> will be responsible for keeping the communication among the three containers.<\/p>\n
Finally, at the end of the listing, you get to see the Kafdrop-related container settings, which are set up with a specific port. This is the same port you\u2019ll use later to access the UI. Feel free to change it as you wish.<\/p>\n
Before you can rerun the docker-compose commands, you need to make sure that the previous images aren\u2019t running nor installed on your machine. The easiest way is through Docker Desktop. Open it and make sure to delete all the containers from the previous article whether they\u2019re running or not.<\/p>\n
Another way to check this out is via command line:<\/p>\n
docker-compose ps<\/pre>\n
Now rerun the docker-compose command to re-download everything and build the images from scratch:<\/p>\n
docker-compose up -d<\/pre>\n
Wait until the command downloads all the images from the internet, as shown in Figure 1.<\/p>\n
<\/p>\n
Figure 1.<\/strong> Downloading new Docker images<\/strong><\/p>\n
You may see that the download was successful, and the images started up, as shown in Figure 2.<\/p>\n
<\/p>\n
Figure 2. New images downloaded and started<\/strong><\/p>\n
To check if the network was created, run the following command:<\/p>\n
docker network ls<\/pre>\n
That\u2019ll print the list of networks including the newly created broker-kafka<\/em> (Figure 3).<\/p>\n
<\/p>\n
Figure 3. Docker network check<\/strong><\/p>\n
To check if the running images include the new Kafdrop, run the command docker ps<\/code> and compare the results with Figure 4.<\/p>\n
<\/p>\n
Figure 4.<\/strong> Checking the running images via docker ps<\/em><\/strong><\/p>\n
Finally, Docker Desktop is the easiest way to check that out. Figure 5 shows how your images should look.<\/p>\n
<\/p>\n
Figure 5.<\/strong> Newly created Docker images on Docker Desktop<\/strong><\/p>\n
That visualization also allows seeing the port in which each image is running. If you click any of the images, Docker Desktop will display the logs of that container live.<\/p>\n
<\/p>\n
Figure 6.<\/strong> Watching Docker image logs<\/strong><\/p>\n
Exploring Kafdrop<\/h2>\n
With the configs set, Kafdrop is started at the http:\/\/localhost:19000\/<\/a> address. Figure 7 shows how the interface looks.<\/p>\n
<\/p>\n
Figure 7.<\/strong> Kafdrop cluster overview<\/strong><\/p>\n
The information is segregated into the broker servers, total topics, and partitions. Kafka is designed to work in a distributed manner, which means that it is usually set up to be within a distributed cluster with more than one bootstrap server. While working locally, however, there\u2019s only one server displayed here.<\/p>\n
Also, since the previous Kafka images are removed, there\u2019s no topic available. To change that, start the producer and consumer applications from the previous article.<\/p>\n
Go ahead and do that. Once you finish the startup, go back to the Kafdrop page and reload it. You will see a result similar to that shown in Figure 8.<\/p>\n
<\/p>\n
Figure 8.<\/strong> Total topics increased by 1 + __consumer_offsets<\/em> topic<\/strong><\/p>\n
Since Kafka keeps the state of the messages in storage, the consumer offsets topic is important to keep track of the sequential order in which the messages arrived at the topics.<\/p>\n
That\u2019s a way Kafka uses to know where it has stopped from the last time in case the infrastructure that holds the broker shuts down, for example.<\/p>\n
This tutorial won\u2019t focus on them, however. As you may guess, besides both producer and consumer are up, the simpletalk_topic<\/em> is still not visible on the list. That happens because it needs to have some interaction with the topic, such as a new message being sent. That\u2019s when Kafka creates the topic in case it still doesn\u2019t exist.<\/p>\n
Send a new message to the topic and see what happens:<\/p>\n
curl -H \"Content-Length: 0\" -X POST \"http:\/\/localhost:51249\/api\/kafka?message=Hello,kafka!\" -k<\/pre>\n
The command you enter may be slightly different depending on your setup. The command line window running the Kafka Producer program should tell you what to enter here. When you refresh the Kafdrop page, Figure 9 shows how it must look.<\/p>\n
<\/p>\n
Figure 9.<\/strong> New topic added to the topics list<\/strong><\/p>\n
Nice, isn\u2019t it? If you click the simpletalk_topic<\/em> link, you\u2019ll be redirected to the page shown in Figure 10.<\/p>\n
<\/p>\n
Figure 10.<\/strong> Visualizing simpletalk_topic on Kafdrop<\/strong><\/p>\n
Among the information provided on this page, the most important is related to the number of available messages for the topic. Kafdrop is a great way to visualize problems in a production environment arising from applications that produce and consume from this topic.<\/p>\n
If you click the View Messages<\/em> button on the top of the page, you\u2019ll be redirected to a second page, as shown in Figure 11.<\/p>\n
<\/p>\n
Figure 11.<\/strong> Visualizing the messages of a topic<\/strong><\/p>\n
You may click the View Messages<\/em> button once more to be able to see the list of messages. Remember that this is just a visualization; it doesn\u2019t mean that a consumer or a group of consumers have consumed the message already.<\/p>\n
To see this, make sure that your consumer project has printed the message stating that it has committed the topic consumption. You may need to restart the consumer project to get this to appear. Then, get back to the topic details page, and you may see it as shown below.<\/p>\n
<\/p>\n
Figure 12. Visualizing new consumers attached to a topic<\/strong><\/p>\n
As you see, a new consumer whose group id is \u201cst_consumer_group<\/em>\u201d is now tied to this topic. If you click it, a more detailed version of the consumer group will show up, as shown in Figure 13.<\/p>\n
<\/p>\n
Figure 13.<\/strong> Details about a specific consumer group<\/strong><\/p>\n
The Last Offset<\/em> column states the current latest offset for the entire topic, regardless of who consumed its messages. The Consumer Offset<\/em>, in turn, stands for the offset at which the current consumer group is at. The value 1 means that all the published messages were already consumed by this consumer.<\/p>\n
If you wish, you\u2019ll be able to create a new topic via Kafdrop by clicking the + New<\/em> button located on the homepage. This will enable the following screen for you to finalize the creation.<\/p>\n
<\/p>\n
Figure 14.<\/strong> Creating a new topic with Kafdrop<\/strong><\/p>\n
Swagger Integration<\/h2>\n
If you wish to document your Kafka topics, consumers, etc., in an API-related way, Kafdrop also provides you with integration with Swagger<\/a>.<\/p>\n
Towards the endpoint localhost:19000\/v2\/api-docs<\/a>, you\u2019ll be able to see all the Swagger 2.0 Open Specification JSON that you can use to document and run endpoints on your Kafdrop environment. That\u2019s another alternative for those who don\u2019t want to explicitly allow access to Kafdrop UI for all users.<\/p>\n
<\/p>\n
Figure 15.<\/strong> Swagger API docs for Kafdrop<\/strong><\/p>\n
If you\u2019re used to working with micro metrics, and since Kafdrop is made with Spring and Java, it also provides an endpoint for the app metrics at localhost:19000\/actuator<\/a>.<\/p>\n
<\/p>\n
Figure 16.<\/strong> Kafdrop micro metrics via Actuator<\/strong><\/p>\n
That\u2019s very useful when you already have automatic monitor tools that check the health of your infrastructure apps as well as other factors and alarm based on failing scenarios.<\/p>\n
Setting Up a Kafka Test Environment with Kafdrop<\/h1>\n
Kafdrop is a great option for allowing a better-integrated environment not only for testing purposes but also for development and operational activities within your company. With it, you\u2019re able to visualize your entire Kafka cluster, including the topics, consumer groups, messages, offsets, and more.<\/p>\n
How about you? Have you used any similar tool in the past? Let me know in the comments about your experience with them.<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
Kafdrop is n open source project that allows you to view information from Kafka brokers. In this article, Diogo Souza explains setting up a Kafka test environment with Kafdrop.…<\/p>\n","protected":false},"author":320401,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[143538,53],"tags":[143553,143552,5134],"coauthors":[60461],"class_list":["post-90870","post","type-post","status-publish","format-standard","hentry","category-dotnet-development","category-featured","tag-kafdrop","tag-kafka","tag-sql-prompt"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/90870","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/320401"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=90870"}],"version-history":[{"count":2,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/90870\/revisions"}],"predecessor-version":[{"id":90888,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/90870\/revisions\/90888"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=90870"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=90870"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=90870"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=90870"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}