IBM Event Streams cheat sheet for developers

In this article, we present some top tips for implementing, debugging, and operating applications with Event Streams.

Enable IBM Cloud Monitoring

Event Streams supports IBM Cloud Monitoring. You can use IBM Cloud Monitoring to monitor metrics and gain operational visibility into the performance and health of your applications, services, and platforms. It offers administrators, DevOps teams, and developers full stack telemetry with advanced features to monitor and troubleshoot, define alerts, and design custom dashboards.

For more details about the Event Streams metrics you can monitor, see Monitoring Event Streams metrics using IBM Cloud Monitoring in the documentation.

Client behavior and diagnostics

Most Kafka clients generate logs and metrics. When running applications, it's important to capture logs and metrics as they often help with investigating an issue, if one occurs.

For example, for the Java client:

logs. Logs are configured via log4j. The minimal recommended configuration file is:

log4j.rootLogger=INFO, stdout, kafkaAppender

  log4j.appender.stdout=org.apache.log4j.ConsoleAppender
  log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
  log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n

Then pass the following JVM argument when starting the application:

-Dlog4j.configuration=file:<PATH_TO_LOG4J_PROPERTIES_FILE>

metrics: By default, metrics are emitted via JMX. You can also set the metric.reporters configuration property to use other reporters appropriate to your metrics pipeline. For recommendations on the key Java client metrics to monitor, see: Getting started with Kafka client metrics

Debugging connection problems

When debugging connection issues, we recommend completing the following steps:

Verify your client configuration, including:
1. bootstrap servers are configured in the Kafka client.
2. Verify the security protocol is set to SASL and SSL.
3. Verify the password configured for SASL.
Verify the ServiceID is associated with policies that have the appropriate IAM permissions.
Verify the maximum number of connected clients is not reached. There are metrics available to help you understand your usage.

For more details, see the Troubleshooting section in the documentation.

Use the Event Streams sample repository

The Event Streams sample GitHub repository contains sample apps that demonstrate best practices for building Kafka applications in multiple languages, including Java, Node.js, and Python.

This repository also contains instructions and Docker images to do the following:

Run Kafka Connect. This sample contains the artifacts required to build the ibmcom/eventstreams-kafkaconnect Docker image. This image contains the Kafka Connect runtime and the IBM Cloud Object Storage sink connector and the IBM MQ source connector alongside instructions about how to run it on Kubernetes.
Run MirrorMaker. This repository contains the artifacts required to build the ibmcom/eventstreams-kafkamirrormaker Docker image. This image contains Kafka Mirror Maker and can be used to replicate data between clusters.

Use the Event Streams CLI

The Event Streams CLI is a quick and easy way to gather details about your cluster. It supports:

Creating, listing, updating, and deleting topics.
Listing, deleting, and resetting consumer groups.
Describing the cluster.

For more details, see the Event Streams CLI reference.

Event Streams must-gather information

If you'd like help from the Event Streams team, when opening a ticket, it's best to provide the following information:

What is the CRN ID of the Event Streams service you are using? You can provide this ID by pasting the full IBM Cloud console URL after clicking on the service, or by pasting the output from the following CLI command:
```
ibmcloud resource service-instance NAME
```
When did the problem first occur (specifically time, date, and timezone)? How long was your app running before this?
Is the problem still occurring? Can you replicate it?
Which Kafka client is your application using? What are the version details?
What are your client configuration details? We need to know your producer and consumer settings, so please list any non-default options you have passed to your producer or consumer creation.
Do you have application log snippets that display the problem?
What is the issue you are seeing? Which topics, client IDs, group IDs, and transaction IDs are affected?
What impact is the problem having on your service?

For more details, see Reporting a problem to the Event Streams team.

Summary and next steps

Using all these tips, you should be able to manage your Event Streams instances and applications like a pro. And, if any issues happen, you will be able to debug them quickly!

There is an FAQs section in the Event Streams documentation that covers some of the most common questions users ask. We recommend you to check it out whenever you have a question.

If you’re all done and ready to take the quiz and earn your IBM Event Streams Developer Essentials badge, go back to the IBM Event Streams Developer Essentials badge page, and click through to the quiz!