1. Introduction

This tutorial will show, step by step, how to set up a DataDog custom provider that would allow us to pick logs stored in the file on filesystem. After the proper configuration, changes on the log file will be visible in DataDog.

We'll walk you through the example of NiFi - DataDog integration. This example was created on the Windows operating system, but I guess that it's similar for Unix-based systems.

For the purpose of this tutorial, we used a trial version of DataDog.

2. Steps

From a general perspective, these are the steps:

  1. Install the DataDog agent on our machine
  2. Locate the file where logs are stored
  3. Add and configure a custom provider for DataDog

2.1. Install the DataDog agent

First, we need to install the DataDog agent on our machine. For the Windows system, we can get it here.

After installation, if everything is fine you should be able to see the DataDog manager tray icon. The manager is used for managing DataDog agents. It informs us about their status. It’s running on http://host:5002.

2.2. Locate the file where logs are stored

NiFi application logs are stored in {nifiRoot}/logs/nifi-app.log. So, we need to configure the DataDog to listen for changes on that log file.
To send these logs to DataDog we need to enable the log listener in the DataDog configuration file datadog.yaml found in C:\ProgramData\Datadog on Windows.
We enable logs by setting the following parameter to true:

logs_enabled: true

This parameter is false by default and the manager will also show in the Logs Agent section that this agent is not running.

2.3. Add and configure a custom provider for DataDog

Also, we need to configure the custom provider for DataDog to know which file to listen to. For this we need to go to the conf.d folder in our DataDog installation and create a newProviderName.d folder where we’ll create config.yaml file for our provider.
The content for this file should be as follows:


logs:
- type: file
path: "C:/nifi-1.12.1/logs/nifi-app.log"
service: "nifi"
source: "nifi"

Since in this example we’re listening for changes in a file, type should be file and path should contain the location of the log file we’re listening to.

After we do all the setup steps,  we need to restart the agent so that our new configuration is applied.
After the restart Logs Agent section on Data should show that Logs Agent is up with listed statuses and running log listeners. There we should also see our configured listener and it’s status.

DataDog Logs Agent

Now we can go to the DataDog UI, and check if our logs are there:

3. Conclusion

In this tutorial, we showed how to make Apache NiFi logs visible on DataDog.

DataDog is a handy tool that could be easily configured to support custom use cases. For more learning materials on DataDog you can check their official documentation. 

Check also...