Frequent Fluent Bit Restart Events

Problem

The New Relic infrastructure agent is sending excessive Fluent Bit Started and Fluent Bit Stopped events without actual Fluent Bit restarts. This can clutter logs, affect monitoring accuracy, and increase data ingestion, potentially leading to unexpected billing charges.

Cause

This is caused by a bug in the infrastructure agent's hostname resolution component. The agent incorrectly detects frequent hostname changes, which triggers the LogForwarderSupervisor to send these false Fluent Bit restart events. You might see thousands of debug messages similar to the following example:

time="2024-04-20T02:46:47Z" level=debug msg="Notifying observers." change=2 component=HostnameResolver
time="2024-04-20T02:46:47Z" level=debug msg="Observer notified." component=HostnameResolver name=LogForwarderSupervisor

Solution

To resolve this issue, modify the hostname resolution behavior in your infrastructure agent's configuration. This fix applies to both Windows and Linux systems.

Add the following configuration to your infrastructure agent configuration file:

Linux: /etc/newrelic-infra.yml Windows: C:\Program Files\New Relic\newrelic-infra\newrelic-infra.yml

dns_hostname_resolution: false
override_hostname: "your-server-hostname"

Replace "your-server-hostname" with your actual server hostname.

This configuration:

• Disables reverse DNS lookups that may be causing unstable hostname detection
• Forces the agent to use a specific hostname, preventing any hostname change detection

Apply the configuration

After you add the configuration:

1. Save the configuration file.

2. Restart the New Relic infrastructure agent:

   Linux:

   bash

   Copier

   $
   sudo systemctl restart newrelic-infra

   Windows:

   Restart-Service newrelic-infra

   Copier

Verification

To confirm the issue is resolved:

1. Monitor your infrastructure agent logs for the absence of repeated HostnameResolver messages.
2. Check the New Relic dashboard to ensure that Fluent Bit Started and Fluent Bit Stopped events are no longer appearing continuously.
3. Verify that log forwarding continues to function normally and that logs appear as configured.

Additional resources

• Infrastructure agent hostname resolution documentation
• Infrastructure agent configuration settings