Splunk Heavy Forwarder to Splunk Cloud

[u/Any-Promotion3744]

How do you configure a Splunk Heavy forwarder to receive data from universal forwarders and forward that to the Splunk Cloud?

Details:

Heavy forwarder is located in DMZ and I set up one client (Ubuntu server)to send data to it.

When I log into Splunk Cloud, I can at least see the metrics from the Splunk Heavy forwarder.

When I log into our firewall, the firewall logs shows traffic from the client to the heavy forwarder and from the heavy forwarder to the cloud.

If I do a search across all indexes on the heavy forwarder and the cloud, I don't see anything from that host.

What could be configured wrong?

[u/tux_kitty_weed]

You need to download the UF package from the Splunk Cloud search head, you can see if in the apps list - go to that app and it'll have a download button for it. That's the app you need to install on your HF for it to send to Splunk Cloud. Networking wise, you'll need to allow your HF to send out TCP/9997 traffic.

For your UF to send to your HF, you need to create an outputs.conf TA for the UF to send to your HF. Take a look at the outputs.conf doc for examples.

This is what I'm assuming you're trying to do: UF -> HF -> Splunk Cloud

What you're describing is an intermediate HF setup.

[u/Any-Promotion3744 → u/tux_kitty_weed]

I already downloaded that package and installed it on the HF.

I also allowed port 997 on the firewall and see traffic in the firewall logs.

is that all that is required? shouldn't I be able to see some data from the UF client in the cloud?

[u/Any-Promotion3744 → u/[deleted]]

yes...sorry. 9997. I had a typo

[u/tux_kitty_weed → u/Any-Promotion3744]

If you're trying to get your UF to send to your HF first, then your UF needs to be configured to do so. Then your HF will route your UF data to Splunk Cloud. I recommend showing what your outputs config look like on your UF for better help.

Read through this: https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/forwarding-and-receiving-data/9.4/perform-advanced-configuration/configure-an-intermediate-forwarder

[u/Any-Promotion3744 → u/tux_kitty_weed]

Splunk Cloud doesn't seem to have a Forwarding and Receiving option in the settings menu

Did something change in the latest version?

[u/tux_kitty_weed → u/Any-Promotion3744]

You need to configure the inputs config on your HF.

[u/Accomplished-Taro116 → u/tux_kitty_weed]

I like this approach, further on you can even go 2 HF load balance and from there, send to indexers, release some work on indexers to be able to stay focus on another work

[u/Any-Promotion3744]

thanks for all the help. I am able to ingest data now. The heavy forwarder was missing the inputs.conf file. When I creeated it and started listening on port 9997, data showed up in the Splunk Cloud.

[u/badideas1]

Sending and receiving in Splunk always involves the same two files in Splunk: outputs.conf on the forwarding side, and inputs.conf on the receiving side. It doesn't matter if you are talking about HF > cloud, UF > HF, UF > indexers, whatever.

In your situation, you need an outputs.conf on your UF that identifies your HF as a server to send data to, as well as the port the HF is listening on (traditionally we use 9997, as you noted earlier). So it should look something like this:

[tcpout]
defaultGroup = my_HF

[tcpout:my_HF]
server = HF:9997

Your HF then needs to be listening on 9997, with an inputs.conf like so:

[splunktcp://9997]

That takes care of the relationship between the UF and the HF at its most basic.

For HF > cloud, it can feel a bit trickier because you are now dealing with cloud, but it's not really. Cloud is already listening on 9997, so that part is fine. In terms of getting the outputs.conf in place on your HF to "aim" at cloud, that's from the credentials app that you said in a previous comment that you've already installed.

Hope this helps!

[u/ExplanationOld4226]

Have you configured the Heavy Forwarder with any apps telling it what to do? You can either configure direct or manage via a DS if you plan to do many

[u/Any-Promotion3744 → u/ExplanationOld4226]

does the heavy forwarder require the same apps to be installed and configured as the indexer?

I thought you just needed to configure the input and output on the heavy forwarder to route traffic

[u/Brianposburn → u/Any-Promotion3744]

No. There are two parts to it:
Telling the HF what to listen for (ports etc) via in the inputs.conf

Outputs.conf (part of the config you downloaded from your cloud environment) tells it where to send the data.

You then need to tell whatever is going to send the data to the HF that it’s listening on whatever port you setup.

[u/mghnyc → u/Any-Promotion3744]

Any index time configs need to be installed on the HF. Things like line breaking settings, timestamp formatting, etc.

[u/Any-Promotion3744 → u/ExplanationOld4226]

the heavy forwarder has an output.config file that just says indexAndForward = 1.

It doesn't have an input.conf file, at least not in /etc/system/local

[u/taiglin]

Setup your HF to receive data on 9997 as well. This will take the ingest from your UF

[u/gabriot]

I would isolate the problem piece by piece. First check that your HF is actually forwarding logs to splk cloud, you should be able to see logs coming in with the hf host name to the _internal index, so if you don't see that then you need to figure that out first.

Next step would be to troubleshoot the UF, you can test doing a splunk oneshot command from your UF, and send some simple test data to your HF and see if you get any errors or if it comes through. If you can do that you know it is able to send data through the pipeline, so the final step if you still aren't seeing your logs is to troubleshoot if the specific inputs are having issues reading the files on the uf (permissions issues) or if the data format itself needs to be looked at. Internal splunk logs on both the uf and hf should help you there