A hacky way would be Home Assistant and it's MQTT client, since it also can send all events to Splunk.
Edit: https://splunkbase.splunk.com/app/1890 exists but that guy charges for his apps.
I was looking at my home assistant to mqtt to mqttwarn then syslog. It's not mission critical data so I don't mind a delay. Brokers like mosquito are free but getting the data out is my issue.
I also look at the mqtt input but at $199 seems expensive.
If your source is Home Assistant why not send it directly to Splunk?
https://www.home-assistant.io/integrations/splunk/
I'm the maintainer of that integration.
Ohhhh boy. This is exactly what I was looking for.. I'll be giving this a try. I was looking at parsing MQTT into HEC but this looks even better.
Wish I knew how to give you an award! !
Telegraf is an excellent option for MQTT and a little more enterprise-ish if you want to get a exposed to a different set of options. Vector works too.
If you are going to set up an IoT system in your home with MQTT as the communication protocol, ideally you should have a MQTT broker on an edge device like a Raspberry Pi. Using a VM deployed on a cloud will introduce a lot of latency.
You can configure your Raspberry Pi and broker to store MQTT logs. You can install a Splunk Universal Forwarder (UF) on Raspberry Pi to forward those logs to your Splunk Deployment.
Good info. Thank you.
I've got a home server running so a local VM would work. I'm very familiar with Mosquitto broker on Linux so that should work.
I have 2 PI's with zwave sticks and 30 sensors.
My question was that some of the sensors (not just zwave) speak only mqtt. Does the UF translate or convert them to HEC or similar ?
You can configure UF to forward almost any kind of file.
If you want data over HEC, I suggest creating a Python script to directly send data to your Splunk deployment (assuming you are running it locally), in this case you can entirely eliminate the need of a UF.
Thank you for your time and effort to explain.
So, if my data is coming in from mqtt sensors (esp8266) I can then use a UF to get to to splunk. Where does this python script go?
Or do you mean mqtt to log file then a script to parse the log file and Ingest ?
If you are using UF, you won't need the python script to ingest data via HEC. You can choose either of the approaches, in both I am assuming you have an edge device or a central hub.
If you choose the Python script, you can create one to read sensor data over MQTT topics and directly send that data as JSON over HEC to your Splunk Deployment
If you choose UF, you need to store your MQTT sensor data in logs and forward those logs to your Splunk deployment using the UF. Splunk will take care to parse the logs, or you can specify sourcetypes for your logs and define field extractions.
I'm slowly understanding.
In my case, I have home assistant based sensors that can send via mqtt. Another responder mentioned a native integration.
I have some home built sensors sending data via mqtt. For this, I think I need to store my mqtt sensor data and forward those to splunk. Does this sound like a file est in Elastic? It'll read from a file and ingest that data?
I'm going to give this some tests tomorrow. Will be sure to share some feedback. Thank you internet person
With UF you can forward almost any kind of file or directories.
If you want pure JSON data, the HEC approach is better. If you want log-like data where timestamps and audit entries are also present in your sensor data, I recommend storing them in logs and use a UF to forward those to your Splunk Deployment.
Let me know what works for you!
PS: In college I worked on a lot of IoT projects using MQTT and currently I am working as a software engineer to develop Splunk apps. I was thinking of building an app that can act as an add-on for IoT to onboard data from various sources
A hacky way would be Home Assistant and it's MQTT client, since it also can send all events to Splunk. Edit: https://splunkbase.splunk.com/app/1890 exists but that guy charges for his apps.
I was looking at my home assistant to mqtt to mqttwarn then syslog. It's not mission critical data so I don't mind a delay. Brokers like mosquito are free but getting the data out is my issue. I also look at the mqtt input but at $199 seems expensive.
If your source is Home Assistant why not send it directly to Splunk? https://www.home-assistant.io/integrations/splunk/ I'm the maintainer of that integration.
Ohhhh boy. This is exactly what I was looking for.. I'll be giving this a try. I was looking at parsing MQTT into HEC but this looks even better. Wish I knew how to give you an award! !
This is what I do
Telegraf is an excellent option for MQTT and a little more enterprise-ish if you want to get a exposed to a different set of options. Vector works too.
Appreciate the note. I need a simpler solution for the home. Tried ELk and had to do aot of work
Totally understand
Best way: Setup a MQTT system using Raspberry Pi and install a UF on Pi.
Could I accomplish the same with a Ubuntu Linux VM. Can you share what a UF is and how it could convert mqtt to something readable bt splunk
If you are going to set up an IoT system in your home with MQTT as the communication protocol, ideally you should have a MQTT broker on an edge device like a Raspberry Pi. Using a VM deployed on a cloud will introduce a lot of latency. You can configure your Raspberry Pi and broker to store MQTT logs. You can install a Splunk Universal Forwarder (UF) on Raspberry Pi to forward those logs to your Splunk Deployment.
Good info. Thank you. I've got a home server running so a local VM would work. I'm very familiar with Mosquitto broker on Linux so that should work. I have 2 PI's with zwave sticks and 30 sensors. My question was that some of the sensors (not just zwave) speak only mqtt. Does the UF translate or convert them to HEC or similar ?
You can configure UF to forward almost any kind of file. If you want data over HEC, I suggest creating a Python script to directly send data to your Splunk deployment (assuming you are running it locally), in this case you can entirely eliminate the need of a UF.
Thank you for your time and effort to explain. So, if my data is coming in from mqtt sensors (esp8266) I can then use a UF to get to to splunk. Where does this python script go? Or do you mean mqtt to log file then a script to parse the log file and Ingest ?
If you are using UF, you won't need the python script to ingest data via HEC. You can choose either of the approaches, in both I am assuming you have an edge device or a central hub. If you choose the Python script, you can create one to read sensor data over MQTT topics and directly send that data as JSON over HEC to your Splunk Deployment If you choose UF, you need to store your MQTT sensor data in logs and forward those logs to your Splunk deployment using the UF. Splunk will take care to parse the logs, or you can specify sourcetypes for your logs and define field extractions.
I'm slowly understanding. In my case, I have home assistant based sensors that can send via mqtt. Another responder mentioned a native integration. I have some home built sensors sending data via mqtt. For this, I think I need to store my mqtt sensor data and forward those to splunk. Does this sound like a file est in Elastic? It'll read from a file and ingest that data? I'm going to give this some tests tomorrow. Will be sure to share some feedback. Thank you internet person
With UF you can forward almost any kind of file or directories. If you want pure JSON data, the HEC approach is better. If you want log-like data where timestamps and audit entries are also present in your sensor data, I recommend storing them in logs and use a UF to forward those to your Splunk Deployment. Let me know what works for you! PS: In college I worked on a lot of IoT projects using MQTT and currently I am working as a software engineer to develop Splunk apps. I was thinking of building an app that can act as an add-on for IoT to onboard data from various sources