Smahub MQTT issue

[solarboy]

I have a new SMA Energy Meter (Home Manager) for my SMA Sunny Boy PV system to control my feed in. I am using Smahub to publish the data (multicast) to MQTT with Home Assistant Autodiscovery as SBFspot does not see/collect this data as far as I know.

It works well but CPU usage is huge and I have narrowed this down to Domoticz if the "hardware" is enabled. If I have Smahub publishing the data to MQTT including HA Discovery and the Domoticz "hardware" entry disabled I do not have this excessive CPU load. As soon as I enable the hardware the CPU load triples and the system becomes a lot less responsive.

It seems there is something about this admittedly very regular MQTT data that causes Domoticz to choke a bit (I checked with "top" and it's definitely Domoticz). There doesn't seem to be a way to restrict the publish interval in Smahub (only a complete data refresh) as the muticast data from the SMA meter is extremely regular.

I was wondering if perhaps I am missing something or if someone else has found a different solution to get the useful data from the SMA Sunny Energy Meter/Home Manager into Domoticz. I don't require it to update with a frequency greater than once every 30-60 seconds.

[FlyingDomotic]

What you can eventually do is a small Python code that read MQTT data of your hub, and publish to another topic last received data every 30 to 60 seconds. You can then read this topic, with less updates. Load of this small Python code will be small.

[solarboy]

Great, thanks for that advice. I had a little "chat" with chatGpt and I arrived at the following python script to run as a service. I then create a new "MQTT Client Gateway" hardware instance with the discovery topic "smahub" and I should get the consumption data.

Code: Select all

import time
import paho.mqtt.client as mqtt
import json

# MQTT Configuration
MQTT_BROKER = "localhost"  # Broker IP
MQTT_PORT = 1883
TOPIC_IN = "SHM2/1901403936/p/1/p1consume"  # Subscription topic for power consumption
TOPIC_OUT_CONFIG = "smahub/config"  # Discovery topic for configuration
TOPIC_STATE = "smahub/state"  # State topic for republished data
PUBLISH_INTERVAL = 60  # Seconds

# Variables to store the latest message
last_message = None

# Callback when a new message is received
def on_message(client, userdata, message):
    global last_message
    last_message = message.payload.decode("utf-8")

# Set up MQTT client and subscribe
client = mqtt.Client()
client.on_message = on_message
client.connect(MQTT_BROKER, MQTT_PORT)
client.subscribe(TOPIC_IN)
client.loop_start()

# Publish the configuration for Home Assistant auto-discovery
def publish_discovery():
    config = {
        "name": "Power Consumption",
        "state_topic": TOPIC_STATE,
        "unit_of_measurement": "W",  # Watts
        "value_template": "{{ value }}",
        "device_class": "power",  # Power measurement
        "unique_id": "unique_id_for_power_consumption"  # Ensure this is unique
    }
    client.publish(TOPIC_OUT_CONFIG, json.dumps(config), retain=True)

# Periodically publish the last received message
try:
    publish_discovery()  # Publish the configuration once at start
    while True:
        if last_message is not None:
            # Publish the state of the last message (power consumption)
            client.publish(TOPIC_STATE, last_message, retain=True)
        time.sleep(PUBLISH_INTERVAL)
except KeyboardInterrupt:
    client.loop_stop()
    client.disconnect()

I have never learned/written python so it will be interesting how this works out.

[waltervl]

It seems you have a lot going on on your MQTT server when you enable smahub.

What is your configuration for smahub? For MQTT AD you only need smahub ha_mqtt enabled with for example update every 60 seconds. That would definitely not overload Domoticz.

https://github.com/AnotherDaniel/smahub ... ks/ha_mqtt

[solarboy]

Smahub appears to publish and incredible amount of messages (78,000 in the last hour compared to 5430 for my Zigbee2mqtt) , the power measurements are every 2 seconds and there is data for 3 phases, power, apparant power and reactive power, so 9 values. I have configured smahub to only update every 60 seconds but that is only a "full" update wether or not items have changed, but they are always changing so the config doesn't do much. I have also deleted the unused devices from Domoticz.

What is odd is that if I don't enable the hardware in Domoticz, CPU usage remains normal despite all these MQTT messages in Mosquitto.

It should be noted that this SMA device's main use is limiting grid feed in and has a 1000ms refresh rate and uses multicast to communicate with the inverters.

[waltervl]

What was your setting of smahub plugin HA_MQTT_UPDATEFREQ ? Should be 30 or 60
What was your setting for the smahub Mqtt plugin UPDATEFREQ ? Should be 30 or 60 seconds delta.

[solarboy]

ha_mqtt.conf "sink"

Code: Select all

[plugin]
enabled = false

[server]
address = 192.0.2.1
username = ""
password = ""

[behavior]
# this determines the number of seconds before a full republish of all collected data, even if there's been no value updates
updatefreq = 60
sensorprefix = homeassistant

shm2.conf "sink" (the energy meter)

Code: Select all

[plugin]
enabled = true

[behavior]
updateFreq = 60
sensorPrefix = SHM2.

mqtt.conf

Code: Select all

[plugin]
enabled = true

[server]
address = 192.0.2.1
port = 1883
username = ""
password = ""
# false to disable tls, 1 for tls v1.1, 2 for tls v1.2
tls = false
tls_insecure = false
ssl_ca = <path-to-file>
ssl_cert = <path-to-file>
ssl_key = <path-to-file>

[behavior]
# this determines the number of seconds before a full republish of all collected data, even if there's been no value updates
updatefreq = 60
# whether to include data units in publication (will publish value-unit pairs as a tuple)
publish_units = false

[FireWizard]

Hi,

In your post, I see that you use as "server" address for MQTT:

Code: Select all

192.0.2.1

This is NOT a private IP address. IP addresses should be in the range of

Code: Select all

192.168.0.0 to 192.168.255.255.

I do not know if this has impact on your issue, but it might be, as your address is in an IANA reserved range.

Regards

[waltervl]

And seeing your config, smahub should not send 78000 mqtt messages in an hour so something seems wrong.

You do not use the docker version of smahub? As that uses another configuration method.

[solarboy]

All working now with this script and trust me smahub truly does spam this much. 128k messages in 24 hours. The config was as delivered.

Code: Select all

import time
import paho.mqtt.client as mqtt
import json

# MQTT Configuration
MQTT_BROKER = "localhost"
MQTT_PORT = 1883
TOPIC_IN = "SHM2/1901403936/p/1/p1consume"  # SMA topic
TOPIC_STATE = "domoticz/in"  # Topic for Domoticz input
PUBLISH_INTERVAL = 60  # Publish every 60 seconds

# Replace with the actual IDX for "usage-electric" in Domoticz
DOMOTICZ_IDX = 3153  # Update this with the correct IDX

# Variable to store the last received message
last_message = None

# Callback for when a message is received from the SMA topic
def on_message(client, userdata, message):
    global last_message
    last_message = message.payload.decode("utf-8")

# Set up MQTT client and subscribe to the input topic
client = mqtt.Client()
client.on_message = on_message
client.connect(MQTT_BROKER, MQTT_PORT)
client.subscribe(TOPIC_IN)
client.loop_start()

# Periodically publish the last received message to Domoticz input topic
try:
    while True:
        if last_message is not None:
            # Format message for Domoticz
            state_message = {
                "idx": DOMOTICZ_IDX,
                "nvalue": 0,
                "svalue": last_message  # Directly use received power consumption value
            }
            client.publish(TOPIC_STATE, json.dumps(state_message), retain=True)
        time.sleep(PUBLISH_INTERVAL)
except KeyboardInterrupt:
    client.loop_stop()
    client.disconnect()

This clones 1 device "consumption" to a new virtual sensor and back down to sensible CPU use (16%)

[solarboy]

Hi,

In your post, I see that you use as "server" address for MQTT:

Code: Select all

192.0.2.1

This is NOT a private IP address. IP addresses should be in the range of

Code: Select all

192.168.0.0 to 192.168.255.255.

I do not know if this has impact on your issue, but it might be, as your address is in an IANA reserved range.

Regards

Well spotted. That configuration is as it came from github and even like that somehow it was publishing to my mosquitto server at 192.168.2.9 so I didn't even check that as I could see the messages and topics being dynamically created in Mqtt explorer. I have now changed them to 192.168.2.9 but there are still about 20 messages being published every 5 seconds, however with the script above this doesn't cause Domoticz to panic anymore.

[solarboy]

Just noticed I'd configured the mqtt broker correctly in the docker-compose.yml which overides these settings.

[FireWizard]

Hello @solarboy,

but there are still about 20 messages being published every 5 seconds

That is really nothing for MQTT as Mosquitto can handle 18,000 + messages per second.
So your Mosquitto broker will not be the limiting factor.

Regards

[solarboy]

Agreed. The system doesn't bog down until I try and subscribe to the topic with auto discovery in Domoticz. This is when CPU load goes through the roof and become extremely unresponsive. However the work around to run a small python script to republish just the desired topics at a lower frequency works perfectly. The interval setting in smahub seems to do nothing.

[solarboy]

Here is the final python script I ended up with which I will run as a service. This uses a single virtual "Usage (electric)" device (I named mine "exporting") is bi-directional for supply and consumption.

Code: Select all

import time
import paho.mqtt.client as mqtt
import json

# MQTT Configuration, change to suit
MQTT_BROKER = "localhost"
MQTT_PORT = 1883
TOPIC_CONSUME = "SHM2/<your SMA energy meter/home manager serial number>/p/pconsume"  # SMA topic for power consumption
TOPIC_SUPPLY = "SHM2/<your SMA energy meter/home manager serial number>/p/psupply"    # SMA topic for power supply
TOPIC_STATE = "domoticz/in"                   # Topic for Domoticz input
PUBLISH_INTERVAL = 5                          # Publish every 5 seconds

# Domoticz IDX value (same for both supply and consume, create a virtual "Usage (electric)" sensor and use it's IDX here)
DOMOTICZ_IDX = 1234

# Variables to store the last received messages
last_message_consume = None
last_message_supply = None

# Callback for when a message is received from the SMA topics
def on_message(client, userdata, message):
    global last_message_consume, last_message_supply
    if message.topic == TOPIC_CONSUME:
        last_message_consume = float(message.payload.decode("utf-8"))
    elif message.topic == TOPIC_SUPPLY:
        last_message_supply = float(message.payload.decode("utf-8"))

# Set up MQTT client and subscribe to the input topics
client = mqtt.Client()
client.on_message = on_message
client.connect(MQTT_BROKER, MQTT_PORT)
client.subscribe([(TOPIC_CONSUME, 0), (TOPIC_SUPPLY, 0)])
client.loop_start()

# Periodically publish the last received message to Domoticz input topic
try:
    while True:
        # Check if either message has been received
        if last_message_consume is not None or last_message_supply is not None:
            # Sanity check: if both values are non-None and positive, skip this iteration
            if (last_message_consume is not None and last_message_consume > 0) and \
               (last_message_supply is not None and last_message_supply > 0):
                print("Sanity check failed: both consume and supply are positive. Skipping this update.")
            else:
                # Determine the svalue to send to Domoticz
                if last_message_consume is not None and last_message_consume > 0:
                    svalue = -last_message_consume  # Negative for consumption
                elif last_message_supply is not None and last_message_supply > 0:
                    svalue = last_message_supply   # Positive for supply
                else:
                    svalue = 0  # Default to zero if neither is active

                # Format message for Domoticz
                state_message = {
                    "idx": DOMOTICZ_IDX,
                    "nvalue": 0,
                    "svalue": str(svalue)
                }
                client.publish(TOPIC_STATE, json.dumps(state_message), retain=True)

        # Wait for the next publish interval
        time.sleep(PUBLISH_INTERVAL)
except KeyboardInterrupt:
    client.loop_stop()
    client.disconnect()

[solarboy]

And seeing your config, smahub should not send 78000 mqtt messages in an hour so something seems wrong.

You do not use the docker version of smahub? As that uses another configuration method.

It is in docker yes, the interval is in the docker-compose file

[waltervl]

If smahub is spamming mqtt so much with these settings something is wrong there. I would check your sma docker settings again as it seems something is wrong there.

You can do the python workaround but is is not really helping others using smahub with MQTT. Perhaps make an issue on that repository why it creates 1000 messages in a very short time.
It should not send a message every second if your settings say it should send a message every 60 seconds. So you did something wrong or smahub has a bug.

[solarboy]

I opened an issue on github. Apologies if I am not helping anyone, just following the useful (to me) advice of a previous poster.