Opentherm gateway + esp8266 weekend project

Domotibart · Post by **Domotibart** » Sunday 02 December 2018 21:40

I have been using the OTGW in domoticz for 2 years now. The first year by USB connection. To get more flexibility i added a esp8266 to connect the OTGW via LAN.
Unfortunately i never got this to work flawless. In the beginning i could not even connect domoticz to the esp8266. For some reason the otmonitor tool worked at once without any problems.
After experimenting with some different firmware for the esp8266 i got it working. At this time use the latest espeasy firmware. But ..... stil i get a lot of errors. It works at startup but after a few hours i reguelarly get zero valuesto eventualy lose the connection. Setting the room temperature from domoticz never worked.
So why does everything work perfect with the otmonitor software and does not in domoticz. I beleive the big difference is that domoticz uses PS=1 to control OTGW and otmonitor does not.
So here my weekend project was born. Let domoticz do the same as otmonitor does. This is how;
The plan is that the ESP is going to parse all the opentherm codes and put the all the sensor data in its dummy sensors. The ESPeasy firmware takes care of sending all the data to Domoticz. Bonus with this method is that i can use domoticz and otmonitor at the same time.
1- First step is to change the ser2net ino from the easyesp firmware. I added some code to parse the opentherm data and create events in the ESP8266. I had never done this so this took some time with trial and error.
2 - Second step make some rules in espeasy to add the parsed data in the corresponding dummy sensors. This is a standard procedure and was easy to do.
3 - Third step was to create virtual sensors in domoticz so that the esp8266 transfers the data to domoticz.
4 - fouth step was to make a virtual setpoint device in domoticz and send the new temperature setpoint to the esp8266 so that the esp8266 can send a serial command to the OTGW setting a temporary temperature. This was the most challenging part becouse i had to make a lua script in domoticz so send a command to the esp8266.
Its now sunday and i have got three values in domoticz; room temperature, room setpoint , boiler temperature and a setpoint temperature thermostat.
It is working for a few hours but it look promising, its the first time that i can set a the therostat from domoticz!
I will put the codes from the ser2net.ino , the lua script, and the esp rules in my next post maybe someone who has more coding skills can optimize them. If not, for me it is a nice workaround to get opentherm working without any errors in domoticz.

Domotibart · Post by **Domotibart** » Sunday 02 December 2018 22:04

Here's the first part of my project. The INO code from the easyesp firmware. I made a new firmware.bin so that you can download it on the esp8266.

Code: Select all

#ifdef USES_P020
//#######################################################################################################
//#################################### Plugin 020: Ser2Net ##############################################
//#######################################################################################################

#define PLUGIN_020
#define PLUGIN_ID_020         20
#define PLUGIN_NAME_020       "Communication - Serial Server"
#define PLUGIN_VALUENAME1_020 "Ser2Net"

#define P020_BUFFER_SIZE 128
boolean Plugin_020_init = false;
byte Plugin_020_SerialProcessing = 0;

WiFiServer *ser2netServer;
WiFiClient ser2netClient;

boolean Plugin_020(byte function, struct EventStruct *event, String& string)
{
  boolean success = false;
  static byte connectionState = 0;

  switch (function)
  {

    case PLUGIN_DEVICE_ADD:
      {
        Device[++deviceCount].Number = PLUGIN_ID_020;
        Device[deviceCount].Type = DEVICE_TYPE_SINGLE;
        Device[deviceCount].Custom = true;
        Device[deviceCount].TimerOption = false;
        break;
      }

    case PLUGIN_GET_DEVICENAME:
      {
        string = F(PLUGIN_NAME_020);
        break;
      }

    case PLUGIN_GET_DEVICEVALUENAMES:
      {
        strcpy_P(ExtraTaskSettings.TaskDeviceValueNames[0], PSTR(PLUGIN_VALUENAME1_020));
        break;
      }

    case PLUGIN_GET_DEVICEGPIONAMES:
      {
        event->String1 = formatGpioName_bidirectional(F("Reset"));
        break;
      }

    case PLUGIN_WEBFORM_LOAD:
      {
      	addFormNumericBox(F("TCP Port"), F("p020_port"), ExtraTaskSettings.TaskDevicePluginConfigLong[0]);
      	addFormNumericBox(F("Baud Rate"), F("p020_baud"), ExtraTaskSettings.TaskDevicePluginConfigLong[1]);
      	addFormNumericBox(F("Data bits"), F("p020_data"), ExtraTaskSettings.TaskDevicePluginConfigLong[2]);

        byte choice = ExtraTaskSettings.TaskDevicePluginConfigLong[3];
        String options[3];
        options[0] = F("No parity");
        options[1] = F("Even");
        options[2] = F("Odd");
        int optionValues[3];
        optionValues[0] = 0;
        optionValues[1] = 2;
        optionValues[2] = 3;
        addFormSelector(F("Parity"), F("p020_parity"), 3, options, optionValues, choice);

      	addFormNumericBox(F("Stop bits"), F("p020_stop"), ExtraTaskSettings.TaskDevicePluginConfigLong[4]);

      	addFormPinSelect(F("Reset target after boot"), F("taskdevicepin1"), Settings.TaskDevicePin1[event->TaskIndex]);

      	addFormNumericBox(F("RX Receive Timeout (mSec)"), F("p020_rxwait"), Settings.TaskDevicePluginConfig[event->TaskIndex][0]);


        byte choice2 = Settings.TaskDevicePluginConfig[event->TaskIndex][1];
        String options2[3];
        options2[0] = F("None");
        options2[1] = F("Generic");
        options2[2] = F("OpenTherm");
        addFormSelector(F("Event processing"), F("p020_events"), 3, options2, NULL, choice2);

        success = true;
        break;
      }

    case PLUGIN_WEBFORM_SAVE:
      {
        ExtraTaskSettings.TaskDevicePluginConfigLong[0] = getFormItemInt(F("p020_port"));
        ExtraTaskSettings.TaskDevicePluginConfigLong[1] = getFormItemInt(F("p020_baud"));
        ExtraTaskSettings.TaskDevicePluginConfigLong[2] = getFormItemInt(F("p020_data"));
        ExtraTaskSettings.TaskDevicePluginConfigLong[3] = getFormItemInt(F("p020_parity"));
        ExtraTaskSettings.TaskDevicePluginConfigLong[4] = getFormItemInt(F("p020_stop"));
        Settings.TaskDevicePluginConfig[event->TaskIndex][0] = getFormItemInt(F("p020_rxwait"));
        Settings.TaskDevicePluginConfig[event->TaskIndex][1] = getFormItemInt(F("p020_events"));
        success = true;
        break;
      }

    case PLUGIN_INIT:
      {
        LoadTaskSettings(event->TaskIndex);
        if ((ExtraTaskSettings.TaskDevicePluginConfigLong[0] != 0) && (ExtraTaskSettings.TaskDevicePluginConfigLong[1] != 0))
        {
          #if defined(ESP8266)
            byte serialconfig = 0x10;
          #endif
          #if defined(ESP32)
            uint32_t serialconfig = 0x8000010;
          #endif
          serialconfig += ExtraTaskSettings.TaskDevicePluginConfigLong[3];
          serialconfig += (ExtraTaskSettings.TaskDevicePluginConfigLong[2] - 5) << 2;
          if (ExtraTaskSettings.TaskDevicePluginConfigLong[4] == 2)
            serialconfig += 0x20;
          #if defined(ESP8266)
            Serial.begin(ExtraTaskSettings.TaskDevicePluginConfigLong[1], (SerialConfig)serialconfig);
          #endif
          #if defined(ESP32)
            Serial.begin(ExtraTaskSettings.TaskDevicePluginConfigLong[1], serialconfig);
          #endif
          ser2netServer = new WiFiServer(ExtraTaskSettings.TaskDevicePluginConfigLong[0]);
          ser2netServer->begin();

          if (Settings.TaskDevicePin1[event->TaskIndex] != -1)
          {
            pinMode(Settings.TaskDevicePin1[event->TaskIndex], OUTPUT);
            digitalWrite(Settings.TaskDevicePin1[event->TaskIndex], LOW);
            delay(500);
            digitalWrite(Settings.TaskDevicePin1[event->TaskIndex], HIGH);
            pinMode(Settings.TaskDevicePin1[event->TaskIndex], INPUT_PULLUP);
          }

          Plugin_020_init = true;
        }
        Plugin_020_SerialProcessing = Settings.TaskDevicePluginConfig[event->TaskIndex][1];
        success = true;
        break;
      }

    case PLUGIN_TEN_PER_SECOND:
      {
        if (Plugin_020_init)
        {
          size_t bytes_read;
          if (ser2netServer->hasClient())
          {
            if (ser2netClient) ser2netClient.stop();
            ser2netClient = ser2netServer->available();
            addLog(LOG_LEVEL_ERROR, F("Ser2N: Client connected!"));
          }

          if (ser2netClient.connected())
          {
            connectionState = 1;
            uint8_t net_buf[P020_BUFFER_SIZE];
            int count = ser2netClient.available();
            if (count > 0)
            {
              if (count > P020_BUFFER_SIZE)
                count = P020_BUFFER_SIZE;
              bytes_read = ser2netClient.read(net_buf, count);
              Serial.write(net_buf, bytes_read);
              Serial.flush(); // Waits for the transmission of outgoing serial data to complete

              if (count == P020_BUFFER_SIZE) // if we have a full buffer, drop the last position to stuff with string end marker
              {
                count--;
                addLog(LOG_LEVEL_ERROR, F("Ser2N: network buffer full!"));
              }
              net_buf[count] = 0; // before logging as a char array, zero terminate the last position to be safe.
              char log[P020_BUFFER_SIZE + 40];
              sprintf_P(log, PSTR("Ser2N: N>: %s"), (char*)net_buf);
              addLog(LOG_LEVEL_DEBUG, log);
            }
          }
          else
          {
            if (connectionState == 1) // there was a client connected before...
            {
              connectionState = 0;
              // workaround see: https://github.com/esp8266/Arduino/issues/4497#issuecomment-373023864
              ser2netClient = WiFiClient();
              addLog(LOG_LEVEL_ERROR, F("Ser2N: Client disconnected!"));
            }

            while (Serial.available())
              Serial.read();
          }

          success = true;
        }
        break;
      }

    case PLUGIN_SERIAL_IN:
      {
        if (Plugin_020_init)
        {
        uint8_t serial_buf[P020_BUFFER_SIZE];
        int RXWait = Settings.TaskDevicePluginConfig[event->TaskIndex][0];
        if (RXWait == 0)
          RXWait = 1;
        int timeOut = RXWait;
        size_t bytes_read = 0;
        while (timeOut > 0)
        {
          while (Serial.available()) {
            if (bytes_read < P020_BUFFER_SIZE) {
              serial_buf[bytes_read] = Serial.read();
              bytes_read++;
            }
            else
              Serial.read();  // when the buffer is full, just read remaining input, but do not store...

            timeOut = RXWait; // if serial received, reset timeout counter
          }
          delay(1);
          timeOut--;
        }

        if (bytes_read != P020_BUFFER_SIZE)
        {
          if (bytes_read > 0) {
            if (Plugin_020_init && ser2netClient.connected())
            {
              ser2netClient.write((const uint8_t*)serial_buf, bytes_read);
              ser2netClient.flush();
            }
          }
        }
        else // if we have a full buffer, drop the last position to stuff with string end marker
        {
          while (Serial.available()) // read possible remaining data to avoid sending rubbish...
            Serial.read();
          bytes_read--;
          // and log buffer full situation
          addLog(LOG_LEVEL_ERROR, F("Ser2N: serial buffer full!"));
        }
        serial_buf[bytes_read] = 0; // before logging as a char array, zero terminate the last position to be safe.
        char log[P020_BUFFER_SIZE + 40];
        sprintf_P(log, PSTR("Ser2N: S>: %s"), (char*)serial_buf);
        addLog(LOG_LEVEL_DEBUG, log);

        // We can also use the rules engine for local control!
        if (Settings.UseRules)
        {
          String message = (char*)serial_buf;
          String message1 = "";
          String dot = ".";
          int NewLinePos = message.indexOf(F("\r\n"));
          if (NewLinePos > 0)
            message = message.substring(0, NewLinePos);
          String eventString = "";

          switch (Plugin_020_SerialProcessing)
          {
            case 0:
              {
                break;
              }

            case 1: // Generic
              {
                eventString = F("!Serial#");
                eventString += message;
                break;
              }

            case 2: // opentherm
              {
                if (message.substring(3,5) == "10")
                {
                  message1 = strtol(message.substring(7,9).c_str(),0,16);
                  message = strtol(message.substring(5,7).c_str(),0,16);
                  eventString = ("room#setpoint=");
                  eventString += message += dot += message1;
                }
                if (message.substring(3,5) == "18")
                {
                  message1 = strtol(message.substring(7,9).c_str(),0,16);
                  message = strtol(message.substring(5,7).c_str(),0,16);
                  eventString = F("room#temp=");
                  eventString += message += dot += message1;
                }
                if (message.substring(3,5) == "1A")
                {
                  message1 = strtol(message.substring(7,9).c_str(),0,16);
                  message = strtol(message.substring(5,7).c_str(),0,16);
                  eventString = F("tapwater#temp=");
                  eventString += message += dot += message1;
                }
                if (message.startsWith("B4019"))
                {
                  message1 = strtol(message.substring(7,9).c_str(),0,16);
                  message = strtol(message.substring(5,7).c_str(),0,16);
                  eventString = ("boiler#temp=");
                  eventString += message += dot += message1;
                }
                break;
              }
          } // switch

          if (eventString.length() > 0)
            rulesProcessing(eventString);

        } // if rules
        success = true;
        break;
      }
      }

    case PLUGIN_WRITE:
      {
        String command = parseString(string, 1);
        if (command == F("serialsend"))
        {
          success = true;
          String tmpString = string.substring(11);
          Serial.println(tmpString); // FIXME TD-er: Should this also use the serial write buffer?
        }
        break;
      }

  }
  return success;
}
#endif // USES_P020

Xenomes · Post by **Xenomes** » Friday 14 December 2018 11:56

This is very interesting! I have issue with the OTGW to Domoticz to. I am using the OTGW with a NodeMCU from the Nodoshop. My communication stops and the log fills with "Status: OTGW: *" messages, then i open the OTMonitor everything looks fine. I hope you fix will help me too.
Running on a Pi3B+ Ubuntu Mate Domoticz 4.10.272

Domotibart · Post by **Domotibart** » Sunday 16 December 2018 11:59

I Made a new firmware for the esp8266. It is the ESPEasy firmware with the a modified ser2net plug-in.
At this moment it parses the 4 values from the gateway, roomtemp, boiler temp, hotwater temp and roomtemp setpoint.
The values are stored in Virtual sensors on the esp8266. The espeasy firmware sends the values to domoticz (standaard function in the espeasy firmware).
This SETUP is running for a week now without any problems. Nice bonus, this approach makes it possible to use Domoticz and the Otmonitor at the same time!
Also the TT command is working fine now, which I never got working with the original setup.
Next step is to add the rest of the sensor data. I added a screenshot from the ESPEasy sensor data

B7en9 · Post by **B7en9** » Tuesday 08 January 2019 20:43

I have exactly the same problem, thank you for trying to make a working concept!

Can you add some screenshots how it looks in Domotics? Can you share the bin-file?

Will you add more functions?

mvzut · Post by **mvzut** » Tuesday 08 January 2019 22:55

I also stepped over from a wired OTGW (with a USB-to-serial cable) to a wireless setup (with a D1 Mini ESP8266 board) a few weeks ago. I flashed the latest ESPEasy firmware and set up a ser2net device with the correct serial parameters. After some initial hickups (I only saw weird codes instead of OTGW updates in the log, which was solved by a reset of just the OTGW), everything is working practically flawless. Actually even better than before, my old setup sometimes gave timeout errors. How can it be that you guys have a different experience? If you want I can explain details about the process I followed, maybe this can help you find the root cause of the issues.

P.S. Still very interested in your alternative route, always nice to have more than one option! Although I would need support of water pressure and actual water & return temperatures before I would consider trying out your solution.

Domotibart · Post by **Domotibart** » Wednesday 09 January 2019 23:22

A status update.
Still working to make the program easy to use. I made a screenshot from the esp8266 firmware.
The only thing that you have to do is make virtual sensors in domoticz.
Once this is done you only have to put the corresponding idx numbers in the esp configuration.
This weekend I will add more sensor data in the firmware. Eventually I will put all data from the opentherm gateway in the firmware.

B7en9 · Post by **B7en9** » Saturday 12 January 2019 9:01

This looks great!

Can you share your latest bin file?

How can you select the temperature in Domoticz you want to have?

Domotibart · Post by **Domotibart** » Saturday 12 January 2019 20:40

A little update.
I added the water pressure and the return water temp. My CV does not have these sensors so i could not test them.
I changed the domoticz settings. it uses the settings that you can setup in the controller tab of easyesp. For now it uses the first controller from the list!
I uploaded the firmware. Use it at your own risk. Its for the ESP8266_4096 only!

I think it is not possible to upload it here so heres a link;
https://drive.google.com/file/d/1QuDcLy ... sp=sharing

B7en9 · Post by **B7en9** » Sunday 13 January 2019 10:51

Thank you for sharing, flashing did go well.

If I go to to the IP http://adres:500 I see the Opentherm codes, so that is working. There is communication between Opentherm gateway and the ESPmodule.

But the weird thing is that Domotics doesn't get data. What are your other settings in EspEasy and Domoticz?

sincze · Post by **sincze** » Sunday 13 January 2019 11:09

Domotibart wrote: ↑Saturday 12 January 2019 20:40 A little update.
I added the water pressure and the return water temp. My CV does not have these sensors so i could not test them.
I changed the domoticz settings. it uses the settings that you can setup in the controller tab of easyesp. For now it uses the first controller from the list!
I uploaded the firmware. Use it at your own risk. Its for the ESP8266_4096 only!

I think it is not possible to upload it here so heres a link;
https://drive.google.com/file/d/1QuDcLy ... sp=sharing

Very nice. And ofcourse we like images just as "mvzut" did

What is the big difference /advantage with your approach and not using the regular ESPEasy plugin: "Communication - Serial Server" ?

Did you make any modifications to power the ESP from the OTG board? So I heard the default power supply was not sufficient to power such a device? https://gathering.tweakers.net/forum/li ... 1653967/11

mvzut · Post by **mvzut** » Sunday 13 January 2019 11:25

sincze wrote: ↑Sunday 13 January 2019 11:09 Did you make any modifications to power the ESP from the OTG board? So I heard the default power supply was not sufficient to power such a device? https://gathering.tweakers.net/forum/li ... 1653967/11

I also read this, but only after I had already tried powering it directly from the OTGW board. It seems to work well, although I recently found out that the ESP8266 reboots every few days or sometimes even after a few hours, probably because of insufficient power... This doesn't seem to have any negative impact on the functioning in Domoticz though. On the internet there are suggestions for better power components on the OTGW, maybe I'll try that at some point.

Domotibart · Post by **Domotibart** » Sunday 13 January 2019 11:45

I also had a lot of problems with the esp rebooting. First step I did was a separate power supply for the esp.
But after that it keep having problems with the connection. Sometimes it worked for a day or two and than it suddenly stops functioning. Another strange thing was that I regularly got zero values for all the sensors. This is very annoying when you want to see the temperature graph.
I think it has something to do with the WiFi stability of the esp8266. I have a very crowded WiFi network with a lot of devices, I suspect this is not helping keeping the telnet connection stable.
This is why I made this firmware. The esp is not only transferring all the serial data from the OTgateway to domoticz. It reads all data and only if there is a new valua for one of the sensors it send this data to domoticz using JSon.
This way there is no constant telnet connection. But also the gateway does not need to be in the PS=1 status.
Another extra bonus is that you can use Otmonitor and domoticz at the same time!

Domotibart · Post by **Domotibart** » Sunday 13 January 2019 12:16

B7en9 wrote: ↑Sunday 13 January 2019 10:51 Thank you for sharing, flashing did go well.

If I go to to the IP http://adres:500 I see the Opentherm codes, so that is working. There is communication between Opentherm gateway and the ESPmodule.

But the weird thing is that Domotics doesn't get data. What are your other settings in EspEasy and Domoticz?

That is correct, in the Esp you also have to configure the IP address and the port of you’re domoticz server. You can do this at the “controllers” tab. At this moment my firmware uses the config off the first controller in this tab.
After that you have to make all the virtual sensors in domoticz so do not use the domoticz - remote server anymore!
Making virtual sensors in domoticz is not very complicated there is plenty of information on how to do this.

B7en9 · Post by **B7en9** » Sunday 13 January 2019 13:14

That is correct, in the Esp you also have to configure the IP address and the port of you’re domoticz server. You can do this at the “controllers” tab. At this moment my firmware uses the config off the first controller in this tab.
After that you have to make all the virtual sensors in domoticz so do not use the domoticz - remote server anymore!
Making virtual sensors in domoticz is not very complicated there is plenty of information on how to do this.

Thank you for helpen, my house is full of sensors, so it is quite %^$#%$ that it is not working. Domoticz doesn't get info, is there a way how you can see what the ESP sends out?

Domotibart · Post by **Domotibart** » Sunday 13 January 2019 13:29

There is. On the esp at the “tools” “advanced” tab you can set logging options. Set the “web log level” to “info”
Now you can see the data that the esp is sending to domoticz. Look at “tools” “log”.
After a minute or so you should see things like in my attachement.

B7en9 · Post by **B7en9** » Sunday 13 January 2019 13:38

ofcourse stupid, knew that.

get only:
1832203: WD : Uptime 31 ConnectFailures 0 FreeMem 17792
1862203: WD : Uptime 31 ConnectFailures 0 FreeMem 17792
1892203: WD : Uptime 32 ConnectFailures 0 FreeMem 17768

Certainly something wrong, no communication happening. Weird, something is happening because I get info when i go to IP:500

Domotibart · Post by **Domotibart** » Sunday 13 January 2019 13:42

Did you put you’re IP address and port of you’re domoticz server in the “controllers” tab of the esp?

B7en9 · Post by **B7en9** » Sunday 13 January 2019 13:52

this is what I did:

Domotibart · Post by **Domotibart** » Sunday 13 January 2019 13:56

Should be ok. The address and port are the same address that you put in you’re browser when you are using domoticz.
Second check, are the idx numbers of you’re virtual sensors the same as the idx numbers that you have to put in the esp Otha config?