Email problems

willemd · Post by **willemd** » Wednesday 17 January 2024 19:34

in short: the Ziggo SMTP servers don't accept SMTPUTF8 anymore, only 8BITMIME. This is also visible in the debug info received from the server.

I now might have an entry to the systems people at Ziggo. I left them all the info and hope I will receive some feedback.

glsf91 · Post by **glsf91** » Wednesday 17 January 2024 21:14

willemd wrote: ↑Wednesday 17 January 2024 19:34 in short: the Ziggo SMTP servers don't accept SMTPUTF8 anymore, only 8BITMIME. This is also visible in the debug info received from the server.

Why do you think it is sending with SMTPUTF8 ?

I have my doubts (sorry).
I have a postfix mailserver installed relaying to smtp.ziggo.nl. When sending from domoticz to this mailserver it is working again through ziggo.
And when I disable SMTPUTF8 (to domoticz) on this mailserver it is still working !

willemd · Post by **willemd** » Wednesday 17 January 2024 22:27

When I tried sending via ziggo using mail option smtputf8 I received an error message that the server does not support it (unlike gmail)

When I sent via ziggo I have to use 8bitmime currently.

glsf91 · Post by **glsf91** » Thursday 18 January 2024 7:23

willemd wrote: ↑Wednesday 17 January 2024 22:27 When I tried sending via ziggo using mail option smtputf8 I received an error message that the server does not support it (unlike gmail)
When I sent via ziggo I have to use 8bitmime currently.

Ok, but why do you think domoticz/libcurl is using smtputf8 ?

willemd · Post by **willemd** » Thursday 18 January 2024 9:18

glsf91 wrote: ↑Thursday 18 January 2024 7:23
willemd wrote: ↑Wednesday 17 January 2024 22:27 When I tried sending via ziggo using mail option smtputf8 I received an error message that the server does not support it (unlike gmail)
When I sent via ziggo I have to use 8bitmime currently.
Ok, but why do you think domoticz/libcurl is using smtputf8 ?

I had the impression that if the message is sent as plain text (line 318 in SMTPclinet.cpp) that as a consequence it is sent using SMPTUTF8.

I can see gmail is supporting that but ziggo is not.

Anyway, the core of the problem still seems to be that Ziggo is rejecting a from-adress although it is correct, and then is expecting to retrieve it from the mime message instead.

glsf91 · Post by **glsf91** » Thursday 18 January 2024 9:44

willemd wrote: ↑Thursday 18 January 2024 9:18 I had the impression that if the message is sent as plain text (line 318 in SMTPclinet.cpp) that as a consequence it is sent using SMPTUTF8.

I don't think so. The test message for example has this format:

Code: Select all

From: <email address>
To: <email address>
Subject: Domoticz test
Date: Wed, 17 Jan 2024 18:14:12 +0100
MIME-Version: 1.0
Content-Type: multipart/alternative;
        boundary="--------------601143517647029104953591"
This is a multipart message in MIME format.
----------------601143517647029104953591
Content-type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
<body>
<b>Domoticz test message!</body>
</html>

----------------601143517647029104953591--

With curl there is no problem sending this.
I also plan to try to compile domoitcz by myself so I can add debug info

willemd · Post by **willemd** » Thursday 18 January 2024 9:53

Now I am lost ... since all my testing showed the mime messages were sent correctly.
So why did this domoticz test message fail?
Or was it only the domoticz.notify() that failed, which is without a mail body?

glsf91 · Post by **glsf91** » Thursday 18 January 2024 19:39

I found the cause of the problem. The ziggo mailser is responding on a test messages from domoticz with:

Code: Select all

Message contains bare LF and is violating 822.bis section 2.3

It turns out that the HTMLBody contains only LF instead of CRLF. That is this part:

Code: Select all

<html>
<body>
<b>Domoticz test message!</body>
</html>

When I replace the LF with CRLF the messages is accepted.
There is also a PlainBody possible. I don't know how to trigger this and if this have the same problem of only LF.

Again: I'm testing this on version 2023.1.

willemd · Post by **willemd** » Thursday 18 January 2024 20:56

Below is my python test code.
It tries to send three mails of which the first two are accepted and the third one is not.
In the past, the third one was sent without problems.
Is that explained by your finding? What to change to get the last one working? I can only get it to work by adding the "From:[email protected]" to the message
The error feedback is that the From address is missing or malformed, although the debug also shows that the From address and the To address were accepted.

Maybe they have implemented a more strict protocol compliance check at Ziggo (which explains your finding of \r\n need) and in the process got something else wrong (which explains my finding that the processing of the From address fails)

Code: Select all

import smtplib
from email.mime.multipart import MIMEMultipart
emailData = MIMEMultipart()
emailData['Subject'] = "test mime"
emailData['To'] = "[email protected]"
emailData['From'] = "[email protected]"
smtpserver=smtplib.SMTP('smtp.ziggo.nl',587)
smtpserver.set_debuglevel(1)
smtpserver.starttls()
print("--------------\n",emailData.as_string(),"---------------\n")
smtpserver.login("[email protected]","passwordsender")
smtpserver.sendmail("[email protected]","[email protected]", emailData.as_string())   # this mail is sent without problems
smtpserver.sendmail("[email protected]","[email protected]", "From: [email protected]\r\nTo: [email protected]\r\nSubject: test string with From\r\n") #this mail is sent without problems
smtpserver.sendmail("[email protected]","[email protected]", "test string without From\r\n") # this mail fails with message that From address is missing or malformed
smtpserver.quit()

glsf91 · Post by **glsf91** » Friday 19 January 2024 9:58

willemd wrote: ↑Thursday 18 January 2024 20:56 The error feedback is that the From address is missing or malformed, although the debug also shows that the From address and the To address were accepted.

Maybe there is more changed at ziggo which is causing your python problem now. But this has nothing to do with the problem in domoticz. Domoticz is always adding a From in the message so your third example is not applicable.

Domoticz is using libcurl for sending email notifications. Behaviour in the communication with the smtp server in python can be (and probably will) different. So using python to try to solve a problem in Domoticz makes not much sense in this case.

With all respect for your effort of course.

willemd · Post by **willemd** » Friday 19 January 2024 10:27

glsf91 wrote: ↑Friday 19 January 2024 9:58
willemd wrote: ↑Thursday 18 January 2024 20:56 The error feedback is that the From address is missing or malformed, although the debug also shows that the From address and the To address were accepted.
Maybe there is more changed at ziggo which is causing your python problem now. But this has nothing to do with the problem in domoticz. Domoticz is always adding a From in the message so your third example is not applicable.

Domoticz is using libcurl for sending email notifications. Behaviour in the communication with the smtp server in python can be (and probably will) different. So using python to try to solve a problem in Domoticz makes not much sense in this case.

With all respect for your effort of course.

I used python because Domoticz was not he only system that suddenly had issues.
As a final test I did the following:
Within the mime-message I used a different sender and recipient from the ones I used in the send command, so something like
sendmail(fromaddressnr1,toaddressnr1,"mime-message with fromaddressnr2 and toadressnr2")
Guess what happens?
The mail arrives at the mailbox of toaddressnr1, with in the header fromaddressnr2 and toadressnr2....

Anyway, probably time for you to create an issue on Github with your finding so it can be solved?

glsf91 · Post by **glsf91** » Saturday 20 January 2024 11:03

Created issue: https://github.com/domoticz/domoticz/issues/5993

mm10 · Post by **mm10** » Saturday 20 January 2024 11:09

I too have this issue, but with 2023.2 on debian 11.1

WillemD opened https://github.com/domoticz/domoticz/issues/5989 on the domoticz GitHub but closed it again.
I do not agree that the issue does not need involvement from Domoticz.

While it might be true that other mailservers accept the domoticz mail still, this does not mean that ziggo is incorrectly rejecting them.

Until the real reason is found, this should remain an opportunity for Domoticz to become even better.

I can add some experiments if needed, but what is the current consensus about what happens here in all technical detail?

2Bcontinued...

mm10 · Post by **mm10** » Saturday 20 January 2024 11:11

typical crossover of messages.
When I prepared my message, glsf91 just opened a GitHub issue.
Sorry for the fork.

glsf91 · Post by **glsf91** » Saturday 20 January 2024 16:22

Solution is committed to the development branch of domoticz.

mm10 · Post by **mm10** » Saturday 20 January 2024 17:56

Great work!

Any idea when this is included in a formal release?

BobKooi · Post by **BobKooi** » Tuesday 06 August 2024 17:15

Hello, I have a similar problem with Vimexx mail (mail.zxcs.nl).
2024-08-06 16:55:01.816 Error: libcurl: (67)
2024-08-06 16:55:01.816 Error: Login denied
I am running the latest Domoticz version: 2024.07 on the latest Raspberry Pi OS (based on Debian 12 ' Bookworm” ).
Should this problem be solved in this latest Domoticz formal release?

Post by **waltervl** » Tuesday 06 August 2024 19:09

BobKooi wrote: ↑Tuesday 06 August 2024 17:15
Should this problem be solved in this latest Domoticz formal release?

Yes, it was already in 2024.3 or 2024.4

BobKooi · Post by **BobKooi** » Wednesday 07 August 2024 11:07

Ok Waltervl, thanks for your response. I will continue to look for the cause in my configuration.

Found the problem, it was an authentication problem. The data that was entered automatically turned out to be incorrect. Problem solved!

frank666 · Post by **frank666** » Friday 08 November 2024 20:48

email does not work if I use gmail or hotmail SMTP.

On the other hand, it works if I use other mail handlers.

So I deduce that the mail servers that do not accept domoticz have implemented some devilry to block traffic that does not come from mail clients they do not like.

Has anyone managed to solve or work around this problem or get more information about the new system adopted by the SMTP servers?

Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Re: Email problems

Who is online