[606] Clickatell SMS encoding option

[Kouseri]

Hi,

I've been using Clickatell SMS functionality for a while and it's working very well from Domoticz point of view.
But I've noticed that there are nasty encoding issues which seem to be related to the Clickatell's service. I do live in Finland and we do have letters like ä and ö
which do not show correctly in mobile phone. Please take a look at the picture below for more information.

Clickatell encoding issue

Clickatell_encoding_issue.png (93.42 KiB) Viewed 3628 times

Correct text should be:
-SMS1: Varashälytys!
-SMS2: Varashälytin kytketty päälle

I did more research about this one because my wife received SMS messages from the Domoticz but I didn't. I found out that her network operator DNA passes messages through without issues even encoding is messed up but for some reason my network operator Elisa has started blocking SMS messages if there are ä and/or ö letters that encoding has been messed up. I did more tests with the Clickatell's SMS HTTP API and according to the tests, I receive only SMS messages if there are no ä and/or ö letters. I also found out that it's possible to overcome this issue by using unicode encoding. In the following screen capture, last SMS message has been sent with unicode encoding and it's flawless!

Clickatell encoding text

Clickatell_encoding_test.png (96.11 KiB) Viewed 3628 times

So I propose to have an option (tick box for example) to select unicode encoding in the Clickatell SMS settings in the Domoticz. If tick box is checked then the Domoticz automatically adds following parameter to the Clickatell SMS command query according to the Clickatell's HTTP spec:

Code: Select all

&unicode=1

Of course text which will be sent needs to be in UTF-16BE format. Conversion can be done manually for example using the the Clickatell's Unicode converter (requires Clickatell account) if TEXT -> UTF-16BE conversion cannot be done automatically by the Domoticz.

Edit: Tracker number added to the subject

[BakSeeDaa]

Hello @Kouseri

To use unicode is probably not the best option in your case. It would shorten the max length of each SMS with 50%.
You can just convert your SMS text to the GSM 03.38 character set and encode it for URL.

I made a Lua function yesterday that does exactly that. I share it here in case you want to use it.

Code: Select all

function smsEncode(str)
	-- Convert to GSM 03.38 character set and URL-encode
	local utf8Chars={"%%","\n"," ",'"',"&",",","%.","/",":",";","<","=",">","?","¡","£","#","¥","§","Ä","Å","à","ä","å","Æ","Ç","É","è","é","ì","Ñ","ñ","ò","ö","Ø","Ö","Ü","ù","ü","ß", "\\", "*", "'", "%(", "%)", "@", "%+", "%$", "%[", "%]", "%^", "{", "|",  "}", "~"}
	local gsmChars={"%%25","%%0D","%%20","%%22","%%26","%%2C","%%2E","%%2F","%%3A","%%3B","%%3C","%%3D","%%3E","%%3F","%%A1","%%A3","%%A4","%%A5","%%A7","%%C4","%%C5","%%E0","%%E4","%%E5","%%C6","%%C7","%%C9","%%E8","%%E9","%%EC","%%D1","%%F1","%%F2","%%F6","%%D8","%%D6","%%DC","%%F9","%%FC","%%DF","%%5C","%%2A","%%27","%%28","%%29","%%40","%%2B","%%24","%%5B","%%5D","%%5E","%%7B","%%7C","%%7D","%%7E"}
	for i = 1, #utf8Chars, 1 do
		str = string.gsub(str, utf8Chars[i], gsmChars[i])
	end
	return str 
end

Below is 2 more Lua function that You can use to send SMS. Create user variable "SMS-Away" and set it to the default receiver number first.

Code: Select all

function OpenURL(url)
	-- The commandArray also supports nested tables to send the same command type multiple times.
	-- To do this, give the command a numeric index
	commandArray[#commandArray + 1]={['OpenURL']=url }
end

Code: Select all

-- SMS messages are sent by making HTTP calls through the Clickatell API
function sendSMS(SMSText, PhoneNumber)
	if ((PhoneNumber == "") or (PhoneNumber == nil)) then PhoneNumber = uservariables["SMS-Away"] end
	OpenURL("https://api.clickatell.com/http/sendmsg?user="..uservariables["ClickatellAPIUser"].."&password="..uservariables["ClickatellAPIPassw"].."&api_id="..uservariables["ClickatellAPIId"].."&from=+"..uservariables["ClickatellSender"].."&to="..PhoneNumber.."&text=".. smsEncode(SMSText))
end

A better alternative ... If You like, You can download the GCAlDom-*-*-*.zip from this topic extract it and save the file named bakseeda.lua in your ~/domoticz/scripts/lua/ folder. Create user variable "SMS-Away" and set it to the default SMS receiver phone number. After doing that you should be able to use the functions in your Lua scripts just like this:

Code: Select all

local b = require("scripts/lua/bakseeda")
b.sendSMS("Puhun vain vähän suomea")

Cheers

Edit: updated the smsEncode script to include all characters that should be URL-encoded.

[Kouseri]

Hi BakSeeDaa

Thanks for the interesting information and tips. I think I could use your smsEncode function in my scripts. I'll give a try and let's see how it goes.
By the way I didn't know that using unicode would reduce max length of each SMS so radically, so I agree using unicode doesn't make sense.

PS. Your finnish seems to be just fine...

[Kouseri]

I played a bit with the smsEncode() function it works great so thanks for the help. But there are still some odd things happening in Domoticz related to Clickatell's SMS sending.
For example when I send the SMS with the Clickatell's HTTP API using following text "Paloh%E4lytys", received message is "Palohälytys" which correct from encoding point of view.

Code: Select all

http://api.clickatell.com/http/sendmsg?user=USERNAME&password=PASSWORD&api_id=API_ID&to=RECIPIENT&text=Paloh%E4lytys

But when sending the SMS from lua with same content, encoding gets messed up. Received message is in this case "Paloh%E4lytys".

Code: Select all

commandArray["SendSMS"] = "Paloh%E4lytys"

Does anyone know is Domoticz sending the SMS messages via the Clickatell's HTTP API using same syntax as shown above?

[BakSeeDaa]

An option to be able to send SMS in Domoticz using unicode characters would be useless for most of us unless we are sending text in Hebrew, Chinese, Japanese, Arabic or Korean.

As I wrote before, we'd want to avoid using unicode because the max length of the message will be reduced from 160 down to 80 characters per SMS. Unicode is a 8 bit character code while GSM 03.38 is 7 bit.

German, Italian, French and Scandinavian languages are using accented characters. My suggestion is that Domoticz always encode SMS messages into GSM 03.38 before passing them to ClickaTell. Here is a C# function that does that. I believe that Domoticz is currently just URL-encoding the messages before passing them to ClickaTell.

[Kouseri]

If I got it right Domoticz probably now does an URL encoding for my text and treates %, E and 4 from text "Paloh%E4lytys" as an individual characters and encodes them as the individual characters?
I agree it could make sense if the Domoticz encodes SMS messages into GSM 03.38 before passing them to Clickatell.

Maybe the easiest workaround is use OpenURL functionality in lua...

[Kouseri]

I took a look at the Domoticz source code related to SMS notifications but as far as I understood the code, no encoding is done in that piece of code (I'm not a programmer though).
Another thing is, does following function has something to do with URL encoding....

Code: Select all

HTTPClient::POST("https://api.clickatell.com/http/sendmsg", sPostData.str(), ExtraHeaders, sResult)

I look the source code from the following locations:
https://github.com/domoticz/domoticz/bl ... ationSMS.h
https://github.com/domoticz/domoticz/bl ... ionSMS.cpp

[BakSeeDaa]

If I got it right Domoticz probably now does an URL encoding for my text and treates %, E and 4 from text "Paloh%E4lytys" as an individual characters and encodes them as the individual characters?
I agree it could make sense if the Domoticz encodes SMS messages into GSM 03.38 before passing them to Clickatell.

Maybe the easiest workaround is use OpenURL functionality in lua...

You can't use the Domoticz built in function for sending SMS with accented characters, not even if you manually GSM 03.38 encode the text first. (The GSM 03.38 and URL encoding must be done in one single step as in my LUA script above)

Until Domoticz eventually implements GSM 03.38 encoding for SMS sending, I think the easiest way is to download and use my script bakseeda.lua as described at the end of my posting above.

Code: Select all

local b = require("scripts/lua/bakseeda")
b.sendSMS("Puhun vain vähän suomea")

Edit:
Connection Software is another company that also just like ClickaTell offers to send SMS via a http API. Their documentation about Special characters that must be encoded in HTTP POST submissions was very easy to understand for me and it seems to be working well also with ClickaTell.