How to get sources from last stable (2.2563)?

[DaaNMaGeDDoN]

Hi i am struggling a bit with the performance of the new stable (2.3530). It might be just my setup, so i am trying to compare how it works on a vm instead of a Rpi. So far it doesn't look promising; without the eventsystem and only the rfxtrx443e recieving/sending commands it still is a load on the host. I have a disaster recovery of the old rpi sd card that is based on the previous stable.
Now i need to get the sources for that one to compare how that runs in my vm.
I am not new to compiling, however i'm fairly new to github. I can see i should get my hands on the hash of the commit that went with 2.2563 and use that to clone the commit, compile and test.
I am stuck at getting the commit hash. Can somebody help me here? Any easy way of finding that? I think i can handle it from there on, thanks!
/Daan.

[gizmocuz]

the new stable is actually quite faster...

but...

the history file says

Version 2.2563 (June 14th 2015)

so i guess you checkout a branch on github from this date ?

specifically this one ?

https://github.com/domoticz/domoticz/co ... cc56ca0c59

[DaaNMaGeDDoN]

the new stable is actually quite faster...

It should be, but in my case it isnt
http://www.domoticz.com/forum/viewtopic.php?f=4&t=8737 warning: thread has gone off topic
CPU usage increased a bit, responsiveness has gone down. No idea why, maybe its the otgw data (i noticed spikes in cpu load when otgw data comes in), but im not convinced. Lights change _after_ i walked through the room, not when i enter!
Lets see how this runs on my vm, your hash looks good, so the answer was quite simple, just pick the hash from the commit corresponding with the date in the changelog.....sorry for bothering you

Code: Select all

daanmageddon@homecontrol:~/Downloads/domoticz$ git checkout 85d7b702da84537d962156607ae45ecc56ca0c59
Note: checking out '85d7b702da84537d962156607ae45ecc56ca0c59'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 85d7b70... stable release #2563

Thanks!

[DaaNMaGeDDoN]

Hmmm, now after apt-get install svn i still get a svn error:

Code: Select all

inking C static library libmqtt.a
[ 29%] Built target mqtt
[ 30%] Building C object sqlite/CMakeFiles/sqlite.dir/sqlite3.c.o
Linking C static library libsqlite.a
[ 30%] Built target sqlite
-- Found Subversion: /usr/bin/svn (found version "1.8.10")
CMake Error at /usr/share/cmake-3.0/Modules/FindSubversion.cmake:110 (message):
  Command "/usr/bin/svn info /home/daanmageddon/Downloads/domoticz" failed
  with output:

  svn: E155007: '/home/daanmageddon/Downloads/domoticz' is not a working copy

Call Stack (most recent call first):
  getsvn.cmake:9 (Subversion_WC_INFO)


CMakeFiles/revisiontag.dir/build.make:52: recipe for target 'revisiontag' failed
make[2]: *** [revisiontag] Error 1
CMakeFiles/Makefile2:130: recipe for target 'CMakeFiles/revisiontag.dir/all' failed
make[1]: *** [CMakeFiles/revisiontag.dir/all] Error 2
Makefile:137: recipe for target 'all' failed
make: *** [all] Error 2

[gizmocuz]

Yep, we are not on svn anymore, better to remove all references to svn from the cmakelist.txt file

[DaaNMaGeDDoN]

Thanks, that worked. First time i started it, I forgot the copy of the domoticz.db was from the updated stable. I was suprised to see how well that still works, until i saw a couple of sql errors appear (no scenes defined when i go there etc.)
I'll retrieve the one from the backup i made when i still ran the older stable.
Anyway, see now how quick the webinterface responds with the old version. I will have to set up a fair comparison though: the only difference should be the domoticz version.
The new version on the PI2 is really slow; it takes about 2 seconds to switch tabs. I made a clip on my phone i can share if your are interested.

[DaaNMaGeDDoN]

OK i had a good comparison: 3 instances of domoticz running:
1 - my main domoticz, on the rpi2, recently updated from the former stable to the current one, which runs all the lua scripts, has the trxrfx connected and is having performance issues.
2 - domoticz 2.2563 running on the vm with a backup copy of the database from just before (1) got upgraded.
3 - domoticz 2.3530 also running on the vm with a copy of the database from about 15 minutes ago.
2&3 run in parallel on the same vm on different ports, the older version on 8080, the newer one on 9090. I disabled the xfrtrx443 scripts/events for both of them, they still have something todo: OTGW and a youless.
2 and 3 should be a good comparison. And i see a terrible truth: 3 is significantly slower when going through the tabs (Devices/Dashboard/etc).
If (and i believe it should) the new version should perform better, there can only be one conclusion: there is something wrong with my database.
So next test: stop 2&3, copy db from 2->3 and repeat the test.

[DaaNMaGeDDoN]

Ok, that approach worked, confirmation through a couple of red bars on some sensors for the outdated sensor data, now in both instances (2/3).
But things don't look good. For 1 ('production') things are getting worse. Im uploading some clips, links will follow.
Load on the cpu isnt even that high, not more that normal i would say. But this speed is frustrating.
For 2 (testing old stable): perfect! nothing wrong with that (except for the outdated database o/c)
And 3 (testing new stable, same as 1): terrible, just like 1 getting worse over time.
Also when i hit ctrl-c in the screen session i had for 3, it wouldn't die, had to kill it SIGKILL, hardware monitor wouldn't die.

I am sorry you had so much work on this and im only complaining, but you have to admit there's something quite different between what to expect and reality here
I will try to get 2.3 running with an updated database by setting 2 up as a slave to 1, hope that will work...i dont expect it though.
Links showing what i have to deal with will follow, i cant imagine im the only one with this. Cheers dd.

[DaaNMaGeDDoN]

Edits: mentions of3.3530 should have been 2.3530, corrected.
Links to clips:
production running new stable (1): https://drive.google.com/file/d/0B9xiR5 ... sp=sharing
testing running old stable (2): https://drive.google.com/file/d/0B9xiR5 ... sp=sharing
testing running new stable (3): https://drive.google.com/file/d/0B9xiR5 ... sp=sharing

[gizmocuz]

Did you checkout the latest beta with

Code: Select all

mkdir domoticz
cd domoticz
wget http://domoticz.sourceforge.net/domoticz_linux_armv7l.tgz
tar xvfz domoticz_linux_armv7l.tgz
rm domoticz_linux_armv7l.tgz

and copy your compiled domoticz executable over it, and it is much faster ?

(this way we can rule out it has todo with html)

i know there have been made some changes in the routine that gets the sensors to be displayed on the web gui


void CWebServer::GetJSonDevices

[DaaNMaGeDDoN]

Like the way you think!
Yesterday late i reverted the pi back to the backup i had from before the upgrade.
Did what you suggested and even copied over the database i backed up from production running the newer stable, just started it and the webinterface behaves like it should
Have not had good experiences with beta's but who knows this might be a solution, i might just miss a bit of logged data....let keep it running for an hour and see.
Thanks for now!
DD.

[gizmocuz]

Maybe you could checkout the code from the latest beta in a new folder
then replace the function

void CWebServer::GetJSonDevices

with the function from 2.2563

then we know if we have to look at this function, else it might be something else

[DaaNMaGeDDoN]

I guess you mean i could check to see what happens if i replace that function in the new stable from the old one?

EDIT: i see now what you mean, do it the other way around and if it's back to whining we know what the culprit is, np

[DaaNMaGeDDoN]

That function does not seem to exist in the beta:

Code: Select all

daanmageddon@homecontrol:~/Downloads$ ls
domoticz_2.2563  domoticz_2.3530  domoticz_beta
daanmageddon@homecontrol:~/Downloads$ cd domoticz_beta/
daanmageddon@homecontrol:~/Downloads/domoticz_beta$ grep -rnw "CWebServer::GetJSonDevices" ./
daanmageddon@homecontrol:~/Downloads/domoticz_beta$ echo "raar"
raar
daanmageddon@homecontrol:~/Downloads/domoticz_beta$ cd ../domoticz_2.2563/
daanmageddon@homecontrol:~/Downloads/domoticz_2.2563$ grep -rnw "CWebServer::GetJSonDevices" ./
./main/WebServer.cpp:7958:              void CWebServer::GetJSonDevices(Json::Value &root, const std::string &rused, const std::string &rfilter, const std::string &order, const std::string &rowid, const std::string &planID, const std::string &floorID, const bool bDisplayHidden, const time_t LastUpdate, const bool bSkipUserCheck)
daanmageddon@homecontrol:~/Downloads/domoticz_2.2563$

[DaaNMaGeDDoN]

Forget i said that, the beta folder isnt holding any source code, i'll get back on it.

[DaaNMaGeDDoN]

I get a lot of compilation errors when i copy-paste the function CWebServer::GetJSonDevices (WebServer.cpp) and and 'void GetJSonDevices' (WebServer.h) from 2.2563->Latest(beta?).
I dont understand: the beta webcode (pre-compiled) seems to work for the new stable, how is that test helping?
Wouldnt it be a better test if i do the same thing but then copy the sections from Latest(beta?)->New stable and see if that resolves it? I'll try anyway

[DaaNMaGeDDoN]

Ok, that worked.
Just to be sure we understand each other: the new stable was having huge performance issues, whether i ran it on my Pi or vm. Both in responsiveness (events) and over time the webinterface (like the clips i linked show).

I noticed in the changelog you updated the eventsystem code in regards to unused devices. I happen to know i had allmost always enabled to accept new devices, and it had been running for just over a year like that.
My 'unused devices' have 199 entries, maybe that is the issue with the eventsystem being slow?

By the way this is good to notice: i have abused that 'feature' for some time: i didnt want all those switches (i actually use) on the webinterface, but i wanted to give them names so i could use them in lua, that worked fine. So i cannot name any unused devices and use them in lua from here on? No problem, i know there are ways to hide them.

The webinterface issue: we will know about that in an hour or so. I am now running the latest stable with the updated CWebServer::GetJSonDevices from the beta.

I am getting quite happy with the results

[DaaNMaGeDDoN]

34 minutes in, webinterface back to a crawl, i cheered to early.
I'll see how the current beta runs.

[DaaNMaGeDDoN]

I think i found something: also the beta, after about 20-30 minutes started to get sluggish, especially when you switch a lot between switches and scenes, back and forth.
I restarted domoticz on the server, but that didn't help. But when i do a full page refresh response speed is back to when i just started it and opened the web interface for the first time!
I use chrome Version 46.0.2490.80 m, have not tested other browsers.
Could this be a caching issue?

[DaaNMaGeDDoN]

Manly addressed to gizmocuz:
I just wanted to follow up on this because things have improved greatly and i am thankful for your help/hints.
I think the main reason my (newer) domoticz versions, may that be the new stable or last beta, ran sluggish on the pi is maybe down to how much devices and scripts are configured. Maybe it had become a little too much for the pi.
Anyway i moved to my Debian/Jessie machine which has plenty of processing power and is connected to a UPS, so the pi had become a little redundant anyway. I have been running the beta since and i have not once found domoticz had crashed Also responses are great, however the webpage issue seems to persist, maybe i did find some kind of caching issue
Domoticz is running under different credentials (hail the -pidfile parameter), seperate data dir and all looks great. See here for more detail: viewtopic.php?f=4&t=8851&p=62068#p62068

And maybe usefull to everybody: i found domoticz needs a "unit" file together with the domoticz.sh init file to comply with LSB. The main issue without the unit file is that "service domoticz status" returns a 0 even after domoticz died. I found an attribute "RemainAfterExit" should be set to "no". I took the systemd-sysv-generated unit file and changed the attribute.

The following should saved as /etc/systemd/system/domoticz.service:

Code: Select all

[Unit]
SourcePath=/etc/init.d/domoticz
Description=LSB: Domoticz Home Automation Daemon
Before=runlevel2.target runlevel3.target runlevel4.target runlevel5.target shutdown.target
After=remote-fs.target systemd-journald-dev-log.socket network-online.target
Wants=network-online.target
Conflicts=shutdown.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
SysVStartPriority=3
ExecStart=/etc/init.d/domoticz start
ExecStop=/etc/init.d/domoticz stop

[Install]
WantedBy=multi-user.target
systemctl enable domoticz

Note i renamed /etc/init.d/domoticz.sh to domoticz (no extention) seemed more appropriate. To completely comply with LSB more changes are needed: force-reload is missing for instance and i believe some exit codes are missing/wrong.
I think systemd could also replace the need to use monit. I have written a different init script that also allows to run under different credentials and allows multiple instances.
Once i found out how github works i could try to contribute if you'd like that.