2 instances of mosquitto after reboot

[roblom]

Everytime I restart my raspberry I have the problem that mosquitto isn't running. I know how to solve it manually but I want to find the root cause but cannot find it. Maybe someone here can help?

After a reboot:

Code: Select all

pi@P1-Logger:~ $ sudo service mosquitto status
	? mosquitto.service - Mosquitto MQTT Broker
	Loaded: loaded (/etc/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
	Active: failed (Result: exit-code) since Sat 2022-05-28 02:02:16 CEST; 58s ago
		Docs: man:mosquitto.conf(5)
		man:mosquitto(8)
	Process: 619 ExecStartPre=/bin/mkdir -m 740 -p /var/log/mosquitto (code=exited, status=0/SUCCESS)
	Process: 620 ExecStartPre=/bin/chown mosquitto /var/log/mosquitto (code=exited, status=0/SUCCESS)
	Process: 621 ExecStartPre=/bin/mkdir -m 740 -p /run/mosquitto (code=exited, status=0/SUCCESS)
	Process: 622 ExecStartPre=/bin/chown mosquitto /run/mosquitto (code=exited, status=0/SUCCESS)
	Process: 624 ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf (code=exited, status=1/FAILURE)
	Main PID: 624 (code=exited, status=1/FAILURE)

	mei 28 02:02:15 P1-Logger systemd[1]: mosquitto.service: Service RestartSec=100ms expired, scheduling restart.
	mei 28 02:02:15 P1-Logger systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5.
	mei 28 02:02:16 P1-Logger systemd[1]: Stopped Mosquitto MQTT Broker.
	mei 28 02:02:16 P1-Logger systemd[1]: mosquitto.service: Start request repeated too quickly.
	mei 28 02:02:16 P1-Logger systemd[1]: mosquitto.service: Failed with result 'exit-code'.
	mei 28 02:02:16 P1-Logger systemd[1]: Failed to start Mosquitto MQTT Broker.

When I then try to restart it:

Code: Select all

pi@P1-Logger:~ $ sudo service mosquitto restart
	Job for mosquitto.service failed because the control process exited with error code.
	See "systemctl status mosquitto.service" and "journalctl -xe" for details.

Wen I check the status:

Code: Select all

pi@P1-Logger:~ $ systemctl status mosquitto.service
	? mosquitto.service - Mosquitto MQTT Broker
	Loaded: loaded (/etc/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
	   Active: failed (Result: exit-code) since Sat 2022-05-28 02:04:21 CEST; 22s ago
	   Docs: man:mosquitto.conf(5)
	man:mosquitto(8)
	Process: 1656 ExecStartPre=/bin/mkdir -m 740 -p /var/log/mosquitto (code=exited, status=0/SUCCESS)
	Process: 1657 ExecStartPre=/bin/chown mosquitto /var/log/mosquitto (code=exited, status=0/SUCCESS)
	Process: 1658 ExecStartPre=/bin/mkdir -m 740 -p /run/mosquitto (code=exited, status=0/SUCCESS)
	Process: 1659 ExecStartPre=/bin/chown mosquitto /run/mosquitto (code=exited, status=0/SUCCESS)
	Process: 1660 ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf (code=exited, status=1/FAILURE)
	Main PID: 1660 (code=exited, status=1/FAILURE)

	mei 28 02:04:21 P1-Logger systemd[1]: mosquitto.service: Service RestartSec=100ms expired, scheduling restart.
	mei 28 02:04:21 P1-Logger systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5.
	mei 28 02:04:21 P1-Logger systemd[1]: Stopped Mosquitto MQTT Broker.
	mei 28 02:04:21 P1-Logger systemd[1]: mosquitto.service: Start request repeated too quickly.
	mei 28 02:04:21 P1-Logger systemd[1]: mosquitto.service: Failed with result 'exit-code'.
	mei 28 02:04:21 P1-Logger systemd[1]: Failed to start Mosquitto MQTT Broker.

Then check which process is using it, then there are 2 processes using it.

Code: Select all

pi@P1-Logger:~ $ ps -ef | grep mosquitto
	mosquit+   569     1  0 02:02 ?        00:00:00 /usr/sbin/mosquitto -d
	pi        2067   967  0 02:05 pts/0    00:00:00 grep --color=auto mosquitto

Then when I kill the first one.

Code: Select all

pi@P1-Logger:~ $ sudo kill 569

And then restart mosquitto.

Code: Select all

pi@P1-Logger:~ $ sudo service mosquitto restart

Mosquitto is running.

Code: Select all

pi@P1-Logger:~ $ sudo service mosquitto status
	? mosquitto.service - Mosquitto MQTT Broker
	Loaded: loaded (/etc/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
	   Active: active (running) since Sat 2022-05-28 02:05:36 CEST; 6s ago
	  Docs: man:mosquitto.conf(5)
	man:mosquitto(8)
	Process: 2195 ExecStartPre=/bin/mkdir -m 740 -p /var/log/mosquitto (code=exited, status=0/SUCCESS)
	Process: 2196 ExecStartPre=/bin/chown mosquitto /var/log/mosquitto (code=exited, status=0/SUCCESS)
	Process: 2197 ExecStartPre=/bin/mkdir -m 740 -p /run/mosquitto (code=exited, status=0/SUCCESS)
	Process: 2198 ExecStartPre=/bin/chown mosquitto /run/mosquitto (code=exited, status=0/SUCCESS)
	Main PID: 2199 (mosquitto)
	Tasks: 1 (limit: 2059)
	CGroup: /system.slice/mosquitto.service
	  +-2199 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

	mei 28 02:05:36 P1-Logger systemd[1]: Starting Mosquitto MQTT Broker...
	mei 28 02:05:36 P1-Logger mosquitto[2199]: 1653696336: Loading config file /etc/mosquitto/conf.d/connect.conf
	mei 28 02:05:36 P1-Logger systemd[1]: Started Mosquitto MQTT Broker.

[roblom]

Nobody got a tip where to look for?

[FireWizard]

Hello @roblom,

I see that you use something like P1-logger.
Could it be that this application tries to start its own instance of Mosquitto?

You can check this by disabling the automatic start of Mosquitto during boot.

Code: Select all

sudo service mosquitto disable

I see also 2 places where a mosquitto config file exist.

Code: Select all

/etc/mosquitto/mosquitto.conf

and

Code: Select all

/etc/mosquitto/conf.d/connect.conf

So I think you try to start mosquitto 2 times, 1 normal way and 1 by P1-logger.

Check this,

Regards

[roblom]

P1-logger is the name of the system, so it's not a process or something.

/etc/mosquitto/mosquitto.conf
contains

Code: Select all

# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.gz

pid_file /var/run/mosquitto/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

listener 1883
allow_anonymous true

/etc/mosquitto/conf.d/connect.conf
contains

Code: Select all

#listener 1883
allow_anonymous true

[FireWizard]

Hi, @roblom,

Are there any other *.conf files in the /etc/mosquitto/conf.d directory?

And also put a # in front of allow_anonymous true. Not only listener 1883.

Regards

[roblom]

No there are no more *.conf files in that folder.

[roblom]

Also put an # in front of allow_anonymous true but mosquitto won't start after a reboot, and I first have to kill the first mosquitto process in order to start mosquitto.

[roblom]

Still I think for some reason mosquitto is trying to start twice. Does someone know how I can check this?

[FireWizard]

Hi, @roblom,

You said:

I first have to kill the first mosquitto process in order to start mosquitto.

So you have a mosquitto process started. Why are you killing that and then start mosquitto again.
For sure you have 2 processes running.

The first one is because you have a configuration in your /etc/mosquitto/mosquitto conf file and the second one is because you have a configuration in /etc/mosquittoconf.d directory.

In my case the mosquitto conf contains the following:

Code: Select all

# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example

port 1883

pid_file /var/run/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

and my conf.d directory contains a file mymosquitto.conf, with the following contents:

Code: Select all

port 1883
persistence_file mosquitto.db
log_dest syslog
log_dest stdout
log_dest topic
log_type error
log_type warning
log_type notice
log_type information
connection_messages true
log_timestamp true
allow_anonymous false
password_file /etc/mosquitto/passwd

As you can see in both files I have port 1883, but that does no harm. ( it is a leftover from version 1)
As I have my mosquitto server password protected you will also see a password configuration

This starts 1 instance of mosquitto

Perhaps this example will help you

Regards

[roblom]

Hi, @roblom,

You said:

I first have to kill the first mosquitto process in order to start mosquitto.

So you have a mosquitto process started. Why are you killing that and then start mosquitto again.
For sure you have 2 processes running.

Because after a restart mosquitto isn't running.

The first one is because you have a configuration in your /etc/mosquitto/mosquitto conf file and the second one is because you have a configuration in /etc/mosquittoconf.d directory.

The conf file in /etc/mosquitto/conf.d was empty but just to be sure i deleted it and still after a reboot mosquitto is not running and will not start after a "sudo systemctl start mosquitto". The error that is given.

Job for mosquitto.service failed because the control process exited with error code.
See "systemctl status mosquitto.service" and "journalctl -xe" for details.

systemctl status mosquitto.service gives

● mosquitto.service - Mosquitto MQTT Broker
Loaded: loaded (/etc/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2022-07-23 20:18:43 CEST; 49s ago
Docs: man:mosquitto.conf(5)
man:mosquitto(8)
Process: 2667 ExecStartPre=/bin/mkdir -m 740 -p /var/log/mosquitto (code=exited, status=0/SUCCESS)
Process: 2668 ExecStartPre=/bin/chown mosquitto /var/log/mosquitto (code=exited, status=0/SUCCESS)
Process: 2669 ExecStartPre=/bin/mkdir -m 740 -p /run/mosquitto (code=exited, status=0/SUCCESS)
Process: 2670 ExecStartPre=/bin/chown mosquitto /run/mosquitto (code=exited, status=0/SUCCESS)
Process: 2671 ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf (code=exited, status=1/FAILURE)
Main PID: 2671 (code=exited, status=1/FAILURE)

jul 23 20:18:43 P1-Logger systemd[1]: mosquitto.service: Service RestartSec=100ms expired, scheduling restart.
jul 23 20:18:43 P1-Logger systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5.
jul 23 20:18:43 P1-Logger systemd[1]: Stopped Mosquitto MQTT Broker.
jul 23 20:18:43 P1-Logger systemd[1]: mosquitto.service: Start request repeated too quickly.
jul 23 20:18:43 P1-Logger systemd[1]: mosquitto.service: Failed with result 'exit-code'.
jul 23 20:18:43 P1-Logger systemd[1]: Failed to start Mosquitto MQTT Broker.

To get mosquitto running
i use "ps -ef | grep mosquitto"

mosquit+ 581 1 0 20:15 ? 00:00:00 /usr/sbin/mosquitto -d
pi 4529 1030 0 20:22 pts/0 00:00:00 grep --color=auto mosquitto

then kill 581

then "systemctl restart mosquitto.service"

and then "systemctl status mosquitto.service" gives

● mosquitto.service - Mosquitto MQTT Broker
Loaded: loaded (/etc/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2022-07-23 20:24:12 CEST; 29s ago
Docs: man:mosquitto.conf(5)
man:mosquitto(8)
Process: 5343 ExecStartPre=/bin/mkdir -m 740 -p /var/log/mosquitto (code=exited, status=0/SUCCESS)
Process: 5344 ExecStartPre=/bin/chown mosquitto /var/log/mosquitto (code=exited, status=0/SUCCESS)
Process: 5345 ExecStartPre=/bin/mkdir -m 740 -p /run/mosquitto (code=exited, status=0/SUCCESS)
Process: 5347 ExecStartPre=/bin/chown mosquitto /run/mosquitto (code=exited, status=0/SUCCESS)
Main PID: 5349 (mosquitto)
Tasks: 1 (limit: 2059)
CGroup: /system.slice/mosquitto.service
└─5349 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

jul 23 20:24:11 P1-Logger systemd[1]: Starting Mosquitto MQTT Broker...
jul 23 20:24:12 P1-Logger systemd[1]: Started Mosquitto MQTT Broker.

[roblom]

Is it possible that mosquitto is started (so there is a pid file) but not active after boot (so it does not work)?

[FireWizard]

Hi, @roblom,

See: http://www.steves-internet-guide.com/mosquitto-broker

Regards

[roblom]

Found my problem after a long chat with chatGPT, the problem was that there was also a mosquitto start command in "etc/local.rc" which I probably added myself while using trial and error to get mosquitto up and running.