Datatransfer between 2 Raspberries [FTP or RSync]

[Toulon7559]

In other thread discussing how to get PySFTP operational:
that is most interesting for data-transfer to/from a server outside the LAN = remote server.

However, (much simpler and 'more protected') at a LAN also data transfer between 2 Raspberries of equal stature [not Master & Slave].
For such configuration it seems simplest to just use FTP or RSYNC from one Raspberry to the other.
Have tried with 2 setups, but both 'have a problem', for which suggestions requested for resolution.

First setup is a variation of the FTP-script included in the first message of the other thread mentioned above, obviously with UN+PW set for what is needed now at the LAN.
'Destination' ServerAdress inserted as found in internet-information for insertion of IP-adresses as destination.
Both the 'Source' FTP-server at 192.168.0.186 and the 'Destination' FTP-Server 192.168.0.185 run same version of vsftp-software, and always have.
Obviously the <username> and <password> filled in with actual contents as required to access the 'Destination' FTP-Server.

Code: Select all

#!/usr/bin/python
# -*- coding = utf-8 to enable reading by simple editors -*-
# (c)2020 script compiled by Toulon7559 from various material from forums, version 0.0 for upload of *.rrd to /home/pi/
# --------------------------------------------------
# Line005 = PREPARATION & SETTING
# --------------------------------------------------
# Imports for script-operation
import json
import urllib
# FTP from 192.168.0.186 towards 192.168.0.185
# Raspberry3
print
print ('Start of script Misc1_Upload0186')

# --------------------------------------------------
# Line016 = Function for FTP_UPLOAD to Server
# --------------------------------------------------
# Imports for script-operation
import ftplib
import os
# Definition of Upload_function
def upload(ftp, file):
    ext = os.path.splitext(file)[1]
    if ext in (".txt", ".htm", ".html"):
        ftp.storlines("STOR " + file, open(file))
    else:
        ftp.storbinary("STOR " + file, open(file, "rb"), 1024)

# --------------------------------------------------
# Line030 = Actual FTP-Login & -Upload
# --------------------------------------------------
ftp = ftplib.FTP("192.168.0.185")
ftp.login("<username>", "<password>")
# set path to destination directory
ftp.cwd('/home/pi/')
# set path to source directory
os.chdir("/home/pi/")
# upload of files
upload(ftp, "piTempest1.rrd")
upload(ftp, "piTempest2.rrd")
# reset path to root
ftp.cwd('/')
print ('End of script Misc1_Upload0186')

Have run that script for python2 and for python3, but makes no difference in response
CLI=

Code: Select all

sudo python /home/pi/domoticz/scripts/python/Misc1_upload0186.py

or

Code: Select all

sudo python3 /home/pi/domoticz/scripts/python/Misc1_upload0186.py

In latter case also first line of script is

Code: Select all

#!/usr/bin/python3

Response=

Code: Select all

Start of script Misc1_Upload0186
Traceback (most recent call last):
  File "/home/pi/domoticz/scripts/python/Misc1_upload0186.py", line 39, in <module>
    upload(ftp, "piTempest1.rrd")
  File "/home/pi/domoticz/scripts/python/Misc1_upload0186.py", line 27, in upload
    ftp.storbinary("STOR " + file, open(file, "rb"), 1024)
  File "/usr/lib/python2.7/ftplib.py", line 479, in storbinary
    conn = self.transfercmd(cmd, rest)
  File "/usr/lib/python2.7/ftplib.py", line 378, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/usr/lib/python2.7/ftplib.py", line 341, in ntransfercmd
    resp = self.sendcmd(cmd)
  File "/usr/lib/python2.7/ftplib.py", line 251, in sendcmd
    return self.getresp()
  File "/usr/lib/python2.7/ftplib.py", line 226, in getresp
    raise error_perm, resp
ftplib.error_perm: 550 Permission denied.

Question:
Because most errors reported relate to the ftp-function, what's wrong with this script?
Probably something related to the ServerAdress-setting, because that is the difference in contents, although the errors are pointing to line 027 with function ftp.storebinary (which has not changed, neither the file-types to be transferred)
The script was OK (obviously with applicable, other ServerAddress, UN+PW) when previously used for the upload to the FTP-server at ftp.strato.com
In this setup also tried ftp://192.168.0.185 as 'Destination'-address, but without success.

Second setup is an RSYNC-scriptline.
This datatransfer is all on LAN, and therefore no ssh needed, and also safe-enough [?] to include username/password in a command-string.
For WLAN may be considered less safe.
CLI=

Code: Select all

export RSYNC_PASSWORD="<password>"; rsync -avz /home/pi/piTempest*.rrd [email protected]:/home/pi

Response=

Code: Select all

[email protected]'s password:

Question=
Although password included in the above CLI-string, still asking for operator insertion, which hinders automatic execution in a timed cronjob.
Many constructions possible, but what is the SIMPLEST & working automation-construction, including the password in the CLI-string?

[erem]

Question:
Because most errors reported relate to the ftp-function, what's wrong with this script?

answer
i see: ftplib.error_perm: 550 Permission denied.
does the userid have sufficient rights on the target to write?

Question
Although password included in the above CLI-string, still asking for operator insertion, which hinders automatic execution in a timed cronjob.

answer
follow this tutoirial for rsync: https://www.howtoforge.com/mirroring_with_rsync

[Toulon7559]

Rob,

Your question pointing to 'sufficient permissions' in combination with the hinted URL triggered a thought at my side.
Checking on internet I found this description of RSYNC in combination with sudo.
Reading that description, it dawned that I possibly ask something impossible/difficult.
Background for that opinion:
if in both the source and destination machine Raspian in advance has been set with active SSH (for good reasons like use of PuttySSH, WinSCP, etc.), then trying to sync/transfer between these 2 machines without SSH is contradictary.
Is in practise the same as trying to communicate from a Raspberry-with-ssh to the internet-server ssh.strato.com or similar setup.
Not really surprised anymore related to the 550-error.

Just a pity that the application of SSH-keys in Raspian is not easier:
'pampered' by the Windows-software.
Therefore back to the other thread and swallow the pills to implement SSH in the transfer-protocol .......

[Toulon7559]

Many descriptions for filetransfer from RPI to PC under application of 'auxiliary' software, such as Filezilla, but that is dependent on an external 'man-in-the-middle'

Stil wondering:
what is easiest way to periodically, directly read a file from 1 RPI-with-ssh by another RPI-with-ssh?
Or periodically, directly put a file from 1 RPI-with-ssh to another RPI-with-ssh?
Somebody a very simple script for those 2 functions?

[Have a working construction to FTP from RPI to external server, but that fails if trying to modify for application between 2 RPIs-with-ssh on same LAN.
Seems to be 'something wrong' with adressing, username and password, but unclear what.
Tried to find answers in this forum etc., but apparently wrong searchkeywords]

[erem]

instead of ftp, use sftp
setup ssh for passwordless entry, (see here: viewtopic.php?f=63&t=35209&p=266393#p266393)
the use sftp to move stuff effortless.

[Toulon7559]

Rob,

Thanks!
That is one important hurdle less ....
Just check & recheck:
the actions described in the referred weblink probably need to be performed as operator action for each 'channel' to be activated.
To be repeated if you reinstall after crash etc.
Correct?

Next is testing a script for SFTP for getting/putting files from respectively to other clients at the LAN.
The first example of the pysftp Cookbook looks applicable and surely simple enough:
such script as part of a cronjob and periodic file transfer becomes a 'no-brain'er with very light load on the system.

[erem]

the actions described in the referred weblink probably need to be performed as operator action for each 'channel' to be activated.
To be repeated if you reinstall after crash etc.
Correct?

yes for each system you will be using sftp with you will need to setup the key exchange one time.

[Toulon7559]

;-( For 'simple' single file-reading on your LAN best to use 'simple' methods .....

Reverting to the basics as described here, with this setup one Raspberry can read a file from another Raspberry at the same LAN.
1. Install VSFTP in both Raspberries [to ensure that you have 'simple' ftp-function available]
2. Install this kind of Python-script at the reading Raspberry, obviously with adaptation to your case for the login and for the files

Code: Select all

# --------------------------------------------------
# Imports for script-operation
import ftplib
import os
# --------------------------------------------------
# login-info for the remote Raspberry
ftp = ftplib.FTP("<ip-address>")
ftp.login("<username>", "<password>")

# remotefile is source at remote Raspberry
remotefile='/home/pi/piTempest1.rrd'
# download is destination at reading Raspberry
download='/home/pi/piTempest1.rrd'
with open(download,'wb') as file:
    ftp.retrbinary('RETR %s' % remotefile, file.write)
ftp.quit()

3. Don't forget to set the Xes in the properties of the script-file
4. For periodic reading include a related call to this script in the

Code: Select all

crontab -e

Dilemma if you need to read a few more files:
- either multiply this script
- or do something more clever in the script, to handle multiple files

[Toulon7559]

Because LAN should be 'closed', FTP-application as simple solution might be acceptable at your private LAN, but has risks for transfers on the internet.
As indicated earlier in this thread, with that mindset I installed on purpose VSFTPD at my Raspberries to ensure that FTP-functionality is present.

With help from StackOverflow got a solution for a FTP-uploadscript operating with Python3.x
With the script earlier listed, the pragmatic solution is to switch to all_binary transfer.
The construction shown at new lines18+19 also properly closes the files.
The script transfers some example txt-files between folder /home/pi/ and the root-folder of a destination-server.
During testing operated with all types of files.

Code: Select all

#!/usr/bin/python3
# (c)2017 script compiled by Toulon7559 from various material from forums, version 0.0 for upload of *.txt to /
# Version 02P3 is the adaptation for Python3, by extending line 1 for linking to python3, commenting lines 14 - 17, and putting adapted copy of line 17 in lines 18+19
print ('Start of script Misc_Upload_02P3')
# --------------------------------------------------
# Line006 = Function for FTP_UPLOAD to Server
# --------------------------------------------------
# Imports for script-operation
import ftplib
import os
# Definition of Upload_function
def upload(ftp, file):
    ext = os.path.splitext(file)[1]
#    if ext in (".txt", ".htm", ".html"):
#       ftp.storlines("STOR " + file, open(file))
#    else:
#         ftp.storbinary("STOR " + file, open(file, "rb"), 1024)
    with open(file, "rb") as f:
        ftp.storbinary("STOR " + file, f, 1024)

# --------------------------------------------------
# Line022 = Actual FTP-Login & -Upload
# --------------------------------------------------
ftp = ftplib.FTP("<destination_FTPserver>")
ftp.login("<destination_Username>", "<destination_Password>")
# set path to destination directory
ftp.cwd('/')
# set path to source directory
os.chdir("/home/pi/")
# upload of TXT-files
upload(ftp, "PVTemp_Info.txt")
upload(ftp, "PVOutput_Info.txt")
# reset path to root
ftp.cwd('/')
print ('End of script Misc_Upload_02P3')
print

[FlyingDomotic]

Here's an example to sync selected files from a remote folder on a remote node to a local folder. Sync is done if local file don't exists or not the same size as remote file. Nothing is done to delete local files not existing on remote node.Password are not needed should you provide SSH local key to remote node.

Code: Select all

#!/usr/bin/python3
# Synchronize remote folder to locate folder, copying file not existing or with different sizes.
# Local files not existing on remote folder are not deleted.

import sys
import os
import pathlib
import paramiko

# Get list of remote files
def listRemoteFiles(remoteNode, remoteUser, remotePsw, remotePath):
    # Close previous connection
    ssh.connect(remoteNode, username=remoteUser, password=remotePsw, look_for_keys=True)
    remoteCommand = 'ls -gG --time=c --full-time ' + remotePath
    ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(remoteCommand)
    output = ssh_stdout.readlines()
    error = ssh_stderr.readlines()
    if error != []:
        print('Error in "'+remoteCommand+'" on '+remoteNode)
        print(error)
    else:
        fileList = list()
        for l in output:
            while '  ' in l:
                l = l.replace('  ', ' ')
            l = l.replace('\n', '')
            elements = l.split(' ')
            if elements[0] != 'total':
                size = int(elements[2])
                fileLine = []
                fileLine.append(elements[6])
                fileLine.append(elements[2])
                fileLine.append(elements[3]+' '+elements[4]+' '+elements[5])
                fileList.append(fileLine)
        return fileList
    exit(2)

# Copy a remote file locally
def copyRemoteFile(remoteFile, localFile):
    print('Copying '+remoteFile+' to '+ localFile)
    ftp_client=ssh.open_sftp()
    ftp_client.get(remoteFile, localFile)
    ftp_client.close()

# Sync selected files from a remote folder on a remote node to a local path
def synchronizeRemote(remoteNode, remoteUser, remotePsw, remotePath, remoteFile, localPath):
    # List remote files
    for remoteSpec in listRemoteFiles(remoteNode, remoteUser, remotePsw, remotePath + remoteFile):
        # Get file name
        remoteFile = remoteSpec[0]
        localParts = remoteFile.split('/')
        localFile = localPath + localParts[len(localParts)-1]
        fileFound = False   
        # Test if local file exists in localPath and its sub-folders
        for selectedFile in pathlib.Path(pathlib.Path(localFile).parent).rglob(pathlib.Path(localFile).name):
            fileFound = True
            # Check both file size
            if int(os.path.getsize(selectedFile)) != int(remoteSpec[1]):
                # File sizes are different
                print(remoteFile + ' and ' + str(selectedFile) + ' sizes are different')
                copyRemoteFile(remoteFile, str(selectedFile))
        if not fileFound:
            # Local file don't exists
            print(localFile + ' don\'t exists')
            copyRemoteFile(remoteFile, localFile)

# Create ssh object
ssh = paramiko.SSHClient()
# Load SSH host keys.
ssh.load_system_host_keys()
# Add SSH host key automatically if needed.
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

synchronizeRemote('node1','pi', '', '/var/log/', '*.log', '/home/pi/node1/var/log/')

exit(0)

[Toulon7559]

'Devil is in details'

As indicated in first message and subsequent additions in this thread, for simple, closed circuit, mutual data transfer have installed VSFTPD on all Raspberries in my LAN, all of older generation with Buster or Bullseye for OS.
Now added a Raspberry 4B with Bookworm as OS, and therefore revisited the FTP-issue.

An observation pops up, which seems an 'extension' of previously described observations.

Description of events&observations.
1. Scanning with Advance IPScanner provides info confirming that RPI4B has Domoticz and VSFTPD like all other RPIs.
2. In the display of AdvancedIPScanner clicking VSFTP for testing, get question which software to use for access => Filezilla
3. Result in Filezilla:

Code: Select all

Status:	Verbinden met 192.168.2.184:21...
Status:	Verbinding gemaakt, welkomstbericht afwachten...
Status:	Onveilige server, die geen FTP over TLS ondersteunt.
Opdracht:	USER anonymous
Antwoord:	331 Please specify the password.
Opdracht:	PASS *********************
Antwoord:	530 Login incorrect.
Fout:	Fatale fout: Kan niet verbinden met server

Just wondering, because with VSFTPD on all RPIs a 'simple' FTP-handshake should always be possible between my LAN's RPIs, without fuzz for setup .....
Or something simple to be changed in de FTP-setup to enable easier operation?