Upgrade to SFTP-Upload

[Toulon7559]

Tiil October 20th 2020 STRATO allowed basic, simple ftp-upload, and announced a required shift to SFTP.
;-( grace period now expired, and upload mandatorily to be changed to sftp-application .......

[Addition March15, 2021:
Apparently problems between Strato and it's Customers to get SFTP up & running for the usual applications.
FTP-functionality is still active at this moment:
under 'SFTP-Management' it is possible to select either SFTP, FTP, or the combination of both]

Below python-script performs required ftp-upload without much hazzle, in this version for upload of a file for AWEKAS.
Just puzzled how to change this script for application with sftp.
It is not sufficient to change <ftp_servername> towards the new sftp-servername.
The 'problem' can pragmatically be solved under application of external Windows' sync- or SFTP-software ( as described at
website Raspberry.org) and at many other websites.
My own variant is the application of sync-software AllwaySync which has multiple capabilities for upload & download:
only one negative aspect is that this software uses fixed values for intervals.
That has the consequence that peakloads may occur on 'nice' times such as whole hour, half hour etc. => 'trouble'
WinSCP's Sync is better in that perspective, but the Sync-function of that program has other weaknesses.

Prefered to have an integrated Linux-solution running in Domoticz, Python etc. as part of the application (or using an SFTP-server at that same Raspberry) for below argumentation.
As example, for periodic upload of files to a destination, an external SFTP-software has to pull the designated files from the Raspberry on which the application is running.
If you have various Raspberries, it means that the external software must periodically poll the Raspberries, and the external software is a nodal point (= single point of failure) in the route to the destination. To avoid/suppress the problem described in the first section, the external software should be able to poll at flexibly variable times and must have high transferspeed, because otherwise a 'choking point' in time, not only at the external software, but also at the Raspberries.
To reduce the latter aspect, it would be nice if each application at Raspberry would have it's own 'pushing' SFTP-uploadfunction
= per Raspberry&application an independent timing & channel for upload => each with relatively low volume, purposely spread over time, not only at 'nice' times, but also at 'odd' times.

Linux has an SFTP-command, but that seems for commandline only: while my application is aimed at an automated script for periodic upload etc..
Anybody experience/hints/alternatives to convert the below script into an SFTP-variant?
No preference for programming language: Python-application is nice, but if alternative setups exists as lua-script, as dzVents-script, or as PHP-script, equally welcome

Code: Select all

#!/usr/bin/python
# -*- coding = utf-8 to enable reading by simple editors -*-
# (c)2017 script compiled by Toulon7559 from various material from forums, version 0.2 for upload of awekas_wl.html
# --------------------------------------------------
# Line005 = PREPARATION & SETTING
# --------------------------------------------------
# Assure that ftplib has been installed, and make this script part of cronjob for periodic execution
# This script assumes the html-file to be saved to folder /home/pi/
# This script uploads the html-file to the root-folder of the remote ftp-server
# Lines 32 and 33 to be filled with your applicable account-info for the ftp-account
# Linenumbers and Printlines are for testing.
print
print ('Start of script')

# --------------------------------------------------
# Line016 = Function for FTP_UPLOAD to Server
# --------------------------------------------------
# Imports for script-operation
import ftplib
import os
# Definition of Upload_function
def upload(ftp, file):
    ext = os.path.splitext(file)[1]
    if ext in (".txt", ".htm", ".html"):
        ftp.storlines("STOR " + file, open(file))
    else:
        ftp.storbinary("STOR " + file, open(file, "rb"), 1024)

# --------------------------------------------------
# Line030 = Actual FTP-Login & -Upload
# --------------------------------------------------
ftp = ftplib.FTP("<ftp_servername>")
ftp.login("<ftp_username>", "<ftp_password>")
upload(ftp, "awekas_wl.html")
print ('End of script')

[Toulon7559]

Info in StackOverflow is clear related to the question whether it is possible to apply ftplib for SFTP-transfer:
ftplib does not support SFTP.
Following the links in the Answer:
they recommend the application of Paramiko.
The other options mentioned in this (very old) thread all apply Paramiko under the hood:
according to comments in StackOverflow, some of these 'derivatives' apparently are no longer maintained.
Digging a bit deeper, in the related Github it can be seen that the Paramiko-software is maintained/uptodate.
As part of the dependencies, required addition of cryptography is mentioned, and seems part of the automatic install.
However, first efforts to install paramiko or derivatives give error reports related to the dependencies, resulting in 'no installation':
puzzles to solve ......

Looking around found that vsftpd-software also might be applicable, but needs significant mod to get SFTP-capability.

Yet easier, complete, more modern answers to the question 'SFTP-Upload & Download performed at Raspberry'?

[Toulon7559]

Not making much progress into the direction of an SFTP-uploader-function, perhaps the following construction may be a pragmatic solution:
hints & improvements appreciated!!!

Uploading from Raspberries by the external software through SFTP may become slightly easier if 'leaner'
= polling from least number of sources/Raspberries, preferably only 1 location.

Would the following structure work?
1) All applications at Raspberries requiring upload, upload over the local LAN to one designated Raspberry as FTP-Collector/Server.
Because local transfer on LAN, for those uploads existing FTP seems still useful and least change (just change the addressing).
Result: files in 1 set of folders at dedicated Raspberry
2) SFTP-upload from that Raspberry-Server to the 'outside world' to be realized by an external SFTP-software.
Result: pulling files from 1 set of folders at 1 dedicated Raspberry

Advantage: local transfers and external upload may each have own, independent schedule, no need to be 100% synchronous
[better not, to avoid peak loads on the LAN]
Disadvantage: the LAN is occupied by the transfers from the Raspberries, plus the transfers from the FTP_Collector to the destination
[simply 100% extra occupation relative to an external software 'sucking' the files from the Raspberries].
No problem if your LAN is cabled-Gigabit with related EthernetSwitches, but may be different story for many WLANs.

Just an idea:
how to change the script in the first message of this thread to realise the FTP-transfer over (W)LAN from the source to the dedicated Raspberry?
Setting <username> & <password> is simple, but stumbling on

Code: Select all

ftplib.FTP("<ftp_servername>")

, lacking some knowledge on semantics for application of ftplib, and not finding the answer in examples.
Simply inserting the Raspberry's IP-address + UN + PW results in short errorreport

Code: Select all

ftplib.error_perm: 550 Permission denied.

= apparently wrong access rights.
Inserting the Raspberry's computername, results in report that no socket is associated

How to apply IP-address for <ftp_servername>, or what other name to be used to get the files from the Raspberry into that local FTP-Server?

Addition 23rd of December 2020:
In hindsight the error reports have good reason => see other thread

[Toulon7559]

Looking around at internet, for Raspberry the 'usual' SFTP-solution seems:
enable SSH at the Raspberry and then use some external software like WinSCP or FileZilla to do the onetime or periodic filetransfer-job.
My own (interim) solution is equivalent, same for the onetimes, and with AllwaySync for the periodic file-transfer.

My aim is to eliminate the man-in-the-middle, getting more flexible timing, and by shorter path reducing overhead & delay.
Like the FTP-script described at the start of this thread, looking for a solution which at desired time at the source-Raspberry 'copies/pushes' the file to the destination-computer, but now by SFTP instead of FTP.

At one side Raspberry with SSH and at other side another computer with SSH, should not be a big deal, and therefore surprised not seeing a software simply realising such transfer function [or am I missing something related to applicable standard capabilities in Raspian_Buster?]

On an alternative approach, the sshfs python-package seems quite applicable, but a bit short in description.
Anybody seen a solution with more extended description-with-examples?

[jannl]

A short google action showed several solutions, as far as I can see even with secure password access with something called sshpass. Does not look that hard in a simple bash script.

[Toulon7559]

Janni,

You are right: the solution seems very simple.
But the problem is in practical realisation (on a configuration consisting of Raspian Buster and latest Domoticz).

For pysftp the instruction for installation is simplicity itself

Code: Select all

pip install pysftp

The installation nicely proceeds, until the following report

Code: Select all

      building '_cffi_backend' extension
      creating build/temp.linux-armv7l-2.7
      creating build/temp.linux-armv7l-2.7/c
      arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-InigCj/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-armv7l-2.7/c/_cffi_backend.o
      c/_cffi_backend.c:15:10: fatal error: ffi.h: Bestand of map bestaat niet
       #include <ffi.h>
                ^~~~~~~
      compilation terminated.
      error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

      ----------------------------------------
  Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-8DZupq/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-9ZoNBT/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-yiSADe --compile" failed with error code 1 in /tmp/pip-install-8DZupq/cffi/

  ----------------------------------------
Command "/usr/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yiSADe --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --extra-index-url https://www.piwheels.org/simple -- setuptools>=40.8.0 wheel "cffi>=1.1; python_implementation != 'PyPy'"" failed with error code 1 in None

Installation with

Code: Select all

pip install paramiko

or with

Code: Select all

pip install fs.sshfs

ends in the same way.

Question:
why the error-report, and how to remedy?

[jannl]

Looks like a file is missing. Search for the error or the file. I still would go for the much more simple bash script and do not that much care about error handling.

[Toulon7559]

Janni,

Agreed that the signs seem to be pointing to a missing file.
Surprised, because the instruction is very general and the (negative) effect common for the 3 installation-examples:
cannot imagine that I am only user having this effect.

Required Bash-script may become complex and unique per Raspberry, because at each Raspberry I have files in various folders needing transfer to the 'common' ssh-server (and reverse).

Perhaps the solution suggested in a previous message is a solution:
per Raspberry concentrate the files-to-be-exchanged in one folder and sync that folder (both ways) with the related folder at the ssh-server.

Anyway, worth trying to apply a bash-script .....

Or perhaps I stay lazy/relaxed and accept the use of 'external' sync software on a Windows-server:
not 'lean, as desired' and requiring man-in-the-middle, but effective. Temporary/interim solution becoming permanent.

[jannl]

You could use tar to first create a single file to transfer with sftp

[erem]

which version of python?

python2: pip install pysftp
python3: pip3 install pysftp

to install global on unix/linux
python2: sudo pip install pysftp
python3: sudo pip3 install pysftp

[Toulon7559]

Rob,

Have tried both python 2.x and python3.x with apt-get and pip as entries:
missed/forgot the pip3-variations
Next CLI-inputs at least install pysftp 'somewhere':

Code: Select all

pip3 install pysftp

Code: Select all

sudo pip3 install pysftp

Next hurdle is making a draft Python-script run for file-reading, upload and download:
apparently difficulty to get/set the ssh-keys:
- first line at CLI calls the script.
- next line from CLI is first errorreport of a long string.

Code: Select all

 sudo python3 /home/pi/domoticz/scripts/python/SFTP_Download_PySFTP.py

/usr/local/lib/python3.7/dist-packages/pysftp/__init__.py:61: UserWarning: Failed to load HostKeys from /root/.ssh/known_hosts.  You will need to explicitly load HostKeys (cnopts.hostkeys.load(filename)) or disableHostKey checking (cnopts.hostkeys = None).

Next puzzle.

Considering this additional information only gave small step forward:
still same warning as above related to ssh-key-handling, but all other code accepted (no long error string anymore).
Seems that in some way /root/.ssh/known_hosts needs to be tuned for start of PySFTP.

Not as simple as with Windows SFTP-software .........

[Toulon7559]

Many roads lead to Rome.

Just wondering about an alternative approach:
could rsync do the job (both without ssh and with ssh)?
See https://www.tecmint.com/rsync-local-rem ... -commands/
or https://www.linuxtechi.com/rsync-comman ... les-linux/
in combination with https://www.thegeekstuff.com/2011/07/rs ... -password/

[jannl]

Google learns it could be something simple like this:

Code: Select all

export SSHPASS=your-password-here
sshpass -e sftp -oBatchMode=no -b - sftp-user@remote-host << !
   cd incoming
   put your-log-file.log
   bye
!

[Toulon7559]

< Contents of this message not directly related to this thread:
cut & shifted to other, separate thread>

[erem]

@Toulon7559

sorry for the late reply, i missed your post

>>Seems that in some way /root/.ssh/known_hosts needs to be tuned for start of PySFTP.

the easy way is to setup the ssh connection manually before using it in an automation.

here is how to do that (not an original work )

Spoiler: show

how to setup automatic login for ssh & sftp

1) setup ssh keys
- generate private and public key

You want to use Linux and OpenSSH to automate your tasks. Therefore you need an
automatic login from host A / user a to Host B / user b.
You don't want to enter any passwords, because you want to call
ssh from a within a shell script.

How to do it

first log in on A as user a and generate a pair of authentication keys.

a@A:~> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/a/.ssh/id_rsa):
Created directory '/home/a/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/a/.ssh/id_rsa.
Your public key has been saved in /home/a/.ssh/id_rsa.pub.
The key fingerprint is:
3e:4f:05:79:3a:9f:96:7c:3b:ad:e9:58:37:bc:37:e4 a@A
Now use ssh to create a directory ~/.ssh as user b on B. (The directory may already exist, which is fine):

a@A:~> ssh b@B mkdir -p .ssh
b@B's password:
Finally append a's new public key to b@B:.ssh/authorized_keys and enter b's password one last time:

a@A:~> cat .ssh/id_rsa.pub | ssh b@B 'cat >> .ssh/authorized_keys'
b@B's password:

or use ssh-copy-id b@B

From now on you can log into B as b from A as a without password:

a@A:~> ssh b@B

A note from one of our readers: Depending on your version of SSH you might also have to do the following changes:

Put the public key in .ssh/authorized_keys2
Change the permissions of .ssh to 700
Change the permissions of .ssh/authorized_keys2 to 640 (644?)

==================================
on windows
ssh is part of win10
usng powershell, but ssh-keygen also works in command window
============================
PS C:\Users\Christopher> ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (C:\Users\Christopher/.ssh/id_rsa):
Created directory 'C:\Users\Christopher/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\Christopher/.ssh/id_rsa.
Your public key has been saved in C:\Users\Christopher/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:/mjkrJOQbRzCAwlSPYVBNcuxntm/Ms5/MMC15dCRrMc christopher@Christopher-Win10-VM-01
The key's randomart image is:
+---[RSA 2048]----+
|oo.+o== o.o |
|. o +. = o = |
| o .+. . B |
| +..+o o E |
| *+.S. . |
| o +...o |
| o =. .o |
| o.*o .. |
| .=+++. |
+----[SHA256]-----+
PS C:\Users\Christopher>

then copy the key to the target machine

in powershell: type $env:USERPROFILE\.ssh\id_rsa.pub | ssh IP-ADDRESS-OR-FQDN "cat >> .ssh/authorized_keys"
or
in command window: type C:\Users\scott\.ssh\id_rsa.pub | ssh pi@crowpi 'cat >> .ssh/authorized_keys'

test: ssh user@target should not ask password
========================================================

[jannl]

@toulon,

Why ask about SFTP when you want to use ftp?
SFTP is ftp over ssh, with some different commands.

Jan

[Toulon7559]

Janni,

Sorry, have mixed 2 issues in 1 thread:
1) SFTP from LAN to remote server requiring SSH
2) local data-transfer not requiring SSH (for which simple FTP will be sufficient).

Will take out the FTP-related questions and shift to other thread.
This message remains Placeholder not to disrupt the sequence of this thread.

Addition January 03, 2021:
With a bit longer thinking:
if all Raspberries on the (W)LAN have been set to use SSH, no wonder that FTP is not without errors.
SFTP then is the better way to go also for data-transfer between the Raspberries.
Proven by experiments:
AllwaySync with SFTP set left and right (or at least for the 'destination'-RPI) is trouble-free.

Now just the job to replace the 'external' AllwaySync (or equivalent) by a short SFTP-function embedded in the scripts .....

[Toulon7559]

In the derived & parallel-running 'other thread' Rob/ [erem] hinted me to a very simple setup to get ssh and sftp.

Now to make a script aimed at SFTP-upload to ssh.strato.com, to replace the interim SFTP-support by the external PC-software AllwaySync.

simplicity & results count .....

[Toulon7559]

'Simplicity & results' is easier said then done ....

Tried to make from a raspberry-CLI a variant aimed at ssh.strato.com,
and it starts nicely, but in the end do not get opportunity to fill-in a password to get access:
instead strato.com blocks the port 22, apparently considering this call an offense.

Code: Select all

root@Dom1234:/home/pi# ssh ssh.strato.com
The authenticity of host 'ssh.strato.com (81.169.145.126)' can't be established.
ECDSA key fingerprint is SHA256:uBEcHvlYrltumYxZxLSAGLNWM1Tch91sAG7/kMvP6R0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ssh.strato.com,81.169.145.126' (ECDSA) to the list of known hosts.
Connection closed by 81.169.145.126 port 22
root@Dom1234:/home/pi#

Comparing, for access with FTP to ftp.strato.com I would need 3 components
Host = ftp.strato.com
Username = <ftp-username>
Password = <ftp-password>
in which the <ftp-username> = <first-section>@<domain-name>

Get the feeling that the 'problem' for this setup is in the application of Host and Username:
Username is required to arrive at the specific account in Strato, but without previous insertion of Host (= ssh.strato.com), strato does not process the Username

Insertion of all 3 components is possible using the example in the pysftp Cookbook, but when running the simple script then complaints that no keys are found.
[Which in fact does not seem a wonder, because the chance is very slim that calling ssh.strato.com you would arrive at exactly the same ip-port at which the ssh-key was requested]

Somebody an idea how to combine the Host and Username in 1 string which under command ssh is accepted by Strato, and processed in such way to accept a subsequent SFTP script-call?

[erem]

some research yields

doesn't work

Code: Select all

$ ssh ssh.strato.de
Connection closed by 81.169.145.126

works

Code: Select all

$ ssh [email protected]
[email protected]'s password:

to copy key

Code: Select all

ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]

all from here: https://www.dirk-hagedorn.de/?p=1460

hope this helps...