Rest-Server

[FrankM]

Was das?

Rest Server is a high performance HTTP server that implements restic's REST backend API. It provides secure and efficient way to backup data remotely, using restic backup client via the rest: URL.

GitHub - restic/rest-server: Rest Server is a high performance HTTP server that implements restic's REST backend API.

Rest Server is a high performance HTTP server that implements restic's REST backend API. - restic/rest-server

GitHub (github.com)

Preparing a new repository — restic 0.17.2-dev documentation

(restic.readthedocs.io)

Installation

Mittels git das Repo clonen.

root@rockpro64:~# git clone https://github.com/restic/rest-server.git
Cloning into 'rest-server'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3180 (delta 0), reused 2 (delta 0), pack-reused 3177
Receiving objects: 100% (3180/3180), 5.63 MiB | 2.39 MiB/s, done.
Resolving deltas: 100% (1189/1189), done.

In das Verzeichnis wechseln und mittels go das File bauen. Moment, go fehlt uns noch.

apt install golang-go

Danach

 cd rest-server

File bauen

root@rockpro64:~/rest-server# go run build.go
go: finding github.com/prometheus/client_golang v0.8.0
go: finding github.com/prometheus/procfs v0.0.0-20180212145926-282c8707aa21
go: finding github.com/beorn7/perks v0.0.0-20160804104726-4c0e84591b9a
go: finding github.com/golang/protobuf v1.0.0
go: finding github.com/gorilla/handlers v1.3.0
go: finding github.com/inconshreveable/mousetrap v1.0.0
go: finding github.com/miolini/datacounter v0.0.0-20171104152933-fd4e42a1d5e0
go: finding golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a
go: finding golang.org/x/crypto v0.0.0-20180214000028-650f4a345ab4
go: finding github.com/spf13/pflag v1.0.0
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.0
go: finding github.com/spf13/cobra v0.0.1
go: finding github.com/prometheus/client_model v0.0.0-20171117100541-99fa1f4be8e5
go: finding github.com/prometheus/common v0.0.0-20180110214958-89604d197083
go: finding goji.io v2.0.2+incompatible
go: downloading github.com/spf13/cobra v0.0.1
go: downloading github.com/gorilla/handlers v1.3.0
go: downloading goji.io v2.0.2+incompatible
go: downloading golang.org/x/crypto v0.0.0-20180214000028-650f4a345ab4
go: downloading github.com/prometheus/client_golang v0.8.0
go: downloading github.com/miolini/datacounter v0.0.0-20171104152933-fd4e42a1d5e0
go: downloading github.com/spf13/pflag v1.0.0
go: downloading github.com/prometheus/common v0.0.0-20180110214958-89604d197083
go: downloading github.com/prometheus/client_model v0.0.0-20171117100541-99fa1f4be8e5
go: downloading github.com/golang/protobuf v1.0.0
go: downloading github.com/prometheus/procfs v0.0.0-20180212145926-282c8707aa21
go: downloading github.com/beorn7/perks v0.0.0-20160804104726-4c0e84591b9a
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.0

Das resultierende File rest-server an seinen Platz verschieben.

cp rest-server /usr/local/bin   

Test

root@rockpro64:/# rest-server --help
Run a REST server for use with restic

Usage:
  rest-server [flags]

Flags:
      --append-only          enable append only mode
      --cpu-profile string   write CPU profile to file
      --debug                output debug messages
  -h, --help                 help for rest-server
      --listen string        listen address (default ":8000")
      --log string           log HTTP requests in the combined log format
      --max-size int         the maximum size of the repository in bytes
      --no-auth              disable .htpasswd authentication
      --path string          data directory (default "/tmp/restic")
      --private-repos        users can only access their private repo
      --prometheus           enable Prometheus metrics
      --tls                  turn on TLS support
      --tls-cert string      TLS certificate path
      --tls-key string       TLS key path
  -V, --version              output version and exit

Gut, das Programm läuft

Rest-Server starten

root@rockpro64:/# rest-server --path /home/rock64/backup --no-auth
Data directory: /home/rock64/backup
Authentication disabled
Private repositories disabled
Starting server on :8000

OK, der Rest-Server läuft. Dann mal ausprobieren.

Backup erstellen und Test

Init

root@frank-MS-7C37:~# restic -r rest:http://192.168.3.11:8000/ init
enter password for new repository: 
enter password again: 
created restic repository 394364201c at rest:http://192.168.3.11:8000/

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

backup

root@frank-MS-7C37:~# restic -r rest:http://192.168.3.11:8000/ backup /home/frank/Dokumente/
enter password for repository: 
repository 39436420 opened successfully, password is correct
created new cache in /root/.cache/restic

Files:          27 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added to the repo: 6.218 MiB

processed 27 files, 6.217 MiB in 0:00
snapshot ad49233b saved

und noch einer

root@frank-MS-7C37:~# restic -r rest:http://192.168.3.11:8000/ backup /home/frank/Bilder/
enter password for repository: 
repository 39436420 opened successfully, password is correct

Files:         432 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added to the repo: 1.180 GiB

processed 432 files, 1.237 GiB in 0:16
snapshot ed35bfe7 saved

snapshots

root@frank-MS-7C37:~# restic -r rest:http://192.168.3.11:8000/ snapshots
enter password for repository: 
repository 39436420 opened successfully, password is correct
ID        Time                 Host           Tags        Paths
-------------------------------------------------------------------------------
ad49233b  2020-05-10 09:48:10  frank-MS-7C37              /home/frank/Dokumente
ed35bfe7  2020-05-10 09:48:38  frank-MS-7C37              /home/frank/Bilder
-------------------------------------------------------------------------------
2 snapshots

Rest-Server

Auf dem Rest-Server liegen die Daten in dem angegebenen Verzeichnis --path /home/rock64/backup

root@rockpro64:/# ls -lha /home/rock64/backup/
total 32K
drwxr-xr-x   7 root   root   4.0K May 10 09:47 .
drwxr-xr-x   5 rock64 rock64 4.0K May 10 09:43 ..
-rw-------   1 root   root    155 May 10 09:47 config
drwx------ 258 root   root   4.0K May 10 09:47 data
drwx------   2 root   root   4.0K May 10 09:49 index
drwx------   2 root   root   4.0K May 10 09:47 keys
drwx------   2 root   root   4.0K May 10 09:49 locks
drwx------   2 root   root   4.0K May 10 09:49 snapshots

User Authentifizierung

Zur Benutzerauthentifizierung benutzt der Rest-Server das Tool htpasswd, dazu muss folgendes installiert sein.

apt install apache2-utils

Danach kann man im Backup Pfad die .htaccess anlegen.

root@rockpro64:/home/rock64/backup# htpasswd -B -c .htpasswd frank   
New password: 
Re-type new password: 
Adding password for user frank

Sieht dann im Verzeichnis so aus.

root@rockpro64:/home/rock64/backup# ls -lha
total 36K
drwxr-xr-x   7 root   root   4.0K May 10 10:16 .
drwxr-xr-x   5 rock64 rock64 4.0K May 10 09:43 ..
-rw-r--r--   1 root   root     67 May 10 10:17 .htpasswd
-rw-------   1 root   root    155 May 10 09:47 config
drwx------ 258 root   root   4.0K May 10 09:47 data
drwx------   2 root   root   4.0K May 10 09:49 index
drwx------   2 root   root   4.0K May 10 09:47 keys
drwx------   2 root   root   4.0K May 10 10:18 locks
drwx------   2 root   root   4.0K May 10 09:49 snapshots

Man kann jetzt, auf dieses Verzeichnis, nur noch drauf zugreifen wenn man sich mit USER und PASSWORD authentifiziert.

Server Start

root@rockpro64:/home/rock64/backup# rest-server --path /home/rock64/backup          
Data directory: /home/rock64/backup
Authentication enabled
Private repositories disabled
Starting server on :8000

Server Zugriff

root@frank-MS-7C37:~# restic -r rest:http://USER:PASSWORD@192.168.3.11:8000/ snapshots
enter password for repository: 
repository 39436420 opened successfully, password is correct
ID        Time                 Host           Tags        Paths
-------------------------------------------------------------------------------
ad49233b  2020-05-10 09:48:10  frank-MS-7C37              /home/frank/Dokumente
ed35bfe7  2020-05-10 09:48:38  frank-MS-7C37              /home/frank/Bilder
-------------------------------------------------------------------------------
2 snapshots

[FrankM]

Und noch zwei Dinge, die interessant sind und noch ausprobiert werden müssen

The --append-only mode allows creation of new backups but prevents deletion and modification of existing backups. This can be useful when backing up systems that have a potential of being hacked.

To prevent your users from accessing each others' repositories, you may use the --private-repos flag which grants access only when a subdirectory with the same name as the user is specified in the repository URL. For example, user "foo" using the repository URLs rest:https://foo:pass@host:8000/foo or rest:https://foo:pass@host:8000/foo/ would be granted access, but the same user using repository URLs rest:https://foo:pass@host:8000/ or rest:https://foo:pass@host:8000/foobar/ would be denied access.

[FrankM]

--append-only

Schützt Backups vor diesen fiesen Dinger, wie heißen die noch in der Windows Welt? Emotet oder so?

Server starten

root@rockpro64:/home/rock64/backup# rest-server --path /home/rock64/backup --append-only
Data directory: /home/rock64/backup
Authentication enabled
Private repositories disabled
Starting server on :8000

Snapshots

root@frank-MS-7C37:~# restic -r rest:http://USER:PASSWORD@192.168.3.11:8000/ snapshots
enter password for repository: 
repository 39436420 opened successfully, password is correct
ID        Time                 Host           Tags        Paths
-------------------------------------------------------------------------------
ad49233b  2020-05-10 09:48:10  frank-MS-7C37              /home/frank/Dokumente
ed35bfe7  2020-05-10 09:48:38  frank-MS-7C37              /home/frank/Bilder
-------------------------------------------------------------------------------
2 snapshots

Snapshot entfernen

root@frank-MS-7C37:~# restic -r rest:http://USER:PASSWORD@192.168.3.11:8000/ forget ad49233b
enter password for repository: 
repository 39436420 opened successfully, password is correct
Remove(<snapshot/ad49233b1a>) returned error, retrying after 682.09481ms: blob not removed, server response: 403 Forbidden (403)
Remove(<snapshot/ad49233b1a>) returned error, retrying after 1.019858552s: blob not removed, server response: 403 Forbidden (403)
Remove(<snapshot/ad49233b1a>) returned error, retrying after 1.508077523s: blob not removed, server response: 403 Forbidden (403)
Remove(<snapshot/ad49233b1a>) returned error, retrying after 2.060637198s: blob not removed, server response: 403 Forbidden (403)
Remove(<snapshot/ad49233b1a>) returned error, retrying after 1.967534989s: blob not removed, server response: 403 Forbidden (403)
Remove(<snapshot/ad49233b1a>) returned error, retrying after 5.102318508s: blob not removed, server response: 403 Forbidden (403)
Remove(<snapshot/ad49233b1a>) returned error, retrying after 7.165967493s: blob not removed, server response: 403 Forbidden (403)
Remove(<snapshot/ad49233b1a>) returned error, retrying after 6.624453285s: blob not removed, server response: 403 Forbidden (403)
Remove(<snapshot/ad49233b1a>) returned error, retrying after 11.304339401s: blob not removed, server response: 403 Forbidden (403)

Ok, funktioniert

[FrankM]

IPv6

root@frank-MS-7C37:~# restic -r rest:http://USER:PASSWORD@[2a02:908:1265:fbf0:xxxx:xxxx:xxxx:xxxx]:8000/ snapshots
enter password for repository: 
repository 39436420 opened successfully, password is correct
ID        Time                 Host           Tags        Paths
-------------------------------------------------------------------------------
ad49233b  2020-05-10 09:48:10  frank-MS-7C37              /home/frank/Dokumente
ed35bfe7  2020-05-10 09:48:38  frank-MS-7C37              /home/frank/Bilder
-------------------------------------------------------------------------------
2 snapshots

[FrankM]

iptables

        #=========================
        # Rest-Server zur Firewall erlauben (IN)
        #=========================
        $IP4TABLES -A INPUT -p tcp --dport 8000 -j ACCEPT
        $IP6TABLES -A INPUT -p tcp --dport 8000 -j ACCEPT

[FrankM]

Heute mal was Zeit gehabt zum Spielen. Rest-Server aufgesetzt, mittels Letsencrypt die TLS Verbindung eingerichtet und läuft alles. Bin gerade mal mein Home-Verzeichnis am hochladen. Evt. schreibe ich dazu einen separaten Beitrag.

[FrankM]

Rest-Server in Version v0.10.0 released. Test folgt die Tage..

Release v0.10.0 · restic/rest-server

Rest Server is a high performance HTTP server that implements restic's REST backend API. - Release v0.10.0 · restic/rest-server

GitHub (github.com)

[FrankM]

Dann mal eben ausprobiert. Auf meinem Server war die Version 0.9.7 selber, mit go, gebaut. Dann mache ich das auch mit der v0.10.0 so. Aber bevor ich anfange, wird die v0.9.7 gesichert.

mv /usr/local/bin/rest-server /usr/local/bin/rest-server_0_9_7

So erspare ich mir im Problemfall das selber bauen.

Ok, dann die neue Version bauen.

git clone https://github.com/restic/rest-server.git
cd rest-server
go run build.go

Danach befindet sich im Verzeichnis die Binärdatei rest-server

Die kopieren wir jetzt

cp rest-server /usr/local/bin

Danach kurzer Test

# rest-server --version
rest-server 0.10.0 (v0.10.0-6-g037fe06) compiled with go1.11.6 on linux/amd64

Gut Version passt

Dann ein Backup gestartet. Das sichert einen Teil meines Home-Verzeichnis

Files:         153 new,   100 changed, 177857 unmodified
Dirs:            0 new,     1 changed,     0 unmodified
Added to the repo: 81.881 MiB

processed 178110 files, 80.571 GiB in 0:28
snapshot 607e0027 saved
Applying Policy: keep the last 3 snapshots, 3 monthly snapshots
keep 5 snapshots:
ID        Time                 Host           Tags        Reasons           Paths
---------------------------------------------------------------------------------------
fa97890e  2020-07-25 21:02:05  frank-XXX              monthly snapshot  /home/frank
5b073bbb  2020-08-30 10:17:27  frank-XXX              monthly snapshot  /home/frank
f7cf37ef  2020-09-06 15:13:03  frank-XXX              last snapshot     /home/frank
0157462c  2020-09-13 13:32:12  frank-XXX              last snapshot     /home/frank
607e0027  2020-09-14 08:09:34  frank-XXX              last snapshot     /home/frank
                                                          monthly snapshot
---------------------------------------------------------------------------------------
5 snapshots

remove 1 snapshots:
ID        Time                 Host           Tags        Paths
---------------------------------------------------------------------
3010b7cc  2020-09-06 11:39:27  frank-XXX             /home/frank
---------------------------------------------------------------------
1 snapshots

1 snapshots have been removed, running prune
counting files in repo
building new index for repo
[1:34] 100.00%  17351 / 17351 packs

So weit funktioniert das genau wie vorher. Im Changelog stand ja was von Subfoldern. Das betrifft mich nicht, weil ich für jeden User genau ein Verzeichnis habe.

So mit alles Gut Dann warte ich mal morgen ab, ob die täglichen Backups der Server rund laufen.