Note: Object Storage Service Redundancy is only available on Windows and Linux.
The Object Storage Service is used for storing objects, such as profile and event images, as well as ephemeral data, such as event reply messages.
The service can operate in a redundant configuration when you have multiple SAFR servers running. All redundant secondary servers are load-balanced by the primary server for all Object Storage Service requests it receives.
By default all redundant servers will save objects locally, and ask other Object Storage Servers for objects it does not have locally.
When you’re using local object storage, you will lose access to all objects that are only stored by an offline Object Storage Server until the server becomes healthy again. If that server’s objects are lost, and you do not have backups, they will be unrecoverable.
Backups must be run on every redundant server that has Object Storage enabled.
Using network storage (NFS, SMB, etc) provides a shared location for each server to save and retrieve objects from. This provides each Object Storage Server with access to all of the objects, rather than just objects saved to its local storage.
Shared storage also provides an easier backup process, as you only have to run it from the primary server.
On simple secondary servers, the Object Storage Service will operate in proxy mode.
Object Storage Servers operating in proxy mode will not attempt to use their own storage for objects, but will instead proxy the request to Object Storage Services that are running on either the primary server or on a redundant secondary server. If the redundant server it contacts doesn’t have the object, the contacted redundant server will ask all other redundant servers for the object.
The list of servers that run the Object Storage Service is stored in the database and updated every minute. If a host does not respond within a timeout, it is de-prioritized.
On both the primary server and on redundant secondary servers the Object Storage Service stores new objects in storage.
When a server receives a request for a file it does not find in its storage, it will request the object from other Object Storage Servers via HTTPS, and return the object if found. (The same applies for DELETEs.) This allows multiple Object Storage Servers to operate without using shared network storage, with each server saving a subset of the total objects, and relaying requests for other objects to its neighbors.
Even when using shared network storage, sometimes a request will come in for a new object before it is visible to all systems on the shared storage. The Object Storage Service will ask all the other Object Storage Servers for the object until it finds one that has the object.
All objects are stored on a single server, and no proxying requests occur.
All objects are stored on a primary server. Any object requests sent to the secondary server are proxied back to the primary server.
Objects are saved to whichever server receives the POST request. Objects requested in GET requests are facilitated from either system object storage, if found, or requested from other Object Storage Servers if not.
Objects are saved locally on the host that services the POST request. GET requests are served from local storage if found, or requested from other Object Storage Servers if not. All requests to server C are proxied to servers A and B.
Objects are saved to shared storage on the host that services the POST request. GET requests are served from local storage if found, or requested from other Object Storage Servers if not. All requests to server C are proxied to servers A and B.
If you start with local storage but later decide to move to shared storage, you will need to consolidate all of your objects to the new shared storage solution, delete the local copies, and then mount the shared storage to the right location. To do this, do the following:
python /opt/RealNetworks/SAFR/bin/backup.py
python /opt/RealNetworks/SAFR/bin/backup.py -o
python "C:\Program Files\RealNetworks\SAFR\bin\backup.py"
python "C:\Program Files\RealNetworks\SAFR\bin\backup.py" -o
/opt/RealNetworks/SAFR/bin/stop
"C:\Program Files\RealNetworks\SAFR\bin\stop.bat"
/opt/RealNetworks/SAFR/cv-storage
C:\ProgramData\RealNetworks\SAFR\cv-storage
/opt/RealNetworks/SAFR/cv-storage
C:\ProgramData\RealNetworks\SAFR\cv-storage
/opt/RealNetworks/SAFR/bin/start
"C:\Program Files\RealNetworks\SAFR\bin\start.bat"
The SAFR backup and restore process when using shared network storage is straightforward - you just need to back up the primary server. This will back up all configs, database content, and Object Storage objects.
When using local storage, the objects are distributed to multiple servers, so the backup must be run on the primary server as well as any redundantly secondary servers.
The primary server should run a regular backup, while the redundant secondary servers run an ‘objects only’ backup. The difference is just the addition of the “-o” flag to the backup script.
When restoring multiple backups, you can restore them all to the primary server, or you can restore the ‘object only’ backups back to the same servers that they were backed up from.
python /opt/RealNetworks/SAFR/bin/backup.py
python /opt/RealNetworks/SAFR/bin/backup.py -o
python "C:\Program Files\RealNetworks\SAFR\bin\backup.py"
python "C:\Program Files\RealNetworks\SAFR\bin\backup.py" -o
python /opt/RealNetworks/SAFR/bin/restore.py BACKUPFILENAME
python /opt/RealNetworks/SAFR/bin/restore.py -o BACKUPFILENAME
python "C:\Program Files\RealNetworks\SAFR\bin\restore.py" BACKUPFILENAME
python "C:\Program Files\RealNetworks\SAFR\bin\restore.py" -o BACKUPFILENAME
Below are two example shared storage solutions.
Shared storage on Linux is very straightforward. Simply mount NFS or some other shared storage to the /opt/RealNetworks/SAFR/cv-storage
location.
Stop SAFR.
/opt/RealNetworks/SAFR/bin/stop
Create NFS or some other shared storage location. The example below uses AWS EFS.
Edit /etc/fstab
to create a mount point of /opt/RealNetworks/SAFR/cv-storage
for your shared storage. The specific mount options should be provided by your specific storage service/device.
fs-12345678.efs.us-west-2.amazonaws.com:/ /opt/RealNetworks/SAFR/cv-storage nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0
Mount the NFS share.
sudo mount -a
Start SAFR.
/opt/RealNetworks/SAFR/bin/start
Windows cannot mount a shared storage location directly to the C:\ProgramData\RealNetworks\SAFR\cv-storage
location. Instead it must create a symbolic link from that location to the shared storage location (either mapped network drive or SMB share).
"C:\Program Files\RealNetworks\SAFR\bin\stop.bat"
C:\ProgramData\RealNetworks\SAFR\cv-storage
by running rmdir /q /s C:\ProgramData\RealNetworks\SAFR\cv-storage
in an administrative command prompt. Deleting the existing cv-storage allows you to create a symbolic link from the ‘cv-storage’ location to your shared storage location. Note: Be sure you are either doing this on a new system without any data, or that you’ve followed the migration steps above to consolidate your objects onto the new shared storage location.cv-storage
location to your shared storage location. To do this, run one of the following commands in an administrative command prompt:
mklink /d C:\ProgramData\RealNetworks\SAFR\cv-storage Z:\
mklink /d C:\ProgramData\RealNetworks\SAFR\cv-storage \\servername\share
"C:\Program Files\RealNetworks\SAFR\bin\start.bat"