Shiloh/frappe_docker
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #174 from revant/backup-restore

Browse Source 
feat: cloud backup and restore
This commit is contained in:
Revant Nandgaonkar 2020-03-27 16:43:16 +05:30 committed by GitHub
commit d2216100a3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 465 additions and 24 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

68
README.md
View File

@ -224,6 +224,36 @@ docker exec -it \
The backup will be available in the `sites` mounted volume.
#### Push backup to s3 compatible storage
Environment Variables
- `BUCKET_NAME`, Required to set bucket created on S3 compatible storage.
- `ACCESS_KEY_ID`, Required to set access key.
- `SECRET_ACCESS_KEY`, Required to set secret access key.
- `ENDPOINT_URL`, Required to set URL of S3 compatible storage.
- `BUCKET_DIR`, Required to set directory in bucket where sites from this deployment will be backed up.
- `BACKUP_LIMIT`, Optionally set this to limit number of backups in bucket directory. Defaults to 3.
```sh
docker run \
-e "BUCKET_NAME=backups" \
-e "ACCESS_KEY_ID=access_id_from_provider" \
-e "SECRET_ACCESS_KEY=secret_access_from_provider" \
-e "ENDPOINT_URL=https://region.storage-provider.com" \
-e "BUCKET_DIR=frappe-bench-v12" \
-v ./installation/sites:/home/frappe/frappe-bench/sites \
--network <project-name>_default \
frappe/frappe-worker:v12 push-backup
```
Note:
- Above example will backup files in bucket called `backup` at location `frappe-bench-v12/site.name.com/DATE_TIME/DATE_TIME-site_name_com-{filetype}.{extension}`,
- example DATE_TIME: 20200325_042020.
- example filetype: database, files or private-files
- example extension: sql.gz or tar
#### Updating and Migrating Sites
Switch to the root of the `frappe_docker` directory before running the following commands:
@ -251,6 +281,44 @@ docker exec -it \
<project-name>_erpnext-python_1 docker-entrypoint.sh migrate
```
#### Restore backups
Environment Variables
- `MYSQL_ROOT_PASSWORD`, Required to restore mariadb backups.
- `BUCKET_NAME`, Required to set bucket created on S3 compatible storage.
- `ACCESS_KEY_ID`, Required to set access key.
- `SECRET_ACCESS_KEY`, Required to set secret access key.
- `ENDPOINT_URL`, Required to set URL of S3 compatible storage.
- `BUCKET_DIR`, Required to set directory in bucket where sites from this deployment will be backed up.
```sh
docker run \
-e "MYSQL_ROOT_PASSWORD=admin" \
-e "BUCKET_NAME=backups" \
-e "ACCESS_KEY_ID=access_id_from_provider" \
-e "SECRET_ACCESS_KEY=secret_access_from_provider" \
-e "ENDPOINT_URL=https://region.storage-provider.com" \
-e "BUCKET_DIR=frappe-bench-v12" \
-v ./installation/sites:/home/frappe/frappe-bench/sites \
-v ./backups:/home/frappe/backups \
--network <project-name>_default \
frappe/frappe-worker:v12 restore-backup
```
Note:
- Volume must be mounted at location `/home/frappe/backups` for restoring sites
- If no backup files are found in volume, it will use s3 credentials to pull backups
- Backup structure for mounted volume or downloaded from s3:
- /home/frappe/backups
- site1.domain.com
- 20200420_162000
- 20200420_162000-site1_domain_com-*
- site2.domain.com
- 20200420_162000
- 20200420_162000-site2_domain_com-*
### Custom apps
To add your own Frappe/ERPNext apps to the image, we'll need to create a custom image with the help of a unique wrapper script

21
build/common/commands/new.py
View File

@ -29,36 +29,27 @@ def main():
site_config = get_site_config(site_name)
# update User's host to '%' required to connect from any container
command = 'mysql -h{db_host} -u{mariadb_root_username} -p{mariadb_root_password} -e '.format(
mysql_command = 'mysql -h{db_host} -u{mariadb_root_username} -p{mariadb_root_password} -e '.format(
db_host=config.get('db_host'),
mariadb_root_username=mariadb_root_username,
mariadb_root_password=mariadb_root_password
)
command += "\"UPDATE mysql.user SET Host = '%' where User = '{db_name}'; FLUSH PRIVILEGES;\"".format(
# update User's host to '%' required to connect from any container
command = mysql_command + "\"UPDATE mysql.user SET Host = '%' where User = '{db_name}'; FLUSH PRIVILEGES;\"".format(
db_name=site_config.get('db_name')
)
os.system(command)
# Set db password
command = 'mysql -h{db_host} -u{mariadb_root_username} -p{mariadb_root_password} -e '.format(
db_host=config.get('db_host'),
mariadb_root_username=mariadb_root_username,
mariadb_root_password=mariadb_root_password
)
command += "\"SET PASSWORD FOR '{db_name}'@'%' = PASSWORD('{db_password}'); FLUSH PRIVILEGES;\"".format(
command = mysql_command + "\"UPDATE mysql.user SET authentication_string = PASSWORD('{db_password}') WHERE User = \'{db_name}\' AND Host = \'%\';\"".format(
db_name=site_config.get('db_name'),
db_password=site_config.get('db_password')
)
os.system(command)
# Grant permission to database
command = 'mysql -h{db_host} -u{mariadb_root_username} -p{mariadb_root_password} -e '.format(
db_host=config.get('db_host'),
mariadb_root_username=mariadb_root_username,
mariadb_root_password=mariadb_root_password
)
command += "\"GRANT ALL PRIVILEGES ON \`{db_name}\`.* TO '{db_name}'@'%'; FLUSH PRIVILEGES;\"".format(
command = mysql_command + "\"GRANT ALL PRIVILEGES ON \`{db_name}\`.* TO '{db_name}'@'%'; FLUSH PRIVILEGES;\"".format(
db_name=site_config.get('db_name')
)
os.system(command)

182
build/common/commands/push_backup.py Normal file
View File

@ -0,0 +1,182 @@
import os
import time
import boto3
import datetime
from glob import glob
from frappe.utils import get_sites
DATE_FORMAT = "%Y%m%d_%H%M%S"
def get_file_ext():
return {
"database": "-database.sql.gz",
"private_files": "-private-files.tar",
"public_files": "-files.tar"
}
def get_backup_details(sitename):
backup_details = dict()
file_ext = get_file_ext()
# add trailing slash https://stackoverflow.com/a/15010678
site_backup_path = os.path.join(os.getcwd(), sitename, "private", "backups", "")
if os.path.exists(site_backup_path):
for filetype, ext in file_ext.items():
site_slug = sitename.replace('.', '_')
pattern = site_backup_path + '*-' + site_slug + ext
backup_files = list(filter(os.path.isfile, glob(pattern)))
if len(backup_files) > 0:
backup_files.sort(key=lambda file: os.stat(os.path.join(site_backup_path, file)).st_ctime)
backup_date = datetime.datetime.strptime(time.ctime(os.path.getmtime(backup_files[0])), "%a %b %d %H:%M:%S %Y")
backup_details[filetype] = {
"sitename": sitename,
"file_size_in_bytes": os.stat(backup_files[-1]).st_size,
"file_path": os.path.abspath(backup_files[-1]),
"filename": os.path.basename(backup_files[-1]),
"backup_date": backup_date.date().strftime("%Y-%m-%d %H:%M:%S")
}
return backup_details
def get_s3_config():
check_environment_variables()
bucket = os.environ.get('BUCKET_NAME')
conn = boto3.client(
's3',
aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
endpoint_url=os.environ.get('ENDPOINT_URL')
)
return conn, bucket
def check_environment_variables():
if not 'BUCKET_NAME' in os.environ:
print('Variable BUCKET_NAME not set')
exit(1)
if not 'ACCESS_KEY_ID' in os.environ:
print('Variable ACCESS_KEY_ID not set')
exit(1)
if not 'SECRET_ACCESS_KEY' in os.environ:
print('Variable SECRET_ACCESS_KEY not set')
exit(1)
if not 'ENDPOINT_URL' in os.environ:
print('Variable ENDPOINT_URL not set')
exit(1)
if not 'BUCKET_DIR' in os.environ:
print('Variable BUCKET_DIR not set')
exit(1)
def upload_file_to_s3(filename, folder, conn, bucket):
destpath = os.path.join(folder, os.path.basename(filename))
try:
print("Uploading file:", filename)
conn.upload_file(filename, bucket, destpath)
except Exception as e:
print("Error uploading: %s" % (e))
exit(1)
def delete_old_backups(limit, bucket, site_name):
all_backups = list()
all_backup_dates = list()
backup_limit = int(limit)
check_environment_variables()
bucket_dir = os.environ.get('BUCKET_DIR')
oldest_backup_date = None
s3 = boto3.resource(
's3',
aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
endpoint_url=os.environ.get('ENDPOINT_URL')
)
bucket = s3.Bucket(bucket)
objects = bucket.meta.client.list_objects_v2(
Bucket=bucket.name,
Delimiter='/')
if objects:
for obj in objects.get('CommonPrefixes'):
if obj.get('Prefix') == bucket_dir + '/':
for backup_obj in bucket.objects.filter(Prefix=obj.get('Prefix')):
try:
# backup_obj.key is bucket_dir/site/date_time/backupfile.extension
bucket_dir, site_slug, date_time, backupfile = backup_obj.key.split('/')
date_time_object = datetime.datetime.strptime(
date_time, DATE_FORMAT
)
if site_name in backup_obj.key:
all_backup_dates.append(date_time_object)
all_backups.append(backup_obj.key)
except IndexError as error:
print(error)
exit(1)
if len(all_backup_dates) > 0:
oldest_backup_date = min(all_backup_dates)
if len(all_backups) / 3 > backup_limit:
oldest_backup = None
for backup in all_backups:
try:
# backup is bucket_dir/site/date_time/backupfile.extension
backup_dir, site_slug, backup_dt_string, filename = backup.split('/')
backup_datetime = datetime.datetime.strptime(
backup_dt_string, DATE_FORMAT
)
if backup_datetime == oldest_backup_date:
oldest_backup = backup
except IndexError as error:
print(error)
exit(1)
if oldest_backup:
for obj in bucket.objects.filter(Prefix=oldest_backup):
# delete all keys that are inside the oldest_backup
if bucket_dir in obj.key:
print('Deleteing ' + obj.key)
s3.Object(bucket.name, obj.key).delete()
def main():
details = dict()
sites = get_sites()
conn, bucket = get_s3_config()
for site in sites:
details = get_backup_details(site)
db_file = details.get('database', {}).get('file_path')
folder = os.environ.get('BUCKET_DIR') + '/' + site + '/'
if db_file:
folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(db_file)[:15] + '/'
upload_file_to_s3(db_file, folder, conn, bucket)
public_files = details.get('public_files', {}).get('file_path')
if public_files:
folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(public_files)[:15] + '/'
upload_file_to_s3(public_files, folder, conn, bucket)
private_files = details.get('private_files', {}).get('file_path')
if private_files:
folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(private_files)[:15] + '/'
upload_file_to_s3(private_files, folder, conn, bucket)
delete_old_backups(os.environ.get('BACKUP_LIMIT', '3'), bucket, site)
print('push-backup complete')
exit(0)
if __name__ == "__main__":
main()

188
build/common/commands/restore_backup.py Normal file
View File

@ -0,0 +1,188 @@
import os
import datetime
import tarfile
import hashlib
import frappe
import boto3
from push_backup import DATE_FORMAT, check_environment_variables
from frappe.utils import get_sites, random_string
from frappe.commands.site import _new_site
from frappe.installer import make_conf, get_conf_params, make_site_dirs
from check_connection import get_site_config, get_config
def list_directories(path):
directories = []
for name in os.listdir(path):
if os.path.isdir(os.path.join(path, name)):
directories.append(name)
return directories
def get_backup_dir():
return os.path.join(
os.path.expanduser('~'),
'backups'
)
def decompress_db(files_base, site):
database_file = files_base + '-database.sql.gz'
config = get_config()
site_config = get_site_config(site)
db_root_user = os.environ.get('DB_ROOT_USER', 'root')
command = 'gunzip -c {database_file} > {database_extract}'.format(
database_file=database_file,
database_extract=database_file.replace('.gz','')
)
print('Extract Database GZip for site {}'.format(site))
os.system(command)
def restore_database(files_base, site):
db_root_password = os.environ.get('MYSQL_ROOT_PASSWORD')
if not db_root_password:
print('Variable MYSQL_ROOT_PASSWORD not set')
exit(1)
db_root_user = os.environ.get("DB_ROOT_USER", 'root')
# restore database
database_file = files_base + '-database.sql.gz'
decompress_db(files_base, site)
config = get_config()
site_config = get_site_config(site)
# mysql command prefix
mysql_command = 'mysql -u{db_root_user} -h{db_host} -p{db_password} -e '.format(
db_root_user=db_root_user,
db_host=config.get('db_host'),
db_password=db_root_password
)
# drop db if exists for clean restore
drop_database = mysql_command + "\"DROP DATABASE IF EXISTS \`{db_name}\`;\"".format(
db_name=site_config.get('db_name')
)
os.system(drop_database)
# create db
create_database = mysql_command + "\"CREATE DATABASE IF NOT EXISTS \`{db_name}\`;\"".format(
db_name=site_config.get('db_name')
)
os.system(create_database)
# create user
create_user = mysql_command + "\"CREATE USER IF NOT EXISTS \'{db_name}\'@\'%\' IDENTIFIED BY \'{db_password}\'; FLUSH PRIVILEGES;\"".format(
db_name=site_config.get('db_name'),
db_password=site_config.get('db_password')
)
os.system(create_user)
# create user password
set_user_password = mysql_command + "\"UPDATE mysql.user SET authentication_string = PASSWORD('{db_password}') WHERE User = \'{db_name}\' AND Host = \'%\';\"".format(
db_name=site_config.get('db_name'),
db_password=site_config.get('db_password')
)
os.system(set_user_password)
# grant db privileges to user
grant_privileges = mysql_command + "\"GRANT ALL PRIVILEGES ON \`{db_name}\`.* TO '{db_name}'@'%'; FLUSH PRIVILEGES;\"".format(
db_name=site_config.get('db_name')
)
os.system(grant_privileges)
command = "mysql -u{db_root_user} -h{db_host} -p{db_password} '{db_name}' < {database_file}".format(
db_root_user=db_root_user,
db_host=config.get('db_host'),
db_password=db_root_password,
db_name=site_config.get('db_name'),
database_file=database_file.replace('.gz',''),
)
print('Restoring database for site: {}'.format(site))
os.system(command)
def restore_files(files_base):
public_files = files_base + '-files.tar'
# extract tar
public_tar = tarfile.open(public_files)
print('Extracting {}'.format(public_files))
public_tar.extractall()
def restore_private_files(files_base):
private_files = files_base + '-private-files.tar'
private_tar = tarfile.open(private_files)
print('Extracting {}'.format(private_files))
private_tar.extractall()
def pull_backup_from_s3():
check_environment_variables()
# https://stackoverflow.com/a/54672690
s3 = boto3.resource(
's3',
aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
endpoint_url=os.environ.get('ENDPOINT_URL')
)
bucket_dir = os.environ.get('BUCKET_DIR')
bucket_name = os.environ.get('BUCKET_NAME')
bucket = s3.Bucket(bucket_name)
# Change directory to /home/frappe/backups
os.chdir(get_backup_dir())
for obj in bucket.objects.filter(Prefix = bucket_dir):
backup_file = obj.key.replace(os.path.join(bucket_dir,''),'')
if not os.path.exists(os.path.dirname(backup_file)):
os.makedirs(os.path.dirname(backup_file))
print('Downloading {}'.format(backup_file))
bucket.download_file(obj.key, backup_file)
os.chdir(os.path.join(os.path.expanduser('~'), 'frappe-bench', 'sites'))
def main():
backup_dir = get_backup_dir()
if len(list_directories(backup_dir)) == 0:
pull_backup_from_s3()
for site in list_directories(backup_dir):
site_slug = site.replace('.','_')
backups = [datetime.datetime.strptime(backup, DATE_FORMAT) for backup in list_directories(os.path.join(backup_dir,site))]
latest_backup = max(backups).strftime(DATE_FORMAT)
files_base = os.path.join(backup_dir, site, latest_backup, '')
files_base += latest_backup + '-' + site_slug
if site in get_sites():
restore_database(files_base, site)
restore_private_files(files_base)
restore_files(files_base)
else:
mariadb_root_password = os.environ.get('MYSQL_ROOT_PASSWORD')
if not mariadb_root_password:
print('Variable MYSQL_ROOT_PASSWORD not set')
exit(1)
mariadb_root_username = os.environ.get('DB_ROOT_USER', 'root')
database_file = files_base + '-database.sql.gz'
site_config = get_conf_params(
db_name='_' + hashlib.sha1(site.encode()).hexdigest()[:16],
db_password=random_string(16)
)
frappe.local.site = site
frappe.local.sites_path = os.getcwd()
frappe.local.site_path = os.getcwd() + '/' + site
make_conf(
db_name=site_config.get('db_name'),
db_password=site_config.get('db_password'),
)
make_site_dirs()
restore_database(files_base, site)
restore_private_files(files_base)
restore_files(files_base)
exit(0)
if __name__ == "__main__":
main()

12
build/common/worker/docker-entrypoint.sh
View File

@ -175,6 +175,18 @@ elif [ "$1" = 'console' ]; then
python /home/frappe/frappe-bench/commands/console.py "$2"
fi
elif [ "$1" = 'push-backup' ]; then
su frappe -c ". /home/frappe/frappe-bench/env/bin/activate \
&& python /home/frappe/frappe-bench/commands/push_backup.py"
exit
elif [ "$1" = 'restore-backup' ]; then
su frappe -c ". /home/frappe/frappe-bench/env/bin/activate \
&& python /home/frappe/frappe-bench/commands/restore_backup.py"
exit
else
exec su frappe -c "$@"

6
build/frappe-worker/Dockerfile
View File

@ -21,7 +21,7 @@ RUN install_packages \
RUN wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.5/wkhtmltox_0.12.5-1.stretch_amd64.deb
RUN dpkg -i wkhtmltox_0.12.5-1.stretch_amd64.deb && rm wkhtmltox_0.12.5-1.stretch_amd64.deb
RUN mkdir -p apps logs commands
RUN mkdir -p apps logs commands /home/frappe/backups
RUN virtualenv env \
&& . env/bin/activate \
@ -40,9 +40,9 @@ COPY build/common/worker/install_app.sh /usr/local/bin/install_app
WORKDIR /home/frappe/frappe-bench/sites
RUN chown -R frappe:frappe /home/frappe/frappe-bench/sites
RUN chown -R frappe:frappe /home/frappe/frappe-bench/sites /home/frappe/backups
VOLUME [ "/home/frappe/frappe-bench/sites" ]
VOLUME [ "/home/frappe/frappe-bench/sites", "/home/frappe/backups" ]
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["start"]

6
build/frappe-worker/v11.Dockerfile
View File

@ -18,7 +18,7 @@ RUN install_packages \
RUN wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.5/wkhtmltox_0.12.5-1.stretch_amd64.deb
RUN dpkg -i wkhtmltox_0.12.5-1.stretch_amd64.deb && rm wkhtmltox_0.12.5-1.stretch_amd64.deb
RUN mkdir -p apps logs commands
RUN mkdir -p apps logs commands /home/frappe/backups
RUN virtualenv env \
&& . env/bin/activate \
@ -37,9 +37,9 @@ COPY build/common/worker/install_app.sh /usr/local/bin/install_app
WORKDIR /home/frappe/frappe-bench/sites
RUN chown -R frappe:frappe /home/frappe/frappe-bench/sites
RUN chown -R frappe:frappe /home/frappe/frappe-bench/sites /home/frappe/backups
VOLUME [ "/home/frappe/frappe-bench/sites" ]
VOLUME [ "/home/frappe/frappe-bench/sites", "/home/frappe/backups" ]
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["start"]

6
build/frappe-worker/v12.Dockerfile
View File

@ -21,7 +21,7 @@ RUN install_packages \
RUN wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.5/wkhtmltox_0.12.5-1.stretch_amd64.deb
RUN dpkg -i wkhtmltox_0.12.5-1.stretch_amd64.deb && rm wkhtmltox_0.12.5-1.stretch_amd64.deb
RUN mkdir -p apps logs commands
RUN mkdir -p apps logs commands /home/frappe/backups
RUN virtualenv env \
&& . env/bin/activate \
@ -40,9 +40,9 @@ COPY build/common/worker/install_app.sh /usr/local/bin/install_app
WORKDIR /home/frappe/frappe-bench/sites
RUN chown -R frappe:frappe /home/frappe/frappe-bench/sites
RUN chown -R frappe:frappe /home/frappe/frappe-bench/sites /home/frappe/backups
VOLUME [ "/home/frappe/frappe-bench/sites" ]
VOLUME [ "/home/frappe/frappe-bench/sites", "/home/frappe/backups" ]
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["start"]