Merge pull request #607 from vrslev/run-pre-commit

chore: Run pre-commit

.pre-commit-config.yaml

@@ -37,7 +37,7 @@ repos:
     rev: v2.1.0
     hooks:
       - id: codespell
-        exclude: ".*Dockerfile.*"
+        exclude: "build/bench/Dockerfile"
 
   - repo: local
     hooks:

.vscode/extensions.json

@@ -3,11 +3,7 @@
   // Extension identifier format: ${publisher}.${name}. Example: vscode.csharp
 
   // List of extensions which should be recommended for users of this workspace.
-	"recommendations": [
+  "recommendations": ["ms-vscode-remote.remote-containers"],
-		"ms-vscode-remote.remote-containers"
-	],
   // List of extensions recommended by VS Code that should not be recommended for users of this workspace.
-	"unwantedRecommendations": [
+  "unwantedRecommendations": []
-		
-	]
 }

CODE_OF_CONDUCT.md

@@ -14,21 +14,21 @@ appearance, race, religion, or sexual identity and orientation.
 Examples of behavior that contributes to creating a positive environment
 include:
 
-* Using welcoming and inclusive language
+- Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
+- Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
+- Gracefully accepting constructive criticism
-* Focusing on what is best for the community
+- Focusing on what is best for the community
-* Showing empathy towards other community members
+- Showing empathy towards other community members
 
 Examples of unacceptable behavior by participants include:
 
-* The use of sexualized language or imagery and unwelcome sexual attention or
+- The use of sexualized language or imagery and unwelcome sexual attention or
   advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
+- Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
+- Public or private harassment
-* Publishing others' private information, such as a physical or electronic
+- Publishing others' private information, such as a physical or electronic
   address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
+- Other conduct which could reasonably be considered inappropriate in a
   professional setting
 
 ## Our Responsibilities

CONTRIBUTING.md

@@ -7,9 +7,8 @@ Before publishing a PR, please test builds locally:
 - with VSCode for testing environments (only for frappe/bench image).
 
 On each PR that contains changes relevant to Docker builds, images are being built and tested in our CI (GitHub Actions).
-> :evergreen_tree: Please be considerate when pushing commits and opening PR for multiple branches, as the process of building images uses energy and contributes to global warming.
-> 
 
+> :evergreen_tree: Please be considerate when pushing commits and opening PR for multiple branches, as the process of building images uses energy and contributes to global warming.
 
 ## Lint
 
@@ -38,7 +37,6 @@ To run all the files in repository, run:
 pre-commit run --all-files
 ```
 
-
 ## Build
 
 ```shell
@@ -52,20 +50,25 @@ docker buildx bake -f docker-bake.hcl *...*
 ## Test
 
 ### Ping site
+
 Lightweight test that just checks if site will be available after creation.
 
 Frappe:
+
 ```shell
 ./tests/test-frappe.sh
 ```
 
 ERPNext:
+
 ```shell
 ./tests/test-erpnext.sh
 ```
 
 ### Integration test
+
 Tests frappe-bench-like commands, for example, `backup` and `restore`.
+
 ```shell
 ./tests/integration-test.sh
 ```

README.md

@@ -30,14 +30,14 @@ cd frappe_docker
 
 It takes care of the following:
 
-* Setting up the desired version of Frappe/ERPNext.
+- Setting up the desired version of Frappe/ERPNext.
-* Setting up all the system requirements: eg. MariaDB, Node, Redis.
+- Setting up all the system requirements: eg. MariaDB, Node, Redis.
-* Configure networking for remote access and setting up LetsEncrypt.
+- Configure networking for remote access and setting up LetsEncrypt.
 
 It doesn't take care of the following:
 
-* Cron Job to backup sites is not created by default.
+- Cron Job to backup sites is not created by default.
-* Use `CronJob` on k8s or refer wiki for alternatives.
+- Use `CronJob` on k8s or refer wiki for alternatives.
 
 1. Single Server Installs
    1. [Single bench](docs/single-bench.md). Easiest Install!

build/frappe-nginx/nginx-default.conf.template

@@ -89,7 +89,7 @@ server {
     client_body_buffer_size 16K;
     client_header_buffer_size 1k;
 
-    # enable gzip compresion
+    # enable gzip compression
     # based on https://mattstauffer.co/blog/enabling-gzip-on-nginx-servers-including-laravel-forge
     gzip on;
     gzip_http_version 1.1;

build/frappe-worker/bench

@@ -1,19 +1,22 @@
 #!/home/frappe/frappe-bench/env/bin/python
 
+import os
 import subprocess
 import sys
-import os
-
 
 if __name__ == "__main__":
-    bench_dir = os.path.join(os.sep, 'home', 'frappe', 'frappe-bench')
+    bench_dir = os.path.join(os.sep, "home", "frappe", "frappe-bench")
-    sites_dir = os.path.join(bench_dir, 'sites')
+    sites_dir = os.path.join(bench_dir, "sites")
     bench_helper = os.path.join(
-        bench_dir, 'apps', 'frappe',
+        bench_dir,
-        'frappe', 'utils', 'bench_helper.py',
+        "apps",
+        "frappe",
+        "frappe",
+        "utils",
+        "bench_helper.py",
     )
     cwd = os.getcwd()
     os.chdir(sites_dir)
     subprocess.check_call(
-        [sys.executable, bench_helper, 'frappe'] + sys.argv[1:],
+        [sys.executable, bench_helper, "frappe"] + sys.argv[1:],
     )

build/frappe-worker/commands/auto_migrate.py

@@ -1,14 +1,14 @@
 import os
-import semantic_version
-import git
 
+import git
+import semantic_version
 from migrate import migrate_sites
 from utils import (
-    save_version_file,
     get_apps,
+    get_config,
     get_container_versions,
     get_version_file,
-    get_config
+    save_version_file,
 )
 
 
@@ -30,12 +30,12 @@ def main():
         version_file_hash = None
         container_hash = None
 
-        repo = git.Repo(os.path.join('..', 'apps', app))
+        repo = git.Repo(os.path.join("..", "apps", app))
         branch = repo.active_branch.name
 
-        if branch == 'develop':
+        if branch == "develop":
-            version_file_hash = version_file.get(app+'_git_hash')
+            version_file_hash = version_file.get(app + "_git_hash")
-            container_hash = container_versions.get(app+'_git_hash')
+            container_hash = container_versions.get(app + "_git_hash")
             if container_hash and version_file_hash:
                 if container_hash != version_file_hash:
                     is_ready = True
@@ -54,7 +54,7 @@ def main():
 
     config = get_config()
 
-    if is_ready and config.get('maintenance_mode') != 1:
+    if is_ready and config.get("maintenance_mode") != 1:
         migrate_sites(maintenance_mode=True)
         version_file = container_versions
         save_version_file(version_file)

build/frappe-worker/commands/backup.py

@@ -1,7 +1,8 @@
 import os
+
 import frappe
-from frappe.utils.backups import scheduled_backup
 from frappe.utils import cint, get_sites, now
+from frappe.utils.backups import scheduled_backup
 
 
 def backup(sites, with_files=False):
@@ -13,12 +14,17 @@ def backup(sites, with_files=False):
             backup_path_db=None,
             backup_path_files=None,
             backup_path_private_files=None,
-            force=True
+            force=True,
         )
         print("database backup taken -", odb.backup_path_db, "- on", now())
         if with_files:
             print("files backup taken -", odb.backup_path_files, "- on", now())
-            print("private files backup taken -", odb.backup_path_private_files, "- on", now())
+            print(
+                "private files backup taken -",
+                odb.backup_path_private_files,
+                "- on",
+                now(),
+            )
         frappe.destroy()
 
 

build/frappe-worker/commands/check_connection.py

@@ -1,15 +1,16 @@
 import socket
 import time
+
+from constants import (
+    DB_HOST_KEY,
+    DB_PORT,
+    DB_PORT_KEY,
+    REDIS_CACHE_KEY,
+    REDIS_QUEUE_KEY,
+    REDIS_SOCKETIO_KEY,
+)
 from six.moves.urllib.parse import urlparse
 from utils import get_config
-from constants import (
-    REDIS_QUEUE_KEY,
-    REDIS_CACHE_KEY,
-    REDIS_SOCKETIO_KEY,
-    DB_HOST_KEY,
-    DB_PORT_KEY,
-    DB_PORT
-)
 
 
 def is_open(ip, port, timeout=30):
@@ -29,7 +30,7 @@ def check_host(ip, port, retry=10, delay=3, print_attempt=True):
     ipup = False
     for i in range(retry):
         if print_attempt:
-            print("Attempt {i} to connect to {ip}:{port}".format(ip=ip, port=port, i=i+1))
+            print(f"Attempt {i+1} to connect to {ip}:{port}")
         if is_open(ip, port):
             ipup = True
             break
@@ -40,30 +41,26 @@ def check_host(ip, port, retry=10, delay=3, print_attempt=True):
 
 # Check service
 def check_service(
-    retry=10,
+    retry=10, delay=3, print_attempt=True, service_name=None, service_port=None
-    delay=3,
+):
-    print_attempt=True,
-    service_name=None,
-    service_port=None):
 
     config = get_config()
     if not service_name:
-        service_name = config.get(DB_HOST_KEY, 'mariadb')
+        service_name = config.get(DB_HOST_KEY, "mariadb")
     if not service_port:
         service_port = config.get(DB_PORT_KEY, DB_PORT)
 
     is_db_connected = False
     is_db_connected = check_host(
-        service_name,
+        service_name, service_port, retry, delay, print_attempt
-        service_port,
+    )
-        retry,
-        delay,
-        print_attempt)
     if not is_db_connected:
-        print("Connection to {service_name}:{service_port} timed out".format(
+        print(
+            "Connection to {service_name}:{service_port} timed out".format(
                 service_name=service_name,
                 service_port=service_port,
-        ))
+            )
+        )
         exit(1)
 
 
@@ -71,14 +68,13 @@ def check_service(
 def check_redis_queue(retry=10, delay=3, print_attempt=True):
     check_redis_queue = False
     config = get_config()
-    redis_queue_url = urlparse(config.get(REDIS_QUEUE_KEY, "redis://redis-queue:6379")).netloc
+    redis_queue_url = urlparse(
+        config.get(REDIS_QUEUE_KEY, "redis://redis-queue:6379")
+    ).netloc
     redis_queue, redis_queue_port = redis_queue_url.split(":")
     check_redis_queue = check_host(
-        redis_queue,
+        redis_queue, redis_queue_port, retry, delay, print_attempt
-        redis_queue_port,
+    )
-        retry,
-        delay,
-        print_attempt)
     if not check_redis_queue:
         print("Connection to redis queue timed out")
         exit(1)
@@ -88,14 +84,13 @@ def check_redis_queue(retry=10, delay=3, print_attempt=True):
 def check_redis_cache(retry=10, delay=3, print_attempt=True):
     check_redis_cache = False
     config = get_config()
-    redis_cache_url = urlparse(config.get(REDIS_CACHE_KEY, "redis://redis-cache:6379")).netloc
+    redis_cache_url = urlparse(
+        config.get(REDIS_CACHE_KEY, "redis://redis-cache:6379")
+    ).netloc
     redis_cache, redis_cache_port = redis_cache_url.split(":")
     check_redis_cache = check_host(
-        redis_cache,
+        redis_cache, redis_cache_port, retry, delay, print_attempt
-        redis_cache_port,
+    )
-        retry,
-        delay,
-        print_attempt)
     if not check_redis_cache:
         print("Connection to redis cache timed out")
         exit(1)
@@ -105,14 +100,13 @@ def check_redis_cache(retry=10, delay=3, print_attempt=True):
 def check_redis_socketio(retry=10, delay=3, print_attempt=True):
     check_redis_socketio = False
     config = get_config()
-    redis_socketio_url = urlparse(config.get(REDIS_SOCKETIO_KEY, "redis://redis-socketio:6379")).netloc
+    redis_socketio_url = urlparse(
+        config.get(REDIS_SOCKETIO_KEY, "redis://redis-socketio:6379")
+    ).netloc
     redis_socketio, redis_socketio_port = redis_socketio_url.split(":")
     check_redis_socketio = check_host(
-        redis_socketio,
+        redis_socketio, redis_socketio_port, retry, delay, print_attempt
-        redis_socketio_port,
+    )
-        retry,
-        delay,
-        print_attempt)
     if not check_redis_socketio:
         print("Connection to redis socketio timed out")
         exit(1)
@@ -123,7 +117,7 @@ def main():
     check_redis_queue()
     check_redis_cache()
     check_redis_socketio()
-    print('Connections OK')
+    print("Connections OK")
 
 
 if __name__ == "__main__":

build/frappe-worker/commands/constants.py

@@ -1,13 +1,13 @@
-REDIS_QUEUE_KEY = 'redis_queue'
+REDIS_QUEUE_KEY = "redis_queue"
-REDIS_CACHE_KEY = 'redis_cache'
+REDIS_CACHE_KEY = "redis_cache"
-REDIS_SOCKETIO_KEY = 'redis_socketio'
+REDIS_SOCKETIO_KEY = "redis_socketio"
-DB_HOST_KEY = 'db_host'
+DB_HOST_KEY = "db_host"
-DB_PORT_KEY = 'db_port'
+DB_PORT_KEY = "db_port"
 DB_PORT = 3306
-APP_VERSIONS_JSON_FILE = 'app_versions.json'
+APP_VERSIONS_JSON_FILE = "app_versions.json"
-APPS_TXT_FILE = 'apps.txt'
+APPS_TXT_FILE = "apps.txt"
-COMMON_SITE_CONFIG_FILE = 'common_site_config.json'
+COMMON_SITE_CONFIG_FILE = "common_site_config.json"
 DATE_FORMAT = "%Y%m%d_%H%M%S"
-RDS_DB = 'rds_db'
+RDS_DB = "rds_db"
 RDS_PRIVILEGES = "SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, CREATE VIEW, EVENT, TRIGGER, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EXECUTE, LOCK TABLES"
-ARCHIVE_SITES_PATH = '/home/frappe/frappe-bench/sites/archive_sites'
+ARCHIVE_SITES_PATH = "/home/frappe/frappe-bench/sites/archive_sites"

build/frappe-worker/commands/doctor.py

@@ -1,20 +1,20 @@
 import argparse
 
 from check_connection import (
-    check_service,
     check_redis_cache,
     check_redis_queue,
     check_redis_socketio,
+    check_service,
 )
 
 
 def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument(
-        '-p',
+        "-p",
-        '--ping-service',
+        "--ping-service",
-        dest='ping_services',
+        dest="ping_services",
-        action='append',
+        action="append",
         type=str,
         help='list of services to ping, e.g. doctor -p "postgres:5432" --ping-service "mariadb:3306"',
     )
@@ -33,15 +33,15 @@ def main():
     check_redis_socketio(retry=1, delay=0, print_attempt=False)
     print("Redis SocketIO Connected")
 
-    if(args.ping_services):
+    if args.ping_services:
         for service in args.ping_services:
             service_name = None
             service_port = None
 
             try:
-                service_name, service_port = service.split(':')
+                service_name, service_port = service.split(":")
             except ValueError:
-                print('Service should be in format host:port, e.g postgres:5432')
+                print("Service should be in format host:port, e.g postgres:5432")
                 exit(1)
 
             check_service(
@@ -51,7 +51,7 @@ def main():
                 service_name=service_name,
                 service_port=service_port,
             )
-            print("{0}:{1} Connected".format(service_name, service_port))
+            print(f"{service_name}:{service_port} Connected")
 
     print("Health check successful")
     exit(0)

build/frappe-worker/commands/gevent_patch.py

@@ -1,2 +1,3 @@
 import gevent.monkey
+
 gevent.monkey.patch_all()

build/frappe-worker/commands/migrate.py

@@ -1,6 +1,6 @@
 import os
-import frappe
 
+import frappe
 from frappe.utils import cint, get_sites
 from utils import get_config, save_config
 
@@ -27,11 +27,12 @@ def migrate_sites(maintenance_mode=False):
         set_maintenance_mode(True)
 
     for site in sites:
-        print('Migrating', site)
+        print("Migrating", site)
         frappe.init(site=site)
         frappe.connect()
         try:
             from frappe.migrate import migrate
+
             migrate()
         finally:
             frappe.destroy()

build/frappe-worker/commands/new.py

@@ -1,15 +1,10 @@
 import os
+
 import frappe
 import semantic_version
-
-from frappe.installer import update_site_config
 from constants import COMMON_SITE_CONFIG_FILE, RDS_DB, RDS_PRIVILEGES
-from utils import (
+from frappe.installer import update_site_config
-    run_command,
+from utils import get_config, get_password, get_site_config, run_command
-    get_config,
-    get_site_config,
-    get_password,
-)
 
 # try to import _new_site from frappe, which could possibly
 # exist in either commands.py or installer.py, and so we need
@@ -24,33 +19,43 @@ except ImportError:
 
 def main():
     config = get_config()
-    db_type = 'mariadb'
+    db_type = "mariadb"
-    db_port = config.get('db_port', 3306)
+    db_port = config.get("db_port", 3306)
-    db_host = config.get('db_host')
+    db_host = config.get("db_host")
-    site_name = os.environ.get("SITE_NAME", 'site1.localhost')
+    site_name = os.environ.get("SITE_NAME", "site1.localhost")
-    db_root_username = os.environ.get("DB_ROOT_USER", 'root')
+    db_root_username = os.environ.get("DB_ROOT_USER", "root")
-    mariadb_root_password = get_password("MYSQL_ROOT_PASSWORD", 'admin')
+    mariadb_root_password = get_password("MYSQL_ROOT_PASSWORD", "admin")
     postgres_root_password = get_password("POSTGRES_PASSWORD")
     db_root_password = mariadb_root_password
 
     if postgres_root_password:
-        db_type = 'postgres'
+        db_type = "postgres"
         db_host = os.environ.get("POSTGRES_HOST")
         db_port = 5432
         db_root_password = postgres_root_password
         if not db_host:
-            db_host = config.get('db_host')
+            db_host = config.get("db_host")
-            print('Environment variable POSTGRES_HOST not found.')
+            print("Environment variable POSTGRES_HOST not found.")
-            print('Using db_host from common_site_config.json')
+            print("Using db_host from common_site_config.json")
 
         sites_path = os.getcwd()
         common_site_config_path = os.path.join(sites_path, COMMON_SITE_CONFIG_FILE)
-        update_site_config("root_login", db_root_username, validate = False, site_config_path = common_site_config_path)
+        update_site_config(
-        update_site_config("root_password", db_root_password, validate = False, site_config_path = common_site_config_path)
+            "root_login",
+            db_root_username,
+            validate=False,
+            site_config_path=common_site_config_path,
+        )
+        update_site_config(
+            "root_password",
+            db_root_password,
+            validate=False,
+            site_config_path=common_site_config_path,
+        )
 
     force = True if os.environ.get("FORCE", None) else False
     install_apps = os.environ.get("INSTALL_APPS", None)
-    install_apps = install_apps.split(',') if install_apps else []
+    install_apps = install_apps.split(",") if install_apps else []
     frappe.init(site_name, new_site=True)
 
     if semantic_version.Version(frappe.__version__).major > 11:
@@ -59,7 +64,7 @@ def main():
             site_name,
             mariadb_root_username=db_root_username,
             mariadb_root_password=db_root_password,
-            admin_password=get_password("ADMIN_PASSWORD", 'admin'),
+            admin_password=get_password("ADMIN_PASSWORD", "admin"),
             verbose=True,
             install_apps=install_apps,
             source_sql=None,
@@ -75,7 +80,7 @@ def main():
             site_name,
             mariadb_root_username=db_root_username,
             mariadb_root_password=db_root_password,
-            admin_password=get_password("ADMIN_PASSWORD", 'admin'),
+            admin_password=get_password("ADMIN_PASSWORD", "admin"),
             verbose=True,
             install_apps=install_apps,
             source_sql=None,
@@ -83,16 +88,23 @@ def main():
             reinstall=False,
         )
 
-
     if db_type == "mariadb":
         site_config = get_site_config(site_name)
-        db_name = site_config.get('db_name')
+        db_name = site_config.get("db_name")
-        db_password = site_config.get('db_password')
+        db_password = site_config.get("db_password")
 
-        mysql_command = ["mysql", f"-h{db_host}", f"-u{db_root_username}", f"-p{mariadb_root_password}", "-e"]
+        mysql_command = [
+            "mysql",
+            f"-h{db_host}",
+            f"-u{db_root_username}",
+            f"-p{mariadb_root_password}",
+            "-e",
+        ]
 
         # Drop User if exists
-        command = mysql_command + [f"DROP USER IF EXISTS '{db_name}'; FLUSH PRIVILEGES;"]
+        command = mysql_command + [
+            f"DROP USER IF EXISTS '{db_name}'; FLUSH PRIVILEGES;"
+        ]
         run_command(command)
 
         # Grant permission to database and set password
@@ -102,10 +114,12 @@ def main():
         if config.get(RDS_DB) or site_config.get(RDS_DB):
             grant_privileges = RDS_PRIVILEGES
 
-        command = mysql_command + [f"\
+        command = mysql_command + [
+            f"\
             CREATE USER IF NOT EXISTS '{db_name}'@'%' IDENTIFIED BY '{db_password}'; \
             GRANT {grant_privileges} ON `{db_name}`.* TO '{db_name}'@'%'; \
-            FLUSH PRIVILEGES;"]
+            FLUSH PRIVILEGES;"
+        ]
         run_command(command)
 
     if frappe.redis_server:

build/frappe-worker/commands/push_backup.py

@@ -1,16 +1,12 @@
+import datetime
 import os
 import time
-import boto3
-
-import datetime
 from glob import glob
-from frappe.utils import get_sites
+
+import boto3
 from constants import DATE_FORMAT
-from utils import (
+from frappe.utils import get_sites
-    get_s3_config,
+from utils import check_s3_environment_variables, get_s3_config, upload_file_to_s3
-    upload_file_to_s3,
-    check_s3_environment_variables,
-)
 
 
 def get_file_ext():
@@ -18,7 +14,7 @@ def get_file_ext():
         "database": "-database.sql.gz",
         "private_files": "-private-files.tar",
         "public_files": "-files.tar",
-        "site_config": "-site_config_backup.json"
+        "site_config": "-site_config_backup.json",
     }
 
 
@@ -31,19 +27,26 @@ def get_backup_details(sitename):
 
     if os.path.exists(site_backup_path):
         for filetype, ext in file_ext.items():
-            site_slug = sitename.replace('.', '_')
+            site_slug = sitename.replace(".", "_")
-            pattern = site_backup_path + '*-' + site_slug + ext
+            pattern = site_backup_path + "*-" + site_slug + ext
             backup_files = list(filter(os.path.isfile, glob(pattern)))
 
             if len(backup_files) > 0:
-                backup_files.sort(key=lambda file: os.stat(os.path.join(site_backup_path, file)).st_ctime)
+                backup_files.sort(
-                backup_date = datetime.datetime.strptime(time.ctime(os.path.getmtime(backup_files[0])), "%a %b %d %H:%M:%S %Y")
+                    key=lambda file: os.stat(
+                        os.path.join(site_backup_path, file)
+                    ).st_ctime
+                )
+                backup_date = datetime.datetime.strptime(
+                    time.ctime(os.path.getmtime(backup_files[0])),
+                    "%a %b %d %H:%M:%S %Y",
+                )
                 backup_details[filetype] = {
                     "sitename": sitename,
                     "file_size_in_bytes": os.stat(backup_files[-1]).st_size,
                     "file_path": os.path.abspath(backup_files[-1]),
                     "filename": os.path.basename(backup_files[-1]),
-                    "backup_date": backup_date.date().strftime("%Y-%m-%d %H:%M:%S")
+                    "backup_date": backup_date.date().strftime("%Y-%m-%d %H:%M:%S"),
                 }
 
     return backup_details
@@ -54,31 +57,34 @@ def delete_old_backups(limit, bucket, site_name):
     all_backup_dates = list()
     backup_limit = int(limit)
     check_s3_environment_variables()
-    bucket_dir = os.environ.get('BUCKET_DIR')
+    bucket_dir = os.environ.get("BUCKET_DIR")
     oldest_backup_date = None
 
     s3 = boto3.resource(
-        's3',
+        "s3",
-        region_name=os.environ.get('REGION'),
+        region_name=os.environ.get("REGION"),
-        aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
+        aws_access_key_id=os.environ.get("ACCESS_KEY_ID"),
-        aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
+        aws_secret_access_key=os.environ.get("SECRET_ACCESS_KEY"),
-        endpoint_url=os.environ.get('ENDPOINT_URL')
+        endpoint_url=os.environ.get("ENDPOINT_URL"),
     )
 
     bucket = s3.Bucket(bucket)
-    objects = bucket.meta.client.list_objects_v2(
+    objects = bucket.meta.client.list_objects_v2(Bucket=bucket.name, Delimiter="/")
-        Bucket=bucket.name,
-        Delimiter='/')
 
     if objects:
-        for obj in objects.get('CommonPrefixes'):
+        for obj in objects.get("CommonPrefixes"):
-            if obj.get('Prefix') == bucket_dir + '/':
+            if obj.get("Prefix") == bucket_dir + "/":
-                for backup_obj in bucket.objects.filter(Prefix=obj.get('Prefix')):
+                for backup_obj in bucket.objects.filter(Prefix=obj.get("Prefix")):
                     if backup_obj.get()["ContentType"] == "application/x-directory":
                         continue
                     try:
                         # backup_obj.key is bucket_dir/site/date_time/backupfile.extension
-                        bucket_dir, site_slug, date_time, backupfile = backup_obj.key.split('/')
+                        (
+                            bucket_dir,
+                            site_slug,
+                            date_time,
+                            backupfile,
+                        ) = backup_obj.key.split("/")
                         date_time_object = datetime.datetime.strptime(
                             date_time, DATE_FORMAT
                         )
@@ -98,7 +104,7 @@ def delete_old_backups(limit, bucket, site_name):
         for backup in all_backups:
             try:
                 # backup is bucket_dir/site/date_time/backupfile.extension
-                backup_dir, site_slug, backup_dt_string, filename = backup.split('/')
+                backup_dir, site_slug, backup_dt_string, filename = backup.split("/")
                 backup_datetime = datetime.datetime.strptime(
                     backup_dt_string, DATE_FORMAT
                 )
@@ -113,7 +119,7 @@ def delete_old_backups(limit, bucket, site_name):
                 for obj in bucket.objects.filter(Prefix=oldest_backup):
                     # delete all keys that are inside the oldest_backup
                     if bucket_dir in obj.key:
-                        print('Deleteing ' + obj.key)
+                        print("Deleting " + obj.key)
                         s3.Object(bucket.name, obj.key).delete()
 
 
@@ -124,31 +130,52 @@ def main():
 
     for site in sites:
         details = get_backup_details(site)
-        db_file = details.get('database', {}).get('file_path')
+        db_file = details.get("database", {}).get("file_path")
-        folder = os.environ.get('BUCKET_DIR') + '/' + site + '/'
+        folder = os.environ.get("BUCKET_DIR") + "/" + site + "/"
         if db_file:
-            folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(db_file)[:15] + '/'
+            folder = (
+                os.environ.get("BUCKET_DIR")
+                + "/"
+                + site
+                + "/"
+                + os.path.basename(db_file)[:15]
+                + "/"
+            )
             upload_file_to_s3(db_file, folder, conn, bucket)
 
             # Archive site_config.json
-            site_config_file = details.get('site_config', {}).get('file_path')
+            site_config_file = details.get("site_config", {}).get("file_path")
             if not site_config_file:
-                site_config_file = os.path.join(os.getcwd(), site, 'site_config.json')
+                site_config_file = os.path.join(os.getcwd(), site, "site_config.json")
             upload_file_to_s3(site_config_file, folder, conn, bucket)
 
-        public_files = details.get('public_files', {}).get('file_path')
+        public_files = details.get("public_files", {}).get("file_path")
         if public_files:
-            folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(public_files)[:15] + '/'
+            folder = (
+                os.environ.get("BUCKET_DIR")
+                + "/"
+                + site
+                + "/"
+                + os.path.basename(public_files)[:15]
+                + "/"
+            )
             upload_file_to_s3(public_files, folder, conn, bucket)
 
-        private_files = details.get('private_files', {}).get('file_path')
+        private_files = details.get("private_files", {}).get("file_path")
         if private_files:
-            folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(private_files)[:15] + '/'
+            folder = (
+                os.environ.get("BUCKET_DIR")
+                + "/"
+                + site
+                + "/"
+                + os.path.basename(private_files)[:15]
+                + "/"
+            )
             upload_file_to_s3(private_files, folder, conn, bucket)
 
-        delete_old_backups(os.environ.get('BACKUP_LIMIT', '3'), bucket, site)
+        delete_old_backups(os.environ.get("BACKUP_LIMIT", "3"), bucket, site)
 
-    print('push-backup complete')
+    print("push-backup complete")
     exit(0)
 
 

build/frappe-worker/commands/restore_backup.py

@@ -1,93 +1,88 @@
-import os
 import datetime
-import tarfile
 import hashlib
-import frappe
+import os
-import boto3
+import tarfile
 
-from frappe.utils import get_sites, random_string
+import boto3
-from frappe.installer import (
+import frappe
-    make_conf,
-    get_conf_params,
-    make_site_dirs,
-    update_site_config
-)
 from constants import COMMON_SITE_CONFIG_FILE, DATE_FORMAT, RDS_DB, RDS_PRIVILEGES
+from frappe.installer import (
+    get_conf_params,
+    make_conf,
+    make_site_dirs,
+    update_site_config,
+)
+from frappe.utils import get_sites, random_string
 from utils import (
-    run_command,
+    check_s3_environment_variables,
-    list_directories,
-    set_key_in_site_config,
-    get_site_config,
     get_config,
     get_password,
-    check_s3_environment_variables,
+    get_site_config,
+    list_directories,
+    run_command,
+    set_key_in_site_config,
 )
 
 
 def get_backup_dir():
-    return os.path.join(
+    return os.path.join(os.path.expanduser("~"), "backups")
-        os.path.expanduser('~'),
-        'backups'
-    )
 
 
 def decompress_db(database_file, site):
     command = ["gunzip", "-c", database_file]
     with open(database_file.replace(".gz", ""), "w") as db_file:
-        print('Extract Database GZip for site {}'.format(site))
+        print(f"Extract Database GZip for site {site}")
         run_command(command, stdout=db_file)
 
 
 def restore_database(files_base, site_config_path, site):
     # restore database
-    database_file = files_base + '-database.sql.gz'
+    database_file = files_base + "-database.sql.gz"
     decompress_db(database_file, site)
     config = get_config()
 
     # Set db_type if it exists in backup site_config.json
-    set_key_in_site_config('db_type', site, site_config_path)
+    set_key_in_site_config("db_type", site, site_config_path)
     # Set db_host if it exists in backup site_config.json
-    set_key_in_site_config('db_host', site, site_config_path)
+    set_key_in_site_config("db_host", site, site_config_path)
     # Set db_port if it exists in backup site_config.json
-    set_key_in_site_config('db_port', site, site_config_path)
+    set_key_in_site_config("db_port", site, site_config_path)
 
     # get updated site_config
     site_config = get_site_config(site)
 
     # if no db_type exists, default to mariadb
-    db_type = site_config.get('db_type', 'mariadb')
+    db_type = site_config.get("db_type", "mariadb")
     is_database_restored = False
 
-    if db_type == 'mariadb':
+    if db_type == "mariadb":
         restore_mariadb(
-            config=config,
+            config=config, site_config=site_config, database_file=database_file
-            site_config=site_config,
+        )
-            database_file=database_file)
         is_database_restored = True
-    elif db_type == 'postgres':
+    elif db_type == "postgres":
         restore_postgres(
-            config=config,
+            config=config, site_config=site_config, database_file=database_file
-            site_config=site_config,
+        )
-            database_file=database_file)
         is_database_restored = True
 
     if is_database_restored:
         # Set encryption_key if it exists in backup site_config.json
-        set_key_in_site_config('encryption_key', site, site_config_path)
+        set_key_in_site_config("encryption_key", site, site_config_path)
 
 
 def restore_files(files_base):
-    public_files = files_base + '-files.tar'
+    public_files = files_base + "-files.tar"
     # extract tar
     public_tar = tarfile.open(public_files)
-    print('Extracting {}'.format(public_files))
+    print(f"Extracting {public_files}")
     public_tar.extractall()
 
 
 def restore_private_files(files_base):
-    private_files = files_base + '-private-files.tar'
+    private_files = files_base + "-private-files.tar"
     private_tar = tarfile.open(private_files)
-    print('Extracting {}'.format(private_files))
+    print(f"Extracting {private_files}")
     private_tar.extractall()
 
 
@@ -96,15 +91,15 @@ def pull_backup_from_s3():
 
     # https://stackoverflow.com/a/54672690
     s3 = boto3.resource(
-        's3',
+        "s3",
-        region_name=os.environ.get('REGION'),
+        region_name=os.environ.get("REGION"),
-        aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
+        aws_access_key_id=os.environ.get("ACCESS_KEY_ID"),
-        aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
+        aws_secret_access_key=os.environ.get("SECRET_ACCESS_KEY"),
-        endpoint_url=os.environ.get('ENDPOINT_URL')
+        endpoint_url=os.environ.get("ENDPOINT_URL"),
     )
 
-    bucket_dir = os.environ.get('BUCKET_DIR')
+    bucket_dir = os.environ.get("BUCKET_DIR")
-    bucket_name = os.environ.get('BUCKET_NAME')
+    bucket_name = os.environ.get("BUCKET_NAME")
     bucket = s3.Bucket(bucket_name)
 
     # Change directory to /home/frappe/backups
@@ -118,10 +113,10 @@ def pull_backup_from_s3():
     for obj in bucket.objects.filter(Prefix=bucket_dir):
         if obj.get()["ContentType"] == "application/x-directory":
             continue
-        backup_file = obj.key.replace(os.path.join(bucket_dir, ''), '')
+        backup_file = obj.key.replace(os.path.join(bucket_dir, ""), "")
         backup_files.append(backup_file)
-        site_name, timestamp, backup_type = backup_file.split('/')
+        site_name, timestamp, backup_type = backup_file.split("/")
-        site_timestamp = site_name + '/' + timestamp
+        site_timestamp = site_name + "/" + timestamp
         sites.add(site_name)
         site_timestamps.add(site_timestamp)
 
@@ -129,13 +124,11 @@ def pull_backup_from_s3():
     for site in sites:
         backup_timestamps = []
         for site_timestamp in site_timestamps:
-            site_name, timestamp = site_timestamp.split('/')
+            site_name, timestamp = site_timestamp.split("/")
             if site == site_name:
-                timestamp_datetime = datetime.datetime.strptime(
+                timestamp_datetime = datetime.datetime.strptime(timestamp, DATE_FORMAT)
-                    timestamp, DATE_FORMAT
-                )
                 backup_timestamps.append(timestamp)
-        download_backups.append(site + '/' + max(backup_timestamps))
+        download_backups.append(site + "/" + max(backup_timestamps))
 
     # Only download latest backups
     for backup_file in backup_files:
@@ -143,21 +136,21 @@ def pull_backup_from_s3():
             if backup in backup_file:
                 if not os.path.exists(os.path.dirname(backup_file)):
                     os.makedirs(os.path.dirname(backup_file))
-                print('Downloading {}'.format(backup_file))
+                print(f"Downloading {backup_file}")
-                bucket.download_file(bucket_dir + '/' + backup_file, backup_file)
+                bucket.download_file(bucket_dir + "/" + backup_file, backup_file)
 
-    os.chdir(os.path.join(os.path.expanduser('~'), 'frappe-bench', 'sites'))
+    os.chdir(os.path.join(os.path.expanduser("~"), "frappe-bench", "sites"))
 
 
 def restore_postgres(config, site_config, database_file):
     # common config
     common_site_config_path = os.path.join(os.getcwd(), COMMON_SITE_CONFIG_FILE)
 
-    db_root_user = config.get('root_login')
+    db_root_user = config.get("root_login")
     if not db_root_user:
-        postgres_user = os.environ.get('DB_ROOT_USER')
+        postgres_user = os.environ.get("DB_ROOT_USER")
         if not postgres_user:
-            print('Variable DB_ROOT_USER not set')
+            print("Variable DB_ROOT_USER not set")
             exit(1)
 
         db_root_user = postgres_user
@@ -165,13 +158,14 @@ def restore_postgres(config, site_config, database_file):
             "root_login",
             db_root_user,
             validate=False,
-            site_config_path=common_site_config_path)
+            site_config_path=common_site_config_path,
+        )
 
-    db_root_password = config.get('root_password')
+    db_root_password = config.get("root_password")
     if not db_root_password:
-        root_password = get_password('POSTGRES_PASSWORD')
+        root_password = get_password("POSTGRES_PASSWORD")
         if not root_password:
-            print('Variable POSTGRES_PASSWORD not set')
+            print("Variable POSTGRES_PASSWORD not set")
             exit(1)
 
         db_root_password = root_password
@@ -179,53 +173,72 @@ def restore_postgres(config, site_config, database_file):
             "root_password",
             db_root_password,
             validate=False,
-            site_config_path=common_site_config_path)
+            site_config_path=common_site_config_path,
+        )
 
     # site config
-    db_host = site_config.get('db_host')
+    db_host = site_config.get("db_host")
-    db_port = site_config.get('db_port', 5432)
+    db_port = site_config.get("db_port", 5432)
-    db_name = site_config.get('db_name')
+    db_name = site_config.get("db_name")
-    db_password = site_config.get('db_password')
+    db_password = site_config.get("db_password")
 
     psql_command = ["psql"]
     psql_uri = f"postgres://{db_root_user}:{db_root_password}@{db_host}:{db_port}"
 
-    print('Restoring PostgreSQL')
+    print("Restoring PostgreSQL")
-    run_command(psql_command + [psql_uri, "-c", f"DROP DATABASE IF EXISTS \"{db_name}\""])
+    run_command(psql_command + [psql_uri, "-c", f'DROP DATABASE IF EXISTS "{db_name}"'])
     run_command(psql_command + [psql_uri, "-c", f"DROP USER IF EXISTS {db_name}"])
-    run_command(psql_command + [psql_uri, "-c", f"CREATE DATABASE \"{db_name}\""])
+    run_command(psql_command + [psql_uri, "-c", f'CREATE DATABASE "{db_name}"'])
-    run_command(psql_command + [psql_uri, "-c", f"CREATE user {db_name} password '{db_password}'"])
+    run_command(
-    run_command(psql_command + [psql_uri, "-c", f"GRANT ALL PRIVILEGES ON DATABASE \"{db_name}\" TO {db_name}"])
+        psql_command
-    with open(database_file.replace('.gz', ''), 'r') as db_file:
+        + [psql_uri, "-c", f"CREATE user {db_name} password '{db_password}'"]
+    )
+    run_command(
+        psql_command
+        + [psql_uri, "-c", f'GRANT ALL PRIVILEGES ON DATABASE "{db_name}" TO {db_name}']
+    )
+    with open(database_file.replace(".gz", "")) as db_file:
         run_command(psql_command + [f"{psql_uri}/{db_name}", "<"], stdin=db_file)
 
 
 def restore_mariadb(config, site_config, database_file):
-    db_root_password = get_password('MYSQL_ROOT_PASSWORD')
+    db_root_password = get_password("MYSQL_ROOT_PASSWORD")
     if not db_root_password:
-        print('Variable MYSQL_ROOT_PASSWORD not set')
+        print("Variable MYSQL_ROOT_PASSWORD not set")
         exit(1)
 
-    db_root_user = os.environ.get("DB_ROOT_USER", 'root')
+    db_root_user = os.environ.get("DB_ROOT_USER", "root")
 
-    db_host = site_config.get('db_host', config.get('db_host'))
+    db_host = site_config.get("db_host", config.get("db_host"))
-    db_port = site_config.get('db_port', config.get('db_port', 3306))
+    db_port = site_config.get("db_port", config.get("db_port", 3306))
-    db_name = site_config.get('db_name')
+    db_name = site_config.get("db_name")
-    db_password = site_config.get('db_password')
+    db_password = site_config.get("db_password")
 
     # mysql command prefix
-    mysql_command = ["mysql", f"-u{db_root_user}", f"-h{db_host}", f"-p{db_root_password}", f"-P{db_port}"]
+    mysql_command = [
+        "mysql",
+        f"-u{db_root_user}",
+        f"-h{db_host}",
+        f"-p{db_root_password}",
+        f"-P{db_port}",
+    ]
 
     # drop db if exists for clean restore
     drop_database = mysql_command + ["-e", f"DROP DATABASE IF EXISTS `{db_name}`;"]
     run_command(drop_database)
 
     # create db
-    create_database = mysql_command + ["-e", f"CREATE DATABASE IF NOT EXISTS `{db_name}`;"]
+    create_database = mysql_command + [
+        "-e",
+        f"CREATE DATABASE IF NOT EXISTS `{db_name}`;",
+    ]
     run_command(create_database)
 
     # create user
-    create_user = mysql_command + ["-e", f"CREATE USER IF NOT EXISTS '{db_name}'@'%' IDENTIFIED BY '{db_password}'; FLUSH PRIVILEGES;"]
+    create_user = mysql_command + [
+        "-e",
+        f"CREATE USER IF NOT EXISTS '{db_name}'@'%' IDENTIFIED BY '{db_password}'; FLUSH PRIVILEGES;",
+    ]
     run_command(create_user)
 
     # grant db privileges to user
@@ -236,11 +249,14 @@ def restore_mariadb(config, site_config, database_file):
     if config.get(RDS_DB) or site_config.get(RDS_DB):
         grant_privileges = RDS_PRIVILEGES
 
-    grant_privileges_command = mysql_command + ["-e", f"GRANT {grant_privileges} ON `{db_name}`.* TO '{db_name}'@'%' IDENTIFIED BY '{db_password}'; FLUSH PRIVILEGES;"]
+    grant_privileges_command = mysql_command + [
+        "-e",
+        f"GRANT {grant_privileges} ON `{db_name}`.* TO '{db_name}'@'%' IDENTIFIED BY '{db_password}'; FLUSH PRIVILEGES;",
+    ]
     run_command(grant_privileges_command)
 
-    print('Restoring MariaDB')
+    print("Restoring MariaDB")
-    with open(database_file.replace('.gz', ''), 'r') as db_file:
+    with open(database_file.replace(".gz", "")) as db_file:
         run_command(mysql_command + [f"{db_name}"], stdin=db_file)
 
 
@@ -251,35 +267,38 @@ def main():
         pull_backup_from_s3()
 
     for site in list_directories(backup_dir):
-        site_slug = site.replace('.', '_')
+        site_slug = site.replace(".", "_")
-        backups = [datetime.datetime.strptime(backup, DATE_FORMAT) for backup in list_directories(os.path.join(backup_dir, site))]
+        backups = [
+            datetime.datetime.strptime(backup, DATE_FORMAT)
+            for backup in list_directories(os.path.join(backup_dir, site))
+        ]
         latest_backup = max(backups).strftime(DATE_FORMAT)
-        files_base = os.path.join(backup_dir, site, latest_backup, '')
+        files_base = os.path.join(backup_dir, site, latest_backup, "")
-        files_base += latest_backup + '-' + site_slug
+        files_base += latest_backup + "-" + site_slug
-        site_config_path = files_base + '-site_config_backup.json'
+        site_config_path = files_base + "-site_config_backup.json"
         if not os.path.exists(site_config_path):
-            site_config_path = os.path.join(backup_dir, site, 'site_config.json')
+            site_config_path = os.path.join(backup_dir, site, "site_config.json")
         if site in get_sites():
-            print('Overwrite site {}'.format(site))
+            print(f"Overwrite site {site}")
             restore_database(files_base, site_config_path, site)
             restore_private_files(files_base)
             restore_files(files_base)
         else:
             site_config = get_conf_params(
-                db_name='_' + hashlib.sha1(site.encode()).hexdigest()[:16],
+                db_name="_" + hashlib.sha1(site.encode()).hexdigest()[:16],
-                db_password=random_string(16)
+                db_password=random_string(16),
             )
 
             frappe.local.site = site
             frappe.local.sites_path = os.getcwd()
-            frappe.local.site_path = os.getcwd() + '/' + site
+            frappe.local.site_path = os.getcwd() + "/" + site
             make_conf(
-                db_name=site_config.get('db_name'),
+                db_name=site_config.get("db_name"),
-                db_password=site_config.get('db_password'),
+                db_password=site_config.get("db_password"),
             )
             make_site_dirs()
 
-            print('Create site {}'.format(site))
+            print(f"Create site {site}")
             restore_database(files_base, site_config_path, site)
             restore_private_files(files_base)
             restore_files(files_base)

build/frappe-worker/commands/utils.py

@@ -1,15 +1,12 @@
 import json
 import os
 import subprocess
+
 import boto3
 import git
-
+from constants import APP_VERSIONS_JSON_FILE, APPS_TXT_FILE, COMMON_SITE_CONFIG_FILE
 from frappe.installer import update_site_config
-from constants import (
+
-    APP_VERSIONS_JSON_FILE,
-    APPS_TXT_FILE,
-    COMMON_SITE_CONFIG_FILE
-)
 
 def run_command(command, stdout=None, stdin=None, stderr=None):
     stdout = stdout or subprocess.PIPE
@@ -26,7 +23,7 @@ def run_command(command, stdout=None, stdin=None, stderr=None):
 
 
 def save_version_file(versions):
-    with open(APP_VERSIONS_JSON_FILE, 'w') as f:
+    with open(APP_VERSIONS_JSON_FILE, "w") as f:
         return json.dump(versions, f, indent=1, sort_keys=True)
 
 
@@ -58,10 +55,10 @@ def get_container_versions(apps):
             pass
 
         try:
-            path = os.path.join('..', 'apps', app)
+            path = os.path.join("..", "apps", app)
             repo = git.Repo(path)
             commit_hash = repo.head.object.hexsha
-            versions.update({app+'_git_hash': commit_hash})
+            versions.update({app + "_git_hash": commit_hash})
         except Exception:
             pass
 
@@ -94,18 +91,22 @@ def get_config():
 
 def get_site_config(site_name):
     site_config = None
-    with open('{site_name}/site_config.json'.format(site_name=site_name)) as site_config_file:
+    with open(f"{site_name}/site_config.json") as site_config_file:
         site_config = json.load(site_config_file)
     return site_config
 
 
 def save_config(config):
-    with open(COMMON_SITE_CONFIG_FILE, 'w') as f:
+    with open(COMMON_SITE_CONFIG_FILE, "w") as f:
         return json.dump(config, f, indent=1, sort_keys=True)
 
 
 def get_password(env_var, default=None):
-    return os.environ.get(env_var) or get_password_from_secret(f"{env_var}_FILE") or default
+    return (
+        os.environ.get(env_var)
+        or get_password_from_secret(f"{env_var}_FILE")
+        or default
+    )
 
 
 def get_password_from_secret(env_var):
@@ -128,14 +129,14 @@ def get_password_from_secret(env_var):
 
 def get_s3_config():
     check_s3_environment_variables()
-    bucket = os.environ.get('BUCKET_NAME')
+    bucket = os.environ.get("BUCKET_NAME")
 
     conn = boto3.client(
-        's3',
+        "s3",
-        region_name=os.environ.get('REGION'),
+        region_name=os.environ.get("REGION"),
-        aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
+        aws_access_key_id=os.environ.get("ACCESS_KEY_ID"),
-        aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
+        aws_secret_access_key=os.environ.get("SECRET_ACCESS_KEY"),
-        endpoint_url=os.environ.get('ENDPOINT_URL')
+        endpoint_url=os.environ.get("ENDPOINT_URL"),
     )
 
     return conn, bucket
@@ -164,7 +165,7 @@ def list_directories(path):
 def get_site_config_from_path(site_config_path):
     site_config = dict()
     if os.path.exists(site_config_path):
-        with open(site_config_path, 'r') as sc:
+        with open(site_config_path) as sc:
             site_config = json.load(sc)
     return site_config
 
@@ -173,32 +174,35 @@ def set_key_in_site_config(key, site, site_config_path):
     site_config = get_site_config_from_path(site_config_path)
     value = site_config.get(key)
     if value:
-        print('Set {key} in site config for site: {site}'.format(key=key, site=site))
+        print(f"Set {key} in site config for site: {site}")
-        update_site_config(key, value,
+        update_site_config(
-                            site_config_path=os.path.join(os.getcwd(), site, "site_config.json"))
+            key,
+            value,
+            site_config_path=os.path.join(os.getcwd(), site, "site_config.json"),
+        )
 
 
 def check_s3_environment_variables():
-    if 'BUCKET_NAME' not in os.environ:
+    if "BUCKET_NAME" not in os.environ:
-        print('Variable BUCKET_NAME not set')
+        print("Variable BUCKET_NAME not set")
         exit(1)
 
-    if 'ACCESS_KEY_ID' not in os.environ:
+    if "ACCESS_KEY_ID" not in os.environ:
-        print('Variable ACCESS_KEY_ID not set')
+        print("Variable ACCESS_KEY_ID not set")
         exit(1)
 
-    if 'SECRET_ACCESS_KEY' not in os.environ:
+    if "SECRET_ACCESS_KEY" not in os.environ:
-        print('Variable SECRET_ACCESS_KEY not set')
+        print("Variable SECRET_ACCESS_KEY not set")
         exit(1)
 
-    if 'ENDPOINT_URL' not in os.environ:
+    if "ENDPOINT_URL" not in os.environ:
-        print('Variable ENDPOINT_URL not set')
+        print("Variable ENDPOINT_URL not set")
         exit(1)
 
-    if 'BUCKET_DIR' not in os.environ:
+    if "BUCKET_DIR" not in os.environ:
-        print('Variable BUCKET_DIR not set')
+        print("Variable BUCKET_DIR not set")
         exit(1)
 
-    if 'REGION' not in os.environ:
+    if "REGION" not in os.environ:
-        print('Variable REGION not set')
+        print("Variable REGION not set")
         exit(1)

development/README.md

@@ -94,6 +94,7 @@ code Procfile
 ```
 
 Or running the following command:
+
 ```shell
 sed -i '/redis/d' ./Procfile
 ```
@@ -105,6 +106,7 @@ You can create a new site with the following command:
 ```shell
 bench new-site sitename --no-mariadb-socket
 ```
+
 sitename MUST end with .localhost for trying deployments locally.
 
 for example:
@@ -234,7 +236,7 @@ The first step is installing and updating the required software. Usually the fra
 /workspace/development/frappe-bench/env/bin/python -m pip install --upgrade jupyter ipykernel ipython
 ```
 
-Then, run the commmand `Python: Show Python interactive window` from the VSCode command palette.
+Then, run the command `Python: Show Python interactive window` from the VSCode command palette.
 
 Replace `mysite.localhost` with your site and run the following code in a Jupyter cell:
 
@@ -259,7 +261,6 @@ Example shows the queries to be executed for site `localhost`
 
 Open sites/localhost/site_config.json:
 
-
 ```shell
 code sites/localhost/site_config.json
 ```
@@ -286,6 +287,7 @@ EXIT;
 In case you don't use VSCode, you may start the containers manually with the following command:
 
 ### Running the containers
+
 ```shell
 docker-compose -f .devcontainer/docker-compose.yml up -d
 ```

development/vscode-example/launch.json

@@ -10,7 +10,12 @@
       "request": "launch",
       "program": "${workspaceFolder}/frappe-bench/apps/frappe/frappe/utils/bench_helper.py",
       "args": [
-                "frappe", "serve", "--port", "8000", "--noreload", "--nothreading"
+        "frappe",
+        "serve",
+        "--port",
+        "8000",
+        "--noreload",
+        "--nothreading"
       ],
       "pythonPath": "${workspaceFolder}/frappe-bench/env/bin/python",
       "cwd": "${workspaceFolder}/frappe-bench/sites",
@@ -23,9 +28,7 @@
       "type": "python",
       "request": "launch",
       "program": "${workspaceFolder}/frappe-bench/apps/frappe/frappe/utils/bench_helper.py",
-            "args": [
+      "args": ["frappe", "worker", "--queue", "default"],
-                "frappe", "worker", "--queue", "default"
-            ],
       "pythonPath": "${workspaceFolder}/frappe-bench/env/bin/python",
       "cwd": "${workspaceFolder}/frappe-bench/sites",
       "env": {
@@ -37,9 +40,7 @@
       "type": "python",
       "request": "launch",
       "program": "${workspaceFolder}/frappe-bench/apps/frappe/frappe/utils/bench_helper.py",
-            "args": [
+      "args": ["frappe", "worker", "--queue", "short"],
-                "frappe", "worker", "--queue", "short"
-            ],
       "pythonPath": "${workspaceFolder}/frappe-bench/env/bin/python",
       "cwd": "${workspaceFolder}/frappe-bench/sites",
       "env": {
@@ -51,9 +52,7 @@
       "type": "python",
       "request": "launch",
       "program": "${workspaceFolder}/frappe-bench/apps/frappe/frappe/utils/bench_helper.py",
-            "args": [
+      "args": ["frappe", "worker", "--queue", "long"],
-                "frappe", "worker", "--queue", "long"
-            ],
       "pythonPath": "${workspaceFolder}/frappe-bench/env/bin/python",
       "cwd": "${workspaceFolder}/frappe-bench/sites",
       "env": {
@@ -69,7 +68,13 @@
       "cwd": "${workspaceFolder}/frappe-bench",
       "console": "internalConsole",
       "args": [
-                "start", "socketio", "watch", "schedule", "worker_short", "worker_long", "worker_default"
+        "start",
+        "socketio",
+        "watch",
+        "schedule",
+        "worker_short",
+        "worker_long",
+        "worker_default"
       ]
     }
   ]

docs/docker-swarm.md

@@ -42,7 +42,7 @@ version: "3.7"
 
 services:
   mariadb-master:
-    image: 'bitnami/mariadb:10.3'
+    image: "bitnami/mariadb:10.3"
     deploy:
       restart_policy:
         condition: on-failure
@@ -54,7 +54,7 @@ services:
     secrets:
       - frappe-mariadb-root-password
     volumes:
-      - 'mariadb_master_data:/bitnami/mariadb'
+      - "mariadb_master_data:/bitnami/mariadb"
     environment:
       - MARIADB_REPLICATION_MODE=master
       - MARIADB_REPLICATION_USER=repl_user
@@ -62,7 +62,7 @@ services:
       - MARIADB_ROOT_PASSWORD_FILE=/run/secrets/frappe-mariadb-root-password
 
   mariadb-slave:
-    image: 'bitnami/mariadb:10.3'
+    image: "bitnami/mariadb:10.3"
     deploy:
       restart_policy:
         condition: on-failure
@@ -74,7 +74,7 @@ services:
     secrets:
       - frappe-mariadb-root-password
     volumes:
-      - 'mariadb_slave_data:/bitnami/mariadb'
+      - "mariadb_slave_data:/bitnami/mariadb"
     environment:
       - MARIADB_REPLICATION_MODE=slave
       - MARIADB_REPLICATION_USER=repl_user
@@ -265,6 +265,7 @@ Use environment variables:
 - `FRAPPE_VERSION` variable to be set to desired version of Frappe Framework. e.g. 12.7.0
 - `MARIADB_HOST=frappe-mariadb_mariadb-master`
 - `SITES` variable is list of sites in back tick and separated by comma
+
 ```
 SITES=`site1.example.com`,`site2.example.com`
 ```
@@ -292,4 +293,3 @@ SITES=`site1.example.com`,`site2.example.com`
 6. Env variables:
    - MAINTENANCE_MODE=1
 7. Start container
-

docs/multi-bench.md

@@ -123,46 +123,49 @@ Notes:
 ## Docker containers
 
 This repository contains the following docker-compose files, each one containing the described images:
-* docker-compose-common.yml
-    * redis-cache
-        * volume: redis-cache-vol
-    * redis-queue
-        * volume: redis-queue-vol
-    * redis-socketio
-        * volume: redis-socketio-vol
-    * mariadb: main database
-        * volume: mariadb-vol
-* docker-compose-erpnext.yml
-    * erpnext-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
-        * volume: assets-vol
-    * erpnext-python: main application code
-    * frappe-socketio: enables realtime communication to the user interface through websockets
-    * frappe-worker-default: background runner
-    * frappe-worker-short: background runner for short-running jobs
-    * frappe-worker-long: background runner for long-running jobs
-    * frappe-schedule
 
-* docker-compose-frappe.yml
+- docker-compose-common.yml
-    * frappe-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
+  - redis-cache
-        * volume: assets-vol, sites-vol
+    - volume: redis-cache-vol
-    * erpnext-python: main application code
+  - redis-queue
-        * volume: sites-vol
+    - volume: redis-queue-vol
-    * frappe-socketio: enables realtime communication to the user interface through websockets
+  - redis-socketio
-        * volume: sites-vol
+    - volume: redis-socketio-vol
-    * frappe-worker-default: background runner
+  - mariadb: main database
-        * volume: sites-vol
+    - volume: mariadb-vol
-    * frappe-worker-short: background runner for short-running jobs
+- docker-compose-erpnext.yml
-        * volume: sites-vol
-    * frappe-worker-long: background runner for long-running jobs
-        * volume: sites-vol
-    * frappe-schedule
-        * volume: sites-vol
 
-* docker-compose-networks.yml: this yaml define the network to communicate with *Letsencrypt Nginx Proxy Companion*.
+  - erpnext-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
+    - volume: assets-vol
+  - erpnext-python: main application code
+  - frappe-socketio: enables realtime communication to the user interface through websockets
+  - frappe-worker-default: background runner
+  - frappe-worker-short: background runner for short-running jobs
+  - frappe-worker-long: background runner for long-running jobs
+  - frappe-schedule
 
-* erpnext-publish.yml: this yml extends erpnext-nginx service to publish port 80, can only be used with docker-compose-erpnext.yml
+- docker-compose-frappe.yml
 
-* frappe-publish.yml: this yml extends frappe-nginx service to publish port 80, can only be used with docker-compose-frappe.yml
+  - frappe-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
+    - volume: assets-vol, sites-vol
+  - erpnext-python: main application code
+    - volume: sites-vol
+  - frappe-socketio: enables realtime communication to the user interface through websockets
+    - volume: sites-vol
+  - frappe-worker-default: background runner
+    - volume: sites-vol
+  - frappe-worker-short: background runner for short-running jobs
+    - volume: sites-vol
+  - frappe-worker-long: background runner for long-running jobs
+    - volume: sites-vol
+  - frappe-schedule
+    - volume: sites-vol
+
+- docker-compose-networks.yml: this yaml define the network to communicate with _Letsencrypt Nginx Proxy Companion_.
+
+- erpnext-publish.yml: this yml extends erpnext-nginx service to publish port 80, can only be used with docker-compose-erpnext.yml
+
+- frappe-publish.yml: this yml extends frappe-nginx service to publish port 80, can only be used with docker-compose-frappe.yml
 
 ## Updating and Migrating Sites
 

docs/single-bench.md

@@ -37,7 +37,7 @@ To get started, copy the existing `env-local` or `env-production` file to `.env`
 - `SITE_NAME=erp.example.com`
   - Creates this site after starting all services and installs ERPNext. Site name must be resolvable by users machines and the ERPNext components. e.g. `erp.example.com` or `mysite.localhost`.
 - `` SITES=`erp.example.com`  ``
-    - List of sites that are part of the deployment "bench" Each site is separated by a comma(,) and quoted in backtick (`). By default site created by ``SITE_NAME`` variable is added here.
+  - List of sites that are part of the deployment "bench" Each site is separated by a comma(,) and quoted in backtick (`). By default site created by `SITE_NAME` variable is added here.
   - If LetsEncrypt is being setup, make sure that the DNS for all the site's domains correctly point to the current instance.
 - `DB_ROOT_USER=root`
   - MariaDB root username
@@ -77,7 +77,7 @@ Make sure to replace `<project-name>` with the desired name you wish to set for
 
 Notes:
 
-- If it is the first time running and site is being initialized, *it can take multiple minutes for the site to be up*. Monitor `site-creator` container logs to check progress. Use command `docker logs <project-name>_site-creator_1 -f`
+- If it is the first time running and site is being initialized, _it can take multiple minutes for the site to be up_. Monitor `site-creator` container logs to check progress. Use command `docker logs <project-name>_site-creator_1 -f`
 - After the site is ready the username is `Administrator` and the password is `$ADMIN_PASSWORD`
 - The local deployment is for testing and REST API development purpose only
 - A complete development environment is available [here](../development)
@@ -86,32 +86,32 @@ Notes:
 
 The docker-compose file contains following services:
 
-* traefik: manages letsencrypt
+- traefik: manages letsencrypt
-    * volume: cert-vol
+  - volume: cert-vol
-* redis-cache: cache store
+- redis-cache: cache store
-    * volume: redis-cache-vol
+  - volume: redis-cache-vol
-* redis-queue: used by workers
+- redis-queue: used by workers
-    * volume: redis-queue-vol
+  - volume: redis-queue-vol
-* redis-socketio: used by socketio service
+- redis-socketio: used by socketio service
-    * volume: redis-socketio-vol
+  - volume: redis-socketio-vol
-* mariadb: main database
+- mariadb: main database
-    * volume: mariadb-vol
+  - volume: mariadb-vol
-* erpnext-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
+- erpnext-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
-    * volume: assets-vol and sites-vol
+  - volume: assets-vol and sites-vol
-* erpnext-python: main application code
+- erpnext-python: main application code
-    * volume: sites-vol and sites-vol
+  - volume: sites-vol and sites-vol
-* frappe-socketio: enables realtime communication to the user interface through websockets
+- frappe-socketio: enables realtime communication to the user interface through websockets
-    * volume: sites-vol
+  - volume: sites-vol
-* erpnext-worker-default: background runner
+- erpnext-worker-default: background runner
-    * volume: sites-vol
+  - volume: sites-vol
-* erpnext-worker-short: background runner for short-running jobs
+- erpnext-worker-short: background runner for short-running jobs
-    * volume: sites-vol
+  - volume: sites-vol
-* erpnext-worker-long: background runner for long-running jobs
+- erpnext-worker-long: background runner for long-running jobs
-    * volume: sites-vol
+  - volume: sites-vol
-* erpnext-schedule
+- erpnext-schedule
-    * volume: sites-vol
+  - volume: sites-vol
-* site-creator: run once container to create new site.
+- site-creator: run once container to create new site.
-    * volume: sites-vol
+  - volume: sites-vol
 
 ## Updating and Migrating Sites
 

docs/site-operations.md

@@ -168,16 +168,16 @@ Note:
   - /home/frappe/backups
     - site1.domain.com
       - 20200420_162000
-                - 20200420_162000-site1_domain_com-*
+        - 20200420_162000-site1_domain_com-\*
     - site2.domain.com
       - 20200420_162000
-                - 20200420_162000-site2_domain_com-*
+        - 20200420_162000-site2_domain_com-\*
 
 ## Edit configs
 
 Editing config manually might be required in some cases,
 one such case is to use Amazon RDS (or any other DBaaS).
-For full instructions, refer to the [wiki](https://github.com/frappe/frappe/wiki/Using-Frappe-with-Amazon-RDS-(or-any-other-DBaaS)). Common question can be found in Issues and on forum.
+For full instructions, refer to the [wiki](<https://github.com/frappe/frappe/wiki/Using-Frappe-with-Amazon-RDS-(or-any-other-DBaaS)>). Common question can be found in Issues and on forum.
 
 `common_site_config.json` or `site_config.json` from `sites-vol` volume has to be edited using following command:
 
@@ -231,7 +231,6 @@ Notes:
 - Use it to install/uninstall custom apps, add system manager user, etc.
 - To run the command as non root user add the command option `--user frappe`.
 
-
 ## Delete/Drop Site
 
 #### MariaDB Site

installation/docker-compose-common.yml

@@ -1,4 +1,4 @@
-version: '3'
+version: "3"
 
 services:
   redis-cache:

installation/docker-compose-erpnext.yml

@@ -1,4 +1,4 @@
-version: '3'
+version: "3"
 
 services:
   erpnext-nginx:

installation/docker-compose-frappe.yml

@@ -1,4 +1,4 @@
-version: '3'
+version: "3"
 
 services:
   frappe-nginx:

installation/docker-compose-networks.yml

@@ -1,4 +1,4 @@
-version: '3'
+version: "3"
 
 networks:
   default: