Shiloh/frappe_docker
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #607 from vrslev/run-pre-commit

Browse Source 
chore: Run pre-commit
This commit is contained in:
Lev 2021-12-10 12:26:45 +03:00 committed by GitHub
commit 3171f212ab
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
39 changed files with 824 additions and 747 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.pre-commit-config.yaml
View File

@ -37,7 +37,7 @@ repos:
rev: v2.1.0
hooks:
- id: codespell
exclude: ".*Dockerfile.*"
exclude: "build/bench/Dockerfile"
- repo: local
hooks:

8
.vscode/extensions.json vendored
View File

@ -3,11 +3,7 @@
// Extension identifier format: ${publisher}.${name}. Example: vscode.csharp
// List of extensions which should be recommended for users of this workspace.
"recommendations": [
"ms-vscode-remote.remote-containers"
],
"recommendations": ["ms-vscode-remote.remote-containers"],
// List of extensions recommended by VS Code that should not be recommended for users of this workspace.
"unwantedRecommendations": [
]
"unwantedRecommendations": []
}

20
CODE_OF_CONDUCT.md
View File

@ -14,21 +14,21 @@ appearance, race, religion, or sexual identity and orientation.
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
- Using welcoming and inclusive language
- Being respectful of differing viewpoints and experiences
- Gracefully accepting constructive criticism
- Focusing on what is best for the community
- Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
- The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
- Trolling, insulting/derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
- Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities

9
CONTRIBUTING.md
View File

@ -7,9 +7,8 @@ Before publishing a PR, please test builds locally:
- with VSCode for testing environments (only for frappe/bench image).
On each PR that contains changes relevant to Docker builds, images are being built and tested in our CI (GitHub Actions).
> :evergreen_tree: Please be considerate when pushing commits and opening PR for multiple branches, as the process of building images uses energy and contributes to global warming.
>
> :evergreen_tree: Please be considerate when pushing commits and opening PR for multiple branches, as the process of building images uses energy and contributes to global warming.
## Lint
@ -38,7 +37,6 @@ To run all the files in repository, run:
pre-commit run --all-files
```
## Build
```shell
@ -52,20 +50,25 @@ docker buildx bake -f docker-bake.hcl *...*
## Test
### Ping site
Lightweight test that just checks if site will be available after creation.
Frappe:
```shell
./tests/test-frappe.sh
```
ERPNext:
```shell
./tests/test-erpnext.sh
```
### Integration test
Tests frappe-bench-like commands, for example, `backup` and `restore`.
```shell
./tests/integration-test.sh
```

10
README.md
View File

@ -30,14 +30,14 @@ cd frappe_docker
It takes care of the following:
* Setting up the desired version of Frappe/ERPNext.
* Setting up all the system requirements: eg. MariaDB, Node, Redis.
* Configure networking for remote access and setting up LetsEncrypt.
- Setting up the desired version of Frappe/ERPNext.
- Setting up all the system requirements: eg. MariaDB, Node, Redis.
- Configure networking for remote access and setting up LetsEncrypt.
It doesn't take care of the following:
* Cron Job to backup sites is not created by default.
* Use `CronJob` on k8s or refer wiki for alternatives.
- Cron Job to backup sites is not created by default.
- Use `CronJob` on k8s or refer wiki for alternatives.
1. Single Server Installs
1. [Single bench](docs/single-bench.md). Easiest Install!

2
build/frappe-nginx/nginx-default.conf.template
View File

@ -89,7 +89,7 @@ server {
client_body_buffer_size 16K;
client_header_buffer_size 1k;
# enable gzip compresion
# enable gzip compression
# based on https://mattstauffer.co/blog/enabling-gzip-on-nginx-servers-including-laravel-forge
gzip on;
gzip_http_version 1.1;

17
build/frappe-worker/bench
View File

@ -1,19 +1,22 @@
#!/home/frappe/frappe-bench/env/bin/python
import os
import subprocess
import sys
import os
if __name__ == "__main__":
bench_dir = os.path.join(os.sep, 'home', 'frappe', 'frappe-bench')
sites_dir = os.path.join(bench_dir, 'sites')
bench_dir = os.path.join(os.sep, "home", "frappe", "frappe-bench")
sites_dir = os.path.join(bench_dir, "sites")
bench_helper = os.path.join(
bench_dir, 'apps', 'frappe',
'frappe', 'utils', 'bench_helper.py',
bench_dir,
"apps",
"frappe",
"frappe",
"utils",
"bench_helper.py",
)
cwd = os.getcwd()
os.chdir(sites_dir)
subprocess.check_call(
[sys.executable, bench_helper, 'frappe'] + sys.argv[1:],
[sys.executable, bench_helper, "frappe"] + sys.argv[1:],
)

18
build/frappe-worker/commands/auto_migrate.py
View File

@ -1,14 +1,14 @@
import os
import semantic_version
import git
import git
import semantic_version
from migrate import migrate_sites
from utils import (
save_version_file,
get_apps,
get_config,
get_container_versions,
get_version_file,
get_config
save_version_file,
)
@ -30,12 +30,12 @@ def main():
version_file_hash = None
container_hash = None
repo = git.Repo(os.path.join('..', 'apps', app))
repo = git.Repo(os.path.join("..", "apps", app))
branch = repo.active_branch.name
if branch == 'develop':
version_file_hash = version_file.get(app+'_git_hash')
container_hash = container_versions.get(app+'_git_hash')
if branch == "develop":
version_file_hash = version_file.get(app + "_git_hash")
container_hash = container_versions.get(app + "_git_hash")
if container_hash and version_file_hash:
if container_hash != version_file_hash:
is_ready = True
@ -54,7 +54,7 @@ def main():
config = get_config()
if is_ready and config.get('maintenance_mode') != 1:
if is_ready and config.get("maintenance_mode") != 1:
migrate_sites(maintenance_mode=True)
version_file = container_versions
save_version_file(version_file)

12
build/frappe-worker/commands/backup.py
View File

@ -1,7 +1,8 @@
import os
import frappe
from frappe.utils.backups import scheduled_backup
from frappe.utils import cint, get_sites, now
from frappe.utils.backups import scheduled_backup
def backup(sites, with_files=False):
@ -13,12 +14,17 @@ def backup(sites, with_files=False):
backup_path_db=None,
backup_path_files=None,
backup_path_private_files=None,
force=True
force=True,
)
print("database backup taken -", odb.backup_path_db, "- on", now())
if with_files:
print("files backup taken -", odb.backup_path_files, "- on", now())
print("private files backup taken -", odb.backup_path_private_files, "- on", now())
print(
"private files backup taken -",
odb.backup_path_private_files,
"- on",
now(),
)
frappe.destroy()

76
build/frappe-worker/commands/check_connection.py
View File

@ -1,15 +1,16 @@
import socket
import time
from constants import (
DB_HOST_KEY,
DB_PORT,
DB_PORT_KEY,
REDIS_CACHE_KEY,
REDIS_QUEUE_KEY,
REDIS_SOCKETIO_KEY,
)
from six.moves.urllib.parse import urlparse
from utils import get_config
from constants import (
REDIS_QUEUE_KEY,
REDIS_CACHE_KEY,
REDIS_SOCKETIO_KEY,
DB_HOST_KEY,
DB_PORT_KEY,
DB_PORT
)
def is_open(ip, port, timeout=30):
@ -29,7 +30,7 @@ def check_host(ip, port, retry=10, delay=3, print_attempt=True):
ipup = False
for i in range(retry):
if print_attempt:
print("Attempt {i} to connect to {ip}:{port}".format(ip=ip, port=port, i=i+1))
print(f"Attempt {i+1} to connect to {ip}:{port}")
if is_open(ip, port):
ipup = True
break
@ -40,30 +41,26 @@ def check_host(ip, port, retry=10, delay=3, print_attempt=True):
# Check service
def check_service(
retry=10,
delay=3,
print_attempt=True,
service_name=None,
service_port=None):
retry=10, delay=3, print_attempt=True, service_name=None, service_port=None
):
config = get_config()
if not service_name:
service_name = config.get(DB_HOST_KEY, 'mariadb')
service_name = config.get(DB_HOST_KEY, "mariadb")
if not service_port:
service_port = config.get(DB_PORT_KEY, DB_PORT)
is_db_connected = False
is_db_connected = check_host(
service_name,
service_port,
retry,
delay,
print_attempt)
service_name, service_port, retry, delay, print_attempt
)
if not is_db_connected:
print("Connection to {service_name}:{service_port} timed out".format(
print(
"Connection to {service_name}:{service_port} timed out".format(
service_name=service_name,
service_port=service_port,
))
)
)
exit(1)
@ -71,14 +68,13 @@ def check_service(
def check_redis_queue(retry=10, delay=3, print_attempt=True):
check_redis_queue = False
config = get_config()
redis_queue_url = urlparse(config.get(REDIS_QUEUE_KEY, "redis://redis-queue:6379")).netloc
redis_queue_url = urlparse(
config.get(REDIS_QUEUE_KEY, "redis://redis-queue:6379")
).netloc
redis_queue, redis_queue_port = redis_queue_url.split(":")
check_redis_queue = check_host(
redis_queue,
redis_queue_port,
retry,
delay,
print_attempt)
redis_queue, redis_queue_port, retry, delay, print_attempt
)
if not check_redis_queue:
print("Connection to redis queue timed out")
exit(1)
@ -88,14 +84,13 @@ def check_redis_queue(retry=10, delay=3, print_attempt=True):
def check_redis_cache(retry=10, delay=3, print_attempt=True):
check_redis_cache = False
config = get_config()
redis_cache_url = urlparse(config.get(REDIS_CACHE_KEY, "redis://redis-cache:6379")).netloc
redis_cache_url = urlparse(
config.get(REDIS_CACHE_KEY, "redis://redis-cache:6379")
).netloc
redis_cache, redis_cache_port = redis_cache_url.split(":")
check_redis_cache = check_host(
redis_cache,
redis_cache_port,
retry,
delay,
print_attempt)
redis_cache, redis_cache_port, retry, delay, print_attempt
)
if not check_redis_cache:
print("Connection to redis cache timed out")
exit(1)
@ -105,14 +100,13 @@ def check_redis_cache(retry=10, delay=3, print_attempt=True):
def check_redis_socketio(retry=10, delay=3, print_attempt=True):
check_redis_socketio = False
config = get_config()
redis_socketio_url = urlparse(config.get(REDIS_SOCKETIO_KEY, "redis://redis-socketio:6379")).netloc
redis_socketio_url = urlparse(
config.get(REDIS_SOCKETIO_KEY, "redis://redis-socketio:6379")
).netloc
redis_socketio, redis_socketio_port = redis_socketio_url.split(":")
check_redis_socketio = check_host(
redis_socketio,
redis_socketio_port,
retry,
delay,
print_attempt)
redis_socketio, redis_socketio_port, retry, delay, print_attempt
)
if not check_redis_socketio:
print("Connection to redis socketio timed out")
exit(1)
@ -123,7 +117,7 @@ def main():
check_redis_queue()
check_redis_cache()
check_redis_socketio()
print('Connections OK')
print("Connections OK")
if __name__ == "__main__":

20
build/frappe-worker/commands/constants.py
View File

@ -1,13 +1,13 @@
REDIS_QUEUE_KEY = 'redis_queue'
REDIS_CACHE_KEY = 'redis_cache'
REDIS_SOCKETIO_KEY = 'redis_socketio'
DB_HOST_KEY = 'db_host'
DB_PORT_KEY = 'db_port'
REDIS_QUEUE_KEY = "redis_queue"
REDIS_CACHE_KEY = "redis_cache"
REDIS_SOCKETIO_KEY = "redis_socketio"
DB_HOST_KEY = "db_host"
DB_PORT_KEY = "db_port"
DB_PORT = 3306
APP_VERSIONS_JSON_FILE = 'app_versions.json'
APPS_TXT_FILE = 'apps.txt'
COMMON_SITE_CONFIG_FILE = 'common_site_config.json'
APP_VERSIONS_JSON_FILE = "app_versions.json"
APPS_TXT_FILE = "apps.txt"
COMMON_SITE_CONFIG_FILE = "common_site_config.json"
DATE_FORMAT = "%Y%m%d_%H%M%S"
RDS_DB = 'rds_db'
RDS_DB = "rds_db"
RDS_PRIVILEGES = "SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, CREATE VIEW, EVENT, TRIGGER, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EXECUTE, LOCK TABLES"
ARCHIVE_SITES_PATH = '/home/frappe/frappe-bench/sites/archive_sites'
ARCHIVE_SITES_PATH = "/home/frappe/frappe-bench/sites/archive_sites"

18
build/frappe-worker/commands/doctor.py
View File

@ -1,20 +1,20 @@
import argparse
from check_connection import (
check_service,
check_redis_cache,
check_redis_queue,
check_redis_socketio,
check_service,
)
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument(
'-p',
'--ping-service',
dest='ping_services',
action='append',
"-p",
"--ping-service",
dest="ping_services",
action="append",
type=str,
help='list of services to ping, e.g. doctor -p "postgres:5432" --ping-service "mariadb:3306"',
)
@ -33,15 +33,15 @@ def main():
check_redis_socketio(retry=1, delay=0, print_attempt=False)
print("Redis SocketIO Connected")
if(args.ping_services):
if args.ping_services:
for service in args.ping_services:
service_name = None
service_port = None
try:
service_name, service_port = service.split(':')
service_name, service_port = service.split(":")
except ValueError:
print('Service should be in format host:port, e.g postgres:5432')
print("Service should be in format host:port, e.g postgres:5432")
exit(1)
check_service(
@ -51,7 +51,7 @@ def main():
service_name=service_name,
service_port=service_port,
)
print("{0}:{1} Connected".format(service_name, service_port))
print(f"{service_name}:{service_port} Connected")
print("Health check successful")
exit(0)

1
build/frappe-worker/commands/gevent_patch.py
View File

@ -1,2 +1,3 @@
import gevent.monkey
gevent.monkey.patch_all()

5
build/frappe-worker/commands/migrate.py
View File

@ -1,6 +1,6 @@
import os
import frappe
import frappe
from frappe.utils import cint, get_sites
from utils import get_config, save_config
@ -27,11 +27,12 @@ def migrate_sites(maintenance_mode=False):
set_maintenance_mode(True)
for site in sites:
print('Migrating', site)
print("Migrating", site)
frappe.init(site=site)
frappe.connect()
try:
from frappe.migrate import migrate
migrate()
finally:
frappe.destroy()

74
build/frappe-worker/commands/new.py
View File

@ -1,15 +1,10 @@
import os
import frappe
import semantic_version
from frappe.installer import update_site_config
from constants import COMMON_SITE_CONFIG_FILE, RDS_DB, RDS_PRIVILEGES
from utils import (
run_command,
get_config,
get_site_config,
get_password,
)
from frappe.installer import update_site_config
from utils import get_config, get_password, get_site_config, run_command
# try to import _new_site from frappe, which could possibly
# exist in either commands.py or installer.py, and so we need
@ -24,33 +19,43 @@ except ImportError:
def main():
config = get_config()
db_type = 'mariadb'
db_port = config.get('db_port', 3306)
db_host = config.get('db_host')
site_name = os.environ.get("SITE_NAME", 'site1.localhost')
db_root_username = os.environ.get("DB_ROOT_USER", 'root')
mariadb_root_password = get_password("MYSQL_ROOT_PASSWORD", 'admin')
db_type = "mariadb"
db_port = config.get("db_port", 3306)
db_host = config.get("db_host")
site_name = os.environ.get("SITE_NAME", "site1.localhost")
db_root_username = os.environ.get("DB_ROOT_USER", "root")
mariadb_root_password = get_password("MYSQL_ROOT_PASSWORD", "admin")
postgres_root_password = get_password("POSTGRES_PASSWORD")
db_root_password = mariadb_root_password
if postgres_root_password:
db_type = 'postgres'
db_type = "postgres"
db_host = os.environ.get("POSTGRES_HOST")
db_port = 5432
db_root_password = postgres_root_password
if not db_host:
db_host = config.get('db_host')
print('Environment variable POSTGRES_HOST not found.')
print('Using db_host from common_site_config.json')
db_host = config.get("db_host")
print("Environment variable POSTGRES_HOST not found.")
print("Using db_host from common_site_config.json")
sites_path = os.getcwd()
common_site_config_path = os.path.join(sites_path, COMMON_SITE_CONFIG_FILE)
update_site_config("root_login", db_root_username, validate = False, site_config_path = common_site_config_path)
update_site_config("root_password", db_root_password, validate = False, site_config_path = common_site_config_path)
update_site_config(
"root_login",
db_root_username,
validate=False,
site_config_path=common_site_config_path,
)
update_site_config(
"root_password",
db_root_password,
validate=False,
site_config_path=common_site_config_path,
)
force = True if os.environ.get("FORCE", None) else False
install_apps = os.environ.get("INSTALL_APPS", None)
install_apps = install_apps.split(',') if install_apps else []
install_apps = install_apps.split(",") if install_apps else []
frappe.init(site_name, new_site=True)
if semantic_version.Version(frappe.__version__).major > 11:
@ -59,7 +64,7 @@ def main():
site_name,
mariadb_root_username=db_root_username,
mariadb_root_password=db_root_password,
admin_password=get_password("ADMIN_PASSWORD", 'admin'),
admin_password=get_password("ADMIN_PASSWORD", "admin"),
verbose=True,
install_apps=install_apps,
source_sql=None,
@ -75,7 +80,7 @@ def main():
site_name,
mariadb_root_username=db_root_username,
mariadb_root_password=db_root_password,
admin_password=get_password("ADMIN_PASSWORD", 'admin'),
admin_password=get_password("ADMIN_PASSWORD", "admin"),
verbose=True,
install_apps=install_apps,
source_sql=None,
@ -83,16 +88,23 @@ def main():
reinstall=False,
)
if db_type == "mariadb":
site_config = get_site_config(site_name)
db_name = site_config.get('db_name')
db_password = site_config.get('db_password')
db_name = site_config.get("db_name")
db_password = site_config.get("db_password")
mysql_command = ["mysql", f"-h{db_host}", f"-u{db_root_username}", f"-p{mariadb_root_password}", "-e"]
mysql_command = [
"mysql",
f"-h{db_host}",
f"-u{db_root_username}",
f"-p{mariadb_root_password}",
"-e",
]
# Drop User if exists
command = mysql_command + [f"DROP USER IF EXISTS '{db_name}'; FLUSH PRIVILEGES;"]
command = mysql_command + [
f"DROP USER IF EXISTS '{db_name}'; FLUSH PRIVILEGES;"
]
run_command(command)
# Grant permission to database and set password
@ -102,10 +114,12 @@ def main():
if config.get(RDS_DB) or site_config.get(RDS_DB):
grant_privileges = RDS_PRIVILEGES
command = mysql_command + [f"\
command = mysql_command + [
f"\
CREATE USER IF NOT EXISTS '{db_name}'@'%' IDENTIFIED BY '{db_password}'; \
GRANT {grant_privileges} ON `{db_name}`.* TO '{db_name}'@'%'; \
FLUSH PRIVILEGES;"]
FLUSH PRIVILEGES;"
]
run_command(command)
if frappe.redis_server:

109
build/frappe-worker/commands/push_backup.py
View File

@ -1,16 +1,12 @@
import datetime
import os
import time
import boto3
import datetime
from glob import glob
from frappe.utils import get_sites
import boto3
from constants import DATE_FORMAT
from utils import (
get_s3_config,
upload_file_to_s3,
check_s3_environment_variables,
)
from frappe.utils import get_sites
from utils import check_s3_environment_variables, get_s3_config, upload_file_to_s3
def get_file_ext():
@ -18,7 +14,7 @@ def get_file_ext():
"database": "-database.sql.gz",
"private_files": "-private-files.tar",
"public_files": "-files.tar",
"site_config": "-site_config_backup.json"
"site_config": "-site_config_backup.json",
}
@ -31,19 +27,26 @@ def get_backup_details(sitename):
if os.path.exists(site_backup_path):
for filetype, ext in file_ext.items():
site_slug = sitename.replace('.', '_')
pattern = site_backup_path + '*-' + site_slug + ext
site_slug = sitename.replace(".", "_")
pattern = site_backup_path + "*-" + site_slug + ext
backup_files = list(filter(os.path.isfile, glob(pattern)))
if len(backup_files) > 0:
backup_files.sort(key=lambda file: os.stat(os.path.join(site_backup_path, file)).st_ctime)
backup_date = datetime.datetime.strptime(time.ctime(os.path.getmtime(backup_files[0])), "%a %b %d %H:%M:%S %Y")
backup_files.sort(
key=lambda file: os.stat(
os.path.join(site_backup_path, file)
).st_ctime
)
backup_date = datetime.datetime.strptime(
time.ctime(os.path.getmtime(backup_files[0])),
"%a %b %d %H:%M:%S %Y",
)
backup_details[filetype] = {
"sitename": sitename,
"file_size_in_bytes": os.stat(backup_files[-1]).st_size,
"file_path": os.path.abspath(backup_files[-1]),
"filename": os.path.basename(backup_files[-1]),
"backup_date": backup_date.date().strftime("%Y-%m-%d %H:%M:%S")
"backup_date": backup_date.date().strftime("%Y-%m-%d %H:%M:%S"),
}
return backup_details
@ -54,31 +57,34 @@ def delete_old_backups(limit, bucket, site_name):
all_backup_dates = list()
backup_limit = int(limit)
check_s3_environment_variables()
bucket_dir = os.environ.get('BUCKET_DIR')
bucket_dir = os.environ.get("BUCKET_DIR")
oldest_backup_date = None
s3 = boto3.resource(
's3',
region_name=os.environ.get('REGION'),
aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
endpoint_url=os.environ.get('ENDPOINT_URL')
"s3",
region_name=os.environ.get("REGION"),
aws_access_key_id=os.environ.get("ACCESS_KEY_ID"),
aws_secret_access_key=os.environ.get("SECRET_ACCESS_KEY"),
endpoint_url=os.environ.get("ENDPOINT_URL"),
)
bucket = s3.Bucket(bucket)
objects = bucket.meta.client.list_objects_v2(
Bucket=bucket.name,
Delimiter='/')
objects = bucket.meta.client.list_objects_v2(Bucket=bucket.name, Delimiter="/")
if objects:
for obj in objects.get('CommonPrefixes'):
if obj.get('Prefix') == bucket_dir + '/':
for backup_obj in bucket.objects.filter(Prefix=obj.get('Prefix')):
for obj in objects.get("CommonPrefixes"):
if obj.get("Prefix") == bucket_dir + "/":
for backup_obj in bucket.objects.filter(Prefix=obj.get("Prefix")):
if backup_obj.get()["ContentType"] == "application/x-directory":
continue
try:
# backup_obj.key is bucket_dir/site/date_time/backupfile.extension
bucket_dir, site_slug, date_time, backupfile = backup_obj.key.split('/')
(
bucket_dir,
site_slug,
date_time,
backupfile,
) = backup_obj.key.split("/")
date_time_object = datetime.datetime.strptime(
date_time, DATE_FORMAT
)
@ -98,7 +104,7 @@ def delete_old_backups(limit, bucket, site_name):
for backup in all_backups:
try:
# backup is bucket_dir/site/date_time/backupfile.extension
backup_dir, site_slug, backup_dt_string, filename = backup.split('/')
backup_dir, site_slug, backup_dt_string, filename = backup.split("/")
backup_datetime = datetime.datetime.strptime(
backup_dt_string, DATE_FORMAT
)
@ -113,7 +119,7 @@ def delete_old_backups(limit, bucket, site_name):
for obj in bucket.objects.filter(Prefix=oldest_backup):
# delete all keys that are inside the oldest_backup
if bucket_dir in obj.key:
print('Deleteing ' + obj.key)
print("Deleting " + obj.key)
s3.Object(bucket.name, obj.key).delete()
@ -124,31 +130,52 @@ def main():
for site in sites:
details = get_backup_details(site)
db_file = details.get('database', {}).get('file_path')
folder = os.environ.get('BUCKET_DIR') + '/' + site + '/'
db_file = details.get("database", {}).get("file_path")
folder = os.environ.get("BUCKET_DIR") + "/" + site + "/"
if db_file:
folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(db_file)[:15] + '/'
folder = (
os.environ.get("BUCKET_DIR")
+ "/"
+ site
+ "/"
+ os.path.basename(db_file)[:15]
+ "/"
)
upload_file_to_s3(db_file, folder, conn, bucket)
# Archive site_config.json
site_config_file = details.get('site_config', {}).get('file_path')
site_config_file = details.get("site_config", {}).get("file_path")
if not site_config_file:
site_config_file = os.path.join(os.getcwd(), site, 'site_config.json')
site_config_file = os.path.join(os.getcwd(), site, "site_config.json")
upload_file_to_s3(site_config_file, folder, conn, bucket)
public_files = details.get('public_files', {}).get('file_path')
public_files = details.get("public_files", {}).get("file_path")
if public_files:
folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(public_files)[:15] + '/'
folder = (
os.environ.get("BUCKET_DIR")
+ "/"
+ site
+ "/"
+ os.path.basename(public_files)[:15]
+ "/"
)
upload_file_to_s3(public_files, folder, conn, bucket)
private_files = details.get('private_files', {}).get('file_path')
private_files = details.get("private_files", {}).get("file_path")
if private_files:
folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(private_files)[:15] + '/'
folder = (
os.environ.get("BUCKET_DIR")
+ "/"
+ site
+ "/"
+ os.path.basename(private_files)[:15]
+ "/"
)
upload_file_to_s3(private_files, folder, conn, bucket)
delete_old_backups(os.environ.get('BACKUP_LIMIT', '3'), bucket, site)
delete_old_backups(os.environ.get("BACKUP_LIMIT", "3"), bucket, site)
print('push-backup complete')
print("push-backup complete")
exit(0)

221
build/frappe-worker/commands/restore_backup.py
View File

@ -1,93 +1,88 @@
import os
import datetime
import tarfile
import hashlib
import frappe
import boto3
import os
import tarfile
from frappe.utils import get_sites, random_string
from frappe.installer import (
make_conf,
get_conf_params,
make_site_dirs,
update_site_config
)
import boto3
import frappe
from constants import COMMON_SITE_CONFIG_FILE, DATE_FORMAT, RDS_DB, RDS_PRIVILEGES
from frappe.installer import (
get_conf_params,
make_conf,
make_site_dirs,
update_site_config,
)
from frappe.utils import get_sites, random_string
from utils import (
run_command,
list_directories,
set_key_in_site_config,
get_site_config,
check_s3_environment_variables,
get_config,
get_password,
check_s3_environment_variables,
get_site_config,
list_directories,
run_command,
set_key_in_site_config,
)
def get_backup_dir():
return os.path.join(
os.path.expanduser('~'),
'backups'
)
return os.path.join(os.path.expanduser("~"), "backups")
def decompress_db(database_file, site):
command = ["gunzip", "-c", database_file]
with open(database_file.replace(".gz", ""), "w") as db_file:
print('Extract Database GZip for site {}'.format(site))
print(f"Extract Database GZip for site {site}")
run_command(command, stdout=db_file)
def restore_database(files_base, site_config_path, site):
# restore database
database_file = files_base + '-database.sql.gz'
database_file = files_base + "-database.sql.gz"
decompress_db(database_file, site)
config = get_config()
# Set db_type if it exists in backup site_config.json
set_key_in_site_config('db_type', site, site_config_path)
set_key_in_site_config("db_type", site, site_config_path)
# Set db_host if it exists in backup site_config.json
set_key_in_site_config('db_host', site, site_config_path)
set_key_in_site_config("db_host", site, site_config_path)
# Set db_port if it exists in backup site_config.json
set_key_in_site_config('db_port', site, site_config_path)
set_key_in_site_config("db_port", site, site_config_path)
# get updated site_config
site_config = get_site_config(site)
# if no db_type exists, default to mariadb
db_type = site_config.get('db_type', 'mariadb')
db_type = site_config.get("db_type", "mariadb")
is_database_restored = False
if db_type == 'mariadb':
if db_type == "mariadb":
restore_mariadb(
config=config,
site_config=site_config,
database_file=database_file)
config=config, site_config=site_config, database_file=database_file
)
is_database_restored = True
elif db_type == 'postgres':
elif db_type == "postgres":
restore_postgres(
config=config,
site_config=site_config,
database_file=database_file)
config=config, site_config=site_config, database_file=database_file
)
is_database_restored = True
if is_database_restored:
# Set encryption_key if it exists in backup site_config.json
set_key_in_site_config('encryption_key', site, site_config_path)
set_key_in_site_config("encryption_key", site, site_config_path)
def restore_files(files_base):
public_files = files_base + '-files.tar'
public_files = files_base + "-files.tar"
# extract tar
public_tar = tarfile.open(public_files)
print('Extracting {}'.format(public_files))
print(f"Extracting {public_files}")
public_tar.extractall()
def restore_private_files(files_base):
private_files = files_base + '-private-files.tar'
private_files = files_base + "-private-files.tar"
private_tar = tarfile.open(private_files)
print('Extracting {}'.format(private_files))
print(f"Extracting {private_files}")
private_tar.extractall()
@ -96,15 +91,15 @@ def pull_backup_from_s3():
# https://stackoverflow.com/a/54672690
s3 = boto3.resource(
's3',
region_name=os.environ.get('REGION'),
aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
endpoint_url=os.environ.get('ENDPOINT_URL')
"s3",
region_name=os.environ.get("REGION"),
aws_access_key_id=os.environ.get("ACCESS_KEY_ID"),
aws_secret_access_key=os.environ.get("SECRET_ACCESS_KEY"),
endpoint_url=os.environ.get("ENDPOINT_URL"),
)
bucket_dir = os.environ.get('BUCKET_DIR')
bucket_name = os.environ.get('BUCKET_NAME')
bucket_dir = os.environ.get("BUCKET_DIR")
bucket_name = os.environ.get("BUCKET_NAME")
bucket = s3.Bucket(bucket_name)
# Change directory to /home/frappe/backups
@ -118,10 +113,10 @@ def pull_backup_from_s3():
for obj in bucket.objects.filter(Prefix=bucket_dir):
if obj.get()["ContentType"] == "application/x-directory":
continue
backup_file = obj.key.replace(os.path.join(bucket_dir, ''), '')
backup_file = obj.key.replace(os.path.join(bucket_dir, ""), "")
backup_files.append(backup_file)
site_name, timestamp, backup_type = backup_file.split('/')
site_timestamp = site_name + '/' + timestamp
site_name, timestamp, backup_type = backup_file.split("/")
site_timestamp = site_name + "/" + timestamp
sites.add(site_name)
site_timestamps.add(site_timestamp)
@ -129,13 +124,11 @@ def pull_backup_from_s3():
for site in sites:
backup_timestamps = []
for site_timestamp in site_timestamps:
site_name, timestamp = site_timestamp.split('/')
site_name, timestamp = site_timestamp.split("/")
if site == site_name:
timestamp_datetime = datetime.datetime.strptime(
timestamp, DATE_FORMAT
)
timestamp_datetime = datetime.datetime.strptime(timestamp, DATE_FORMAT)
backup_timestamps.append(timestamp)
download_backups.append(site + '/' + max(backup_timestamps))
download_backups.append(site + "/" + max(backup_timestamps))
# Only download latest backups
for backup_file in backup_files:
@ -143,21 +136,21 @@ def pull_backup_from_s3():
if backup in backup_file:
if not os.path.exists(os.path.dirname(backup_file)):
os.makedirs(os.path.dirname(backup_file))
print('Downloading {}'.format(backup_file))
bucket.download_file(bucket_dir + '/' + backup_file, backup_file)
print(f"Downloading {backup_file}")
bucket.download_file(bucket_dir + "/" + backup_file, backup_file)
os.chdir(os.path.join(os.path.expanduser('~'), 'frappe-bench', 'sites'))
os.chdir(os.path.join(os.path.expanduser("~"), "frappe-bench", "sites"))
def restore_postgres(config, site_config, database_file):
# common config
common_site_config_path = os.path.join(os.getcwd(), COMMON_SITE_CONFIG_FILE)
db_root_user = config.get('root_login')
db_root_user = config.get("root_login")
if not db_root_user:
postgres_user = os.environ.get('DB_ROOT_USER')
postgres_user = os.environ.get("DB_ROOT_USER")
if not postgres_user:
print('Variable DB_ROOT_USER not set')
print("Variable DB_ROOT_USER not set")
exit(1)
db_root_user = postgres_user
@ -165,13 +158,14 @@ def restore_postgres(config, site_config, database_file):
"root_login",
db_root_user,
validate=False,
site_config_path=common_site_config_path)
site_config_path=common_site_config_path,
)
db_root_password = config.get('root_password')
db_root_password = config.get("root_password")
if not db_root_password:
root_password = get_password('POSTGRES_PASSWORD')
root_password = get_password("POSTGRES_PASSWORD")
if not root_password:
print('Variable POSTGRES_PASSWORD not set')
print("Variable POSTGRES_PASSWORD not set")
exit(1)
db_root_password = root_password
@ -179,53 +173,72 @@ def restore_postgres(config, site_config, database_file):
"root_password",
db_root_password,
validate=False,
site_config_path=common_site_config_path)
site_config_path=common_site_config_path,
)
# site config
db_host = site_config.get('db_host')
db_port = site_config.get('db_port', 5432)
db_name = site_config.get('db_name')
db_password = site_config.get('db_password')
db_host = site_config.get("db_host")
db_port = site_config.get("db_port", 5432)
db_name = site_config.get("db_name")
db_password = site_config.get("db_password")
psql_command = ["psql"]
psql_uri = f"postgres://{db_root_user}:{db_root_password}@{db_host}:{db_port}"
print('Restoring PostgreSQL')
run_command(psql_command + [psql_uri, "-c", f"DROP DATABASE IF EXISTS \"{db_name}\""])
print("Restoring PostgreSQL")
run_command(psql_command + [psql_uri, "-c", f'DROP DATABASE IF EXISTS "{db_name}"'])
run_command(psql_command + [psql_uri, "-c", f"DROP USER IF EXISTS {db_name}"])
run_command(psql_command + [psql_uri, "-c", f"CREATE DATABASE \"{db_name}\""])
run_command(psql_command + [psql_uri, "-c", f"CREATE user {db_name} password '{db_password}'"])
run_command(psql_command + [psql_uri, "-c", f"GRANT ALL PRIVILEGES ON DATABASE \"{db_name}\" TO {db_name}"])
with open(database_file.replace('.gz', ''), 'r') as db_file:
run_command(psql_command + [psql_uri, "-c", f'CREATE DATABASE "{db_name}"'])
run_command(
psql_command
+ [psql_uri, "-c", f"CREATE user {db_name} password '{db_password}'"]
)
run_command(
psql_command
+ [psql_uri, "-c", f'GRANT ALL PRIVILEGES ON DATABASE "{db_name}" TO {db_name}']
)
with open(database_file.replace(".gz", "")) as db_file:
run_command(psql_command + [f"{psql_uri}/{db_name}", "<"], stdin=db_file)
def restore_mariadb(config, site_config, database_file):
db_root_password = get_password('MYSQL_ROOT_PASSWORD')
db_root_password = get_password("MYSQL_ROOT_PASSWORD")
if not db_root_password:
print('Variable MYSQL_ROOT_PASSWORD not set')
print("Variable MYSQL_ROOT_PASSWORD not set")
exit(1)
db_root_user = os.environ.get("DB_ROOT_USER", 'root')
db_root_user = os.environ.get("DB_ROOT_USER", "root")
db_host = site_config.get('db_host', config.get('db_host'))
db_port = site_config.get('db_port', config.get('db_port', 3306))
db_name = site_config.get('db_name')
db_password = site_config.get('db_password')
db_host = site_config.get("db_host", config.get("db_host"))
db_port = site_config.get("db_port", config.get("db_port", 3306))
db_name = site_config.get("db_name")
db_password = site_config.get("db_password")
# mysql command prefix
mysql_command = ["mysql", f"-u{db_root_user}", f"-h{db_host}", f"-p{db_root_password}", f"-P{db_port}"]
mysql_command = [
"mysql",
f"-u{db_root_user}",
f"-h{db_host}",
f"-p{db_root_password}",
f"-P{db_port}",
]
# drop db if exists for clean restore
drop_database = mysql_command + ["-e", f"DROP DATABASE IF EXISTS `{db_name}`;"]
run_command(drop_database)
# create db
create_database = mysql_command + ["-e", f"CREATE DATABASE IF NOT EXISTS `{db_name}`;"]
create_database = mysql_command + [
"-e",
f"CREATE DATABASE IF NOT EXISTS `{db_name}`;",
]
run_command(create_database)
# create user
create_user = mysql_command + ["-e", f"CREATE USER IF NOT EXISTS '{db_name}'@'%' IDENTIFIED BY '{db_password}'; FLUSH PRIVILEGES;"]
create_user = mysql_command + [
"-e",
f"CREATE USER IF NOT EXISTS '{db_name}'@'%' IDENTIFIED BY '{db_password}'; FLUSH PRIVILEGES;",
]
run_command(create_user)
# grant db privileges to user
@ -236,11 +249,14 @@ def restore_mariadb(config, site_config, database_file):
if config.get(RDS_DB) or site_config.get(RDS_DB):
grant_privileges = RDS_PRIVILEGES
grant_privileges_command = mysql_command + ["-e", f"GRANT {grant_privileges} ON `{db_name}`.* TO '{db_name}'@'%' IDENTIFIED BY '{db_password}'; FLUSH PRIVILEGES;"]
grant_privileges_command = mysql_command + [
"-e",
f"GRANT {grant_privileges} ON `{db_name}`.* TO '{db_name}'@'%' IDENTIFIED BY '{db_password}'; FLUSH PRIVILEGES;",
]
run_command(grant_privileges_command)
print('Restoring MariaDB')
with open(database_file.replace('.gz', ''), 'r') as db_file:
print("Restoring MariaDB")
with open(database_file.replace(".gz", "")) as db_file:
run_command(mysql_command + [f"{db_name}"], stdin=db_file)
@ -251,35 +267,38 @@ def main():
pull_backup_from_s3()
for site in list_directories(backup_dir):
site_slug = site.replace('.', '_')
backups = [datetime.datetime.strptime(backup, DATE_FORMAT) for backup in list_directories(os.path.join(backup_dir, site))]
site_slug = site.replace(".", "_")
backups = [
datetime.datetime.strptime(backup, DATE_FORMAT)
for backup in list_directories(os.path.join(backup_dir, site))
]
latest_backup = max(backups).strftime(DATE_FORMAT)
files_base = os.path.join(backup_dir, site, latest_backup, '')
files_base += latest_backup + '-' + site_slug
site_config_path = files_base + '-site_config_backup.json'
files_base = os.path.join(backup_dir, site, latest_backup, "")
files_base += latest_backup + "-" + site_slug
site_config_path = files_base + "-site_config_backup.json"
if not os.path.exists(site_config_path):
site_config_path = os.path.join(backup_dir, site, 'site_config.json')
site_config_path = os.path.join(backup_dir, site, "site_config.json")
if site in get_sites():
print('Overwrite site {}'.format(site))
print(f"Overwrite site {site}")
restore_database(files_base, site_config_path, site)
restore_private_files(files_base)
restore_files(files_base)
else:
site_config = get_conf_params(
db_name='_' + hashlib.sha1(site.encode()).hexdigest()[:16],
db_password=random_string(16)
db_name="_" + hashlib.sha1(site.encode()).hexdigest()[:16],
db_password=random_string(16),
)
frappe.local.site = site
frappe.local.sites_path = os.getcwd()
frappe.local.site_path = os.getcwd() + '/' + site
frappe.local.site_path = os.getcwd() + "/" + site
make_conf(
db_name=site_config.get('db_name'),
db_password=site_config.get('db_password'),
db_name=site_config.get("db_name"),
db_password=site_config.get("db_password"),
)
make_site_dirs()
print('Create site {}'.format(site))
print(f"Create site {site}")
restore_database(files_base, site_config_path, site)
restore_private_files(files_base)
restore_files(files_base)

72
build/frappe-worker/commands/utils.py
View File

@ -1,15 +1,12 @@
import json
import os
import subprocess
import boto3
import git
from constants import APP_VERSIONS_JSON_FILE, APPS_TXT_FILE, COMMON_SITE_CONFIG_FILE
from frappe.installer import update_site_config
from constants import (
APP_VERSIONS_JSON_FILE,
APPS_TXT_FILE,
COMMON_SITE_CONFIG_FILE
)
def run_command(command, stdout=None, stdin=None, stderr=None):
stdout = stdout or subprocess.PIPE
@ -26,7 +23,7 @@ def run_command(command, stdout=None, stdin=None, stderr=None):
def save_version_file(versions):
with open(APP_VERSIONS_JSON_FILE, 'w') as f:
with open(APP_VERSIONS_JSON_FILE, "w") as f:
return json.dump(versions, f, indent=1, sort_keys=True)
@ -58,10 +55,10 @@ def get_container_versions(apps):
pass
try:
path = os.path.join('..', 'apps', app)
path = os.path.join("..", "apps", app)
repo = git.Repo(path)
commit_hash = repo.head.object.hexsha
versions.update({app+'_git_hash': commit_hash})
versions.update({app + "_git_hash": commit_hash})
except Exception:
pass
@ -94,18 +91,22 @@ def get_config():
def get_site_config(site_name):
site_config = None
with open('{site_name}/site_config.json'.format(site_name=site_name)) as site_config_file:
with open(f"{site_name}/site_config.json") as site_config_file:
site_config = json.load(site_config_file)
return site_config
def save_config(config):
with open(COMMON_SITE_CONFIG_FILE, 'w') as f:
with open(COMMON_SITE_CONFIG_FILE, "w") as f:
return json.dump(config, f, indent=1, sort_keys=True)
def get_password(env_var, default=None):
return os.environ.get(env_var) or get_password_from_secret(f"{env_var}_FILE") or default
return (
os.environ.get(env_var)
or get_password_from_secret(f"{env_var}_FILE")
or default
)
def get_password_from_secret(env_var):
@ -128,14 +129,14 @@ def get_password_from_secret(env_var):
def get_s3_config():
check_s3_environment_variables()
bucket = os.environ.get('BUCKET_NAME')
bucket = os.environ.get("BUCKET_NAME")
conn = boto3.client(
's3',
region_name=os.environ.get('REGION'),
aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
endpoint_url=os.environ.get('ENDPOINT_URL')
"s3",
region_name=os.environ.get("REGION"),
aws_access_key_id=os.environ.get("ACCESS_KEY_ID"),
aws_secret_access_key=os.environ.get("SECRET_ACCESS_KEY"),
endpoint_url=os.environ.get("ENDPOINT_URL"),
)
return conn, bucket
@ -164,7 +165,7 @@ def list_directories(path):
def get_site_config_from_path(site_config_path):
site_config = dict()
if os.path.exists(site_config_path):
with open(site_config_path, 'r') as sc:
with open(site_config_path) as sc:
site_config = json.load(sc)
return site_config
@ -173,32 +174,35 @@ def set_key_in_site_config(key, site, site_config_path):
site_config = get_site_config_from_path(site_config_path)
value = site_config.get(key)
if value:
print('Set {key} in site config for site: {site}'.format(key=key, site=site))
update_site_config(key, value,
site_config_path=os.path.join(os.getcwd(), site, "site_config.json"))
print(f"Set {key} in site config for site: {site}")
update_site_config(
key,
value,
site_config_path=os.path.join(os.getcwd(), site, "site_config.json"),
)
def check_s3_environment_variables():
if 'BUCKET_NAME' not in os.environ:
print('Variable BUCKET_NAME not set')
if "BUCKET_NAME" not in os.environ:
print("Variable BUCKET_NAME not set")
exit(1)
if 'ACCESS_KEY_ID' not in os.environ:
print('Variable ACCESS_KEY_ID not set')
if "ACCESS_KEY_ID" not in os.environ:
print("Variable ACCESS_KEY_ID not set")
exit(1)
if 'SECRET_ACCESS_KEY' not in os.environ:
print('Variable SECRET_ACCESS_KEY not set')
if "SECRET_ACCESS_KEY" not in os.environ:
print("Variable SECRET_ACCESS_KEY not set")
exit(1)
if 'ENDPOINT_URL' not in os.environ:
print('Variable ENDPOINT_URL not set')
if "ENDPOINT_URL" not in os.environ:
print("Variable ENDPOINT_URL not set")
exit(1)
if 'BUCKET_DIR' not in os.environ:
print('Variable BUCKET_DIR not set')
if "BUCKET_DIR" not in os.environ:
print("Variable BUCKET_DIR not set")
exit(1)
if 'REGION' not in os.environ:
print('Variable REGION not set')
if "REGION" not in os.environ:
print("Variable REGION not set")
exit(1)

6
development/README.md
View File

@ -94,6 +94,7 @@ code Procfile
```
Or running the following command:
```shell
sed -i '/redis/d' ./Procfile
```
@ -105,6 +106,7 @@ You can create a new site with the following command:
```shell
bench new-site sitename --no-mariadb-socket
```
sitename MUST end with .localhost for trying deployments locally.
for example:
@ -234,7 +236,7 @@ The first step is installing and updating the required software. Usually the fra
/workspace/development/frappe-bench/env/bin/python -m pip install --upgrade jupyter ipykernel ipython
```
Then, run the commmand `Python: Show Python interactive window` from the VSCode command palette.
Then, run the command `Python: Show Python interactive window` from the VSCode command palette.
Replace `mysite.localhost` with your site and run the following code in a Jupyter cell:
@ -259,7 +261,6 @@ Example shows the queries to be executed for site `localhost`
Open sites/localhost/site_config.json:
```shell
code sites/localhost/site_config.json
```
@ -286,6 +287,7 @@ EXIT;
In case you don't use VSCode, you may start the containers manually with the following command:
### Running the containers
```shell
docker-compose -f .devcontainer/docker-compose.yml up -d
```

27
development/vscode-example/launch.json
View File

@ -10,7 +10,12 @@
"request": "launch",
"program": "${workspaceFolder}/frappe-bench/apps/frappe/frappe/utils/bench_helper.py",
"args": [
"frappe", "serve", "--port", "8000", "--noreload", "--nothreading"
"frappe",
"serve",
"--port",
"8000",
"--noreload",
"--nothreading"
],
"pythonPath": "${workspaceFolder}/frappe-bench/env/bin/python",
"cwd": "${workspaceFolder}/frappe-bench/sites",
@ -23,9 +28,7 @@
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/frappe-bench/apps/frappe/frappe/utils/bench_helper.py",
"args": [
"frappe", "worker", "--queue", "default"
],
"args": ["frappe", "worker", "--queue", "default"],
"pythonPath": "${workspaceFolder}/frappe-bench/env/bin/python",
"cwd": "${workspaceFolder}/frappe-bench/sites",
"env": {
@ -37,9 +40,7 @@
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/frappe-bench/apps/frappe/frappe/utils/bench_helper.py",
"args": [
"frappe", "worker", "--queue", "short"
],
"args": ["frappe", "worker", "--queue", "short"],
"pythonPath": "${workspaceFolder}/frappe-bench/env/bin/python",
"cwd": "${workspaceFolder}/frappe-bench/sites",
"env": {
@ -51,9 +52,7 @@
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/frappe-bench/apps/frappe/frappe/utils/bench_helper.py",
"args": [
"frappe", "worker", "--queue", "long"
],
"args": ["frappe", "worker", "--queue", "long"],
"pythonPath": "${workspaceFolder}/frappe-bench/env/bin/python",
"cwd": "${workspaceFolder}/frappe-bench/sites",
"env": {
@ -69,7 +68,13 @@
"cwd": "${workspaceFolder}/frappe-bench",
"console": "internalConsole",
"args": [
"start", "socketio", "watch", "schedule", "worker_short", "worker_long", "worker_default"
"start",
"socketio",
"watch",
"schedule",
"worker_short",
"worker_long",
"worker_default"
]
}
]

10
docs/docker-swarm.md
View File

@ -42,7 +42,7 @@ version: "3.7"
services:
mariadb-master:
image: 'bitnami/mariadb:10.3'
image: "bitnami/mariadb:10.3"
deploy:
restart_policy:
condition: on-failure
@ -54,7 +54,7 @@ services:
secrets:
- frappe-mariadb-root-password
volumes:
- 'mariadb_master_data:/bitnami/mariadb'
- "mariadb_master_data:/bitnami/mariadb"
environment:
- MARIADB_REPLICATION_MODE=master
- MARIADB_REPLICATION_USER=repl_user
@ -62,7 +62,7 @@ services:
- MARIADB_ROOT_PASSWORD_FILE=/run/secrets/frappe-mariadb-root-password
mariadb-slave:
image: 'bitnami/mariadb:10.3'
image: "bitnami/mariadb:10.3"
deploy:
restart_policy:
condition: on-failure
@ -74,7 +74,7 @@ services:
secrets:
- frappe-mariadb-root-password
volumes:
- 'mariadb_slave_data:/bitnami/mariadb'
- "mariadb_slave_data:/bitnami/mariadb"
environment:
- MARIADB_REPLICATION_MODE=slave
- MARIADB_REPLICATION_USER=repl_user
@ -265,6 +265,7 @@ Use environment variables:
- `FRAPPE_VERSION` variable to be set to desired version of Frappe Framework. e.g. 12.7.0
- `MARIADB_HOST=frappe-mariadb_mariadb-master`
- `SITES` variable is list of sites in back tick and separated by comma
```
SITES=`site1.example.com`,`site2.example.com`
```
@ -292,4 +293,3 @@ SITES=`site1.example.com`,`site2.example.com`
6. Env variables:
- MAINTENANCE_MODE=1
7. Start container

75
docs/multi-bench.md
View File

@ -123,46 +123,49 @@ Notes:
## Docker containers
This repository contains the following docker-compose files, each one containing the described images:
* docker-compose-common.yml
* redis-cache
* volume: redis-cache-vol
* redis-queue
* volume: redis-queue-vol
* redis-socketio
* volume: redis-socketio-vol
* mariadb: main database
* volume: mariadb-vol
* docker-compose-erpnext.yml
* erpnext-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
* volume: assets-vol
* erpnext-python: main application code
* frappe-socketio: enables realtime communication to the user interface through websockets
* frappe-worker-default: background runner
* frappe-worker-short: background runner for short-running jobs
* frappe-worker-long: background runner for long-running jobs
* frappe-schedule
* docker-compose-frappe.yml
* frappe-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
* volume: assets-vol, sites-vol
* erpnext-python: main application code
* volume: sites-vol
* frappe-socketio: enables realtime communication to the user interface through websockets
* volume: sites-vol
* frappe-worker-default: background runner
* volume: sites-vol
* frappe-worker-short: background runner for short-running jobs
* volume: sites-vol
* frappe-worker-long: background runner for long-running jobs
* volume: sites-vol
* frappe-schedule
* volume: sites-vol
- docker-compose-common.yml
- redis-cache
- volume: redis-cache-vol
- redis-queue
- volume: redis-queue-vol
- redis-socketio
- volume: redis-socketio-vol
- mariadb: main database
- volume: mariadb-vol
- docker-compose-erpnext.yml
* docker-compose-networks.yml: this yaml define the network to communicate with *Letsencrypt Nginx Proxy Companion*.
- erpnext-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
- volume: assets-vol
- erpnext-python: main application code
- frappe-socketio: enables realtime communication to the user interface through websockets
- frappe-worker-default: background runner
- frappe-worker-short: background runner for short-running jobs
- frappe-worker-long: background runner for long-running jobs
- frappe-schedule
* erpnext-publish.yml: this yml extends erpnext-nginx service to publish port 80, can only be used with docker-compose-erpnext.yml
- docker-compose-frappe.yml
* frappe-publish.yml: this yml extends frappe-nginx service to publish port 80, can only be used with docker-compose-frappe.yml
- frappe-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
- volume: assets-vol, sites-vol
- erpnext-python: main application code
- volume: sites-vol
- frappe-socketio: enables realtime communication to the user interface through websockets
- volume: sites-vol
- frappe-worker-default: background runner
- volume: sites-vol
- frappe-worker-short: background runner for short-running jobs
- volume: sites-vol
- frappe-worker-long: background runner for long-running jobs
- volume: sites-vol
- frappe-schedule
- volume: sites-vol
- docker-compose-networks.yml: this yaml define the network to communicate with _Letsencrypt Nginx Proxy Companion_.
- erpnext-publish.yml: this yml extends erpnext-nginx service to publish port 80, can only be used with docker-compose-erpnext.yml
- frappe-publish.yml: this yml extends frappe-nginx service to publish port 80, can only be used with docker-compose-frappe.yml
## Updating and Migrating Sites

56
docs/single-bench.md
View File

@ -37,7 +37,7 @@ To get started, copy the existing `env-local` or `env-production` file to `.env`
- `SITE_NAME=erp.example.com`
- Creates this site after starting all services and installs ERPNext. Site name must be resolvable by users machines and the ERPNext components. e.g. `erp.example.com` or `mysite.localhost`.
- `` SITES=`erp.example.com` ``
- List of sites that are part of the deployment "bench" Each site is separated by a comma(,) and quoted in backtick (`). By default site created by ``SITE_NAME`` variable is added here.
- List of sites that are part of the deployment "bench" Each site is separated by a comma(,) and quoted in backtick (`). By default site created by `SITE_NAME` variable is added here.
- If LetsEncrypt is being setup, make sure that the DNS for all the site's domains correctly point to the current instance.
- `DB_ROOT_USER=root`
- MariaDB root username
@ -77,7 +77,7 @@ Make sure to replace `<project-name>` with the desired name you wish to set for
Notes:
- If it is the first time running and site is being initialized, *it can take multiple minutes for the site to be up*. Monitor `site-creator` container logs to check progress. Use command `docker logs <project-name>_site-creator_1 -f`
- If it is the first time running and site is being initialized, _it can take multiple minutes for the site to be up_. Monitor `site-creator` container logs to check progress. Use command `docker logs <project-name>_site-creator_1 -f`
- After the site is ready the username is `Administrator` and the password is `$ADMIN_PASSWORD`
- The local deployment is for testing and REST API development purpose only
- A complete development environment is available [here](../development)
@ -86,32 +86,32 @@ Notes:
The docker-compose file contains following services:
* traefik: manages letsencrypt
* volume: cert-vol
* redis-cache: cache store
* volume: redis-cache-vol
* redis-queue: used by workers
* volume: redis-queue-vol
* redis-socketio: used by socketio service
* volume: redis-socketio-vol
* mariadb: main database
* volume: mariadb-vol
* erpnext-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
* volume: assets-vol and sites-vol
* erpnext-python: main application code
* volume: sites-vol and sites-vol
* frappe-socketio: enables realtime communication to the user interface through websockets
* volume: sites-vol
* erpnext-worker-default: background runner
* volume: sites-vol
* erpnext-worker-short: background runner for short-running jobs
* volume: sites-vol
* erpnext-worker-long: background runner for long-running jobs
* volume: sites-vol
* erpnext-schedule
* volume: sites-vol
* site-creator: run once container to create new site.
* volume: sites-vol
- traefik: manages letsencrypt
- volume: cert-vol
- redis-cache: cache store
- volume: redis-cache-vol
- redis-queue: used by workers
- volume: redis-queue-vol
- redis-socketio: used by socketio service
- volume: redis-socketio-vol
- mariadb: main database
- volume: mariadb-vol
- erpnext-nginx: serves static assets and proxies web request to the appropriate container, allowing to offer all services on the same port.
- volume: assets-vol and sites-vol
- erpnext-python: main application code
- volume: sites-vol and sites-vol
- frappe-socketio: enables realtime communication to the user interface through websockets
- volume: sites-vol
- erpnext-worker-default: background runner
- volume: sites-vol
- erpnext-worker-short: background runner for short-running jobs
- volume: sites-vol
- erpnext-worker-long: background runner for long-running jobs
- volume: sites-vol
- erpnext-schedule
- volume: sites-vol
- site-creator: run once container to create new site.
- volume: sites-vol
## Updating and Migrating Sites

7
docs/site-operations.md
View File

@ -168,16 +168,16 @@ Note:
- /home/frappe/backups
- site1.domain.com
- 20200420_162000
- 20200420_162000-site1_domain_com-*
- 20200420_162000-site1_domain_com-\*
- site2.domain.com
- 20200420_162000
- 20200420_162000-site2_domain_com-*
- 20200420_162000-site2_domain_com-\*
## Edit configs
Editing config manually might be required in some cases,
one such case is to use Amazon RDS (or any other DBaaS).
For full instructions, refer to the [wiki](https://github.com/frappe/frappe/wiki/Using-Frappe-with-Amazon-RDS-(or-any-other-DBaaS)). Common question can be found in Issues and on forum.
For full instructions, refer to the [wiki](<https://github.com/frappe/frappe/wiki/Using-Frappe-with-Amazon-RDS-(or-any-other-DBaaS)>). Common question can be found in Issues and on forum.
`common_site_config.json` or `site_config.json` from `sites-vol` volume has to be edited using following command:
@ -231,7 +231,6 @@ Notes:
- Use it to install/uninstall custom apps, add system manager user, etc.
- To run the command as non root user add the command option `--user frappe`.
## Delete/Drop Site
#### MariaDB Site

2
installation/docker-compose-common.yml
View File

@ -1,4 +1,4 @@
version: '3'
version: "3"
services:
redis-cache:

2
installation/docker-compose-erpnext.yml
View File

@ -1,4 +1,4 @@
version: '3'
version: "3"
services:
erpnext-nginx:

2
installation/docker-compose-frappe.yml
View File

@ -1,4 +1,4 @@
version: '3'
version: "3"
services:
frappe-nginx:

2
installation/docker-compose-networks.yml
View File

@ -1,4 +1,4 @@
version: '3'
version: "3"
networks:
default: