Within one day after changing the docker image used in production, the ERPNext client started reporting 404 errors when trying to load certain critical JS bundles. This was because the files they were asking for didn't actually exist. After a lot of digging, I discovered that the files being requested were actually the bundles from the previous docker image.
Every time a docker build is done, a new frappe-bench/sites/assets folder is created. The common cache-busting strategy of appending a value to the end of the bundle was used (e.g. bundle.IQTBPBLP.js). An assets.json file is created in the assets folder that maps the normal name, e.g. desk.bundle.css, to the full path to that file, /assets/frappe/dist/css/desk.bundle.RYMMWJSN.css. Inside frappe-bench/sites/assets, besides assets.json, are mostly just symlinks to the actual files in the frappe-bench/apps folder.
The frappe-bench/sites folder is mounted in docker as a volume, because frappe-bench/sites contains instance-specific information like the host name and the DB creds that need to be persisted. This is awkward, since the frappe-bench/sites/assets should not be persisted and should always come from the current docker build. If you do persist it, you create the opportunity for the assets.json file to "point" to files that don't exist, thus leading to the 404 errors.
Frappe's official docker image and compose file solved this issue by declaring both frappe-bench/sites and frappe-bench/sites/assets as volumes in the Dockerfile, but only connecting frappe-bench/sites to a named volume in the compose file. This means that docker would always create an anonymous volume for frappe-bench/sites/assets. Anonymous volumes are created fresh every time the compose file is brought up. When a volume is created, docker gets it's initial content from the base image. So, when the frappe-bench/sites named volume was first created, it got the assets folder from the original image, and when the frappe-bench/sites/assets is created each time, it picks up the latest assets from the base image and mounts them. In this way, most of the frappe-bench/sites is persisted, but frappe-bench/sites/assets always shows the files from the underlying image instead of the parent volume. A clever hack that takes advantage of how docker anonymous volumes operate. However, if that anonymous volume is unmounted for whatever reason, or ends up pulling it's initial data from the parent volume instead of the base image, the frappe-bench/sites/assets/assets.json will revert to being the first version deployed on this server, and cause the 404 error. Unfortunately, I was not able to determine exactly what went wrong, only that it was related to this anonymous volume.
My safer solution is to move frappe-bench/sites/assets to frappe-bench/assets during the image build and remove the frappe-bench/sites/assets volume from the Dockerfile. Then, in the configurator scripts, I remove anything currently at frappe-bench/sites/assets inside the volume and replace it with a symlink to frappe-bench/assets. In this way, the assets folder will always be in sync with the actual assets, I don't have to worry about docker creating 7 new anonymous volumes every time I update the image, and everything still works. You can see these changes in 5601d97782.