Add docker-compose certbot configurations with backward compatibility (#6702)

Co-authored-by: Your Name <you@example.com>

.gitignore

@@ -155,6 +155,7 @@ docker-legacy/volumes/milvus/*
 docker-legacy/volumes/chroma/*
 
 docker/volumes/app/storage/*
+docker/volumes/certbot/*
 docker/volumes/db/data/*
 docker/volumes/redis/data/*
 docker/volumes/weaviate/*

docker/.env.example

@@ -601,6 +601,22 @@ NGINX_KEEPALIVE_TIMEOUT=65
 NGINX_PROXY_READ_TIMEOUT=3600s
 NGINX_PROXY_SEND_TIMEOUT=3600s
 
+NGINX_ENABLE_CERTBOT_CHALLENGE=false
+
+# ------------------------------
+# Certbot Configuration
+# ------------------------------
+
+# Email address (required to get certificates from Let's Encrypt)
+CERTBOT_EMAIL=your_email@example.com
+
+# Domain name
+CERTBOT_DOMAIN=your_domain.com
+
+# certbot command options
+# i.e: --force-renewal --dry-run --test-cert --debug
+CERTBOT_OPTIONS=
+
 # ------------------------------
 # Environment Variables for SSRF Proxy
 # ------------------------------

docker/README.md

@@ -3,86 +3,105 @@
 Welcome to the new `docker` directory for deploying Dify using Docker Compose. This README outlines the updates, deployment instructions, and migration details for existing users.
 
 ### What's Updated
-- **Persistent Environment Variables**: Environment variables are now managed through a `.env` file, ensuring that your configurations persist across deployments.
 
-   > What is `.env`? </br> </br>
+- **Certbot Container**: `docker-compose.yaml` now contains `certbot` for managing SSL certificates. This container automatically renews certificates and ensures secure HTTPS connections.  
-   > The `.env` file is a crucial component in Docker and Docker Compose environments, serving as a centralized configuration file where you can define environment variables that are accessible to the containers at runtime. This file simplifies the management of environment settings across different stages of development, testing, and production, providing consistency and ease of configuration to deployments.
+  For more information, refer `docker/certbot/README.md`.
 
-- **Unified Vector Database Services**: All vector database services are now managed from a single Docker Compose file `docker-compose.yaml`. You can switch between different vector databases by setting the `VECTOR_STORE` environment variable in your `.env` file.
+- **Persistent Environment Variables
-- **Mandatory .env File**: A `.env` file is now required to run `docker compose up`. This file is crucial for configuring your deployment and for any custom settings to persist through upgrades.
+  **: Environment variables are now managed through a `.env` file, ensuring that your configurations persist across deployments.
-- **Legacy Support**: Previous deployment files are now located in the `docker-legacy` directory and will no longer be maintained.
+
+  > What is `.env`? </br> </br>
+  > The `.env` file is a crucial component in Docker and Docker Compose environments, serving as a centralized configuration file where you can define environment variables that are accessible to the containers at runtime. This file simplifies the management of environment settings across different stages of development, testing, and production, providing consistency and ease of configuration to deployments.
+
+- **Unified Vector Database Services
+  **: All vector database services are now managed from a single Docker Compose file `docker-compose.yaml`. You can switch between different vector databases by setting the `VECTOR_STORE` environment variable in your `.env` file.
+- **Mandatory .env File
+  **: A `.env` file is now required to run `docker compose up`. This file is crucial for configuring your deployment and for any custom settings to persist through upgrades.
+- **Legacy Support
+  **: Previous deployment files are now located in the `docker-legacy` directory and will no longer be maintained.
 
 ### How to Deploy Dify with `docker-compose.yaml`
+
 1. **Prerequisites**: Ensure Docker and Docker Compose are installed on your system.
 2. **Environment Setup**:
-   - Navigate to the `docker` directory.
+    - Navigate to the `docker` directory.
-   - Copy the `.env.example` file to a new file named `.env` by running `cp .env.example .env`.
+    - Copy the `.env.example` file to a new file named `.env` by running `cp .env.example .env`.
-   - Customize the `.env` file as needed. Refer to the `.env.example` file for detailed configuration options.
+    - Customize the `.env` file as needed. Refer to the `.env.example` file for detailed configuration options.
 3. **Running the Services**:
-   - Execute `docker compose up` from the `docker` directory to start the services.
+    - Execute `docker compose up` from the `docker` directory to start the services.
-   - To specify a vector database, set the `VECTOR_store` variable in your `.env` file to your desired vector database service, such as `milvus`, `weaviate`, or `opensearch`.
+    - To specify a vector database, set the `VECTOR_store` variable in your `.env` file to your desired vector database service, such as `milvus`, `weaviate`, or `opensearch`.
+4. **SSL Certificate Setup**:
+    - Rrefer `docker/certbot/README.md` to set up SSL certificates using Certbot.
 
 ### How to Deploy Middleware for Developing Dify
+
 1. **Middleware Setup**:
-   - Use the `docker-compose.middleware.yaml` for setting up essential middleware services like databases and caches.
+    - Use the `docker-compose.middleware.yaml` for setting up essential middleware services like databases and caches.
-   - Navigate to the `docker` directory.
+    - Navigate to the `docker` directory.
-   - Ensure the `middleware.env` file is created by running `cp middleware.env.example middleware.env` (refer to the `middleware.env.example` file).
+    - Ensure the `middleware.env` file is created by running `cp middleware.env.example middleware.env` (refer to the `middleware.env.example` file).
 2. **Running Middleware Services**:
-   - Execute `docker-compose -f docker-compose.middleware.yaml up -d` to start the middleware services.
+    - Execute `docker-compose -f docker-compose.middleware.yaml up -d` to start the middleware services.
 
 ### Migration for Existing Users
+
 For users migrating from the `docker-legacy` setup:
+
 1. **Review Changes**: Familiarize yourself with the new `.env` configuration and Docker Compose setup.
 2. **Transfer Customizations**:
-   - If you have customized configurations such as `docker-compose.yaml`, `ssrf_proxy/squid.conf`, or `nginx/conf.d/default.conf`, you will need to reflect these changes in the `.env` file you create.
+    - If you have customized configurations such as `docker-compose.yaml`, `ssrf_proxy/squid.conf`, or `nginx/conf.d/default.conf`, you will need to reflect these changes in the `.env` file you create.
 3. **Data Migration**:
-   - Ensure that data from services like databases and caches is backed up and migrated appropriately to the new structure if necessary.
+    - Ensure that data from services like databases and caches is backed up and migrated appropriately to the new structure if necessary.
 
 ### Overview of `.env`
 
 #### Key Modules and Customization
 
-- **Vector Database Services**: Depending on the type of vector database used (`VECTOR_STORE`), users can set specific endpoints, ports, and authentication details.
+- **Vector Database Services
-- **Storage Services**: Depending on the storage type (`STORAGE_TYPE`), users can configure specific settings for S3, Azure Blob, Google Storage, etc.
+  **: Depending on the type of vector database used (`VECTOR_STORE`), users can set specific endpoints, ports, and authentication details.
+- **Storage Services
+  **: Depending on the storage type (`STORAGE_TYPE`), users can configure specific settings for S3, Azure Blob, Google Storage, etc.
 - **API and Web Services**: Users can define URLs and other settings that affect how the API and web frontends operate.
 
 #### Other notable variables
+
 The `.env.example` file provided in the Docker setup is extensive and covers a wide range of configuration options. It is structured into several sections, each pertaining to different aspects of the application and its services. Here are some of the key sections and variables:
 
 1. **Common Variables**:
-   - `CONSOLE_API_URL`, `SERVICE_API_URL`: URLs for different API services.
+    - `CONSOLE_API_URL`, `SERVICE_API_URL`: URLs for different API services.
-   - `APP_WEB_URL`: Frontend application URL.
+    - `APP_WEB_URL`: Frontend application URL.
-   - `FILES_URL`: Base URL for file downloads and previews.
+    - `FILES_URL`: Base URL for file downloads and previews.
 
 2. **Server Configuration**:
-   - `LOG_LEVEL`, `DEBUG`, `FLASK_DEBUG`: Logging and debug settings.
+    - `LOG_LEVEL`, `DEBUG`, `FLASK_DEBUG`: Logging and debug settings.
-   - `SECRET_KEY`: A key for encrypting session cookies and other sensitive data.
+    - `SECRET_KEY`: A key for encrypting session cookies and other sensitive data.
 
 3. **Database Configuration**:
-   - `DB_USERNAME`, `DB_PASSWORD`, `DB_HOST`, `DB_PORT`, `DB_DATABASE`: PostgreSQL database credentials and connection details.
+    - `DB_USERNAME`, `DB_PASSWORD`, `DB_HOST`, `DB_PORT`, `DB_DATABASE`: PostgreSQL database credentials and connection details.
 
 4. **Redis Configuration**:
-   - `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD`: Redis server connection settings.
+    - `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD`: Redis server connection settings.
 
 5. **Celery Configuration**:
-   - `CELERY_BROKER_URL`: Configuration for Celery message broker.
+    - `CELERY_BROKER_URL`: Configuration for Celery message broker.
 
 6. **Storage Configuration**:
-   - `STORAGE_TYPE`, `S3_BUCKET_NAME`, `AZURE_BLOB_ACCOUNT_NAME`: Settings for file storage options like local, S3, Azure Blob, etc.
+    - `STORAGE_TYPE`, `S3_BUCKET_NAME`, `AZURE_BLOB_ACCOUNT_NAME`: Settings for file storage options like local, S3, Azure Blob, etc.
 
 7. **Vector Database Configuration**:
-   - `VECTOR_STORE`: Type of vector database (e.g., `weaviate`, `milvus`).
+    - `VECTOR_STORE`: Type of vector database (e.g., `weaviate`, `milvus`).
-   - Specific settings for each vector store like `WEAVIATE_ENDPOINT`, `MILVUS_HOST`.
+    - Specific settings for each vector store like `WEAVIATE_ENDPOINT`, `MILVUS_HOST`.
 
 8. **CORS Configuration**:
-   - `WEB_API_CORS_ALLOW_ORIGINS`, `CONSOLE_CORS_ALLOW_ORIGINS`: Settings for cross-origin resource sharing.
+    - `WEB_API_CORS_ALLOW_ORIGINS`, `CONSOLE_CORS_ALLOW_ORIGINS`: Settings for cross-origin resource sharing.
 
 9. **Other Service-Specific Environment Variables**:
-   - Each service like `nginx`, `redis`, `db`, and vector databases have specific environment variables that are directly referenced in the `docker-compose.yaml`.
+    - Each service like `nginx`, `redis`, `db`, and vector databases have specific environment variables that are directly referenced in the `docker-compose.yaml`.
-
 
 ### Additional Information
-- **Continuous Improvement Phase**: We are actively seeking feedback from the community to refine and enhance the deployment process. As more users adopt this new method, we will continue to make improvements based on your experiences and suggestions.
+
-- **Support**: For detailed configuration options and environment variable settings, refer to the `.env.example` file and the Docker Compose configuration files in the `docker` directory.
+- **Continuous Improvement Phase
+  **: We are actively seeking feedback from the community to refine and enhance the deployment process. As more users adopt this new method, we will continue to make improvements based on your experiences and suggestions.
+- **Support
+  **: For detailed configuration options and environment variable settings, refer to the `.env.example` file and the Docker Compose configuration files in the `docker` directory.
 
 This README aims to guide you through the deployment process using the new Docker Compose setup. For any issues or further assistance, please refer to the official documentation or contact support.

docker/certbot/README.md

@@ -0,0 +1,76 @@
+# Launching new servers with SSL certificates
+
+## Short description
+
+Docker-compose certbot configurations with Backward compatibility (without certbot container).  
+Use `docker-compose --profile certbot up` to use this features.
+
+## The simplest way for launching new servers with SSL certificates
+
+1. Get letsencrypt certs  
+   set `.env` values
+   ```properties
+   NGINX_SSL_CERT_FILENAME=fullchain.pem
+   NGINX_SSL_CERT_KEY_FILENAME=privkey.pem
+   NGINX_ENABLE_CERTBOT_CHALLENGE=true
+   CERTBOT_DOMAIN=your_domain.com
+   CERTBOT_EMAIL=example@your_domain.com
+   ```
+   excecute command:
+   ```shell
+   sudo docker network prune
+   sudo docker-compose --profile certbot up --force-recreate -d
+   ```
+   then after the containers launched:
+   ```shell
+   sudo docker-compose exec -it certbot /bin/sh /update-cert.sh
+   ```
+2. Edit `.env` file and `sudo docker-compose --profile certbot up` again.  
+   set `.env` value additionally
+   ```properties
+   NGINX_HTTPS_ENABLED=true
+   ```
+   excecute command:
+   ```shell
+   sudo docker-compose --profile certbot up -d --no-deps --force-recreate nginx
+   ```
+   Then you can access your serve with HTTPS.  
+   [https://your_domain.com](https://your_domain.com)
+
+## SSL certificates renewal
+
+For SSL certificates renewal, execute commands below:
+
+```shell
+sudo docker-compose exec -it certbot /bin/sh /update-cert.sh
+sudo docker-compose exec nginx nginx -s reload
+```
+
+## Options for certbot
+
+`CERTBOT_OPTIONS` key might be helpful for testing. i.e.,
+
+```properties
+CERTBOT_OPTIONS=--dry-run
+```
+
+To apply changes to `CERTBOT_OPTIONS`, regenerate the certbot container before updating the certificates.
+
+```shell
+sudo docker-compose --profile certbot up -d --no-deps --force-recreate certbot
+sudo docker-compose exec -it certbot /bin/sh /update-cert.sh
+```
+
+Then, reload the nginx container if necessary.
+
+```shell
+sudo docker-compose exec nginx nginx -s reload
+```
+
+## For legacy servers
+
+To use cert files dir `nginx/ssl` as before, simply launch containers WITHOUT `--profile certbot` option.
+
+```shell
+sudo docker-compose up -d
+```

docker/certbot/docker-entrypoint.sh

@@ -0,0 +1,30 @@
+#!/bin/sh
+set -e
+
+printf '%s\n' "Docker entrypoint script is running"
+
+printf '%s\n' "\nChecking specific environment variables:"
+printf '%s\n' "CERTBOT_EMAIL: ${CERTBOT_EMAIL:-Not set}"
+printf '%s\n' "CERTBOT_DOMAIN: ${CERTBOT_DOMAIN:-Not set}"
+printf '%s\n' "CERTBOT_OPTIONS: ${CERTBOT_OPTIONS:-Not set}"
+
+printf '%s\n' "\nChecking mounted directories:"
+for dir in "/etc/letsencrypt" "/var/www/html" "/var/log/letsencrypt"; do
+    if [ -d "$dir" ]; then
+        printf '%s\n' "$dir exists. Contents:"
+        ls -la "$dir"
+    else
+        printf '%s\n' "$dir does not exist."
+    fi
+done
+
+printf '%s\n' "\nGenerating update-cert.sh from template"
+sed -e "s|\${CERTBOT_EMAIL}|$CERTBOT_EMAIL|g" \
+    -e "s|\${CERTBOT_DOMAIN}|$CERTBOT_DOMAIN|g" \
+    -e "s|\${CERTBOT_OPTIONS}|$CERTBOT_OPTIONS|g" \
+    /update-cert.template.txt > /update-cert.sh
+
+chmod +x /update-cert.sh
+
+printf '%s\n' "\nExecuting command:" "$@"
+exec "$@"

docker/certbot/update-cert.template.txt

@@ -0,0 +1,19 @@
+#!/bin/bash
+set -e
+
+DOMAIN="${CERTBOT_DOMAIN}"
+EMAIL="${CERTBOT_EMAIL}"
+OPTIONS="${CERTBOT_OPTIONS}"
+CERT_NAME="${DOMAIN}" # 証明書名をドメイン名と同じにする
+
+# Check if the certificate already exists
+if [ -f "/etc/letsencrypt/renewal/${CERT_NAME}.conf" ]; then
+  echo "Certificate exists. Attempting to renew..."
+  certbot renew --noninteractive --cert-name ${CERT_NAME} --webroot --webroot-path=/var/www/html --email ${EMAIL} --agree-tos --no-eff-email ${OPTIONS}
+else
+  echo "Certificate does not exist. Obtaining a new certificate..."
+  certbot certonly --noninteractive --webroot --webroot-path=/var/www/html --email ${EMAIL} --agree-tos --no-eff-email -d ${DOMAIN} ${OPTIONS}
+fi
+echo "Certificate operation successful"
+# Note: Nginx reload should be handled outside this container
+echo "Please ensure to reload Nginx to apply any certificate changes."

docker/docker-compose.yaml

@@ -295,6 +295,26 @@ services:
       - ssrf_proxy_network
       - default
 
+  # Certbot service
+  # use `docker-compose --profile certbot up` to start the certbot service.
+  certbot:
+    image: certbot/certbot
+    profiles:
+      - certbot
+    volumes:
+      - ./volumes/certbot/conf:/etc/letsencrypt
+      - ./volumes/certbot/www:/var/www/html
+      - ./volumes/certbot/logs:/var/log/letsencrypt
+      - ./volumes/certbot/conf/live:/etc/letsencrypt/live
+      - ./certbot/update-cert.template.txt:/update-cert.template.txt
+      - ./certbot/docker-entrypoint.sh:/docker-entrypoint.sh
+    environment:
+      - CERTBOT_EMAIL=${CERTBOT_EMAIL}
+      - CERTBOT_DOMAIN=${CERTBOT_DOMAIN}
+      - CERTBOT_OPTIONS=${CERTBOT_OPTIONS:-}
+    entrypoint: [ "/docker-entrypoint.sh" ]
+    command: ["tail", "-f", "/dev/null"]
+
   # The nginx reverse proxy.
   # used for reverse proxying the API service and Web service.
   nginx:
@@ -306,7 +326,10 @@ services:
       - ./nginx/https.conf.template:/etc/nginx/https.conf.template
       - ./nginx/conf.d:/etc/nginx/conf.d
       - ./nginx/docker-entrypoint.sh:/docker-entrypoint-mount.sh
-      - ./nginx/ssl:/etc/ssl
+      - ./nginx/ssl:/etc/ssl # cert dir (legacy)
+      - ./volumes/certbot/conf/live:/etc/letsencrypt/live # cert dir (with certbot container)
+      - ./volumes/certbot/conf:/etc/letsencrypt
+      - ./volumes/certbot/www:/var/www/html
     entrypoint: [ "sh", "-c", "cp /docker-entrypoint-mount.sh /docker-entrypoint.sh && sed -i 's/\r$$//' /docker-entrypoint.sh && chmod +x /docker-entrypoint.sh && /docker-entrypoint.sh" ]
     environment:
       NGINX_SERVER_NAME: ${NGINX_SERVER_NAME:-_}
@@ -323,6 +346,8 @@ services:
       NGINX_KEEPALIVE_TIMEOUT: ${NGINX_KEEPALIVE_TIMEOUT:-65}
       NGINX_PROXY_READ_TIMEOUT: ${NGINX_PROXY_READ_TIMEOUT:-3600s}
       NGINX_PROXY_SEND_TIMEOUT: ${NGINX_PROXY_SEND_TIMEOUT:-3600s}
+      NGINX_ENABLE_CERTBOT_CHALLENGE: ${NGINX_ENABLE_CERTBOT_CHALLENGE:-false}
+      CERTBOT_DOMAIN: ${CERTBOT_DOMAIN:-}
     depends_on:
       - api
       - web
@@ -453,7 +478,7 @@ services:
       - ./volumes/milvus/etcd:/etcd
     command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
     healthcheck:
-      test: ["CMD", "etcdctl", "endpoint", "health"]
+      test: [ "CMD", "etcdctl", "endpoint", "health" ]
       interval: 30s
       timeout: 20s
       retries: 3
@@ -472,7 +497,7 @@ services:
       - ./volumes/milvus/minio:/minio_data
     command: minio server /minio_data --console-address ":9001"
     healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
+      test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
       interval: 30s
       timeout: 20s
       retries: 3
@@ -484,7 +509,7 @@ services:
     image: milvusdb/milvus:v2.3.1
     profiles:
       - milvus
-    command: ["milvus", "run", "standalone"]
+    command: [ "milvus", "run", "standalone" ]
     environment:
       ETCD_ENDPOINTS: ${ETCD_ENDPOINTS:-etcd:2379}
       MINIO_ADDRESS: ${MINIO_ADDRESS:-minio:9000}
@@ -492,7 +517,7 @@ services:
     volumes:
       - ./volumes/milvus/milvus:/var/lib/milvus
     healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
+      test: [ "CMD", "curl", "-f", "http://localhost:9091/healthz" ]
       interval: 30s
       start_period: 90s
       timeout: 20s

docker/nginx/conf.d/default.conf.template

@@ -29,6 +29,9 @@ server {
       include proxy.conf;
     }
 
+    # placeholder for acme challenge location
+    ${ACME_CHALLENGE_LOCATION}
+
     # placeholder for https config defined in https.conf.template
     ${HTTPS_CONFIG}
 }

docker/nginx/docker-entrypoint.sh

@@ -1,6 +1,19 @@
 #!/bin/bash
 
 if [ "${NGINX_HTTPS_ENABLED}" = "true" ]; then
+    # Check if the certificate and key files for the specified domain exist
+    if [ -n "${CERTBOT_DOMAIN}" ] && \
+       [ -f "/etc/letsencrypt/live/${CERTBOT_DOMAIN}/${NGINX_SSL_CERT_FILENAME}" ] && \
+       [ -f "/etc/letsencrypt/live/${CERTBOT_DOMAIN}/${NGINX_SSL_CERT_KEY_FILENAME}" ]; then
+        SSL_CERTIFICATE_PATH="/etc/letsencrypt/live/${CERTBOT_DOMAIN}/${NGINX_SSL_CERT_FILENAME}"
+        SSL_CERTIFICATE_KEY_PATH="/etc/letsencrypt/live/${CERTBOT_DOMAIN}/${NGINX_SSL_CERT_KEY_FILENAME}"
+    else
+        SSL_CERTIFICATE_PATH="/etc/ssl/${NGINX_SSL_CERT_FILENAME}"
+        SSL_CERTIFICATE_KEY_PATH="/etc/ssl/${NGINX_SSL_CERT_KEY_FILENAME}"
+    fi
+    export SSL_CERTIFICATE_PATH
+    export SSL_CERTIFICATE_KEY_PATH
+
     # set the HTTPS_CONFIG environment variable to the content of the https.conf.template
     HTTPS_CONFIG=$(envsubst < /etc/nginx/https.conf.template)
     export HTTPS_CONFIG
@@ -8,6 +21,13 @@ if [ "${NGINX_HTTPS_ENABLED}" = "true" ]; then
     envsubst '${HTTPS_CONFIG}' < /etc/nginx/conf.d/default.conf.template > /etc/nginx/conf.d/default.conf
 fi
 
+if [ "${NGINX_ENABLE_CERTBOT_CHALLENGE}" = "true" ]; then
+    ACME_CHALLENGE_LOCATION='location /.well-known/acme-challenge/ { root /var/www/html; }'
+else
+    ACME_CHALLENGE_LOCATION=''
+fi
+export ACME_CHALLENGE_LOCATION
+
 env_vars=$(printenv | cut -d= -f1 | sed 's/^/$/g' | paste -sd, -)
 
 envsubst "$env_vars" < /etc/nginx/nginx.conf.template > /etc/nginx/nginx.conf

docker/nginx/https.conf.template

@@ -1,8 +1,8 @@
 # Please do not directly edit this file. Instead, modify the .env variables related to NGINX configuration.
 
 listen ${NGINX_SSL_PORT} ssl;
-ssl_certificate ./../ssl/${NGINX_SSL_CERT_FILENAME};
+ssl_certificate ${SSL_CERTIFICATE_PATH};
-ssl_certificate_key ./../ssl/${NGINX_SSL_CERT_KEY_FILENAME};
+ssl_certificate_key ${SSL_CERTIFICATE_KEY_PATH};
 ssl_protocols ${NGINX_SSL_PROTOCOLS};
 ssl_prefer_server_ciphers on;
 ssl_session_cache shared:SSL:10m;