feat: configure Gitea deployment and environment-managed WorldQuant credentials
Deploy production / deploy (push) Has been cancelled
Deploy production / deploy (push) Has been cancelled
This commit is contained in:
+3
-1
@@ -8,4 +8,6 @@ ENCRYPTION_KEY=replace-with-generated-fernet-key
|
||||
LOCAL_PORT=8080
|
||||
# Public deployment only; bare DNS hostname, without scheme or path.
|
||||
DOMAIN=alpha.example.com
|
||||
# WorldQuant email/password are configured after system login, not in this file.
|
||||
# WorldQuant credentials; configure both. Single quotes preserve literal $ and #.
|
||||
WQ_EMAIL=
|
||||
WQ_PASSWORD=
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
name: Deploy production
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
# Register a dedicated runner on the target host with label wq-production:host.
|
||||
runs-on: wq-production
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: https://github.com/actions/checkout@v4
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Build, migrate and deploy
|
||||
shell: bash
|
||||
env:
|
||||
DATABASE_URL: ${{ secrets.DATABASE_URL }}
|
||||
ADMIN_PASSWORD: ${{ secrets.ADMIN_PASSWORD }}
|
||||
WQ_EMAIL: ${{ secrets.WQ_EMAIL }}
|
||||
WQ_PASSWORD: ${{ secrets.WQ_PASSWORD }}
|
||||
ENCRYPTION_KEY: ${{ secrets.ENCRYPTION_KEY }}
|
||||
ADMIN_USERNAME: ${{ vars.ADMIN_USERNAME }}
|
||||
DATABASE_NETWORK: ${{ vars.DATABASE_NETWORK }}
|
||||
PUBLIC_ORIGIN: ${{ vars.PUBLIC_ORIGIN }}
|
||||
run: bash scripts/deploy-production.sh
|
||||
@@ -0,0 +1,22 @@
|
||||
# Gitea 生产部署
|
||||
|
||||
采用已选择的 compound 经验 1,经当前项目核验:复用锁定依赖的 Dockerfile、单 worker 和数据库健康接口;新增独立生产 Compose、外部 PostgreSQL 配置、同机 host Runner 工作流。生产凭据通过 Gitea Secrets 注入,非敏感配置通过 Variables 注入,不生成凭据文件。数据库网络为可配置参数,不沿用旧项目常量作为强制约定。
|
||||
|
||||
部署在构建完成后停止写入、执行一次性迁移,再启动健康检查。固定项目名与按提交标记镜像;使用宿主机文件锁串行化。保留本地和独立公网部署的现有行为。不执行远程部署、不修改知识库。
|
||||
|
||||
验证:Compose 展开及缺失配置拒绝、脚本语法、镜像构建、隔离数据库上的迁移和 HTTP 健康检查。真实服务器 Runner、数据库、TLS 入口需首次部署验收。
|
||||
|
||||
## 验证结果(2026-09-09)
|
||||
|
||||
- Docker Engine 29.6.2 / Compose 5.3.1:默认和 jobs profile 配置校验通过;5 个必填配置缺失时均拒绝展开。
|
||||
- 前后端生产镜像构建通过(前端依赖 lottie-web 存在既有 eval 构建警告)。
|
||||
- 独立测试项目、临时 PostgreSQL 17、独立外部网络:首次迁移、页面/API 健康、管理员登录与 Secure/HttpOnly Cookie、重复迁移和重启均通过。测试容器、卷和网络已清理,未操作现有应用数据库。
|
||||
- Bash 语法与 workflow YAML 解析通过;模拟 Docker 验证成功流程及构建、预检、迁移、健康失败分支,确认提前失败不停止服务、迁移失败不启动服务、失败不记录成功版本。
|
||||
- 实际 Gitea/Runner、Linux flock 互斥、公网 TLS、生产库与恢复流程未在目标服务器验证。本机存在用户并行前端改动,本任务未修改这些文件。
|
||||
- 经验 1 已应用;后端无可写 named volume,未添加不适用的卷初始化 Job;凭据按用户最新要求由 Gitea 注入。知识库未修改。
|
||||
|
||||
## Gitea Secrets 调整
|
||||
|
||||
按用户确认改为步骤级 Secrets/Variables 注入;删除生产环境示例文件和相关忽略例外,不再依赖服务器凭据文件。脚本显式使用 `/dev/null` 作为 env-file,5 个必填值缺失时立即停止;锁与版本记录独立保存在 `/opt/wq-alpha`。
|
||||
|
||||
本轮验证:Compose 默认/jobs 展开、5 个缺失配置拒绝、真实隔离容器内特殊字符密码完整性、Bash 语法和工作流映射均通过。模拟 Docker 验证成功、锁冲突、缺失变量和构建/预检/迁移/健康失败分支;未输出密钥。未重新运行未变化的迁移和前端构建;真实 Gitea 注入仍待服务器运行确认。
|
||||
@@ -10,6 +10,7 @@
|
||||
|
||||
```bash
|
||||
python3 scripts/init_env.py
|
||||
# 编辑 .env,填写 WQ_EMAIL 与 WQ_PASSWORD,再启动
|
||||
docker compose up -d --build --wait
|
||||
docker compose ps
|
||||
```
|
||||
@@ -20,7 +21,7 @@ docker compose ps
|
||||
|
||||
首次使用:
|
||||
|
||||
1. 登录系统,在“个人信息”保存 WorldQuant 邮箱和密码,点击“连接 WorldQuant”。
|
||||
1. 登录系统,在“个人信息 → 连接设置”点击“连接 WorldQuant”。
|
||||
2. 如平台要求人工验证,在显示的入口完成操作,再点击“继续验证”;后台保留同一挑战会话。
|
||||
3. 在“Alpha 管理”切换“待提交 / 已提交”。待提交先选创建日期范围,按天同步;已提交可选提交日期范围按天同步,或全量同步。相同起止日期表示单日,日期边界为 UTC,均包含隐藏记录。也可导入指定 Alpha ID。任务面板显示当前日期、进度、错误、取消和重试。
|
||||
4. 点击 Alpha 打开详情。研究记录保存在本地;PnL 点击获取后缓存。详情的“本地自相关”可发起检测,列表也可选中最多 100 条批量检测。建议先全量同步已提交 Alpha,建立比较基准。下次同步会更新平台数据并保留本地研究记录。
|
||||
@@ -83,6 +84,10 @@ Chatbox 来源使用 `kind=chatbox`,会话 ID 为 `reference`,生成轮次 I
|
||||
|
||||
公共接口位于 `/api/v1/backtests`,对接与验证记录见 [实施规格](.scratch/backtest/spec.md) 和 [回测验收记录](.scratch/backtest/verification.md)。真实平台权限、当前协议与限额尚未联调。
|
||||
|
||||
## Gitea 自动部署(复用已有 PostgreSQL)
|
||||
|
||||
使用独立的 `compose.production.yaml` 和 `.gitea/workflows/deploy-production.yaml`。配置步骤、数据库账号密码位置及升级处理见 [Gitea 部署说明](docs/deployment-gitea.md)。
|
||||
|
||||
## 公网 HTTPS 部署
|
||||
|
||||
`compose.public.yaml` 是独立配置,不与本机配置叠加。先在服务器完成上面的密钥初始化,将 `.env` 中 `DOMAIN` 改为自己的域名(无协议、路径、端口)。DNS 指向服务器,允许入站 TCP 80/443,UDP 443 可选。
|
||||
@@ -103,6 +108,7 @@ docker compose -f compose.public.yaml logs --tail=100 web
|
||||
| `ADMIN_USERNAME` / `ADMIN_PASSWORD` | 仅首次空库初始化管理员,重启不会重置现有密码 |
|
||||
| `POSTGRES_PASSWORD` | 数据库密码;初始化脚本使用随机十六进制,避免连接 URL 转义问题 |
|
||||
| `ENCRYPTION_KEY` | 独立 Fernet 密钥,加密数据库中的 WorldQuant 密码和模型 API Key |
|
||||
| `WQ_EMAIL` / `WQ_PASSWORD` | WorldQuant 邮箱和密码,成对设置;本地 `.env`,生产 Gitea Secrets |
|
||||
| `LOCAL_PORT` | 本机入口端口,默认 8080 |
|
||||
| `DOMAIN` | 公网域名 |
|
||||
| `AI_REQUEST_LIMIT` | 每轮模型请求上限,默认 12 |
|
||||
@@ -110,6 +116,8 @@ docker compose -f compose.public.yaml logs --tail=100 web
|
||||
| `AI_OUTPUT_TOKENS` | 每次模型输出上限,默认 4096 |
|
||||
| `AI_TIMEOUT` | 每轮累计活动执行时限(秒),默认 180,等待确认不计入 |
|
||||
|
||||
WorldQuant 凭据不再从 `account.json` 读取。进程环境变量优先于项目根目录 `.env`;启动时更新数据库中的加密凭据,配置后页面禁止覆盖。修改凭据后重建后端容器(Gitea 重新部署);已绑定账户不能更换邮箱。两个变量都未设置时兼容已有页面配置,只有一个时启动失败。
|
||||
|
||||
WorldQuant 密码仅在后端解密。平台 Cookie 仅保存在后端内存,进程重启后重新认证。前端不保存密码或 Cookie 副本;日志与响应不输出平台认证正文。`.env` 不进入 Docker 构建上下文,应与数据库备份分别安全保管。丢失 `ENCRYPTION_KEY` 后须重新输入平台密码和模型 API Key;切勿在正常升级时重新生成它。
|
||||
|
||||
修改系统密码(同时撤销所有系统会话):
|
||||
|
||||
+10
-1
@@ -1,12 +1,16 @@
|
||||
"""Deployment configuration; secrets are required and never included in API responses."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from cryptography.fernet import Fernet
|
||||
from pydantic import Field, SecretStr, model_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_file="../.env", extra="ignore")
|
||||
model_config = SettingsConfigDict(
|
||||
env_file=Path(__file__).resolve().parents[2] / ".env", extra="ignore", hide_input_in_errors=True
|
||||
)
|
||||
|
||||
database_url: str = "postgresql+asyncpg://wq:wq@localhost:5432/wq"
|
||||
admin_username: str = "admin"
|
||||
@@ -15,6 +19,8 @@ class Settings(BaseSettings):
|
||||
public_origin: str = "http://localhost:8080"
|
||||
cookie_secure: bool = False
|
||||
session_hours: int = Field(default=24, ge=1, le=168)
|
||||
wq_email: str = ""
|
||||
wq_password: SecretStr = SecretStr("")
|
||||
wq_base_url: str = "https://api.worldquantbrain.com"
|
||||
request_timeout: float = 30
|
||||
retry_attempts: int = Field(default=4, ge=1, le=8)
|
||||
@@ -26,6 +32,9 @@ class Settings(BaseSettings):
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_secrets(self):
|
||||
self.wq_email = self.wq_email.strip()
|
||||
if bool(self.wq_email) != bool(self.wq_password.get_secret_value()):
|
||||
raise ValueError("WQ_EMAIL and WQ_PASSWORD must be configured together")
|
||||
Fernet(self.encryption_key.get_secret_value().encode())
|
||||
if self.cookie_secure and not self.public_origin.startswith("https://"):
|
||||
raise ValueError("COOKIE_SECURE requires an HTTPS PUBLIC_ORIGIN")
|
||||
|
||||
+7
-4
@@ -53,7 +53,7 @@ from .schemas import (
|
||||
from .security import bootstrap, cipher, issue_session, require_auth, token_hash, valid_password
|
||||
|
||||
|
||||
def account_output(account, client):
|
||||
def account_output(account, client, settings):
|
||||
keys = (
|
||||
"email",
|
||||
"wq_user_id",
|
||||
@@ -70,6 +70,7 @@ def account_output(account, client):
|
||||
return {
|
||||
**{k: getattr(account, k) for k in keys},
|
||||
"configured": bool(account.password_encrypted),
|
||||
"credentials_source": "environment" if settings.wq_email else "database",
|
||||
"session": client.session_info(),
|
||||
}
|
||||
|
||||
@@ -210,10 +211,12 @@ def create_app(settings=None, wq_client=None, ai_model_factory=None):
|
||||
@api.get("/account", response_model=AccountOutput, tags=["account"])
|
||||
async def get_account():
|
||||
async with sessions() as db:
|
||||
return account_output(await db.get(Account, 1), runner.client)
|
||||
return account_output(await db.get(Account, 1), runner.client, settings)
|
||||
|
||||
@api.put("/account/credentials", response_model=AccountOutput, tags=["account"])
|
||||
async def credentials(body: CredentialsInput):
|
||||
if settings.wq_email:
|
||||
raise HTTPException(409, "WorldQuant 凭据由环境变量管理,请修改部署配置并重启服务")
|
||||
async with sessions() as db:
|
||||
account = await db.get(Account, 1)
|
||||
if account.wq_user_id and account.email.casefold() != body.email.casefold():
|
||||
@@ -224,7 +227,7 @@ def create_app(settings=None, wq_client=None, ai_model_factory=None):
|
||||
account.email = body.email
|
||||
account.password_encrypted = cipher(settings).encrypt(body.password.encode()).decode()
|
||||
await db.commit()
|
||||
return account_output(account, runner.client)
|
||||
return account_output(account, runner.client, settings)
|
||||
|
||||
@api.patch("/account/preferences", response_model=AccountOutput, tags=["account"])
|
||||
async def preferences(body: PreferencesInput):
|
||||
@@ -233,7 +236,7 @@ def create_app(settings=None, wq_client=None, ai_model_factory=None):
|
||||
for key, value in body.model_dump().items():
|
||||
setattr(account, key, value)
|
||||
await db.commit()
|
||||
return account_output(account, runner.client)
|
||||
return account_output(account, runner.client, settings)
|
||||
|
||||
async def account_job(kind):
|
||||
async with sessions() as db:
|
||||
|
||||
@@ -287,6 +287,7 @@ class PlatformSessionOutput(BaseModel):
|
||||
class AccountOutput(BaseModel):
|
||||
email: str | None
|
||||
configured: bool
|
||||
credentials_source: Literal["environment", "database"]
|
||||
wq_user_id: str | None
|
||||
profile: dict
|
||||
connection_status: str
|
||||
|
||||
+13
-3
@@ -26,7 +26,7 @@ def cipher(settings) -> Fernet:
|
||||
|
||||
|
||||
async def bootstrap(db, settings):
|
||||
"""Only initialize missing singleton records; deployments never reset existing passwords."""
|
||||
"""Initialize singletons and apply environment credentials without resetting the admin password."""
|
||||
if not await db.get(Admin, 1):
|
||||
db.add(
|
||||
Admin(
|
||||
@@ -35,8 +35,18 @@ async def bootstrap(db, settings):
|
||||
password_hash=password_hasher.hash(settings.admin_password.get_secret_value()),
|
||||
)
|
||||
)
|
||||
if not await db.get(Account, 1):
|
||||
db.add(Account(id=1))
|
||||
account = await db.get(Account, 1)
|
||||
if account is None:
|
||||
account = Account(id=1)
|
||||
db.add(account)
|
||||
if settings.wq_email:
|
||||
# The environment must not bypass the single-account data ownership boundary.
|
||||
if account.wq_user_id and (account.email or "").casefold() != settings.wq_email.casefold():
|
||||
raise ValueError("WQ_EMAIL conflicts with the bound WorldQuant account")
|
||||
account.email = settings.wq_email
|
||||
account.password_encrypted = cipher(settings).encrypt(
|
||||
settings.wq_password.get_secret_value().encode()
|
||||
).decode()
|
||||
await db.execute(delete(LoginSession).where(LoginSession.expires_at < now()))
|
||||
await db.commit()
|
||||
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Environment credential precedence, rotation and account ownership boundaries."""
|
||||
|
||||
import pytest
|
||||
from pydantic import SecretStr, ValidationError
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Account, Admin
|
||||
from app.security import bootstrap, cipher
|
||||
|
||||
|
||||
def test_dotenv_and_process_precedence(settings, tmp_path, monkeypatch):
|
||||
env = tmp_path / '.env'
|
||||
env.write_text("WQ_EMAIL=local@example.com\nWQ_PASSWORD='literal-$value#password'\n")
|
||||
values = settings.model_dump(exclude={'wq_email', 'wq_password'})
|
||||
local = Settings(_env_file=env, **values)
|
||||
assert local.wq_email == 'local@example.com'
|
||||
assert local.wq_password.get_secret_value() == 'literal-$value#password'
|
||||
monkeypatch.setenv('WQ_EMAIL', 'production@example.com')
|
||||
monkeypatch.setenv('WQ_PASSWORD', 'production-secret')
|
||||
production = Settings(_env_file=env, **values)
|
||||
assert production.wq_email == 'production@example.com'
|
||||
assert production.wq_password.get_secret_value() == 'production-secret'
|
||||
assert 'production-secret' not in repr(production)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('email,password', [('person@example.com', ''), ('', 'private-value')])
|
||||
def test_partial_configuration_fails_without_leaking(settings, email, password):
|
||||
values = settings.model_dump(exclude={'wq_email', 'wq_password'})
|
||||
with pytest.raises(ValidationError) as error:
|
||||
Settings(_env_file=None, **values, wq_email=email, wq_password=password)
|
||||
assert 'must be configured together' in str(error.value)
|
||||
assert 'private-value' not in str(error.value)
|
||||
assert 'person@example.com' not in str(error.value)
|
||||
|
||||
|
||||
async def test_env_rotation_api_lock_and_bound_account(app, logged_in):
|
||||
settings = app.state.settings
|
||||
settings.wq_email = 'person@example.com'
|
||||
settings.wq_password = SecretStr('initial-env-secret')
|
||||
async with app.state.sessions() as db:
|
||||
original_admin = (await db.get(Admin, 1)).password_hash
|
||||
await bootstrap(db, settings)
|
||||
account = await db.get(Account, 1)
|
||||
assert cipher(settings).decrypt(account.password_encrypted.encode()) == b'initial-env-secret'
|
||||
account.wq_user_id = 'bound-user'
|
||||
await db.commit()
|
||||
settings.wq_password = SecretStr('rotated-env-secret')
|
||||
async with app.state.sessions() as db:
|
||||
await bootstrap(db, settings)
|
||||
account = await db.get(Account, 1)
|
||||
assert cipher(settings).decrypt(account.password_encrypted.encode()) == b'rotated-env-secret'
|
||||
assert (await db.get(Admin, 1)).password_hash == original_admin
|
||||
response = await logged_in.get('/api/v1/account')
|
||||
assert response.json()['credentials_source'] == 'environment'
|
||||
assert response.json()['configured'] is True
|
||||
assert 'secret' not in response.text and 'password' not in response.text
|
||||
response = await logged_in.put('/api/v1/account/credentials', json={
|
||||
'email': 'person@example.com', 'password': 'manual-secret',
|
||||
})
|
||||
assert response.status_code == 409
|
||||
settings.wq_email = 'other@example.com'
|
||||
async with app.state.sessions() as db:
|
||||
with pytest.raises(ValueError, match='bound WorldQuant account'):
|
||||
await bootstrap(db, settings)
|
||||
await db.rollback()
|
||||
assert (await db.get(Account, 1)).email == 'person@example.com'
|
||||
@@ -0,0 +1,74 @@
|
||||
# Independent production configuration; do not merge with compose.yaml.
|
||||
name: wq-alpha-production
|
||||
|
||||
x-backend: &backend
|
||||
image: wq-alpha-production-backend:${DEPLOY_TAG:-local}
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.backend
|
||||
environment:
|
||||
DATABASE_URL: ${DATABASE_URL:?Set the Gitea DATABASE_URL secret}
|
||||
ADMIN_USERNAME: ${ADMIN_USERNAME:-admin}
|
||||
ADMIN_PASSWORD: ${ADMIN_PASSWORD:?Set ADMIN_PASSWORD}
|
||||
ENCRYPTION_KEY: ${ENCRYPTION_KEY:?Set ENCRYPTION_KEY}
|
||||
PUBLIC_ORIGIN: ${PUBLIC_ORIGIN:?Set the public HTTPS origin}
|
||||
WQ_EMAIL: ${WQ_EMAIL:?Set the Gitea WQ_EMAIL secret}
|
||||
WQ_PASSWORD: ${WQ_PASSWORD:?Set the Gitea WQ_PASSWORD secret}
|
||||
COOKIE_SECURE: "true"
|
||||
AI_REQUEST_LIMIT: ${AI_REQUEST_LIMIT:-12}
|
||||
AI_TOOL_LIMIT: ${AI_TOOL_LIMIT:-12}
|
||||
AI_OUTPUT_TOKENS: ${AI_OUTPUT_TOKENS:-4096}
|
||||
AI_TIMEOUT: ${AI_TIMEOUT:-180}
|
||||
WQ_BASE_URL: ${WQ_BASE_URL:-https://api.worldquantbrain.com}
|
||||
networks:
|
||||
- default
|
||||
- database
|
||||
|
||||
services:
|
||||
backend:
|
||||
<<: *backend
|
||||
# Migration is run separately with all application writers stopped.
|
||||
command: [uvicorn, 'app.main:create_app', --factory, --host, 0.0.0.0, --port, '8000', --workers, '1', --proxy-headers]
|
||||
healthcheck:
|
||||
test: [CMD, python, -c, "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/api/v1/health', timeout=3)"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 12
|
||||
start_period: 20s
|
||||
stop_grace_period: 60s
|
||||
restart: unless-stopped
|
||||
migrate:
|
||||
<<: *backend
|
||||
profiles: [jobs]
|
||||
command: [alembic, upgrade, head]
|
||||
restart: 'no'
|
||||
web:
|
||||
image: wq-alpha-production-web:${DEPLOY_TAG:-local}
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.frontend
|
||||
environment:
|
||||
SITE_ADDRESS: http://:80
|
||||
ports:
|
||||
- '127.0.0.1:${WEB_PORT:-8112}:80'
|
||||
volumes:
|
||||
- caddy_data:/data
|
||||
- caddy_config:/config
|
||||
depends_on:
|
||||
backend:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'wget -q -O /dev/null http://127.0.0.1/api/v1/health && wget -q -O /dev/null http://127.0.0.1/']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 12
|
||||
start_period: 10s
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
database:
|
||||
external: true
|
||||
name: ${DATABASE_NETWORK:?Set the existing PostgreSQL Docker network}
|
||||
volumes:
|
||||
caddy_data:
|
||||
caddy_config:
|
||||
@@ -29,6 +29,8 @@ services:
|
||||
AI_TOOL_LIMIT: ${AI_TOOL_LIMIT:-12}
|
||||
AI_OUTPUT_TOKENS: ${AI_OUTPUT_TOKENS:-4096}
|
||||
AI_TIMEOUT: ${AI_TIMEOUT:-180}
|
||||
WQ_EMAIL: ${WQ_EMAIL:-}
|
||||
WQ_PASSWORD: ${WQ_PASSWORD:-}
|
||||
COOKIE_SECURE: "true"
|
||||
depends_on:
|
||||
db:
|
||||
|
||||
@@ -28,6 +28,8 @@ services:
|
||||
AI_TOOL_LIMIT: ${AI_TOOL_LIMIT:-12}
|
||||
AI_OUTPUT_TOKENS: ${AI_OUTPUT_TOKENS:-4096}
|
||||
AI_TIMEOUT: ${AI_TIMEOUT:-180}
|
||||
WQ_EMAIL: ${WQ_EMAIL:-}
|
||||
WQ_PASSWORD: ${WQ_PASSWORD:-}
|
||||
COOKIE_SECURE: "false"
|
||||
WQ_BASE_URL: ${WQ_BASE_URL:-https://api.worldquantbrain.com}
|
||||
depends_on:
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
# Gitea 生产部署
|
||||
|
||||
本方案在目标 Linux 服务器上由 Gitea host Runner 构建并启动 Docker Compose,复用已有 PostgreSQL。入口绑定 `127.0.0.1:8112`,由宿主机反向代理提供公网 HTTPS。现有 `compose.yaml`、`compose.public.yaml` 保持独立,不与生产文件叠加。
|
||||
|
||||
## 1. 填写配置和数据库账号密码
|
||||
|
||||
进入 **Gitea 仓库 → 设置 → Actions → Secrets / Variables**,按下表创建同名配置。敏感值填写在 Secrets,工作流仅在部署步骤通过环境变量注入,不生成服务器凭据文件。
|
||||
|
||||
| 位置 | 名称 | 填写内容 |
|
||||
| --- | --- | --- |
|
||||
| Secrets(必填) | `DATABASE_URL` | `postgresql+asyncpg://数据库账号:数据库密码@数据库网络别名:5432/数据库名` |
|
||||
| Secrets(必填) | `ADMIN_PASSWORD` | 系统初始管理员密码,至少 12 字符,与数据库密码独立 |
|
||||
| Secrets(必填) | `ENCRYPTION_KEY` | 下面命令生成的 Fernet 密钥,升级保持不变 |
|
||||
| Secrets(必填) | `WQ_EMAIL` | WorldQuant 登录邮箱 |
|
||||
| Secrets(必填) | `WQ_PASSWORD` | WorldQuant 登录密码,按原样填写,不加引号 |
|
||||
| Variables(必填) | `DATABASE_NETWORK` | PostgreSQL 所在的现有 Docker 网络,例如 `1panel-network` |
|
||||
| Variables(必填) | `PUBLIC_ORIGIN` | 实际 HTTPS 来源,例如 `https://alpha.your-domain.com`,不带路径或末尾 `/` |
|
||||
| Variables(可选) | `ADMIN_USERNAME` | 初始管理员账号,默认 `admin` |
|
||||
|
||||
端口与 AI 限制使用 `compose.production.yaml` 的默认值,不需要在 Gitea 配置:`WEB_PORT=8112`、`AI_REQUEST_LIMIT=12`、`AI_TOOL_LIMIT=12`、`AI_OUTPUT_TOKENS=4096`、`AI_TIMEOUT=180`。需要调整时修改 Compose 中对应默认值;工作流不再读取这些同名 Gitea Variables。
|
||||
|
||||
数据库连接串示例(实际填写时替换示例值):
|
||||
|
||||
```text
|
||||
postgresql+asyncpg://wq_user:YOUR_PASSWORD@postgresql:5432/wq_alpha
|
||||
```
|
||||
|
||||
生成加密密钥:
|
||||
|
||||
```bash
|
||||
python3 -c 'import base64,secrets; print(base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())'
|
||||
```
|
||||
|
||||
`DATABASE_URL` 的用户名和密码中的特殊字符须做 URL 百分号编码,例如 `@` → `%40`、`#` → `%23`、`/` → `%2F`、`%` → `%25`。在 Gitea 输入框填写值本身,不添加包裹引号,不填写 `DATABASE_URL=` 前缀。管理员密码中的 `$`、引号等按原样填写,由环境变量传递,无需 shell 转义。不要在日志打印变量,或开启 `set -x`。
|
||||
|
||||
在现有 PostgreSQL 管理界面先创建专用数据库与账号,账号须可连接并拥有该库中应用 schema 的建表和迁移权限。确认网络存在、数据库别名可解析、PostgreSQL 允许该 Docker 网络访问。应用使用异步驱动 `asyncpg`;远程数据库或强制 TLS 的实例需另外按该实例的 TLS 配置调整连接参数,此模板默认同机 Docker 网络。
|
||||
|
||||
如果从已有安装迁移数据,需恢复完整数据库并沿用原 `ENCRYPTION_KEY`;修改管理员环境变量不会重置已有管理员密码。
|
||||
|
||||
WorldQuant 凭据由环境变量管理,启动时加密写入数据库;页面只保留连接操作。修改 `WQ_PASSWORD` 后重新运行部署即可更新。`WQ_EMAIL` 必须与已有绑定账户一致,避免混入其他账户数据;生产不读取 `account.json` 或本地 `.env`。
|
||||
|
||||
## 2. 配置 Gitea Runner
|
||||
|
||||
在目标 Docker 宿主机直接运行专用 Runner,注册标签 **`wq-production:host`**,工作流的 `runs-on` 对应 `wq-production`。建议专用 Runner 配置 `runner.capacity: 1`。不要使用指向其他机器的 Docker context;这里的 host 模式也不能被当成“容器化 Runner 自动进入宿主机”。
|
||||
|
||||
Runner 用户需要 Docker 权限,以及写入预先创建的 `/opt/wq-alpha` 目录的权限。该目录仅保存锁文件和版本记录,不保存凭据。宿主机需具备 Git、Bash、Node.js 20(checkout v4)、`flock`(通常由 util-linux 提供)和支持 `up --wait --wait-timeout` 的 Docker Compose v2 或 v5。镜像使用锁文件构建,服务器需能访问 GitHub checkout action、基础镜像仓库和依赖源。本地验证环境为 Docker Engine 29.6.2、Compose 5.3.1;你的 Gitea/Runner 版本需在首次运行核验。
|
||||
|
||||
在 Gitea 仓库启用 Actions,提交并推送这些配置后,`main` 的 push 或手动运行会部署。Runner 应只接收可信仓库的任务,因为它具备生产主机 Docker 权限。凭据通过上述 Secrets 注入,脚本使用 `--env-file /dev/null` 防止误读检出目录的开发 `.env`。部署脚本用 `/opt/wq-alpha/deploy.lock` 实现跨 checkout 的互斥,冲突部署直接失败,可稍后重新运行。
|
||||
|
||||
## 3. 配置反向代理并首次运行
|
||||
|
||||
为 `PUBLIC_ORIGIN` 对应域名配置 HTTPS,转发至 **`http://127.0.0.1:8112`**(或你的 `WEB_PORT`)。代理须运行在宿主机网络中;独立 bridge 容器中的 `127.0.0.1` 不指向宿主机。若使用容器化 1Panel/OpenResty,先确认其网络模式。AI 流式响应需要关闭代理缓冲,并允许长连接/足够长的读取超时。
|
||||
|
||||
首次部署建议在 Gitea Actions 页面手动运行工作流。需要在服务器排障运行时,须先从安全渠道将上表必填配置注入当前进程环境,再执行:
|
||||
|
||||
```bash
|
||||
bash scripts/deploy-production.sh
|
||||
```
|
||||
|
||||
固定 Compose 项目名为 `wq-alpha-production`,后端保持一个实例、一个 worker。脚本先验证 Compose,构建按 Git 提交标记的镜像,再检查密钥格式和数据库连通性;随后停止 web/backend、执行一次性迁移、启动服务并等待健康检查。Web 健康检查同时覆盖页面和经 Caddy 转发的数据库健康接口。迁移或启动失败时非零退出并显示容器状态,不继续标记成功。
|
||||
|
||||
首次部署完成后,检查公网 `/api/v1/health`,再通过真实域名登录并确认页面、写请求和 Cookie 正常。容器健康通过不能替代 HTTPS、DNS 和真实 Gitea Runner 验收。
|
||||
|
||||
## 4. 升级、备份和失败处理
|
||||
|
||||
升级包含停机窗口;提前结束或暂停长时间任务。每次发布前通过现有数据库管理工具备份专用库,并在独立安全位置备份加密密钥及必要配置,先在独立库验证恢复。此工作流不会自动备份或自动恢复数据库。
|
||||
|
||||
构建及预检失败时旧服务继续运行。停止服务后的迁移或健康检查失败需要人工处理;不要对可能已变更的 schema 直接自动降级。脚本将切换前的镜像 ID/标签记录到 `/opt/wq-alpha/previous-images.txt`,最后成功的提交记录到 `current-release.txt`,保留旧镜像且不执行 prune。失败重试前另存这些记录,避免后续尝试覆盖回退参考。
|
||||
|
||||
如旧代码与当前 schema 兼容,可检出旧提交并指定其镜像标签启动;否则先停止应用,使用经过验证的备份恢复数据库,再用原加密密钥和对应旧版本启动。数据库恢复会丢失备份后的写入,必须人工确认后执行。本配置没有自动数据库降级,也不承诺无停机升级。
|
||||
|
||||
以下排查命令不依赖凭据环境变量(可能含业务信息的日志请勿公开):
|
||||
|
||||
```bash
|
||||
docker ps -a --filter label=com.docker.compose.project=wq-alpha-production
|
||||
docker logs --tail=100 wq-alpha-production-backend-1
|
||||
docker logs --tail=100 wq-alpha-production-web-1
|
||||
```
|
||||
|
||||
普通维护不要使用 `down -v`。这里的数据库由外部管理,备份与恢复应在数据库管理端进行。
|
||||
|
||||
## 配置依据
|
||||
|
||||
采用 compound 经验 1 中的生产/开发隔离、显式外部网络、必填凭据、固定项目名、独立迁移和部署健康检查。未照搬旧项目端口、业务 Job 或卷权限初始化:本应用后端不写 named volume,已有镜像使用 UID 10001。
|
||||
|
||||
凭据存放在 Gitea Secrets,通过步骤级环境变量交给 Compose,不落地到配置文件。Docker 容器仍需持有运行时配置,因此拥有 Runner 或 Docker 管理权限的人仍可能读取它们。若此前手动创建了旧 `.env.production`,新流程不再读取它;确认配置已迁移到 Gitea 并妥善备份密钥后可自行移除旧文件。相关官方资料:[Compose 外部网络](https://docs.docker.com/reference/compose-file/networks/)、[环境变量插值](https://docs.docker.com/compose/how-tos/environment-variables/variable-interpolation/)、[Gitea Runner 标签](https://gitea.com/gitea/runner/src/branch/main/README.md)。
|
||||
@@ -233,9 +233,16 @@ export function AccountPage({
|
||||
aria-label="WorldQuant 连接设置"
|
||||
>
|
||||
<h2>WorldQuant 连接</h2>
|
||||
{account.credentials_source === "environment" && (
|
||||
<p className="muted">
|
||||
凭据由环境变量管理。修改本地 .env 或 Gitea Secrets
|
||||
后重新部署生效。
|
||||
</p>
|
||||
)}
|
||||
<form
|
||||
onSubmit={(e) => {
|
||||
e.preventDefault();
|
||||
if (account.credentials_source === "environment") return;
|
||||
void action("save", async () => {
|
||||
await api("/account/credentials", {
|
||||
method: "PUT",
|
||||
@@ -246,41 +253,45 @@ export function AccountPage({
|
||||
});
|
||||
}}
|
||||
>
|
||||
<div className="credentials-grid">
|
||||
<label>
|
||||
WorldQuant 邮箱
|
||||
<Input
|
||||
aria-label="WorldQuant 邮箱"
|
||||
value={email}
|
||||
onChange={setEmail}
|
||||
autoComplete="off"
|
||||
/>
|
||||
</label>
|
||||
<label>
|
||||
WorldQuant 密码
|
||||
<Input
|
||||
aria-label="WorldQuant 密码"
|
||||
mode="password"
|
||||
value={password}
|
||||
onChange={setPassword}
|
||||
placeholder={
|
||||
account.configured
|
||||
? "已保存,输入新密码可更新"
|
||||
: "输入平台密码"
|
||||
}
|
||||
autoComplete="new-password"
|
||||
/>
|
||||
</label>
|
||||
</div>
|
||||
{account.credentials_source !== "environment" && (
|
||||
<div className="credentials-grid">
|
||||
<label>
|
||||
WorldQuant 邮箱
|
||||
<Input
|
||||
aria-label="WorldQuant 邮箱"
|
||||
value={email}
|
||||
onChange={setEmail}
|
||||
autoComplete="off"
|
||||
/>
|
||||
</label>
|
||||
<label>
|
||||
WorldQuant 密码
|
||||
<Input
|
||||
aria-label="WorldQuant 密码"
|
||||
mode="password"
|
||||
value={password}
|
||||
onChange={setPassword}
|
||||
placeholder={
|
||||
account.configured
|
||||
? "已保存,输入新密码可更新"
|
||||
: "输入平台密码"
|
||||
}
|
||||
autoComplete="new-password"
|
||||
/>
|
||||
</label>
|
||||
</div>
|
||||
)}
|
||||
<div className="inline-actions">
|
||||
<Button
|
||||
type="tertiary"
|
||||
htmlType="submit"
|
||||
disabled={!email || !password || blocked}
|
||||
loading={busy === "save"}
|
||||
>
|
||||
保存凭据
|
||||
</Button>
|
||||
{account.credentials_source !== "environment" && (
|
||||
<Button
|
||||
type="tertiary"
|
||||
htmlType="submit"
|
||||
disabled={!email || !password || blocked}
|
||||
loading={busy === "save"}
|
||||
>
|
||||
保存凭据
|
||||
</Button>
|
||||
)}
|
||||
<Button
|
||||
theme="solid"
|
||||
disabled={!account.configured || blocked}
|
||||
|
||||
@@ -71,6 +71,7 @@ export type AlphaDetail = Alpha & {
|
||||
export type Account = {
|
||||
email: string | null;
|
||||
configured: boolean;
|
||||
credentials_source?: "environment" | "database";
|
||||
wq_user_id: string | null;
|
||||
profile: Record<string, unknown>;
|
||||
connection_status: string;
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
# Deploy on the target Linux Docker host with configuration injected by Gitea.
|
||||
# Returns nonzero on configuration/build/migration/health failure. Never removes data.
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/.."
|
||||
# Fail before touching the host; never read a checkout's local .env file.
|
||||
for key in WQ_EMAIL WQ_PASSWORD DATABASE_URL ADMIN_PASSWORD ENCRYPTION_KEY DATABASE_NETWORK PUBLIC_ORIGIN; do
|
||||
if [[ -z "${!key:-}" ]]; then
|
||||
echo "Missing required Gitea configuration: $key" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
state_dir="${DEPLOY_STATE_DIR:-/opt/wq-alpha}"
|
||||
if [[ "$state_dir" != /* || ! -d "$state_dir" || ! -w "$state_dir" ]]; then
|
||||
echo 'DEPLOY_STATE_DIR must be an existing writable absolute directory.' >&2
|
||||
exit 1
|
||||
fi
|
||||
command -v flock >/dev/null
|
||||
# Stable across checkouts; stores only a lock and release metadata, never credentials.
|
||||
exec 9>"$state_dir/deploy.lock"
|
||||
flock -n 9 || { echo 'Another production deployment is running.' >&2; exit 1; }
|
||||
export DEPLOY_TAG="${DEPLOY_TAG:-$(git rev-parse HEAD)}"
|
||||
compose=(docker compose --env-file /dev/null -p wq-alpha-production -f compose.production.yaml)
|
||||
trap 'rc=$?; "${compose[@]}" --profile jobs ps -a || true; exit "$rc"' EXIT
|
||||
|
||||
"${compose[@]}" config --quiet
|
||||
"${compose[@]}" --profile jobs config --quiet
|
||||
"${compose[@]}" build backend web
|
||||
# Validate secrets and DB connectivity before interrupting the running version.
|
||||
"${compose[@]}" run --rm --no-deps backend python -c '
|
||||
import asyncio
|
||||
from app.config import Settings
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import create_async_engine
|
||||
async def check():
|
||||
settings = Settings()
|
||||
engine = create_async_engine(settings.database_url)
|
||||
try:
|
||||
async with engine.connect() as connection:
|
||||
await connection.execute(text("SELECT 1"))
|
||||
finally:
|
||||
await engine.dispose()
|
||||
try:
|
||||
asyncio.run(check())
|
||||
except Exception:
|
||||
raise SystemExit("Production configuration/database preflight failed; check env and database access.") from None
|
||||
'
|
||||
|
||||
# Record immutable image references before switching; do not prune old images.
|
||||
previous_images=$("${compose[@]}" images --quiet)
|
||||
if [[ -n "$previous_images" ]]; then
|
||||
docker image inspect --format '{{.Id}} {{json .RepoTags}}' $previous_images > "$state_dir/previous-images.txt"
|
||||
fi
|
||||
"${compose[@]}" stop web backend
|
||||
"${compose[@]}" --profile jobs run --rm --no-deps migrate
|
||||
"${compose[@]}" up -d --no-build --remove-orphans --wait --wait-timeout 180 backend web
|
||||
printf '%s\n' "$DEPLOY_TAG" > "$state_dir/current-release.txt"
|
||||
echo "Production is healthy; release $DEPLOY_TAG"
|
||||
@@ -14,6 +14,7 @@ content = "\n".join([
|
||||
f"ADMIN_PASSWORD={secrets.token_urlsafe(24)}",
|
||||
f"POSTGRES_PASSWORD={secrets.token_hex(24)}",
|
||||
f"ENCRYPTION_KEY={base64.urlsafe_b64encode(secrets.token_bytes(32)).decode()}",
|
||||
"WQ_EMAIL=", "WQ_PASSWORD=",
|
||||
"LOCAL_PORT=8080", "DOMAIN=alpha.example.com", "",
|
||||
])
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user