diff --git a/.env.example b/.env.example index 80a0d0e..8ba6cd3 100644 --- a/.env.example +++ b/.env.example @@ -3,5 +3,10 @@ ZHIXING_SERVER_PORT=8000 ZHIXING_WEB_PORT=5173 ZHIXING_APP_ENV=development ZHIXING_LOG_LEVEL=INFO +ZHIXING_POSTGRES_DB=zhixing +ZHIXING_POSTGRES_USER=zhixing +ZHIXING_POSTGRES_PASSWORD=zhixing +ZHIXING_DATABASE_URL=postgresql://zhixing:zhixing@postgres:5432/zhixing +ZHIXING_TUSHARE_TOKEN= +ZHIXING_MARKET_DATA_CSV_ROOT=/app/data/market-data API_UPSTREAM=http://server:8000 - diff --git a/.trellis/spec/backend/index.md b/.trellis/spec/backend/index.md index 2a32c4a..7c454cc 100644 --- a/.trellis/spec/backend/index.md +++ b/.trellis/spec/backend/index.md @@ -8,6 +8,7 @@ | --- | --- | | [目录与模块边界](./directory-structure.md) | 包结构、bounded context 和导入边界 | | [配置与运行时](./configuration-and-runtime.md) | `Settings`、应用工厂和部署环境 | +| [市场数据同步](./market-data-sync.md) | Tushare qfq、PostgreSQL、CSV 快照和一次性 Job 契约 | | [HTTP 契约](./http-api-contracts.md) | 路由组合、响应模型和同源 API 路径 | | [错误处理](./error-handling.md) | 当前 FastAPI 错误行为及跨层错误传递 | | [质量与测试](./quality-guidelines.md) | Ruff、Pyright、pytest 及禁止模式 | diff --git a/.trellis/spec/backend/market-data-sync.md b/.trellis/spec/backend/market-data-sync.md new file mode 100644 index 0000000..a82cb3f --- /dev/null +++ b/.trellis/spec/backend/market-data-sync.md @@ -0,0 +1,79 @@ +# 市场数据同步代码规格 + +## Scenario: Tushare qfq 日线与每日指标一次性同步 + +### 1. Scope / Trigger + +- 触发:新增 `modules/market_data` bounded context、Alembic PostgreSQL schema、Tushare/CSV/Compose Job 集成。 +- 目标:外部调度器每次启动一个进程,向 PostgreSQL 写入当前沪深非 ST A 股的六年 qfq 日线和按交易日归属的 `daily_basic`,并发布可恢复 CSV。 +- 边界:FastAPI 生命周期不启动定时器;本上下文不实现具体选股策略、分钟行情或退市成员资格回溯。 + +### 2. Signatures + +- 应用用例:`SyncMarketData.execute(command: SyncMarketDataCommand | None = None) -> SyncBatchSummary`。 +- CLI:`market-data-sync [--initialize | --retry-batch-id ] [--trade-date YYYY-MM-DD]`。 +- 数据源端口:`fetch_stocks()`、`fetch_open_dates(start, end)`、`fetch_bars(ts_code, window)`、`fetch_daily_basic(trade_date)`。 +- 数据库事实表:`market_stock`、`market_daily_bar(ts_code, trade_date)`、`market_daily_basic(ts_code, trade_date)`;同步审计表:`market_sync_batch`、`market_sync_item`。 + +### 3. Contracts + +- 配置入口只允许 `bootstrap.config.Settings` 读取环境变量: + - `ZHIXING_DATABASE_URL` + - `ZHIXING_TUSHARE_TOKEN` + - `ZHIXING_MARKET_DATA_CSV_ROOT` + - `ZHIXING_MARKET_DATA_COVERAGE_THRESHOLD`,默认 `0.99` + - `ZHIXING_MARKET_DATA_MAX_RETRIES`、`ZHIXING_MARKET_DATA_RETRY_BACKOFF_SECONDS`、`ZHIXING_MARKET_DATA_REQUEST_INTERVAL_SECONDS` + - `ZHIXING_MARKET_DATA_ADVISORY_LOCK_KEY` +- 股票 qfq 快照路径为 `bars/.csv`;每日指标路径为 `daily-basic//.csv`;当前股票主数据为 `stock-basic/current.csv`。 +- `market_daily_bar` 的唯一键是 `(ts_code, trade_date)`,`source_adj` 必须是 `qfq`;所有价格、金额和比率使用有限 `NUMERIC`/`Decimal`。 +- `SyncBatchSummary` 至少返回 `batch_id`、目标交易日、窗口、状态、目标数、有效数、覆盖率、策略资格、插入数、更新数、未变化数和失败列表。 +- 退出码:`0` 表示覆盖率达标的成功批次;`2` 表示部分成功或覆盖率不足;`1` 表示失败或没有可用成功结果。 +- 单股票流程必须保持“临时 CSV → 指纹比较 → PostgreSQL 事务 → `os.replace` 发布”顺序;数据库异常不得替换正式 CSV。 + +### 4. Validation & Error Matrix + +| 条件 | 行为 | +| --- | --- | +| 非 `SH`/`SZ`、非上市、ST/退市名称或北交所股票 | 从当前目标股票池排除 | +| 指定日期不是交易日 | 批次准备失败,不发起行情写入 | +| qfq 快照重复键、空快照、OHLC 非法 | 当前股票 item 失败,旧 CSV 和旧库行保持不变 | +| 重叠区间指纹相同 | 只批量插入旧上界之后的新交易日 | +| 重叠行字段变化、缺失或快照开始边界异常变化 | 当前股票执行六年窗口集合式 upsert/删除 | +| `daily_basic` 没有目标股票行 | 当前指标日期 item 失败,不发布空正式快照 | +| PostgreSQL/COPY/迁移异常 | 转换为安全的 repository error;成功对象不回滚,失败对象保留旧发布版本 | +| 同一环境已有同步锁 | 返回 `failed` 和退出码 `1`,不执行第二个批次 | +| `valid / target < coverage_threshold` | 不触发选股,批次显式标记 `strategy_eligible=false` | + +### 5. Good/Base/Bad Cases + +- Good:相同六年 qfq 返回重复执行,指纹相同,数据库无无意义 update,正式 CSV 可原子替换,覆盖率为 `1`。 +- Base:新增一个开市日且历史重叠不变,只 COPY 新日期;批次成功后清理窗口起点之前的数据库行和日期文件。 +- Bad:一只股票历史 qfq 行变化或数据库事务失败,只修复/失败该股票;其他股票成功结果保留,失败对象正式 CSV 不被替换。 + +### 6. Tests Required + +- 领域单元测试:六年窗口边界、输入顺序稳定性、历史值变化、重叠缺行、窗口提前、ST/北交所过滤;断言 `SnapshotChange` 和股票代码集合。 +- CSV 单元测试:固定表头、临时文件、原子发布、异常/丢弃后旧正式文件仍可读;断言正式文件内容和临时文件清理。 +- 同步编排测试:`success`、覆盖率、幂等重跑、空指标失败、部分失败和失败 item 重试;断言 batch/item 状态与退出码。 +- PostgreSQL 集成测试:设置 `ZHIXING_TEST_DATABASE_URL` 后运行 Alembic upgrade,断言五张业务/审计表、唯一键和迁移头;批量 upsert 测试断言 insert/update/unchanged 计数和事务回滚。 +- 质量门禁:`uv lock --check`、后端 Ruff/Pyright/pytest、前端/root `./dev.sh check` 与 `./dev.sh test`、四种 Compose config。 + +### 7. Wrong vs Correct + +#### Wrong + +```python +# 把最新总市值写入六年前每一根 K 线,产生前视偏差。 +for bar in bars: + bar.total_mv = latest_total_mv +``` + +#### Correct + +```python +# 行情和估值按各自交易日保存;策略只使用同一目标日的两类事实。 +market_daily_bar[(ts_code, trade_date)] = bar +market_daily_basic[(ts_code, trade_date)] = daily_basic +``` + +> **Warning**:CSV 发布不是数据库事务的一部分。只有数据库事务成功后才能执行原子 `os.replace`;发布失败必须记录为 item 失败并依靠幂等重试恢复。 diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 9c72dd4..4511dc2 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -1,18 +1,39 @@ name: ${COMPOSE_PROJECT_NAME:-zhixing-system-dev} services: + postgres: + image: postgres:16-alpine + environment: + POSTGRES_DB: ${ZHIXING_POSTGRES_DB:-zhixing} + POSTGRES_PASSWORD: ${ZHIXING_POSTGRES_PASSWORD:-zhixing} + POSTGRES_USER: ${ZHIXING_POSTGRES_USER:-zhixing} + healthcheck: + test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"] + interval: 5s + timeout: 3s + retries: 10 + volumes: + - postgres-data:/var/lib/postgresql/data + server: build: context: ./zhixing-server target: development + depends_on: + postgres: + condition: service_healthy environment: ZHIXING_APP_ENV: ${ZHIXING_APP_ENV:-development} + ZHIXING_DATABASE_URL: ${ZHIXING_DATABASE_URL:-postgresql://zhixing:zhixing@postgres:5432/zhixing} ZHIXING_LOG_LEVEL: ${ZHIXING_LOG_LEVEL:-INFO} + ZHIXING_MARKET_DATA_CSV_ROOT: /app/data/market-data + ZHIXING_TUSHARE_TOKEN: ${ZHIXING_TUSHARE_TOKEN:-} init: true ports: - "${ZHIXING_SERVER_PORT:-8000}:8000" volumes: - ./zhixing-server:/app + - market-data:/app/data/market-data - server-venv:/app/.venv healthcheck: test: @@ -41,7 +62,41 @@ services: - ./zhixing-web:/app - web-node-modules:/app/node_modules + migrate: + profiles: ["jobs"] + build: + context: ./zhixing-server + target: development + command: ["alembic", "upgrade", "head"] + depends_on: + postgres: + condition: service_healthy + environment: + ZHIXING_DATABASE_URL: ${ZHIXING_DATABASE_URL:-postgresql://zhixing:zhixing@postgres:5432/zhixing} + volumes: + - ./zhixing-server:/app + - server-venv:/app/.venv + + market-sync: + profiles: ["jobs"] + build: + context: ./zhixing-server + target: development + command: ["market-data-sync"] + depends_on: + migrate: + condition: service_completed_successfully + environment: + ZHIXING_DATABASE_URL: ${ZHIXING_DATABASE_URL:-postgresql://zhixing:zhixing@postgres:5432/zhixing} + ZHIXING_MARKET_DATA_CSV_ROOT: /app/data/market-data + ZHIXING_TUSHARE_TOKEN: ${ZHIXING_TUSHARE_TOKEN:-} + volumes: + - ./zhixing-server:/app + - market-data:/app/data/market-data + - server-venv:/app/.venv + volumes: + market-data: + postgres-data: server-venv: web-node-modules: - diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 0f03240..372e815 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -1,14 +1,35 @@ name: ${COMPOSE_PROJECT_NAME:-zhixing-system} services: + postgres: + image: postgres:16-alpine + restart: unless-stopped + environment: + POSTGRES_DB: ${ZHIXING_POSTGRES_DB:-zhixing} + POSTGRES_PASSWORD: ${ZHIXING_POSTGRES_PASSWORD:-zhixing} + POSTGRES_USER: ${ZHIXING_POSTGRES_USER:-zhixing} + healthcheck: + test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"] + interval: 10s + timeout: 5s + retries: 5 + volumes: + - postgres-data:/var/lib/postgresql/data + server: build: context: ./zhixing-server target: production + depends_on: + postgres: + condition: service_healthy restart: unless-stopped environment: ZHIXING_APP_ENV: production + ZHIXING_DATABASE_URL: ${ZHIXING_DATABASE_URL:-postgresql://zhixing:zhixing@postgres:5432/zhixing} ZHIXING_LOG_LEVEL: ${ZHIXING_LOG_LEVEL:-INFO} + ZHIXING_MARKET_DATA_CSV_ROOT: /app/data/market-data + ZHIXING_TUSHARE_TOKEN: ${ZHIXING_TUSHARE_TOKEN:-} init: true expose: - "8000" @@ -22,6 +43,8 @@ services: timeout: 5s retries: 3 start_period: 10s + volumes: + - market-data:/app/data/market-data web: build: @@ -36,3 +59,35 @@ services: init: true ports: - "127.0.0.1:${ZHIXING_WEB_PORT:-8111}:80" + + migrate: + profiles: ["jobs"] + build: + context: ./zhixing-server + target: production + command: ["alembic", "upgrade", "head"] + depends_on: + postgres: + condition: service_healthy + environment: + ZHIXING_DATABASE_URL: ${ZHIXING_DATABASE_URL:-postgresql://zhixing:zhixing@postgres:5432/zhixing} + + market-sync: + profiles: ["jobs"] + build: + context: ./zhixing-server + target: production + command: ["market-data-sync"] + depends_on: + migrate: + condition: service_completed_successfully + environment: + ZHIXING_DATABASE_URL: ${ZHIXING_DATABASE_URL:-postgresql://zhixing:zhixing@postgres:5432/zhixing} + ZHIXING_MARKET_DATA_CSV_ROOT: /app/data/market-data + ZHIXING_TUSHARE_TOKEN: ${ZHIXING_TUSHARE_TOKEN:-} + volumes: + - market-data:/app/data/market-data + +volumes: + market-data: + postgres-data: diff --git a/docs/market-data-sync.md b/docs/market-data-sync.md new file mode 100644 index 0000000..efde67a --- /dev/null +++ b/docs/market-data-sync.md @@ -0,0 +1,53 @@ +# 市场数据同步 Job + +`market-data-sync` 是外部调度器触发的一次性任务。FastAPI 进程不包含定时器;PostgreSQL 是策略查询事实源,`market-data` 卷中的 CSV 只作为 qfq 落地快照和恢复介质。 + +## 首次部署 + +先准备 `.env`,至少设置 `ZHIXING_TUSHARE_TOKEN`、PostgreSQL 凭据和 `ZHIXING_DATABASE_URL`,然后执行迁移: + +```bash +docker compose -f docker-compose.prod.yml --profile jobs run --rm migrate +``` + +首次初始化会回补六年交易日指标,并为当前沪深非 ST A 股获取六年 qfq 行情: + +```bash +docker compose -f docker-compose.prod.yml --profile jobs run --rm market-sync --initialize +``` + +日常收盘后同步默认解析最近的开市日;也可以显式指定日期: + +```bash +docker compose -f docker-compose.prod.yml --profile jobs run --rm market-sync --trade-date 2026-08-05 +``` + +`success` 返回 0;覆盖率低于配置阈值或出现部分失败返回 2;没有可用成功结果或基础设施失败返回 1。CLI 输出不包含 Tushare token 或数据库密码。 + +## Cron 与重试 + +宿主机 cron 只负责启动临时容器,不写入容器内部的 crontab。下面的示例每天工作日 18:00 触发;交易日历、唯一约束和 PostgreSQL advisory lock 使周末、节假日、重复触发和重叠触发保持安全: + +```cron +0 18 * * 1-5 cd /srv/zhixing-system && docker compose -f docker-compose.prod.yml --profile jobs run --rm market-sync >> /var/log/zhixing-market-sync.log 2>&1 +``` + +部分成功批次的输出会包含 `batch_id`。修复凭据、网络或数据库问题后,只重试该批次失败的股票和指标日期: + +```bash +docker compose -f docker-compose.prod.yml --profile jobs run --rm market-sync \ + --retry-batch-id +``` + +单只股票按数据库事务和 CSV 发布作为恢复边界。数据库提交失败时正式 CSV 不会替换;CSV 发布失败时数据库写入保持幂等,重试可以再次发布。成功批次完成后,数据库事实表和正式日期快照清理六年窗口起点以前的数据;失败对象保留上一次成功版本。 + +## 本地检查 + +```bash +docker compose -f docker-compose.dev.yml config +docker compose -f docker-compose.dev.yml --profile jobs config +docker compose -f docker-compose.prod.yml config +docker compose -f docker-compose.prod.yml --profile jobs config +``` + +真实 PostgreSQL 迁移和批量 upsert 集成测试使用 `ZHIXING_TEST_DATABASE_URL` 显式开启;普通单元测试不会访问网络、Tushare 或数据库。 diff --git a/zhixing-server/Dockerfile b/zhixing-server/Dockerfile index b014685..8593aae 100644 --- a/zhixing-server/Dockerfile +++ b/zhixing-server/Dockerfile @@ -25,6 +25,8 @@ COPY pyproject.toml uv.lock ./ RUN uv sync --frozen --no-dev --no-install-project COPY README.md ./ COPY src ./src +COPY alembic.ini ./ +COPY migrations ./migrations RUN uv sync --frozen --no-dev RUN useradd --create-home --uid 10001 appuser diff --git a/zhixing-server/alembic.ini b/zhixing-server/alembic.ini new file mode 100644 index 0000000..6dde0bb --- /dev/null +++ b/zhixing-server/alembic.ini @@ -0,0 +1,37 @@ +[alembic] +script_location = migrations +prepend_sys_path = . +sqlalchemy.url = postgresql://unused + +[loggers] +keys = root,sqlalchemy,alembic + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARN +handlers = console + +[logger_sqlalchemy] +level = WARN +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s +datefmt = %H:%M:%S diff --git a/zhixing-server/migrations/__init__.py b/zhixing-server/migrations/__init__.py new file mode 100644 index 0000000..9eb2c04 --- /dev/null +++ b/zhixing-server/migrations/__init__.py @@ -0,0 +1 @@ +"""Alembic migration package for Zhixing Server.""" diff --git a/zhixing-server/migrations/env.py b/zhixing-server/migrations/env.py new file mode 100644 index 0000000..1b36969 --- /dev/null +++ b/zhixing-server/migrations/env.py @@ -0,0 +1,51 @@ +"""Alembic environment using the server's single Settings entry point.""" + +from logging.config import fileConfig + +from alembic import context +from sqlalchemy import engine_from_config, pool + +from zhixing_server.bootstrap.config import get_settings +from zhixing_server.modules.market_data.infrastructure.schema import metadata + +config = context.config +settings = get_settings() +config.set_main_option("sqlalchemy.url", settings.database_url.replace("%", "%%")) +if config.config_file_name is not None: + fileConfig(config.config_file_name) + +target_metadata = metadata + + +def run_migrations_offline() -> None: + """Run migrations without opening a database connection.""" + + context.configure( + url=settings.database_url, + target_metadata=target_metadata, + literal_binds=True, + dialect_opts={"paramstyle": "named"}, + compare_type=True, + ) + with context.begin_transaction(): + context.run_migrations() + + +def run_migrations_online() -> None: + """Run migrations against the configured PostgreSQL connection.""" + + connectable = engine_from_config( + config.get_section(config.config_ini_section, {}), + prefix="sqlalchemy.", + poolclass=pool.NullPool, + ) + with connectable.connect() as connection: + context.configure(connection=connection, target_metadata=target_metadata, compare_type=True) + with context.begin_transaction(): + context.run_migrations() + + +if context.is_offline_mode(): + run_migrations_offline() +else: + run_migrations_online() diff --git a/zhixing-server/migrations/versions/0001_market_data.py b/zhixing-server/migrations/versions/0001_market_data.py new file mode 100644 index 0000000..7020607 --- /dev/null +++ b/zhixing-server/migrations/versions/0001_market_data.py @@ -0,0 +1,140 @@ +"""Create the market-data bounded-context schema.""" + +from collections.abc import Sequence + +import sqlalchemy as sa +from alembic import op + +revision: str = "0001_market_data" +down_revision: str | None = None +branch_labels: str | Sequence[str] | None = None +depends_on: str | Sequence[str] | None = None + + +def upgrade() -> None: + """Create market master, facts, and auditable synchronization tables.""" + + op.create_table( + "market_stock", + sa.Column("ts_code", sa.String(12), primary_key=True), + sa.Column("name", sa.String(128), nullable=False), + sa.Column("market", sa.String(32), nullable=False), + sa.Column("exchange", sa.String(16), nullable=False), + sa.Column("list_status", sa.String(2), nullable=False), + sa.Column("list_date", sa.Date()), + sa.Column("is_active", sa.Boolean(), nullable=False, server_default=sa.text("true")), + sa.Column( + "updated_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.text("now()"), + ), + ) + op.create_table( + "market_daily_bar", + sa.Column("ts_code", sa.String(12), nullable=False), + sa.Column("trade_date", sa.Date(), nullable=False), + *[ + sa.Column(field, sa.Numeric(20, 6)) + for field in ("open", "high", "low", "close", "pre_close", "change", "pct_chg") + ], + sa.Column("vol", sa.Numeric(24, 6)), + sa.Column("amount", sa.Numeric(24, 6)), + sa.Column("source_adj", sa.String(8), nullable=False, server_default="qfq"), + sa.Column( + "updated_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.text("now()"), + ), + sa.PrimaryKeyConstraint("ts_code", "trade_date"), + ) + op.create_index("ix_market_daily_bar_trade_date", "market_daily_bar", ["trade_date"]) + op.create_table( + "market_daily_basic", + sa.Column("ts_code", sa.String(12), nullable=False), + sa.Column("trade_date", sa.Date(), nullable=False), + *[ + sa.Column(field, sa.Numeric(24, 6)) + for field in ( + "close", + "turnover_rate", + "turnover_rate_f", + "volume_ratio", + "pe", + "pe_ttm", + "pb", + "ps", + "ps_ttm", + "dv_ratio", + "dv_ttm", + "total_share", + "float_share", + "free_share", + "total_mv", + "circ_mv", + ) + ], + sa.Column( + "updated_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.text("now()"), + ), + sa.PrimaryKeyConstraint("ts_code", "trade_date"), + ) + op.create_index("ix_market_daily_basic_trade_date", "market_daily_basic", ["trade_date"]) + op.create_table( + "market_sync_batch", + sa.Column("id", sa.String(36), primary_key=True), + sa.Column("target_trade_date", sa.Date(), nullable=False), + sa.Column("window_start", sa.Date(), nullable=False), + sa.Column("mode", sa.String(16), nullable=False), + sa.Column("status", sa.String(24), nullable=False), + sa.Column("target_count", sa.Integer(), nullable=False), + sa.Column("valid_count", sa.Integer(), nullable=False, server_default="0"), + sa.Column("coverage", sa.Numeric(8, 6), nullable=False, server_default="0"), + sa.Column("strategy_eligible", sa.Boolean(), nullable=False, server_default="false"), + sa.Column("parent_batch_id", sa.String(36)), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.text("now()"), + ), + sa.Column("finished_at", sa.DateTime(timezone=True)), + ) + op.create_table( + "market_sync_item", + sa.Column("batch_id", sa.String(36), nullable=False), + sa.Column("item_kind", sa.String(24), nullable=False), + sa.Column("item_key", sa.String(64), nullable=False), + sa.Column("status", sa.String(24), nullable=False), + sa.Column("inserted_count", sa.Integer(), nullable=False, server_default="0"), + sa.Column("updated_count", sa.Integer(), nullable=False, server_default="0"), + sa.Column("unchanged_count", sa.Integer(), nullable=False, server_default="0"), + sa.Column("fingerprint", sa.String(64)), + sa.Column("error_type", sa.String(64)), + sa.Column("error_message", sa.Text()), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.text("now()"), + ), + sa.PrimaryKeyConstraint("batch_id", "item_kind", "item_key"), + ) + op.create_index("ix_market_sync_item_status", "market_sync_item", ["batch_id", "status"]) + + +def downgrade() -> None: + """Drop market-data tables in dependency-safe order.""" + + op.drop_index("ix_market_sync_item_status", table_name="market_sync_item") + op.drop_table("market_sync_item") + op.drop_table("market_sync_batch") + op.drop_index("ix_market_daily_basic_trade_date", table_name="market_daily_basic") + op.drop_table("market_daily_basic") + op.drop_index("ix_market_daily_bar_trade_date", table_name="market_daily_bar") + op.drop_table("market_daily_bar") + op.drop_table("market_stock") diff --git a/zhixing-server/pyproject.toml b/zhixing-server/pyproject.toml index 0d7c606..8df7d40 100644 --- a/zhixing-server/pyproject.toml +++ b/zhixing-server/pyproject.toml @@ -5,8 +5,12 @@ description = "FastAPI backend and quantitative processing foundation for Zhixin readme = "README.md" requires-python = ">=3.12,<3.13" dependencies = [ + "alembic>=1.18.0", "fastapi>=0.141.1", + "psycopg[binary]>=3.3.2", "pydantic-settings>=2.14.2", + "sqlalchemy>=2.0.46", + "tushare>=1.4.24", "uvicorn[standard]>=0.52.1", ] @@ -26,13 +30,18 @@ build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["src/zhixing_server"] +[project.scripts] +market-data-sync = "zhixing_server.modules.market_data.presentation.cli:main" + [tool.pytest.ini_options] addopts = "-ra --strict-config --strict-markers" pythonpath = ["src"] testpaths = ["tests"] +markers = ["integration: requires a PostgreSQL test database"] [tool.pyright] include = ["src", "tests"] +extraPaths = ["src"] pythonVersion = "3.12" typeCheckingMode = "strict" venvPath = "." @@ -47,4 +56,3 @@ select = ["B", "E", "F", "I", "SIM", "UP"] [tool.ruff.format] quote-style = "double" - diff --git a/zhixing-server/src/zhixing_server/bootstrap/config.py b/zhixing-server/src/zhixing_server/bootstrap/config.py index 7c44779..527b8ba 100644 --- a/zhixing-server/src/zhixing_server/bootstrap/config.py +++ b/zhixing-server/src/zhixing_server/bootstrap/config.py @@ -1,6 +1,8 @@ """Environment-backed application configuration.""" +from decimal import Decimal from functools import lru_cache +from pathlib import Path from typing import Literal from pydantic_settings import BaseSettings, SettingsConfigDict @@ -12,6 +14,15 @@ class Settings(BaseSettings): app_name: str = "Zhixing Server" app_env: Literal["development", "test", "production"] = "development" log_level: str = "INFO" + database_url: str = "postgresql://zhixing:zhixing@localhost:5432/zhixing" + tushare_token: str = "" + market_data_csv_root: Path = Path("./data/market-data") + market_data_coverage_threshold: Decimal = Decimal("0.99") + market_data_max_workers: int = 4 + market_data_request_interval_seconds: float = 0.2 + market_data_max_retries: int = 3 + market_data_retry_backoff_seconds: float = 1.0 + market_data_advisory_lock_key: int = 7_380_521 model_config = SettingsConfigDict( env_file=".env", diff --git a/zhixing-server/src/zhixing_server/modules/market_data/__init__.py b/zhixing-server/src/zhixing_server/modules/market_data/__init__.py new file mode 100644 index 0000000..b19368d --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/__init__.py @@ -0,0 +1,5 @@ +"""Market data bounded context. + +The context owns the current A-share universe, daily qfq bars, daily basic +metrics, and the externally-triggered synchronization use case. +""" diff --git a/zhixing-server/src/zhixing_server/modules/market_data/application/__init__.py b/zhixing-server/src/zhixing_server/modules/market_data/application/__init__.py new file mode 100644 index 0000000..924b661 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/application/__init__.py @@ -0,0 +1,5 @@ +"""Market data synchronization use cases.""" + +from .sync import SyncBatchSummary, SyncMarketData, SyncMarketDataCommand + +__all__ = ["SyncBatchSummary", "SyncMarketData", "SyncMarketDataCommand"] diff --git a/zhixing-server/src/zhixing_server/modules/market_data/application/sync.py b/zhixing-server/src/zhixing_server/modules/market_data/application/sync.py new file mode 100644 index 0000000..c143a18 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/application/sync.py @@ -0,0 +1,403 @@ +"""One-shot market data synchronization orchestration.""" + +from __future__ import annotations + +from collections.abc import Iterable, Sequence +from dataclasses import dataclass, field +from datetime import date, timedelta +from decimal import Decimal +from typing import Literal + +from ..domain.fingerprint import SnapshotChange, compare_snapshots +from ..domain.models import Bar, Stock, SyncWindow +from ..domain.ports import MarketDataRepository, MarketDataSource, SnapshotStore, WriteResult +from ..domain.rules import filter_current_hs_a_stocks + +SyncMode = Literal["daily", "initialize", "retry"] + + +@dataclass(frozen=True, slots=True) +class SyncMarketDataCommand: + """Input for a daily, initial, or failed-item retry synchronization.""" + + mode: SyncMode = "daily" + target_trade_date: date | None = None + parent_batch_id: str | None = None + + +@dataclass(frozen=True, slots=True) +class SyncFailure: + """A redacted, actionable item failure.""" + + item_kind: str + item_key: str + error_type: str + message: str + + +@dataclass(frozen=True, slots=True) +class SyncBatchSummary: + """Stable output contract for CLI, cron, and later strategy callers.""" + + batch_id: str | None + target_trade_date: date | None + window: SyncWindow | None + status: str + target_count: int + valid_count: int + coverage: Decimal + strategy_eligible: bool + inserted_count: int = 0 + updated_count: int = 0 + unchanged_count: int = 0 + failures: tuple[SyncFailure, ...] = field(default_factory=tuple) + + @property + def exit_code(self) -> int: + """Return a cron-friendly code: success, incomplete, or infrastructure failure.""" + + if self.status == "failed": + return 1 + return 0 if self.strategy_eligible else 2 + + def as_dict(self) -> dict[str, object]: + """Serialize the summary without credentials or raw vendor responses.""" + + return { + "batch_id": self.batch_id, + "target_trade_date": self.target_trade_date.isoformat() + if self.target_trade_date + else None, + "window_start": self.window.start.isoformat() if self.window else None, + "window_end": self.window.end.isoformat() if self.window else None, + "status": self.status, + "target_count": self.target_count, + "valid_count": self.valid_count, + "coverage": str(self.coverage), + "strategy_eligible": self.strategy_eligible, + "inserted_count": self.inserted_count, + "updated_count": self.updated_count, + "unchanged_count": self.unchanged_count, + "failures": [ + { + "item_kind": failure.item_kind, + "item_key": failure.item_key, + "error_type": failure.error_type, + "message": failure.message, + } + for failure in self.failures + ], + } + + +class SyncMarketData: + """Coordinate source, snapshot, and repository ports at item boundaries.""" + + def __init__( + self, + source: MarketDataSource, + snapshots: SnapshotStore, + repository: MarketDataRepository, + *, + coverage_threshold: Decimal = Decimal("0.99"), + lock_key: int = 7_380_521, + today: date | None = None, + ) -> None: + if not Decimal("0") <= coverage_threshold <= Decimal("1"): + raise ValueError("coverage_threshold must be between 0 and 1") + self.source = source + self.snapshots = snapshots + self.repository = repository + self.coverage_threshold = coverage_threshold + self.lock_key = lock_key + self.today = today or date.today() + + def execute(self, command: SyncMarketDataCommand | None = None) -> SyncBatchSummary: + """Run one synchronization and retain successful items on partial failure.""" + + command = command or SyncMarketDataCommand() + with self.repository.advisory_lock(self.lock_key) as acquired: + if not acquired: + return SyncBatchSummary( + batch_id=None, + target_trade_date=None, + window=None, + status="failed", + target_count=0, + valid_count=0, + coverage=Decimal("0"), + strategy_eligible=False, + failures=( + SyncFailure("batch", "lock", "sync_locked", "another sync is running"), + ), + ) + return self._execute_locked(command) + + def _execute_locked(self, command: SyncMarketDataCommand) -> SyncBatchSummary: + target_trade_date = self._resolve_target(command.target_trade_date) + window = SyncWindow.from_target(target_trade_date) + all_stocks = filter_current_hs_a_stocks(self.source.fetch_stocks()) + if not all_stocks: + return SyncBatchSummary( + None, + target_trade_date, + window, + "failed", + 0, + 0, + Decimal("0"), + False, + failures=( + SyncFailure("stock", "universe", "empty_universe", "no eligible stocks"), + ), + ) + batch_id = self.repository.create_batch( + target_trade_date, + window, + command.mode, + command.parent_batch_id, + len(all_stocks), + ) + failures: list[SyncFailure] = [] + totals = [0, 0, 0] + stock_codes = {stock.ts_code for stock in all_stocks} + retry_items: set[tuple[str, str]] = ( + self._retry_items(command.parent_batch_id) if command.mode == "retry" else set() + ) + + self._process_stock_master(batch_id, all_stocks, failures, totals) + dates = self._dates_to_process(window, target_trade_date, command.mode, retry_items) + for trade_date in dates: + if ( + command.mode == "retry" + and ("daily_basic", trade_date.isoformat()) not in retry_items + ): + continue + self._process_daily_basic(batch_id, trade_date, stock_codes, window, failures, totals) + for stock in all_stocks: + if command.mode == "retry" and ("bar", stock.ts_code) not in retry_items: + continue + self._process_bar(batch_id, stock, window, failures, totals) + + if not failures: + try: + self.repository.purge_before(window) + self.snapshots.clean_daily_basic_before(window.start) + except (OSError, RuntimeError, TypeError, ValueError) as exc: + failure = self._failure("batch", "retention", exc) + failures.append(failure) + self.repository.record_item( + batch_id, + "batch", + "retention", + "failed", + WriteResult(), + error_type=failure.error_type, + error_message=failure.message, + ) + valid_count = sum( + 1 + for stock in all_stocks + if self.repository.has_bar(stock.ts_code, target_trade_date) + and self.repository.has_daily_basic(stock.ts_code, target_trade_date) + ) + coverage = Decimal(valid_count) / Decimal(len(all_stocks)) + status = "success" if not failures else "partial_success" if valid_count else "failed" + eligible = coverage >= self.coverage_threshold + self.repository.record_batch(batch_id, status, valid_count, coverage, eligible) + return SyncBatchSummary( + batch_id, + target_trade_date, + window, + status, + len(all_stocks), + valid_count, + coverage, + eligible, + totals[0], + totals[1], + totals[2], + tuple(failures), + ) + + def _resolve_target(self, requested: date | None) -> date: + if requested is not None: + open_dates = self.source.fetch_open_dates(requested, requested) + if requested not in open_dates: + raise ValueError(f"target date is not an open trading day: {requested}") + return requested + start = self.today - timedelta(days=14) + open_dates = self.source.fetch_open_dates(start, self.today) + if not open_dates: + raise ValueError("no open trading day found before today") + return max(open_dates) + + def _dates_to_process( + self, + window: SyncWindow, + target: date, + mode: SyncMode, + retry_items: set[tuple[str, str]], + ) -> tuple[date, ...]: + if mode == "initialize": + dates = self.source.fetch_open_dates(window.start, target) + return tuple(sorted(set(dates))) + if mode == "retry": + return tuple( + sorted( + date.fromisoformat(key) for kind, key in retry_items if kind == "daily_basic" + ) + ) + return (target,) + + def _retry_items(self, parent_batch_id: str | None) -> set[tuple[str, str]]: + if not parent_batch_id: + raise ValueError("retry mode requires parent_batch_id") + return set(self.repository.failed_items(parent_batch_id)) + + def _process_stock_master( + self, + batch_id: str, + stocks: Sequence[Stock], + failures: list[SyncFailure], + totals: list[int], + ) -> None: + staged = None + try: + staged = self.snapshots.stage_stocks(stocks) + result = self.repository.upsert_stocks(stocks) + self.snapshots.publish(staged) + totals[0] += result.inserted + totals[1] += result.updated + totals[2] += result.unchanged + self.repository.record_item( + batch_id, + "stock", + "current", + "success", + result, + staged.fingerprint, + ) + except (OSError, RuntimeError, TypeError, ValueError) as exc: + if staged is not None: + self.snapshots.discard(staged) + failure = self._failure("stock", "current", exc) + failures.append(failure) + self.repository.record_item( + batch_id, + "stock", + "current", + "failed", + WriteResult(), + error_type=failure.error_type, + error_message=failure.message, + ) + + def _process_daily_basic( + self, + batch_id: str, + trade_date: date, + stock_codes: set[str], + window: SyncWindow, + failures: list[SyncFailure], + totals: list[int], + ) -> None: + staged = None + key = trade_date.isoformat() + try: + rows = tuple( + row + for row in self.source.fetch_daily_basic(trade_date) + if row.ts_code in stock_codes + ) + if not rows: + raise ValueError(f"daily-basic returned no target rows for {trade_date}") + staged = self.snapshots.stage_daily_basic(trade_date, rows) + result = self.repository.upsert_daily_basic(rows, window) + self.snapshots.publish(staged) + self._add_counts(totals, result) + self.repository.record_item( + batch_id, + "daily_basic", + key, + "success", + result, + staged.fingerprint, + ) + except (OSError, RuntimeError, TypeError, ValueError) as exc: + if staged is not None: + self.snapshots.discard(staged) + failure = self._failure("daily_basic", key, exc) + failures.append(failure) + self.repository.record_item( + batch_id, + "daily_basic", + key, + "failed", + WriteResult(), + error_type=failure.error_type, + error_message=failure.message, + ) + + def _process_bar( + self, + batch_id: str, + stock: Stock, + window: SyncWindow, + failures: list[SyncFailure], + totals: list[int], + ) -> None: + staged = None + try: + rows = tuple(self.source.fetch_bars(stock.ts_code, window)) + old_rows = self.snapshots.read_bars(stock.ts_code) + comparison = compare_snapshots(old_rows, rows) + staged = self.snapshots.stage_bars(stock.ts_code, rows) + if comparison.change is SnapshotChange.UNCHANGED: + result = WriteResult(unchanged=len(rows)) + else: + to_write: Iterable[Bar] = rows + if comparison.change is SnapshotChange.NEW_DATES: + old_end = max((row.trade_date for row in old_rows or ()), default=window.start) + to_write = tuple(row for row in rows if row.trade_date > old_end) + result = self.repository.upsert_bars( + to_write, + window, + full_snapshot=comparison.change + in {SnapshotChange.INITIAL, SnapshotChange.CHANGED}, + ) + self.snapshots.publish(staged) + self._add_counts(totals, result) + self.repository.record_item( + batch_id, + "bar", + stock.ts_code, + "success", + result, + staged.fingerprint, + ) + except (OSError, RuntimeError, TypeError, ValueError) as exc: + if staged is not None: + self.snapshots.discard(staged) + failure = self._failure("bar", stock.ts_code, exc) + failures.append(failure) + self.repository.record_item( + batch_id, + "bar", + stock.ts_code, + "failed", + WriteResult(), + error_type=failure.error_type, + error_message=failure.message, + ) + + @staticmethod + def _add_counts(totals: list[int], result: WriteResult) -> None: + totals[0] += result.inserted + totals[1] += result.updated + totals[2] += result.unchanged + + @staticmethod + def _failure(item_kind: str, item_key: str, error: BaseException) -> SyncFailure: + message = " ".join(str(error).split())[:500] or "synchronization item failed" + return SyncFailure(item_kind, item_key, type(error).__name__, message) diff --git a/zhixing-server/src/zhixing_server/modules/market_data/domain/__init__.py b/zhixing-server/src/zhixing_server/modules/market_data/domain/__init__.py new file mode 100644 index 0000000..b405f83 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/domain/__init__.py @@ -0,0 +1,18 @@ +"""Pure market data domain types and ports.""" + +from .fingerprint import SnapshotChange, SnapshotComparison, compare_snapshots, snapshot_fingerprint +from .models import Bar, DailyBasic, Stock, SyncWindow +from .rules import filter_current_hs_a_stocks, is_current_hs_a_stock + +__all__ = [ + "Bar", + "DailyBasic", + "SnapshotChange", + "SnapshotComparison", + "Stock", + "SyncWindow", + "compare_snapshots", + "filter_current_hs_a_stocks", + "is_current_hs_a_stock", + "snapshot_fingerprint", +] diff --git a/zhixing-server/src/zhixing_server/modules/market_data/domain/fingerprint.py b/zhixing-server/src/zhixing_server/modules/market_data/domain/fingerprint.py new file mode 100644 index 0000000..cb88ed5 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/domain/fingerprint.py @@ -0,0 +1,150 @@ +"""Stable qfq snapshot fingerprinting and change classification.""" + +from __future__ import annotations + +from collections.abc import Iterable +from dataclasses import dataclass +from datetime import date +from enum import StrEnum +from hashlib import sha256 + +from .models import Bar, decimal_text + + +class SnapshotChange(StrEnum): + """Action required after comparing a previous and current bar snapshot.""" + + INITIAL = "initial" + UNCHANGED = "unchanged" + NEW_DATES = "new_dates" + CHANGED = "changed" + + +@dataclass(frozen=True, slots=True) +class SnapshotComparison: + """Result of comparing normalized snapshots for one stock.""" + + change: SnapshotChange + old_fingerprint: str | None + new_fingerprint: str + new_dates: tuple[date, ...] + + +FINGERPRINT_FIELDS = ( + "trade_date", + "open", + "high", + "low", + "close", + "pre_close", + "change", + "pct_chg", + "vol", + "amount", +) + + +def _validate(rows: Iterable[Bar]) -> tuple[Bar, ...]: + """Sort rows and reject duplicate identity keys before hashing or writing.""" + + sorted_rows = tuple(sorted(rows, key=lambda row: (row.ts_code, row.trade_date))) + seen: set[tuple[str, date]] = set() + for row in sorted_rows: + key = (row.ts_code, row.trade_date) + if not row.ts_code or key in seen: + raise ValueError(f"duplicate or empty bar key: {key!r}") + seen.add(key) + for value in (row.open, row.high, row.low, row.close): + if value is not None and value < 0: + raise ValueError(f"OHLC values must be non-negative: {key!r}") + if row.high is not None and row.low is not None and row.high < row.low: + raise ValueError(f"high must not be lower than low: {key!r}") + return sorted_rows + + +def _canonical(row: Bar) -> str: + values = ( + row.trade_date.isoformat(), + decimal_text(row.open), + decimal_text(row.high), + decimal_text(row.low), + decimal_text(row.close), + decimal_text(row.pre_close), + decimal_text(row.change), + decimal_text(row.pct_chg), + decimal_text(row.vol), + decimal_text(row.amount), + ) + return ",".join(values) + + +def snapshot_fingerprint(rows: Iterable[Bar]) -> str: + """Hash sorted, normalized fixed bar columns, excluding runtime metadata.""" + + normalized = _validate(rows) + payload = "\n".join(_canonical(row) for row in normalized).encode("utf-8") + return sha256(payload).hexdigest() + + +def _overlap_fingerprint(rows: dict[date, Bar], dates: set[date]) -> str: + payload = "\n".join(_canonical(rows[day]) for day in sorted(dates)).encode("utf-8") + return sha256(payload).hexdigest() + + +def compare_snapshots( + old_rows: Iterable[Bar] | None, + new_rows: Iterable[Bar], +) -> SnapshotComparison: + """Compare a complete old/new six-year snapshot. + + A shifted rolling start is expected and remains unchanged. Missing rows + or changed values inside the common historical range are repairs, while + rows strictly after the old upper bound are ordinary new dates. + """ + + current = _validate(new_rows) + if not current: + raise ValueError("a bar snapshot must contain at least one row") + new_by_date = {row.trade_date: row for row in current} + new_dates = set(new_by_date) + new_fingerprint = snapshot_fingerprint(current) + if old_rows is None: + return SnapshotComparison( + SnapshotChange.INITIAL, + None, + new_fingerprint, + tuple(sorted(new_dates)), + ) + + previous = _validate(old_rows) + if not previous: + return SnapshotComparison( + SnapshotChange.INITIAL, + None, + new_fingerprint, + tuple(sorted(new_dates)), + ) + old_by_date = {row.trade_date: row for row in previous} + old_dates = set(old_by_date) + overlap_start = max(min(old_dates), min(new_dates)) + overlap_end = min(max(old_dates), max(new_dates)) + overlap = {day for day in new_dates if overlap_start <= day <= overlap_end} + old_overlap = {day for day in old_dates if overlap_start <= day <= overlap_end} + old_fp = _overlap_fingerprint(old_by_date, old_overlap) + new_fp = _overlap_fingerprint(new_by_date, overlap) + + starts_earlier = min(new_dates) < min(old_dates) + changed = ( + old_overlap != overlap + or old_fp != new_fp + or max(new_dates) < max(old_dates) + or starts_earlier + ) + if changed: + kind = SnapshotChange.CHANGED + elif max(new_dates) > max(old_dates): + kind = SnapshotChange.NEW_DATES + else: + kind = SnapshotChange.UNCHANGED + additions = tuple(sorted(day for day in new_dates if day > max(old_dates))) + return SnapshotComparison(kind, old_fp, new_fp, additions) diff --git a/zhixing-server/src/zhixing_server/modules/market_data/domain/models.py b/zhixing-server/src/zhixing_server/modules/market_data/domain/models.py new file mode 100644 index 0000000..7cb38ff --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/domain/models.py @@ -0,0 +1,199 @@ +"""Market data entities and stable scalar normalization helpers.""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass +from datetime import date +from decimal import Decimal, InvalidOperation + + +def parse_date(value: object) -> date: + """Parse a Tushare date value without accepting locale-dependent formats.""" + + if isinstance(value, date): + return value + text = str(value).strip() + if len(text) == 8 and text.isdigit(): + return date.fromisoformat(f"{text[:4]}-{text[4:6]}-{text[6:]}") + return date.fromisoformat(text[:10]) + + +def normalize_decimal(value: object | None) -> Decimal | None: + """Convert vendor numbers to finite, text-stable decimals. + + Tushare may return a float, a decimal, a string, or ``None`` depending on + the transport. Decimal constructed from the textual representation keeps + those variants from changing snapshot fingerprints. + """ + + if value is None or str(value).strip() == "": + return None + try: + result = Decimal(str(value).strip()) + except (InvalidOperation, ValueError) as exc: + raise ValueError(f"invalid numeric value: {value!r}") from exc + if not result.is_finite(): + raise ValueError(f"numeric value must be finite: {value!r}") + return Decimal(0) if result == 0 else result.normalize() + + +def decimal_text(value: Decimal | None) -> str: + """Serialize a decimal in a locale-independent, non-exponent form.""" + + if value is None: + return "" + text = format(value, "f") + if "." in text: + text = text.rstrip("0").rstrip(".") + return text or "0" + + +def _text(row: Mapping[str, object], key: str, default: str = "") -> str: + value = row.get(key, default) + return default if value is None else str(value).strip() + + +@dataclass(frozen=True, slots=True) +class Stock: + """Current stock master data used to construct the target universe.""" + + ts_code: str + name: str + market: str = "" + exchange: str = "" + list_status: str = "L" + list_date: date | None = None + + @classmethod + def from_mapping(cls, row: Mapping[str, object]) -> Stock: + """Build a stock from a ``stock_basic`` row.""" + + raw_date = row.get("list_date") + return cls( + ts_code=_text(row, "ts_code"), + name=_text(row, "name"), + market=_text(row, "market"), + exchange=_text(row, "exchange"), + list_status=_text(row, "list_status", "L").upper(), + list_date=parse_date(raw_date) if raw_date else None, + ) + + +@dataclass(frozen=True, slots=True) +class Bar: + """One qfq daily OHLCV record.""" + + ts_code: str + trade_date: date + open: Decimal | None = None + high: Decimal | None = None + low: Decimal | None = None + close: Decimal | None = None + pre_close: Decimal | None = None + change: Decimal | None = None + pct_chg: Decimal | None = None + vol: Decimal | None = None + amount: Decimal | None = None + + @classmethod + def from_mapping(cls, row: Mapping[str, object]) -> Bar: + """Build a normalized bar from a Tushare ``pro_bar`` row.""" + + return cls( + ts_code=_text(row, "ts_code"), + trade_date=parse_date(row["trade_date"]), + open=normalize_decimal(row.get("open")), + high=normalize_decimal(row.get("high")), + low=normalize_decimal(row.get("low")), + close=normalize_decimal(row.get("close")), + pre_close=normalize_decimal(row.get("pre_close")), + change=normalize_decimal(row.get("change")), + pct_chg=normalize_decimal(row.get("pct_chg")), + vol=normalize_decimal(row.get("vol")), + amount=normalize_decimal(row.get("amount")), + ) + + +@dataclass(frozen=True, slots=True) +class DailyBasic: + """One trading-day valuation and liquidity snapshot.""" + + ts_code: str + trade_date: date + close: Decimal | None = None + turnover_rate: Decimal | None = None + turnover_rate_f: Decimal | None = None + volume_ratio: Decimal | None = None + pe: Decimal | None = None + pe_ttm: Decimal | None = None + pb: Decimal | None = None + ps: Decimal | None = None + ps_ttm: Decimal | None = None + dv_ratio: Decimal | None = None + dv_ttm: Decimal | None = None + total_share: Decimal | None = None + float_share: Decimal | None = None + free_share: Decimal | None = None + total_mv: Decimal | None = None + circ_mv: Decimal | None = None + + @classmethod + def from_mapping(cls, row: Mapping[str, object]) -> DailyBasic: + """Build a normalized row from a Tushare ``daily_basic`` response.""" + + decimal_fields = ( + "close", + "turnover_rate", + "turnover_rate_f", + "volume_ratio", + "pe", + "pe_ttm", + "pb", + "ps", + "ps_ttm", + "dv_ratio", + "dv_ttm", + "total_share", + "float_share", + "free_share", + "total_mv", + "circ_mv", + ) + values = {field: normalize_decimal(row.get(field)) for field in decimal_fields} + return cls( + ts_code=_text(row, "ts_code"), + trade_date=parse_date(row["trade_date"]), + **values, + ) + + +@dataclass(frozen=True, slots=True) +class SyncWindow: + """Inclusive rolling window used by both the database and CSV retention.""" + + start: date + end: date + + @classmethod + def from_target(cls, target_trade_date: date, years: int = 6) -> SyncWindow: + """Return a six-calendar-year window including the target date.""" + + if years < 1: + raise ValueError("years must be positive") + try: + start = target_trade_date.replace(year=target_trade_date.year - years) + except ValueError: + # 29 February has no counterpart in a non-leap year. The last + # day of February is the deterministic calendar boundary. + start = target_trade_date.replace( + year=target_trade_date.year - years, + month=2, + day=28, + ) + return cls(start=start, end=target_trade_date) + + def contains(self, value: date) -> bool: + """Return whether a date is within the inclusive window.""" + + return self.start <= value <= self.end diff --git a/zhixing-server/src/zhixing_server/modules/market_data/domain/ports.py b/zhixing-server/src/zhixing_server/modules/market_data/domain/ports.py new file mode 100644 index 0000000..0b2eb28 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/domain/ports.py @@ -0,0 +1,126 @@ +"""Ports used by the market data application layer.""" + +from __future__ import annotations + +from collections.abc import Iterable, Sequence +from contextlib import AbstractContextManager +from dataclasses import dataclass +from datetime import date +from decimal import Decimal +from pathlib import Path +from typing import Protocol + +from .models import Bar, DailyBasic, Stock, SyncWindow + + +class MarketDataRepositoryError(RuntimeError): + """A storage adapter failed without exposing vendor or credential details.""" + + +@dataclass(frozen=True, slots=True) +class WriteResult: + """Counts returned by one idempotent repository write.""" + + inserted: int = 0 + updated: int = 0 + unchanged: int = 0 + + +@dataclass(frozen=True, slots=True) +class StagedSnapshot: + """A temporary file that may be published after its database transaction.""" + + temporary_path: Path + final_path: Path + fingerprint: str + + +class MarketDataSource(Protocol): + """External source port for stock, calendar, bar, and metric data.""" + + def fetch_stocks(self) -> Sequence[Stock]: ... + + def fetch_open_dates(self, start: date, end: date) -> Sequence[date]: ... + + def fetch_bars(self, ts_code: str, window: SyncWindow) -> Sequence[Bar]: ... + + def fetch_daily_basic(self, trade_date: date) -> Sequence[DailyBasic]: ... + + +class SnapshotStore(Protocol): + """Snapshot and atomic-file publication port.""" + + def read_bars(self, ts_code: str) -> tuple[Bar, ...] | None: ... + + def stage_bars(self, ts_code: str, rows: Iterable[Bar]) -> StagedSnapshot: ... + + def stage_daily_basic(self, trade_date: date, rows: Iterable[DailyBasic]) -> StagedSnapshot: ... + + def stage_stocks(self, rows: Iterable[Stock]) -> StagedSnapshot: ... + + def publish(self, staged: StagedSnapshot) -> None: ... + + def discard(self, staged: StagedSnapshot) -> None: ... + + def clean_daily_basic_before(self, boundary: date) -> int: ... + + +class MarketDataRepository(Protocol): + """PostgreSQL repository port. + + ``full_snapshot`` controls whether rows missing from the current six-year + window are deleted for the affected stock. Incremental writes never + remove rows before the database transaction has succeeded. + """ + + def upsert_stocks(self, rows: Iterable[Stock]) -> WriteResult: ... + + def upsert_bars( + self, + rows: Iterable[Bar], + window: SyncWindow, + *, + full_snapshot: bool, + ) -> WriteResult: ... + + def upsert_daily_basic(self, rows: Iterable[DailyBasic], window: SyncWindow) -> WriteResult: ... + + def purge_before(self, window: SyncWindow) -> None: ... + + def has_bar(self, ts_code: str, trade_date: date) -> bool: ... + + def has_daily_basic(self, ts_code: str, trade_date: date) -> bool: ... + + def create_batch( + self, + target_trade_date: date, + window: SyncWindow, + mode: str, + parent_batch_id: str | None, + target_count: int, + ) -> str: ... + + def record_batch( + self, + batch_id: str, + status: str, + valid_count: int, + coverage: Decimal, + strategy_eligible: bool, + ) -> None: ... + + def record_item( + self, + batch_id: str, + item_kind: str, + item_key: str, + status: str, + result: WriteResult, + fingerprint: str | None = None, + error_type: str | None = None, + error_message: str | None = None, + ) -> None: ... + + def failed_items(self, parent_batch_id: str) -> tuple[tuple[str, str], ...]: ... + + def advisory_lock(self, key: int) -> AbstractContextManager[bool]: ... diff --git a/zhixing-server/src/zhixing_server/modules/market_data/domain/rules.py b/zhixing-server/src/zhixing_server/modules/market_data/domain/rules.py new file mode 100644 index 0000000..bc52c88 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/domain/rules.py @@ -0,0 +1,32 @@ +"""Target universe rules for the current listed Shanghai/Shenzhen A shares.""" + +from collections.abc import Iterable + +from .models import Stock + + +def is_current_hs_a_stock(stock: Stock) -> bool: + """Return whether a stock is a current non-ST Shanghai/Shenzhen A share.""" + + code, separator, suffix = stock.ts_code.upper().partition(".") + if not separator or suffix not in {"SH", "SZ"} or not code.isdigit(): + return False + if stock.list_status.upper() != "L": + return False + if stock.exchange and stock.exchange.upper() not in {"SSE", "SZSE", "SH", "SZ"}: + return False + market = stock.market.upper().replace(" ", "") + if "北交所" in market or market in {"B", "B股", "BEIJING"}: + return False + return "ST" not in stock.name.upper().replace(" ", "") and "退" not in stock.name + + +def filter_current_hs_a_stocks(stocks: Iterable[Stock]) -> tuple[Stock, ...]: + """Filter and deterministically sort the current target universe.""" + + return tuple( + sorted( + (stock for stock in stocks if is_current_hs_a_stock(stock)), + key=lambda stock: stock.ts_code, + ) + ) diff --git a/zhixing-server/src/zhixing_server/modules/market_data/domain/window.py b/zhixing-server/src/zhixing_server/modules/market_data/domain/window.py new file mode 100644 index 0000000..11897b4 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/domain/window.py @@ -0,0 +1,5 @@ +"""Compatibility import for the market data rolling window.""" + +from .models import SyncWindow + +__all__ = ["SyncWindow"] diff --git a/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/__init__.py b/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/__init__.py new file mode 100644 index 0000000..45a2c52 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/__init__.py @@ -0,0 +1 @@ +"""Market data infrastructure adapters.""" diff --git a/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/csv_snapshot.py b/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/csv_snapshot.py new file mode 100644 index 0000000..614a1ee --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/csv_snapshot.py @@ -0,0 +1,256 @@ +"""Fixed-layout CSV snapshots with atomic publication semantics.""" + +from __future__ import annotations + +import csv +import os +import re +import tempfile +from collections.abc import Iterable +from contextlib import suppress +from datetime import date +from hashlib import sha256 +from pathlib import Path +from uuid import uuid4 + +from ..domain.fingerprint import snapshot_fingerprint +from ..domain.models import Bar, DailyBasic, Stock, decimal_text +from ..domain.ports import StagedSnapshot + +_SAFE_CODE = re.compile(r"^[0-9]{6}\.(SH|SZ)$") +BAR_COLUMNS = ( + "ts_code", + "trade_date", + "open", + "high", + "low", + "close", + "pre_close", + "change", + "pct_chg", + "vol", + "amount", +) +DAILY_BASIC_COLUMNS = ( + "ts_code", + "trade_date", + "close", + "turnover_rate", + "turnover_rate_f", + "volume_ratio", + "pe", + "pe_ttm", + "pb", + "ps", + "ps_ttm", + "dv_ratio", + "dv_ttm", + "total_share", + "float_share", + "free_share", + "total_mv", + "circ_mv", +) +STOCK_COLUMNS = ("ts_code", "name", "market", "exchange", "list_status", "list_date") +_DAILY_DECIMAL_FIELDS = DAILY_BASIC_COLUMNS[2:] + + +def _code_path(ts_code: str) -> str: + if not _SAFE_CODE.fullmatch(ts_code): + raise ValueError(f"unsupported stock code: {ts_code!r}") + return ts_code + + +def _bar_row(row: Bar) -> dict[str, str]: + return { + "ts_code": row.ts_code, + "trade_date": row.trade_date.isoformat(), + "open": decimal_text(row.open), + "high": decimal_text(row.high), + "low": decimal_text(row.low), + "close": decimal_text(row.close), + "pre_close": decimal_text(row.pre_close), + "change": decimal_text(row.change), + "pct_chg": decimal_text(row.pct_chg), + "vol": decimal_text(row.vol), + "amount": decimal_text(row.amount), + } + + +def _daily_basic_row(row: DailyBasic) -> dict[str, str]: + values: dict[str, str] = { + "ts_code": row.ts_code, + "trade_date": row.trade_date.isoformat(), + } + for field in _DAILY_DECIMAL_FIELDS: + values[field] = decimal_text(getattr(row, field)) + return values + + +def _stock_row(row: Stock) -> dict[str, str]: + return { + "ts_code": row.ts_code, + "name": row.name, + "market": row.market, + "exchange": row.exchange, + "list_status": row.list_status, + "list_date": row.list_date.isoformat() if row.list_date else "", + } + + +class CsvSnapshotStore: + """Store and publish the three market-data CSV layouts. + + Temporary files are created beside their final file. Therefore ``os.replace`` + is a same-filesystem operation and a database failure can leave the previous + published snapshot untouched. + """ + + def __init__(self, root: Path) -> None: + self.root = root + + def bars_path(self, ts_code: str) -> Path: + """Return the formal per-stock qfq path.""" + + return self.root / "bars" / f"{_code_path(ts_code)}.csv" + + def daily_basic_path(self, trade_date: date) -> Path: + """Return the formal per-trading-day metrics path.""" + + return self.root / "daily-basic" / f"{trade_date:%Y}" / f"{trade_date:%Y%m%d}.csv" + + @property + def stock_basic_path(self) -> Path: + """Return the formal current stock-master path.""" + + return self.root / "stock-basic" / "current.csv" + + def read_bars(self, ts_code: str) -> tuple[Bar, ...] | None: + """Read the current formal snapshot, returning ``None`` if absent.""" + + path = self.bars_path(ts_code) + if not path.exists(): + return None + with path.open(newline="", encoding="utf-8") as handle: + reader = csv.DictReader(handle) + if tuple(reader.fieldnames or ()) != BAR_COLUMNS: + raise ValueError(f"unexpected bar CSV header: {path}") + rows = tuple(Bar.from_mapping(row) for row in reader) + return tuple(sorted(rows, key=lambda row: row.trade_date)) + + def stage_bars(self, ts_code: str, rows: Iterable[Bar]) -> StagedSnapshot: + """Validate and write a temporary bar snapshot.""" + + normalized = tuple(sorted(rows, key=lambda row: row.trade_date)) + if not normalized or any(row.ts_code != ts_code for row in normalized): + raise ValueError("bar snapshot must be non-empty and belong to one stock") + final_path = self.bars_path(ts_code) + temp_path = self._write_csv(final_path, BAR_COLUMNS, (_bar_row(row) for row in normalized)) + return StagedSnapshot(temp_path, final_path, snapshot_fingerprint(normalized)) + + def stage_daily_basic(self, trade_date: date, rows: Iterable[DailyBasic]) -> StagedSnapshot: + """Validate and stage one daily-basic date snapshot.""" + + normalized = tuple(sorted(rows, key=lambda row: row.ts_code)) + if any(row.trade_date != trade_date for row in normalized): + raise ValueError("daily-basic snapshot contains an unexpected date") + keys = [row.ts_code for row in normalized] + if len(keys) != len(set(keys)): + raise ValueError("daily-basic snapshot contains duplicate stock codes") + final_path = self.daily_basic_path(trade_date) + payload = "\n".join( + ",".join(_daily_basic_row(row)[column] for column in DAILY_BASIC_COLUMNS) + for row in normalized + ).encode("utf-8") + fingerprint = sha256(payload).hexdigest() + temp_path = self._write_csv( + final_path, + DAILY_BASIC_COLUMNS, + (_daily_basic_row(row) for row in normalized), + ) + return StagedSnapshot(temp_path, final_path, fingerprint) + + def stage_stocks(self, rows: Iterable[Stock]) -> StagedSnapshot: + """Validate and stage the current stock-master snapshot.""" + + normalized = tuple(sorted(rows, key=lambda row: row.ts_code)) + if not normalized: + raise ValueError("stock-master snapshot must not be empty") + keys = [row.ts_code for row in normalized] + if len(keys) != len(set(keys)): + raise ValueError("stock-master snapshot contains duplicate stock codes") + payload = "\n".join( + ",".join(_stock_row(row)[field] for field in STOCK_COLUMNS) for row in normalized + ) + final_path = self.stock_basic_path + temp_path = self._write_csv( + final_path, + STOCK_COLUMNS, + (_stock_row(row) for row in normalized), + ) + return StagedSnapshot(temp_path, final_path, sha256(payload.encode("utf-8")).hexdigest()) + + def publish(self, staged: StagedSnapshot) -> None: + """Atomically promote a staged snapshot and sync the parent directory.""" + + if not staged.temporary_path.exists(): + raise FileNotFoundError(staged.temporary_path) + os.replace(staged.temporary_path, staged.final_path) + try: + directory_fd = os.open(staged.final_path.parent, os.O_RDONLY) + except OSError: + return + try: + os.fsync(directory_fd) + finally: + os.close(directory_fd) + + def discard(self, staged: StagedSnapshot) -> None: + """Remove a staged file without touching its formal counterpart.""" + + with suppress(FileNotFoundError): + staged.temporary_path.unlink() + + def clean_daily_basic_before(self, boundary: date) -> int: + """Remove only formal daily-basic files older than the retention boundary.""" + + removed = 0 + root = self.root / "daily-basic" + if not root.exists(): + return removed + for path in root.glob("*/????????.csv"): + try: + file_date = date.fromisoformat( + path.stem[:4] + "-" + path.stem[4:6] + "-" + path.stem[6:] + ) + except ValueError: + continue + if file_date < boundary: + path.unlink() + removed += 1 + return removed + + def _write_csv( + self, + final_path: Path, + columns: tuple[str, ...], + rows: Iterable[dict[str, str]], + ) -> Path: + final_path.parent.mkdir(parents=True, exist_ok=True) + descriptor, raw_path = tempfile.mkstemp( + prefix=f".{final_path.name}.{uuid4().hex}.", + suffix=".tmp", + dir=final_path.parent, + ) + temp_path = Path(raw_path) + try: + with os.fdopen(descriptor, "w", encoding="utf-8", newline="") as handle: + writer = csv.DictWriter(handle, fieldnames=columns, lineterminator="\n") + writer.writeheader() + writer.writerows(rows) + handle.flush() + os.fsync(handle.fileno()) + except BaseException: + temp_path.unlink(missing_ok=True) + raise + return temp_path diff --git a/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/postgres.py b/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/postgres.py new file mode 100644 index 0000000..f1a5d83 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/postgres.py @@ -0,0 +1,465 @@ +"""Psycopg 3 PostgreSQL adapter using COPY staging and set-based upserts.""" + +from __future__ import annotations + +from collections.abc import Generator, Iterable +from contextlib import contextmanager +from datetime import date +from decimal import Decimal +from typing import Any +from uuid import uuid4 + +import psycopg + +from ..domain.models import Bar, DailyBasic, Stock, SyncWindow +from ..domain.ports import MarketDataRepositoryError, WriteResult + + +class PostgresMarketDataRepository: + """Persist market data without an ORM identity map. + + Each public write opens a short transaction. The caller publishes the + matching CSV only after this method returns successfully. + """ + + def __init__(self, database_url: str) -> None: + self.database_url = database_url + + def upsert_stocks(self, rows: Iterable[Stock]) -> WriteResult: + records = tuple(rows) + if not records: + return WriteResult() + with self._connection() as connection, connection.transaction(): + cursor = connection.cursor() + codes = [row.ts_code for row in records] + cursor.execute( + """ + UPDATE market_stock + SET is_active = false, updated_at = now() + WHERE is_active AND NOT (ts_code = ANY(%s)) + """, + (codes,), + ) + inserted = 0 + updated = 0 + unchanged = 0 + for row in records: + result = cursor.execute( + """ + INSERT INTO market_stock + ( + ts_code, name, market, exchange, list_status, list_date, + is_active, updated_at + ) + VALUES (%s, %s, %s, %s, %s, %s, true, now()) + ON CONFLICT (ts_code) DO UPDATE SET + name = EXCLUDED.name, + market = EXCLUDED.market, + exchange = EXCLUDED.exchange, + list_status = EXCLUDED.list_status, + list_date = EXCLUDED.list_date, + is_active = true, + updated_at = now() + WHERE (market_stock.name, market_stock.market, market_stock.exchange, + market_stock.list_status, market_stock.list_date, market_stock.is_active) + IS DISTINCT FROM (EXCLUDED.name, EXCLUDED.market, EXCLUDED.exchange, + EXCLUDED.list_status, EXCLUDED.list_date, true) + RETURNING (xmax = 0) AS inserted + """, + ( + row.ts_code, + row.name, + row.market, + row.exchange, + row.list_status, + row.list_date, + ), + ).fetchone() + if result is None: + unchanged += 1 + elif bool(result[0]): + inserted += 1 + else: + updated += 1 + return WriteResult(inserted=inserted, updated=updated, unchanged=unchanged) + + def upsert_bars( + self, + rows: Iterable[Bar], + window: SyncWindow, + *, + full_snapshot: bool, + ) -> WriteResult: + records = tuple(rows) + if records and len({row.ts_code for row in records}) != 1: + raise ValueError("a bar write must contain one stock") + if not records: + return WriteResult() + code = records[0].ts_code + with self._connection() as connection, connection.transaction(): + cursor = connection.cursor() + self._create_bar_stage(cursor) + self._copy_bars(cursor, records) + before = self._count_stage_matches(cursor, "market_daily_bar") + inserted, updated = self._upsert_bars_from_stage(cursor) + if full_snapshot: + cursor.execute( + """ + DELETE FROM market_daily_bar AS target + WHERE target.ts_code = %s + AND target.trade_date BETWEEN %s AND %s + AND NOT EXISTS ( + SELECT 1 FROM market_daily_bar_stage AS stage + WHERE stage.ts_code = target.ts_code + AND stage.trade_date = target.trade_date + ) + """, + (code, window.start, window.end), + ) + return WriteResult(inserted=inserted, updated=updated, unchanged=max(0, before - updated)) + + def upsert_daily_basic(self, rows: Iterable[DailyBasic], window: SyncWindow) -> WriteResult: + records = tuple(rows) + if not records: + return WriteResult() + if len({row.trade_date for row in records}) != 1: + raise ValueError("a daily-basic write must contain one trade date") + with self._connection() as connection, connection.transaction(): + cursor = connection.cursor() + self._create_basic_stage(cursor) + self._copy_daily_basic(cursor, records) + before = self._count_stage_matches(cursor, "market_daily_basic") + inserted, updated = self._upsert_daily_basic_from_stage(cursor) + return WriteResult(inserted=inserted, updated=updated, unchanged=max(0, before - updated)) + + def purge_before(self, window: SyncWindow) -> None: + """Apply rolling retention only after the caller has finished a batch.""" + + with self._connection() as connection, connection.transaction(): + cursor = connection.cursor() + cursor.execute("DELETE FROM market_daily_bar WHERE trade_date < %s", (window.start,)) + cursor.execute("DELETE FROM market_daily_basic WHERE trade_date < %s", (window.start,)) + + def has_bar(self, ts_code: str, trade_date: date) -> bool: + return self._exists("market_daily_bar", ts_code, trade_date) + + def has_daily_basic(self, ts_code: str, trade_date: date) -> bool: + return self._exists("market_daily_basic", ts_code, trade_date) + + def create_batch( + self, + target_trade_date: date, + window: SyncWindow, + mode: str, + parent_batch_id: str | None, + target_count: int, + ) -> str: + batch_id = str(uuid4()) + with self._connection() as connection, connection.transaction(): + connection.execute( + """ + INSERT INTO market_sync_batch + (id, target_trade_date, window_start, mode, status, target_count) + VALUES (%s, %s, %s, %s, 'running', %s) + """, + (batch_id, target_trade_date, window.start, mode, target_count), + ) + if parent_batch_id: + connection.execute( + "UPDATE market_sync_batch SET parent_batch_id = %s WHERE id = %s", + (parent_batch_id, batch_id), + ) + return batch_id + + def record_batch( + self, + batch_id: str, + status: str, + valid_count: int, + coverage: Decimal, + strategy_eligible: bool, + ) -> None: + with self._connection() as connection, connection.transaction(): + connection.execute( + """ + UPDATE market_sync_batch + SET status = %s, valid_count = %s, coverage = %s, + strategy_eligible = %s, finished_at = now() + WHERE id = %s + """, + (status, valid_count, coverage, strategy_eligible, batch_id), + ) + + def record_item( + self, + batch_id: str, + item_kind: str, + item_key: str, + status: str, + result: WriteResult, + fingerprint: str | None = None, + error_type: str | None = None, + error_message: str | None = None, + ) -> None: + with self._connection() as connection, connection.transaction(): + connection.execute( + """ + INSERT INTO market_sync_item + (batch_id, item_kind, item_key, status, inserted_count, + updated_count, unchanged_count, fingerprint, error_type, error_message) + VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s) + ON CONFLICT (batch_id, item_kind, item_key) DO UPDATE SET + status = EXCLUDED.status, + inserted_count = EXCLUDED.inserted_count, + updated_count = EXCLUDED.updated_count, + unchanged_count = EXCLUDED.unchanged_count, + fingerprint = EXCLUDED.fingerprint, + error_type = EXCLUDED.error_type, + error_message = EXCLUDED.error_message + """, + ( + batch_id, + item_kind, + item_key, + status, + result.inserted, + result.updated, + result.unchanged, + fingerprint, + error_type, + self._safe_error(error_message), + ), + ) + + def failed_items(self, parent_batch_id: str) -> tuple[tuple[str, str], ...]: + with self._connection() as connection: + rows = connection.execute( + """ + SELECT item_kind, item_key + FROM market_sync_item + WHERE batch_id = %s AND status = 'failed' + ORDER BY item_kind, item_key + """, + (parent_batch_id,), + ).fetchall() + return tuple((str(row[0]), str(row[1])) for row in rows) + + @contextmanager + def advisory_lock(self, key: int) -> Generator[bool, None, None]: + """Hold a PostgreSQL advisory lock for the lifetime of one sync run.""" + + with self._connection() as connection: + row = connection.execute( + "SELECT pg_try_advisory_lock(%s)", + (key,), + ).fetchone() + acquired = bool(row[0]) if row is not None else False + if not acquired: + yield False + return + try: + yield True + finally: + connection.execute("SELECT pg_advisory_unlock(%s)", (key,)) + + def _exists(self, table: str, ts_code: str, trade_date: date) -> bool: + if table == "market_daily_bar": + query = "SELECT 1 FROM market_daily_bar WHERE ts_code = %s AND trade_date = %s LIMIT 1" + elif table == "market_daily_basic": + query = ( + "SELECT 1 FROM market_daily_basic WHERE ts_code = %s AND trade_date = %s LIMIT 1" + ) + else: + raise ValueError(f"unsupported existence table: {table}") + with self._connection() as connection: + result = connection.execute( + query, + (ts_code, trade_date), + ).fetchone() + return result is not None + + @contextmanager + def _connection(self) -> Generator[Any, None, None]: + """Translate driver failures into a safe application-level error.""" + + try: + with psycopg.connect(self.database_url) as connection: + yield connection + except psycopg.Error as exc: + raise MarketDataRepositoryError("market data database operation failed") from exc + + @staticmethod + def _create_bar_stage(cursor: Any) -> None: + cursor.execute( + """ + CREATE TEMP TABLE market_daily_bar_stage ( + ts_code varchar(12) NOT NULL, + trade_date date NOT NULL, + open numeric(20,6), high numeric(20,6), low numeric(20,6), close numeric(20,6), + pre_close numeric(20,6), change numeric(20,6), pct_chg numeric(20,6), + vol numeric(24,6), amount numeric(24,6) + ) ON COMMIT DROP + """ + ) + + @staticmethod + def _create_basic_stage(cursor: Any) -> None: + cursor.execute( + """ + CREATE TEMP TABLE market_daily_basic_stage ( + ts_code varchar(12) NOT NULL, + trade_date date NOT NULL, + close numeric(24,6), turnover_rate numeric(24,6), + turnover_rate_f numeric(24,6), volume_ratio numeric(24,6), + pe numeric(24,6), pe_ttm numeric(24,6), pb numeric(24,6), + ps numeric(24,6), ps_ttm numeric(24,6), dv_ratio numeric(24,6), + dv_ttm numeric(24,6), total_share numeric(24,6), + float_share numeric(24,6), free_share numeric(24,6), + total_mv numeric(24,6), circ_mv numeric(24,6) + ) ON COMMIT DROP + """ + ) + + @staticmethod + def _copy_bars(cursor: Any, rows: Iterable[Bar]) -> None: + copy_sql = ( + "COPY market_daily_bar_stage " + "(ts_code, trade_date, open, high, low, close, pre_close, " + "change, pct_chg, vol, amount) FROM STDIN" + ) + with cursor.copy(copy_sql) as copy: + for row in rows: + copy.write_row( + ( + row.ts_code, + row.trade_date, + row.open, + row.high, + row.low, + row.close, + row.pre_close, + row.change, + row.pct_chg, + row.vol, + row.amount, + ) + ) + + @staticmethod + def _copy_daily_basic(cursor: Any, rows: Iterable[DailyBasic]) -> None: + copy_sql = ( + "COPY market_daily_basic_stage " + "(ts_code, trade_date, close, turnover_rate, turnover_rate_f, " + "volume_ratio, pe, pe_ttm, pb, ps, ps_ttm, dv_ratio, dv_ttm, " + "total_share, float_share, free_share, total_mv, circ_mv) FROM STDIN" + ) + with cursor.copy(copy_sql) as copy: + for row in rows: + copy.write_row( + ( + row.ts_code, + row.trade_date, + row.close, + row.turnover_rate, + row.turnover_rate_f, + row.volume_ratio, + row.pe, + row.pe_ttm, + row.pb, + row.ps, + row.ps_ttm, + row.dv_ratio, + row.dv_ttm, + row.total_share, + row.float_share, + row.free_share, + row.total_mv, + row.circ_mv, + ) + ) + + @staticmethod + def _upsert_bars_from_stage(cursor: Any) -> tuple[int, int]: + rows = cursor.execute( + """ + INSERT INTO market_daily_bar + ( + ts_code, trade_date, open, high, low, close, pre_close, + change, pct_chg, vol, amount, source_adj, updated_at + ) + SELECT + ts_code, trade_date, open, high, low, close, pre_close, + change, pct_chg, vol, amount, 'qfq', now() + FROM market_daily_bar_stage + ON CONFLICT (ts_code, trade_date) DO UPDATE SET + open = EXCLUDED.open, high = EXCLUDED.high, low = EXCLUDED.low, + close = EXCLUDED.close, pre_close = EXCLUDED.pre_close, + change = EXCLUDED.change, pct_chg = EXCLUDED.pct_chg, + vol = EXCLUDED.vol, amount = EXCLUDED.amount, source_adj = 'qfq', updated_at = now() + WHERE ( + market_daily_bar.open, market_daily_bar.high, market_daily_bar.low, + market_daily_bar.close, market_daily_bar.pre_close, + market_daily_bar.change, market_daily_bar.pct_chg, + market_daily_bar.vol, market_daily_bar.amount + ) IS DISTINCT FROM ( + EXCLUDED.open, EXCLUDED.high, EXCLUDED.low, EXCLUDED.close, + EXCLUDED.pre_close, EXCLUDED.change, EXCLUDED.pct_chg, + EXCLUDED.vol, EXCLUDED.amount + ) + RETURNING (xmax = 0) AS inserted + """ + ).fetchall() + inserted = sum(1 for row in rows if bool(row[0])) + return inserted, len(rows) - inserted + + @staticmethod + def _upsert_daily_basic_from_stage(cursor: Any) -> tuple[int, int]: + fields = ( + "close", + "turnover_rate", + "turnover_rate_f", + "volume_ratio", + "pe", + "pe_ttm", + "pb", + "ps", + "ps_ttm", + "dv_ratio", + "dv_ttm", + "total_share", + "float_share", + "free_share", + "total_mv", + "circ_mv", + ) + assignments = ", ".join(f"{field} = EXCLUDED.{field}" for field in fields) + comparison = ", ".join(f"market_daily_basic.{field}" for field in fields) + excluded = ", ".join(f"EXCLUDED.{field}" for field in fields) + query = f""" + INSERT INTO market_daily_basic (ts_code, trade_date, {", ".join(fields)}, updated_at) + SELECT ts_code, trade_date, {", ".join(fields)}, now() FROM market_daily_basic_stage + ON CONFLICT (ts_code, trade_date) DO UPDATE SET {assignments}, updated_at = now() + WHERE ({comparison}) IS DISTINCT FROM ({excluded}) + RETURNING (xmax = 0) AS inserted + """ + rows = cursor.execute(query).fetchall() + inserted = sum(1 for row in rows if bool(row[0])) + return inserted, len(rows) - inserted + + @staticmethod + def _count_stage_matches(cursor: Any, table: str) -> int: + stage = ( + "market_daily_bar_stage" if table == "market_daily_bar" else "market_daily_basic_stage" + ) + return int( + cursor.execute( + f"SELECT count(*) FROM {stage} AS stage JOIN {table} AS target " + "USING (ts_code, trade_date)" + ).fetchone()[0] + ) + + @staticmethod + def _safe_error(message: str | None) -> str | None: + if message is None: + return None + return " ".join(message.split())[:500] diff --git a/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/schema.py b/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/schema.py new file mode 100644 index 0000000..b0e924c --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/schema.py @@ -0,0 +1,116 @@ +"""SQLAlchemy metadata for the market-data migration contract.""" + +from sqlalchemy import ( + Boolean, + Column, + Date, + DateTime, + Integer, + MetaData, + Numeric, + PrimaryKeyConstraint, + String, + Table, + Text, + func, +) + +metadata = MetaData() + +market_stock = Table( + "market_stock", + metadata, + Column("ts_code", String(12), primary_key=True), + Column("name", String(128), nullable=False), + Column("market", String(32), nullable=False), + Column("exchange", String(16), nullable=False), + Column("list_status", String(2), nullable=False), + Column("list_date", Date), + Column("is_active", Boolean, nullable=False, server_default="true"), + Column("updated_at", DateTime(timezone=True), nullable=False, server_default=func.now()), +) + +market_daily_bar = Table( + "market_daily_bar", + metadata, + Column("ts_code", String(12), nullable=False), + Column("trade_date", Date, nullable=False), + Column("open", Numeric(20, 6)), + Column("high", Numeric(20, 6)), + Column("low", Numeric(20, 6)), + Column("close", Numeric(20, 6)), + Column("pre_close", Numeric(20, 6)), + Column("change", Numeric(20, 6)), + Column("pct_chg", Numeric(20, 6)), + Column("vol", Numeric(24, 6)), + Column("amount", Numeric(24, 6)), + Column("source_adj", String(8), nullable=False, server_default="qfq"), + Column("updated_at", DateTime(timezone=True), nullable=False, server_default=func.now()), + PrimaryKeyConstraint("ts_code", "trade_date"), +) + +market_daily_basic = Table( + "market_daily_basic", + metadata, + Column("ts_code", String(12), nullable=False), + Column("trade_date", Date, nullable=False), + *( + [ + Column(field, Numeric(24, 6)) + for field in ( + "close", + "turnover_rate", + "turnover_rate_f", + "volume_ratio", + "pe", + "pe_ttm", + "pb", + "ps", + "ps_ttm", + "dv_ratio", + "dv_ttm", + "total_share", + "float_share", + "free_share", + "total_mv", + "circ_mv", + ) + ] + ), + Column("updated_at", DateTime(timezone=True), nullable=False, server_default=func.now()), + PrimaryKeyConstraint("ts_code", "trade_date"), +) + +market_sync_batch = Table( + "market_sync_batch", + metadata, + Column("id", String(36), primary_key=True), + Column("target_trade_date", Date, nullable=False), + Column("window_start", Date, nullable=False), + Column("mode", String(16), nullable=False), + Column("status", String(24), nullable=False), + Column("target_count", Integer, nullable=False), + Column("valid_count", Integer, nullable=False, server_default="0"), + Column("coverage", Numeric(8, 6), nullable=False, server_default="0"), + Column("strategy_eligible", Boolean, nullable=False, server_default="false"), + Column("parent_batch_id", String(36)), + Column("created_at", DateTime(timezone=True), nullable=False, server_default=func.now()), + Column("finished_at", DateTime(timezone=True)), +) + +market_sync_item = Table( + "market_sync_item", + metadata, + Column("batch_id", String(36), nullable=False), + Column("item_kind", String(24), nullable=False), + Column("item_key", String(64), nullable=False), + Column("status", String(24), nullable=False), + Column("inserted_count", Integer, nullable=False, server_default="0"), + Column("updated_count", Integer, nullable=False, server_default="0"), + Column("unchanged_count", Integer, nullable=False, server_default="0"), + Column("fingerprint", String(64)), + Column("error_type", String(64)), + Column("error_message", Text), + Column("created_at", DateTime(timezone=True), nullable=False, server_default=func.now()), + PrimaryKeyConstraint("batch_id", "item_kind", "item_key"), +) diff --git a/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/tushare.py b/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/tushare.py new file mode 100644 index 0000000..cd47f81 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/infrastructure/tushare.py @@ -0,0 +1,175 @@ +"""Tushare source adapter and current-universe filtering.""" + +from __future__ import annotations + +import random +import time +from collections.abc import Callable, Iterable, Mapping +from datetime import date +from typing import cast + +from ..domain.models import Bar, DailyBasic, Stock, SyncWindow, parse_date +from ..domain.rules import filter_current_hs_a_stocks + + +class TushareSourceError(RuntimeError): + """A vendor request failed after the configured retry budget.""" + + +class TushareAdapter: + """Translate Tushare SDK responses into domain records. + + The SDK is kept behind this adapter so ordinary domain/application tests + can inject a tiny fake client and never need a network token. + """ + + def __init__( + self, + client: object, + *, + pro_bar: Callable[..., object] | None = None, + max_retries: int = 3, + backoff_seconds: float = 1.0, + request_interval_seconds: float = 0.2, + random_fn: Callable[[], float] = random.random, + sleep_fn: Callable[[float], None] = time.sleep, + ) -> None: + self.client = client + self.pro_bar = pro_bar + self.max_retries = max(0, max_retries) + self.backoff_seconds = max(0.0, backoff_seconds) + self.request_interval_seconds = max(0.0, request_interval_seconds) + self.random_fn = random_fn + self.sleep_fn = sleep_fn + + @classmethod + def from_token( + cls, + token: str, + *, + max_retries: int = 3, + backoff_seconds: float = 1.0, + request_interval_seconds: float = 0.2, + ) -> TushareAdapter: + """Create a production adapter from a token without exposing it.""" + + if not token.strip(): + raise ValueError("ZHIXING_TUSHARE_TOKEN is required for market sync") + import tushare as ts # pyright: ignore[reportMissingTypeStubs] + + pro_bar_function = cast( + Callable[..., object], + ts.pro_bar, # pyright: ignore[reportUnknownMemberType] + ) + return cls( + cast(object, ts.pro_api(token)), + pro_bar=pro_bar_function, + max_retries=max_retries, + backoff_seconds=backoff_seconds, + request_interval_seconds=request_interval_seconds, + ) + + def fetch_stocks(self) -> tuple[Stock, ...]: + """Fetch current listings and apply the target-universe rules.""" + + rows = self._records( + "stock_basic", + exchange="", + list_status="L", + fields="ts_code,name,market,exchange,list_status,list_date", + ) + return filter_current_hs_a_stocks(Stock.from_mapping(row) for row in rows) + + def fetch_open_dates(self, start: date, end: date) -> tuple[date, ...]: + """Fetch open trading dates in an inclusive range.""" + + rows = self._records( + "trade_cal", + exchange="", + start_date=start.strftime("%Y%m%d"), + end_date=end.strftime("%Y%m%d"), + is_open=1, + ) + dates: list[date] = [] + for row in rows: + is_open = row.get("is_open") + if str(is_open).strip() not in {"1", "True", "true"}: + continue + dates.append(parse_date(row.get("cal_date"))) + return tuple(sorted(set(dates))) + + def fetch_bars(self, ts_code: str, window: SyncWindow) -> tuple[Bar, ...]: + """Fetch a complete six-year qfq snapshot for one stock.""" + + rows = self._records( + "pro_bar", + ts_code=ts_code, + adj="qfq", + start_date=window.start.strftime("%Y%m%d"), + end_date=window.end.strftime("%Y%m%d"), + freq="D", + ) + bars = tuple(Bar.from_mapping(row) for row in rows) + if any(row.ts_code != ts_code for row in bars): + raise ValueError(f"Tushare returned a different stock for {ts_code}") + return tuple( + sorted( + (row for row in bars if window.contains(row.trade_date)), + key=lambda row: row.trade_date, + ) + ) + + def fetch_daily_basic(self, trade_date: date) -> tuple[DailyBasic, ...]: + """Fetch one full-market daily-basic snapshot by trading date.""" + + rows = self._records("daily_basic", trade_date=trade_date.strftime("%Y%m%d")) + metrics = tuple(DailyBasic.from_mapping(row) for row in rows) + if any(row.trade_date != trade_date for row in metrics): + raise ValueError(f"Tushare returned a different metric date for {trade_date}") + return tuple(sorted(metrics, key=lambda row: row.ts_code)) + + def _records(self, method_name: str, **kwargs: object) -> tuple[Mapping[str, object], ...]: + """Call an SDK method with bounded retry and normalize its tabular output.""" + + def request() -> object: + if method_name == "pro_bar": + if self.pro_bar is not None: + return self.pro_bar(**kwargs) + method = getattr(self.client, method_name, None) + else: + method = getattr(self.client, method_name, None) + if not callable(method): + raise TypeError(f"Tushare client has no callable {method_name}") + return method(**kwargs) + + last_error: BaseException | None = None + for attempt in range(self.max_retries + 1): + try: + result = request() + self.sleep_fn(self.request_interval_seconds) + return self._as_records(result) + except (OSError, RuntimeError, TimeoutError) as exc: + last_error = exc + if attempt == self.max_retries: + break + delay = self.backoff_seconds * (2**attempt) * (0.5 + self.random_fn()) + self.sleep_fn(delay) + raise TushareSourceError(f"Tushare request failed: {method_name}") from last_error + + @staticmethod + def _as_records(result: object) -> tuple[Mapping[str, object], ...]: + if result is None: + return () + to_dict = getattr(result, "to_dict", None) + if callable(to_dict): + result = to_dict("records") + if isinstance(result, Mapping): + return (cast(Mapping[str, object], result),) + if isinstance(result, Iterable) and not isinstance(result, (str, bytes)): + records: list[Mapping[str, object]] = [] + for row in cast(Iterable[object], result): + if not isinstance(row, Mapping): + raise TypeError("Tushare rows must be mappings") + records.append(cast(Mapping[str, object], row)) + return tuple(records) + raise TypeError("unsupported Tushare tabular response") diff --git a/zhixing-server/src/zhixing_server/modules/market_data/presentation/__init__.py b/zhixing-server/src/zhixing_server/modules/market_data/presentation/__init__.py new file mode 100644 index 0000000..be9bc62 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/presentation/__init__.py @@ -0,0 +1 @@ +"""CLI delivery adapter for the market-data use case.""" diff --git a/zhixing-server/src/zhixing_server/modules/market_data/presentation/cli.py b/zhixing-server/src/zhixing_server/modules/market_data/presentation/cli.py new file mode 100644 index 0000000..cf0ec84 --- /dev/null +++ b/zhixing-server/src/zhixing_server/modules/market_data/presentation/cli.py @@ -0,0 +1,80 @@ +"""One-shot ``market-data-sync`` command used by Compose and cron.""" + +from __future__ import annotations + +import argparse +import json +from collections.abc import Sequence +from datetime import date + +from ....bootstrap.config import get_settings +from ..application.sync import SyncMarketData, SyncMarketDataCommand +from ..infrastructure.csv_snapshot import CsvSnapshotStore +from ..infrastructure.postgres import PostgresMarketDataRepository +from ..infrastructure.tushare import TushareAdapter + + +def build_parser() -> argparse.ArgumentParser: + """Build the explicit, repeatable synchronization CLI.""" + + parser = argparse.ArgumentParser( + description="Synchronize Tushare qfq market data into PostgreSQL" + ) + mode = parser.add_mutually_exclusive_group() + mode.add_argument( + "--initialize", + action="store_true", + help="backfill daily-basic for the six-year window", + ) + mode.add_argument("--retry-batch-id", help="retry only failed items from a prior batch") + parser.add_argument( + "--trade-date", + type=_parse_date, + help="target open date in YYYY-MM-DD format", + ) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + """Execute one synchronization and print its redacted JSON summary.""" + + args = build_parser().parse_args(argv) + settings = get_settings() + if args.retry_batch_id: + command = SyncMarketDataCommand( + mode="retry", + target_trade_date=args.trade_date, + parent_batch_id=args.retry_batch_id, + ) + else: + command = SyncMarketDataCommand( + mode="initialize" if args.initialize else "daily", + target_trade_date=args.trade_date, + ) + source = TushareAdapter.from_token( + settings.tushare_token, + max_retries=settings.market_data_max_retries, + backoff_seconds=settings.market_data_retry_backoff_seconds, + request_interval_seconds=settings.market_data_request_interval_seconds, + ) + use_case = SyncMarketData( + source, + CsvSnapshotStore(settings.market_data_csv_root), + PostgresMarketDataRepository(settings.database_url), + coverage_threshold=settings.market_data_coverage_threshold, + lock_key=settings.market_data_advisory_lock_key, + ) + summary = use_case.execute(command) + print(json.dumps(summary.as_dict(), ensure_ascii=False, sort_keys=True)) + return summary.exit_code + + +def _parse_date(value: str) -> date: + try: + return date.fromisoformat(value) + except ValueError as exc: + raise argparse.ArgumentTypeError("trade date must use YYYY-MM-DD") from exc + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/zhixing-server/tests/integration/test_market_data_migration.py b/zhixing-server/tests/integration/test_market_data_migration.py new file mode 100644 index 0000000..1d992b2 --- /dev/null +++ b/zhixing-server/tests/integration/test_market_data_migration.py @@ -0,0 +1,31 @@ +import os +from pathlib import Path + +import pytest +from alembic import command +from alembic.config import Config +from sqlalchemy import Engine, create_engine, inspect + + +@pytest.mark.integration +def test_postgres_migration_creates_market_data_contract() -> None: + database_url = os.getenv("ZHIXING_TEST_DATABASE_URL") + if not database_url: + pytest.skip("set ZHIXING_TEST_DATABASE_URL to run PostgreSQL integration tests") + + server_root = Path(__file__).parents[2] + config = Config(str(server_root / "alembic.ini")) + config.set_main_option("sqlalchemy.url", database_url.replace("%", "%%")) + engine: Engine = create_engine(database_url) + command.upgrade(config, "head") + try: + tables = set(inspect(engine).get_table_names()) + assert { + "market_stock", + "market_daily_bar", + "market_daily_basic", + "market_sync_batch", + "market_sync_item", + } <= tables + finally: + engine.dispose() diff --git a/zhixing-server/tests/unit/market_data/test_csv_snapshot.py b/zhixing-server/tests/unit/market_data/test_csv_snapshot.py new file mode 100644 index 0000000..2b5ab91 --- /dev/null +++ b/zhixing-server/tests/unit/market_data/test_csv_snapshot.py @@ -0,0 +1,46 @@ +from datetime import date +from decimal import Decimal +from pathlib import Path + +import pytest + +from zhixing_server.modules.market_data.domain.models import Bar, DailyBasic +from zhixing_server.modules.market_data.infrastructure.csv_snapshot import CsvSnapshotStore + + +def make_bar(trade_date: date, close: str = "10") -> Bar: + return Bar( + ts_code="000001.SZ", + trade_date=trade_date, + close=Decimal(close), + ) + + +def make_basic(trade_date: date) -> DailyBasic: + return DailyBasic( + ts_code="000001.SZ", + trade_date=trade_date, + close=Decimal("10"), + total_mv=Decimal("100000"), + ) + + +def test_bar_snapshot_is_atomic_and_failed_publish_preserves_old_file(tmp_path: Path) -> None: + store = CsvSnapshotStore(tmp_path) + original = (make_bar(date(2024, 1, 2)),) + staged = store.stage_bars("000001.SZ", original) + store.publish(staged) + + replacement = store.stage_bars("000001.SZ", (make_bar(date(2024, 1, 2), "11"),)) + store.discard(replacement) + + assert store.read_bars("000001.SZ") == original + assert not replacement.temporary_path.exists() + + +def test_daily_basic_snapshot_rejects_duplicate_codes(tmp_path: Path) -> None: + store = CsvSnapshotStore(tmp_path) + target = date(2024, 1, 2) + + with pytest.raises(ValueError, match="duplicate"): + store.stage_daily_basic(target, (make_basic(target), make_basic(target))) diff --git a/zhixing-server/tests/unit/market_data/test_domain.py b/zhixing-server/tests/unit/market_data/test_domain.py new file mode 100644 index 0000000..f278075 --- /dev/null +++ b/zhixing-server/tests/unit/market_data/test_domain.py @@ -0,0 +1,73 @@ +from datetime import date +from decimal import Decimal + +from zhixing_server.modules.market_data.domain.fingerprint import ( + SnapshotChange, + compare_snapshots, + snapshot_fingerprint, +) +from zhixing_server.modules.market_data.domain.models import Bar, Stock, SyncWindow +from zhixing_server.modules.market_data.domain.rules import filter_current_hs_a_stocks + + +def make_bar(trade_date: date, close: str = "10") -> Bar: + return Bar( + ts_code="000001.SZ", + trade_date=trade_date, + open=Decimal("9"), + high=Decimal(close), + low=Decimal("8"), + close=Decimal(close), + pre_close=Decimal("9"), + change=Decimal("1"), + pct_chg=Decimal("11.11"), + vol=Decimal("100"), + amount=Decimal("1000"), + ) + + +def test_window_uses_inclusive_calendar_boundary() -> None: + window = SyncWindow.from_target(date(2024, 2, 29)) + + assert window.start == date(2018, 2, 28) + assert window.end == date(2024, 2, 29) + assert window.contains(date(2018, 2, 28)) + + +def test_universe_keeps_current_non_st_hs_a_stocks() -> None: + stocks = ( + Stock("000001.SZ", "平安银行", exchange="SZSE", list_status="L"), + Stock("600000.SH", "浦发银行", exchange="SSE", list_status="L"), + Stock("300001.SZ", "特锐德", exchange="SZSE", list_status="L"), + Stock("600001.SH", "*ST风险", exchange="SSE", list_status="L"), + Stock("830001.BJ", "北交所", exchange="BSE", list_status="L"), + ) + + assert [stock.ts_code for stock in filter_current_hs_a_stocks(stocks)] == [ + "000001.SZ", + "300001.SZ", + "600000.SH", + ] + + +def test_fingerprint_is_order_independent_and_excludes_runtime_metadata() -> None: + rows = (make_bar(date(2024, 1, 2)), make_bar(date(2024, 1, 3))) + + assert snapshot_fingerprint(rows) == snapshot_fingerprint(tuple(reversed(rows))) + assert compare_snapshots(rows, rows).change is SnapshotChange.UNCHANGED + assert ( + compare_snapshots(rows, (*rows, make_bar(date(2024, 1, 4)))).change + is SnapshotChange.NEW_DATES + ) + + +def test_fingerprint_detects_repairs_missing_rows_and_earlier_rows() -> None: + old = (make_bar(date(2024, 1, 2)), make_bar(date(2024, 1, 3))) + + changed = (make_bar(date(2024, 1, 2), close="11"), make_bar(date(2024, 1, 3))) + missing = (make_bar(date(2024, 1, 2)),) + earlier = (make_bar(date(2024, 1, 1)), *old) + + assert compare_snapshots(old, changed).change is SnapshotChange.CHANGED + assert compare_snapshots(old, missing).change is SnapshotChange.CHANGED + assert compare_snapshots(old, earlier).change is SnapshotChange.CHANGED diff --git a/zhixing-server/tests/unit/market_data/test_sync.py b/zhixing-server/tests/unit/market_data/test_sync.py new file mode 100644 index 0000000..83d13e8 --- /dev/null +++ b/zhixing-server/tests/unit/market_data/test_sync.py @@ -0,0 +1,165 @@ +from collections.abc import Generator, Iterable, Sequence +from contextlib import contextmanager +from datetime import date +from decimal import Decimal +from pathlib import Path + +from zhixing_server.modules.market_data.application.sync import ( + SyncMarketData, + SyncMarketDataCommand, +) +from zhixing_server.modules.market_data.domain.models import Bar, DailyBasic, Stock, SyncWindow +from zhixing_server.modules.market_data.domain.ports import WriteResult +from zhixing_server.modules.market_data.infrastructure.csv_snapshot import CsvSnapshotStore + + +class FakeSource: + def __init__(self, target: date) -> None: + self.target = target + self.stock = Stock("000001.SZ", "平安银行", exchange="SZSE", list_status="L") + + def fetch_stocks(self) -> Sequence[Stock]: + return (self.stock,) + + def fetch_open_dates(self, start: date, end: date) -> Sequence[date]: + return (self.target,) if start <= self.target <= end else () + + def fetch_bars(self, ts_code: str, window: SyncWindow) -> Sequence[Bar]: + return (Bar(ts_code, self.target, close=Decimal("10")),) + + def fetch_daily_basic(self, trade_date: date) -> Sequence[DailyBasic]: + return (DailyBasic("000001.SZ", trade_date, close=Decimal("10")),) + + +class InMemoryRepository: + def __init__(self) -> None: + self.bars: dict[tuple[str, date], Bar] = {} + self.daily_basic: dict[tuple[str, date], DailyBasic] = {} + self.batch_status: str | None = None + self.batch_counter = 0 + + @contextmanager + def advisory_lock(self, key: int) -> Generator[bool, None, None]: + yield True + + def upsert_stocks(self, rows: Iterable[Stock]) -> WriteResult: + records = tuple(rows) + return WriteResult(inserted=len(records)) + + def upsert_bars( + self, + rows: Iterable[Bar], + window: SyncWindow, + *, + full_snapshot: bool, + ) -> WriteResult: + records = tuple(rows) + inserted = 0 + updated = 0 + unchanged = 0 + code = records[0].ts_code + if full_snapshot: + for key in tuple(self.bars): + if ( + key[0] == code + and window.contains(key[1]) + and key[1] not in {row.trade_date for row in records} + ): + del self.bars[key] + for row in records: + key = (row.ts_code, row.trade_date) + previous = self.bars.get(key) + if previous is None: + inserted += 1 + elif previous == row: + unchanged += 1 + else: + updated += 1 + self.bars[key] = row + return WriteResult(inserted=inserted, updated=updated, unchanged=unchanged) + + def upsert_daily_basic(self, rows: Iterable[DailyBasic], window: SyncWindow) -> WriteResult: + inserted = 0 + updated = 0 + unchanged = 0 + for row in rows: + key = (row.ts_code, row.trade_date) + previous = self.daily_basic.get(key) + if previous is None: + inserted += 1 + elif previous == row: + unchanged += 1 + else: + updated += 1 + self.daily_basic[key] = row + return WriteResult(inserted=inserted, updated=updated, unchanged=unchanged) + + def purge_before(self, window: SyncWindow) -> None: + self.bars = {key: row for key, row in self.bars.items() if key[1] >= window.start} + self.daily_basic = { + key: row for key, row in self.daily_basic.items() if key[1] >= window.start + } + + def has_bar(self, ts_code: str, trade_date: date) -> bool: + return (ts_code, trade_date) in self.bars + + def has_daily_basic(self, ts_code: str, trade_date: date) -> bool: + return (ts_code, trade_date) in self.daily_basic + + def create_batch( + self, + target_trade_date: date, + window: SyncWindow, + mode: str, + parent_batch_id: str | None, + target_count: int, + ) -> str: + self.batch_counter += 1 + return f"batch-{self.batch_counter}" + + def record_batch( + self, + batch_id: str, + status: str, + valid_count: int, + coverage: Decimal, + strategy_eligible: bool, + ) -> None: + self.batch_status = status + + def record_item( + self, + batch_id: str, + item_kind: str, + item_key: str, + status: str, + result: WriteResult, + fingerprint: str | None = None, + error_type: str | None = None, + error_message: str | None = None, + ) -> None: + return None + + def failed_items(self, parent_batch_id: str) -> tuple[tuple[str, str], ...]: + return () + + +def test_sync_is_idempotent_and_reports_coverage(tmp_path: Path) -> None: + target = date(2024, 1, 2) + repository = InMemoryRepository() + use_case = SyncMarketData( + FakeSource(target), + CsvSnapshotStore(tmp_path), + repository, + today=target, + ) + + first = use_case.execute(SyncMarketDataCommand(target_trade_date=target)) + second = use_case.execute(SyncMarketDataCommand(target_trade_date=target)) + + assert first.status == "success" + assert first.coverage == Decimal("1") + assert first.strategy_eligible + assert second.status == "success" + assert second.inserted_count == 1 + assert second.unchanged_count >= 2 diff --git a/zhixing-server/uv.lock b/zhixing-server/uv.lock index b3494b2..907193a 100644 --- a/zhixing-server/uv.lock +++ b/zhixing-server/uv.lock @@ -1,6 +1,25 @@ version = 1 revision = 2 requires-python = "==3.12.*" +resolution-markers = [ + "sys_platform == 'win32'", + "sys_platform == 'emscripten'", + "sys_platform != 'emscripten' and sys_platform != 'win32'", +] + +[[package]] +name = "alembic" +version = "1.19.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mako" }, + { name = "sqlalchemy" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/fb/01/a48dab7827ac4421272399f7ed9a2ec17edd12c8bcde4417bd7b6821b71a/alembic-1.19.0.tar.gz", hash = "sha256:6487c612fc719dcfa22b17d2dd5b2b458929641e6aa2f0b65b135727f5e6d501", size = 2069906, upload-time = "2026-08-04T18:57:04.599Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/34/79/59ab6f1fe72eee229de91aa393225389a85df12a0fbbbfa264efcf6d7872/alembic-1.19.0-py3-none-any.whl", hash = "sha256:cf839d3849116aab3cc047e09c6968b9bd6b2fde61b6bb7c1e97352fe5503580", size = 265738, upload-time = "2026-08-04T18:57:06.219Z" }, +] [[package]] name = "annotated-doc" @@ -33,6 +52,62 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" }, ] +[[package]] +name = "beautifulsoup4" +version = "4.15.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "soupsieve" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/43/65/318323f98dbee45d42dff61d8f047181bc6f2268a9068cfad035a46be5af/beautifulsoup4-4.15.0.tar.gz", hash = "sha256:288e3ca7d54b06f2ac191970bc275c1939cb46d450b255bf6718b04aa37ab4f7", size = 632571, upload-time = "2026-06-07T16:44:20.453Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/c6/92fcd42f1ba33e1184263f25bfabf3d27c383410470f169e4b8163bf9c17/beautifulsoup4-4.15.0-py3-none-any.whl", hash = "sha256:d6f88de62e1d4e38ecb1077eb9724cd0eff29d2a08ca16a401e9b9e93f117cf9", size = 109924, upload-time = "2026-06-07T16:44:21.566Z" }, +] + +[[package]] +name = "bs4" +version = "0.0.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "beautifulsoup4" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c9/aa/4acaf814ff901145da37332e05bb510452ebed97bc9602695059dd46ef39/bs4-0.0.2.tar.gz", hash = "sha256:a48685c58f50fe127722417bae83fe6badf500d54b55f7e39ffe43b798653925", size = 698, upload-time = "2024-01-17T18:15:47.371Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/bb/bf7aab772a159614954d84aa832c129624ba6c32faa559dfb200a534e50b/bs4-0.0.2-py2.py3-none-any.whl", hash = "sha256:abf8742c0805ef7f662dce4b51cca104cffe52b835238afc169142ab9b3fbccc", size = 1189, upload-time = "2024-01-17T18:15:48.613Z" }, +] + +[[package]] +name = "certifi" +version = "2026.7.22" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a3/c2/24167ea9858356b47a87a50d39908bfdb72ceeefe0041586e704e5376b3a/certifi-2026.7.22.tar.gz", hash = "sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55", size = 138112, upload-time = "2026-07-22T03:35:12.644Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/a7/71ac2cff56fec219ed242bb11b8efb69fcc4bec75db06fb7bfe35de520e6/certifi-2026.7.22-py3-none-any.whl", hash = "sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775", size = 136983, upload-time = "2026-07-22T03:35:11.276Z" }, +] + +[[package]] +name = "charset-normalizer" +version = "3.4.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bd/2a/23f34ec9d04624958e137efdc394888716353190e75f25dd22c7a2c7a8aa/charset_normalizer-3.4.9.tar.gz", hash = "sha256:673611bbd43f0810bec0b0f028ddeaaa501190339cac411f347ac76917c3ae7b", size = 152439, upload-time = "2026-07-07T14:34:58.454Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/70/4a/ecbd131485c07fcdfad54e28946d513e3da22ef3b4bd854dcafae54ec739/charset_normalizer-3.4.9-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:45b0cc4e3556cd875e09102988d1ab8356c998b596c9fced84547c8138b487a0", size = 319300, upload-time = "2026-07-07T14:33:15.666Z" }, + { url = "https://files.pythonhosted.org/packages/ec/96/5d9364e3342d69f3a045e1777bc47c85c383e6e9466d561b33fdb419d1f9/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9b2aff1c7b3884512b9512c3eaadd9bab39fb45042ffaaa1dd08ff2b9f8109d9", size = 215802, upload-time = "2026-07-07T14:33:17.031Z" }, + { url = "https://files.pythonhosted.org/packages/4b/4c/5361f9aa7f2cb58d94f2ab831b3d493f69efb1d239654b4744e3c09527cb/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9104ed0bd76a429d46f9ec0dbc9b08ad1d2dcdf2b00a5a0daa1c145329b35b44", size = 237171, upload-time = "2026-07-07T14:33:18.576Z" }, + { url = "https://files.pythonhosted.org/packages/50/78/ce342ca4ff30b2eb49fe6d9578df85974f90c67d294113e94efdd9664cbd/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7b86a2b16095d250c6f58b3d9b2eee6f4147754344f3dab0922f7c9bf7d226c9", size = 233075, upload-time = "2026-07-07T14:33:20.084Z" }, + { url = "https://files.pythonhosted.org/packages/01/c4/4fa4c8b3097a11f3c5f09a35b72ed6855fb1d332469504962ab7bafcc702/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5e226f6218febc71f6c1fc2fafb91c226f75bdc1d8fb12d66823716e891608fd", size = 224256, upload-time = "2026-07-07T14:33:21.747Z" }, + { url = "https://files.pythonhosted.org/packages/87/3a/ad914516df7e358a81aae018caa5e0470ba827fa6d763b1d2e87d920a5f6/charset_normalizer-3.4.9-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:90c44bc373b7687f6948b693cceaea1348ae0975d7474746559494468e3c1d84", size = 208784, upload-time = "2026-07-07T14:33:23.313Z" }, + { url = "https://files.pythonhosted.org/packages/d7/74/3c12f9755717dfe5c5c87da63f35d765fa0c00382ec26bf23f7fae34f2ba/charset_normalizer-3.4.9-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9cdef90ae47919cae358d8ab15797a800ed41da7aba5d72419fb510729e2ed4b", size = 219928, upload-time = "2026-07-07T14:33:24.814Z" }, + { url = "https://files.pythonhosted.org/packages/33/9a/895095b83e7907abd6d3d99aad3a38ad0d9686cc186cb0c94c24320fe63e/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:60f44ade2cf573dad7a277e6f8ca9a51a21dda572b13bd7d8539bb3cd5dbedde", size = 218489, upload-time = "2026-07-07T14:33:26.42Z" }, + { url = "https://files.pythonhosted.org/packages/a1/34/ef5c05f412f42520d7709b7d3784d19640839eb7366ded1755511585429f/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:a1786910334ed46ab1dd73222f2cd1e05c2c3bb39f6dddb4f8b36fc382058a39", size = 210267, upload-time = "2026-07-07T14:33:27.952Z" }, + { url = "https://files.pythonhosted.org/packages/83/dc/9b29fa4412b318bf3bfea985c35d67eb55e04b59a7c3f2237168b0e0be6f/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:03d07803992c6c7bbc976327f34b18b6160327fc81cb82c9d504720ac0be3b62", size = 226030, upload-time = "2026-07-07T14:33:29.397Z" }, + { url = "https://files.pythonhosted.org/packages/0e/42/6dbc00b8cd16011691203e33570fa42ed5746599a2e878112d16eab403a3/charset_normalizer-3.4.9-cp312-cp312-win32.whl", hash = "sha256:78841cccf1af7b40f6f716338d50c0902dbe88d9f800b3c973b7a9a0a693a642", size = 151185, upload-time = "2026-07-07T14:33:30.781Z" }, + { url = "https://files.pythonhosted.org/packages/80/cc/f920afd1a23c58ccd53c1d36085a71893a4737ff5e66e0371efab6809850/charset_normalizer-3.4.9-cp312-cp312-win_amd64.whl", hash = "sha256:4b3dac63058cc36820b0dd072f89898604e2d39686fe05321729d00d8ac185a0", size = 162557, upload-time = "2026-07-07T14:33:32.176Z" }, + { url = "https://files.pythonhosted.org/packages/f0/e6/0386d43a261ff4e4b30c5857af7df877254b46bec7b9d1b74b6bf969a90b/charset_normalizer-3.4.9-cp312-cp312-win_arm64.whl", hash = "sha256:78fa18e436a1a0e58dbd7e02fc4473f3f32cceb12df9dfca542d075961c307d2", size = 152665, upload-time = "2026-07-07T14:33:33.711Z" }, + { url = "https://files.pythonhosted.org/packages/98/2b/f97f1c193fb855c345d678f5077d6926034db0722df74c8f057020e05a25/charset_normalizer-3.4.9-py3-none-any.whl", hash = "sha256:68e5f26a1ad57ded6d1cfb85331d1c1a195314756471d97758c48498bb4dcdf5", size = 64538, upload-time = "2026-07-07T14:34:56.993Z" }, +] + [[package]] name = "click" version = "8.4.2" @@ -94,6 +169,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/03/10388a42375ee7e4ac9b94eb2c5c569c8b5795e377e701c9ac3ad63de890/fastapi-0.141.1-py3-none-any.whl", hash = "sha256:bfb91aa2d334c61cb35ba9a116fc123b3d3df31640b801cf57a7a78ec3f603b3", size = 131954, upload-time = "2026-07-29T17:18:04.364Z" }, ] +[[package]] +name = "greenlet" +version = "3.5.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a3/74/b13368064b09053253555d3f2839cc2684d22d5aed0d2ccffbf7a6736558/greenlet-3.5.4.tar.gz", hash = "sha256:0232ae1de90a8e07867bb127d7a6ba2301e859145489f25cda8a6096dabe1d20", size = 206538, upload-time = "2026-07-22T12:47:14.468Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f3/04/81bd731d6d1e3a469d9a4c36f5eb069bcf0cbb2d5d342c9fec22245b91fc/greenlet-3.5.4-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:3d66250e8b09f182ede05490998c818b5961f7a3640332d44c4927caec7bbfe4", size = 295909, upload-time = "2026-07-22T11:38:09.261Z" }, + { url = "https://files.pythonhosted.org/packages/cc/dd/f5f22903a6ae70f5ea328ed0beaec92ad903f0e3b7d2845133b354abc4b8/greenlet-3.5.4-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c90e930c9c192e5b3ee9fb8bcd920ea3926155e2e3ded39fc697323addecee17", size = 612011, upload-time = "2026-07-22T12:26:40.69Z" }, + { url = "https://files.pythonhosted.org/packages/8e/10/92a4a88d12b915d74ea5b6d288e4afefda4771647caa34442c156f7a454f/greenlet-3.5.4-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:791fdfeeb9c6e0c7b10fa151bf110d2a6974866f13dcb5b1c7efae698245893a", size = 624299, upload-time = "2026-07-22T12:29:02.089Z" }, + { url = "https://files.pythonhosted.org/packages/6c/f9/03e26be3487c5238e81f2b84714959a86ea8515a869828cf41f4fc54b34e/greenlet-3.5.4-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b7c895310363f310361e0fe2072af85269d2a2a285cd04c0c59e79a5e3670dcf", size = 629603, upload-time = "2026-07-22T12:43:43.456Z" }, + { url = "https://files.pythonhosted.org/packages/50/6d/0b14bb9db2989f32cd9fe7f76afedea01ee8bee3f87c07e69f24adfe7e63/greenlet-3.5.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f88193799d43dbf8c8a806d6405c9c52fe2af40bf75072a606357b33cc336c7f", size = 621541, upload-time = "2026-07-22T11:51:09.464Z" }, + { url = "https://files.pythonhosted.org/packages/57/6b/7c55ca72ef80d57c16c4a55210f82582622462dc4485799a30f4ec6f3372/greenlet-3.5.4-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:13b980043cb1b3134e81ea469da1250ddcc6bfe6d245bbaa59168d9cdc8f228f", size = 432554, upload-time = "2026-07-22T12:39:51.379Z" }, + { url = "https://files.pythonhosted.org/packages/48/3d/25e9a2d9eb6b2e8b7ca4e80a3a26cb887cce6c8e0a87c921164f11bc5574/greenlet-3.5.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:b7a5f095767c4493afcd06067f2bb3b8716e3f3f9e92b99c88e7e99f885b3d4d", size = 1581444, upload-time = "2026-07-22T12:25:03.818Z" }, + { url = "https://files.pythonhosted.org/packages/b9/96/4c9bf2e2c408dcc0556edce69efa9f802e82223573c53240136a086821f1/greenlet-3.5.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:42afdc1ab5f66da8c586c32af9224a74a706b4f0ea0dc3a4188a0860a09c65c9", size = 1645842, upload-time = "2026-07-22T11:51:12.295Z" }, + { url = "https://files.pythonhosted.org/packages/b5/41/303ecb26a3a56122c0f4d4073ee078881847bd6b6f463ae0ec57ec20223b/greenlet-3.5.4-cp312-cp312-win_amd64.whl", hash = "sha256:60149df8f462d1b230038e6590c23c3b4768bb5d6c022b3b6e82532b34b0b8a3", size = 247169, upload-time = "2026-07-22T11:38:19.893Z" }, + { url = "https://files.pythonhosted.org/packages/a4/e3/ef56864b4c35fcb3eb3b41b869f6cc46f4cd3f5e2c68e74acde8ac433951/greenlet-3.5.4-cp312-cp312-win_arm64.whl", hash = "sha256:77d6ce04fed0d9aeed42e0f37923cc43eba9b027bdd9c34546bb4ccd143d0fe0", size = 245565, upload-time = "2026-07-22T11:38:27.061Z" }, +] + [[package]] name = "h11" version = "0.16.0" @@ -165,6 +258,63 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] +[[package]] +name = "lxml" +version = "6.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/3b/aab6728cae887456f409b4d75e8a01856e4f04bd510de38052a47768b680/lxml-6.1.1.tar.gz", hash = "sha256:ba96ae44888e0185281e937633a743ea90d5a196c6000f82565ebb0580012d40", size = 4197430, upload-time = "2026-05-18T19:19:06.424Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6a/6e/c4add832b6fc1e887125b96f880d7b9b70aae5248718e046b1704bcac4b9/lxml-6.1.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:104c09bda8d2a562824c0e319d0768ce26a779b7601e0931d33b09b53c392ef7", size = 8570821, upload-time = "2026-05-18T19:17:42.068Z" }, + { url = "https://files.pythonhosted.org/packages/22/00/ff3009c88e65de8011630acf8ab5a09cb2becd2aaf47fba2f3449f6224e9/lxml-6.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:25c6997a9a534e016695a0ba06b2f07945de682731ff01065b6d5a4474179da1", size = 4624252, upload-time = "2026-05-18T19:17:47.897Z" }, + { url = "https://files.pythonhosted.org/packages/42/95/bb63f0fd62e554fe078e1fb3c8fe9083c14ddc7ad7fa178d10e57e071ac7/lxml-6.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c921ba5c51e4e9f63b8b00267d06566e1f63407408a0496da2d1d0bfc819c7fc", size = 4930746, upload-time = "2026-05-18T19:18:29.637Z" }, + { url = "https://files.pythonhosted.org/packages/eb/99/0013e8d9b5960f4f041cf0b73e2f80c23eb5205b1f7bfb20203243651359/lxml-6.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:54a7f95e4de5fb94e2f9f4b9055c6ba33bf3d628fd77a1d647c5923caa2cdcdc", size = 5093723, upload-time = "2026-05-18T19:18:34.168Z" }, + { url = "https://files.pythonhosted.org/packages/29/91/317b332636bfc7bddcff828d41b3307f50043f4b237e40849c333d80fa1a/lxml-6.1.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f2ec43df44b1f76249ee0a615334f9b5b060e1c8bd90e706dad2d14d02f383", size = 5005557, upload-time = "2026-05-18T19:18:39.798Z" }, + { url = "https://files.pythonhosted.org/packages/42/2f/cc9bf06afe70f9c9093ae60855d9759da9db601ec4080f7473319666ffd7/lxml-6.1.1-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:70ef8a7e102a1508f8121aae5b0867abd663f72c14f0a9c937e6554cb4587b7b", size = 5631036, upload-time = "2026-05-18T19:18:44.858Z" }, + { url = "https://files.pythonhosted.org/packages/08/f6/af32e23e563971ffb0fb86be52bc5be5c2c118858ffc119bf6a9039b173d/lxml-6.1.1-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ebe6af670449830d6d9b752c256a983291c766a1365ba5d5460048f9e33a7818", size = 5240367, upload-time = "2026-05-18T19:18:49.217Z" }, + { url = "https://files.pythonhosted.org/packages/78/83/8555d40948b09ce86f1bd0c68a7ac31d07b1929f92cc1b074006c97ef2d2/lxml-6.1.1-cp312-cp312-manylinux_2_28_i686.whl", hash = "sha256:27acc820660aaffa4f7c087f29120e12980f7779d56d8492d263170111284740", size = 5350171, upload-time = "2026-05-18T19:18:52.779Z" }, + { url = "https://files.pythonhosted.org/packages/63/75/5d92da93729b7bad783689e6496049fa40927b45bec7bf183c981de3ca70/lxml-6.1.1-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:1db753c9115ec7100d073b744d17e25e88a8f90f5c39b2f5dd878149af59671f", size = 4694874, upload-time = "2026-05-18T19:18:55.139Z" }, + { url = "https://files.pythonhosted.org/packages/c5/b5/3aad415a9a25b822e783f15deeb4dffccf5113030f1afa2222dd929313d9/lxml-6.1.1-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c4f469aebd783bb741c2ecb2a681008fd26bfe5c16a9a72ed5467f834e810df2", size = 5244492, upload-time = "2026-05-18T19:19:01.28Z" }, + { url = "https://files.pythonhosted.org/packages/f1/a1/5fcf7eb9904b80086aa47dcf0027de07b1bb990afad2e6823144c368ae04/lxml-6.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:766b010012d59470072c1816b5b6c69f1d243e5db36ea5968e94accf430a4635", size = 5048232, upload-time = "2026-05-18T19:18:12.67Z" }, + { url = "https://files.pythonhosted.org/packages/77/74/1f601b63c7a69fcdf10fa9b148c81da8442204194f6c55509cc485c786b9/lxml-6.1.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:b8d812c6011c08b8111a15e54dd990b8923692d80adf35488bee34026c35accf", size = 4777023, upload-time = "2026-05-18T19:18:15.928Z" }, + { url = "https://files.pythonhosted.org/packages/a2/b9/7a78f51aec95b1bf780d78e12705a9f6533284f8693dc5c0e6724fa53d3f/lxml-6.1.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:fe0306bd29505a9177aac19f1877174b0e7422c222a59f70b2cd41633448c3dc", size = 5645773, upload-time = "2026-05-18T19:18:23.223Z" }, + { url = "https://files.pythonhosted.org/packages/a5/6e/98a7b7ad54e4e74fa1f20fff776913980619d0ebe5558232d7da6580bdd8/lxml-6.1.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:5ba186ad207446c65d3bb3d3e0412b032b1d9f595e59861e2354798c5703d955", size = 5233088, upload-time = "2026-05-18T19:18:31.433Z" }, + { url = "https://files.pythonhosted.org/packages/65/d1/bc0ed2427bf609f2ee10da303a6a226f9c8bce94f945dc29a32ce55de6e4/lxml-6.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:aa366a1e55b8ebfe8ca8ddc3cfe75c8ebade181aeb0f661d0cb05986b647f72a", size = 5260995, upload-time = "2026-05-18T19:18:37.091Z" }, + { url = "https://files.pythonhosted.org/packages/69/8b/6772e1a4b513fc50a8d931f19edde0e13ae6918510a1e13ff67864f3e5ed/lxml-6.1.1-cp312-cp312-win32.whl", hash = "sha256:126c93f7f56f0eda92f6d8c619edc463a4f23d9252f1c9d0405a76f25fa9f11a", size = 3596382, upload-time = "2026-05-18T19:17:18.37Z" }, + { url = "https://files.pythonhosted.org/packages/1b/89/45198e9624762af2dfd2cb8782598477ceb29f6e59caab560388ae1f4ec1/lxml-6.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:26e6eda8d38c1fcab1090dd196ee87cbd13788e531937610e2589085de074e77", size = 3997255, upload-time = "2026-05-18T19:17:56.781Z" }, + { url = "https://files.pythonhosted.org/packages/90/a9/7a54b6834088d9ae528a7b780584ba6a39a9457b0ac330479f20ffbc9449/lxml-6.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:6540377fbd53fe1b629172288c464fb18db11ce1fa7dc15891da10aa9dcc3e7f", size = 3659610, upload-time = "2026-05-19T19:22:50.843Z" }, +] + +[[package]] +name = "mako" +version = "1.4.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2a/12/b5fa2353e2754cd67fb9f83793fa48ff42c213a5da7e719869d2301f6ab8/mako-1.4.1.tar.gz", hash = "sha256:d7904710b662996425a21627710c4777c45053146942cf8a7aebf757c92b8c27", size = 410165, upload-time = "2026-08-05T06:10:56.611Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a5/54/12ed58d458474aaab5c3d180173e745a4fe131bb330370596876d19ff60f/mako-1.4.1-py3-none-any.whl", hash = "sha256:a359d9a94a541213958742b2698d0a7757bb83551767bc468a74b9905aba9617", size = 80010, upload-time = "2026-08-05T06:10:58.248Z" }, +] + +[[package]] +name = "markupsafe" +version = "3.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7e/99/7690b6d4034fffd95959cbe0c02de8deb3098cc577c67bb6a24fe5d7caa7/markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698", size = 80313, upload-time = "2025-09-27T18:37:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5a/72/147da192e38635ada20e0a2e1a51cf8823d2119ce8883f7053879c2199b5/markupsafe-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e", size = 11615, upload-time = "2025-09-27T18:36:30.854Z" }, + { url = "https://files.pythonhosted.org/packages/9a/81/7e4e08678a1f98521201c3079f77db69fb552acd56067661f8c2f534a718/markupsafe-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce", size = 12020, upload-time = "2025-09-27T18:36:31.971Z" }, + { url = "https://files.pythonhosted.org/packages/1e/2c/799f4742efc39633a1b54a92eec4082e4f815314869865d876824c257c1e/markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d", size = 24332, upload-time = "2025-09-27T18:36:32.813Z" }, + { url = "https://files.pythonhosted.org/packages/3c/2e/8d0c2ab90a8c1d9a24f0399058ab8519a3279d1bd4289511d74e909f060e/markupsafe-3.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d6dd0be5b5b189d31db7cda48b91d7e0a9795f31430b7f271219ab30f1d3ac9d", size = 22947, upload-time = "2025-09-27T18:36:33.86Z" }, + { url = "https://files.pythonhosted.org/packages/2c/54/887f3092a85238093a0b2154bd629c89444f395618842e8b0c41783898ea/markupsafe-3.0.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:94c6f0bb423f739146aec64595853541634bde58b2135f27f61c1ffd1cd4d16a", size = 21962, upload-time = "2025-09-27T18:36:35.099Z" }, + { url = "https://files.pythonhosted.org/packages/c9/2f/336b8c7b6f4a4d95e91119dc8521402461b74a485558d8f238a68312f11c/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:be8813b57049a7dc738189df53d69395eba14fb99345e0a5994914a3864c8a4b", size = 23760, upload-time = "2025-09-27T18:36:36.001Z" }, + { url = "https://files.pythonhosted.org/packages/32/43/67935f2b7e4982ffb50a4d169b724d74b62a3964bc1a9a527f5ac4f1ee2b/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:83891d0e9fb81a825d9a6d61e3f07550ca70a076484292a70fde82c4b807286f", size = 21529, upload-time = "2025-09-27T18:36:36.906Z" }, + { url = "https://files.pythonhosted.org/packages/89/e0/4486f11e51bbba8b0c041098859e869e304d1c261e59244baa3d295d47b7/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:77f0643abe7495da77fb436f50f8dab76dbc6e5fd25d39589a0f1fe6548bfa2b", size = 23015, upload-time = "2025-09-27T18:36:37.868Z" }, + { url = "https://files.pythonhosted.org/packages/2f/e1/78ee7a023dac597a5825441ebd17170785a9dab23de95d2c7508ade94e0e/markupsafe-3.0.3-cp312-cp312-win32.whl", hash = "sha256:d88b440e37a16e651bda4c7c2b930eb586fd15ca7406cb39e211fcff3bf3017d", size = 14540, upload-time = "2025-09-27T18:36:38.761Z" }, + { url = "https://files.pythonhosted.org/packages/aa/5b/bec5aa9bbbb2c946ca2733ef9c4ca91c91b6a24580193e891b5f7dbe8e1e/markupsafe-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:26a5784ded40c9e318cfc2bdb30fe164bdb8665ded9cd64d500a34fb42067b1c", size = 15105, upload-time = "2025-09-27T18:36:39.701Z" }, + { url = "https://files.pythonhosted.org/packages/e5/f1/216fc1bbfd74011693a4fd837e7026152e89c4bcf3e77b6692fba9923123/markupsafe-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f", size = 13906, upload-time = "2025-09-27T18:36:40.689Z" }, +] + [[package]] name = "nodeenv" version = "1.10.0" @@ -174,6 +324,25 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl", hash = "sha256:5bb13e3eed2923615535339b3c620e76779af4cb4c6a90deccc9e36b274d3827", size = 23438, upload-time = "2025-12-20T14:08:52.782Z" }, ] +[[package]] +name = "numpy" +version = "2.5.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/22/fd/89965aa4ac08c74998539fcbf24fa3540f3e15237fbeb6bcf9c908f4aade/numpy-2.5.1.tar.gz", hash = "sha256:a48a113e6afea91f5608793bafa7ef2ad481fefbda87ec5069f483de61cb9fa3", size = 20755553, upload-time = "2026-07-04T17:08:00.933Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/7b/14687aa674250e5e546f616f486b0d56d3631cd5b2415739141ce40bdcea/numpy-2.5.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2c889b56fe48b1018f764b0eec8df59ab654e9148aa91faa12596043500de277", size = 16801574, upload-time = "2026-07-04T17:06:12.423Z" }, + { url = "https://files.pythonhosted.org/packages/e1/19/cc5bb2a3f2913d27d6dbb2c78d25921fabaedc6741d4a5a615a11f3c5bf3/numpy-2.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ab451b59c5643c570974c43aef780703ef1d3b4965d2be07afd530615a9358d1", size = 11772250, upload-time = "2026-07-04T17:06:15.726Z" }, + { url = "https://files.pythonhosted.org/packages/42/77/fdf34a71dd30f54979b18603bee915e0aaf825b07afe79acd60b04b691e2/numpy-2.5.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:78798bd5b9ad744056af8efa90e3b9ddaa53272a0848a483084a1cc0a13b2dc0", size = 5331516, upload-time = "2026-07-04T17:06:17.913Z" }, + { url = "https://files.pythonhosted.org/packages/ce/e2/eb7efa015b4cce41e2517bf182a7fce0d7d5b9d9ed76a29bfa0f4fe4505c/numpy-2.5.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:2ae0ca40bcb22d6ba59c1dfd5446f49940b0f2d821fde133f10dda11f816b84e", size = 6664863, upload-time = "2026-07-04T17:06:20.02Z" }, + { url = "https://files.pythonhosted.org/packages/a9/4b/a2b32dd94ee9ffbeecb28152240042a3949db33b1c834d44090b80e1b3b8/numpy-2.5.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:61ac47e772e6b8ea489e1d2f441a34c5c3ac17327e7ce294cbdf535795ad4e75", size = 15167977, upload-time = "2026-07-04T17:06:21.621Z" }, + { url = "https://files.pythonhosted.org/packages/b8/a9/6e73d68500f80773f65f0654ea932019d6694329a0eb0ed0533de38df376/numpy-2.5.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:59fda5e192b570217ec2580c96f00e9a7e12ef6866a900eb089b62c1a32545ca", size = 16672469, upload-time = "2026-07-04T17:06:24.064Z" }, + { url = "https://files.pythonhosted.org/packages/24/7d/ad3e59015135f5261c95fd4cafeff159c955febd83a99a1d9250c4233815/numpy-2.5.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f7119ebff1a9829e9f431a4f9d28e703023bb6b9fe7c8f724467dbfc27c94ab3", size = 16527531, upload-time = "2026-07-04T17:06:26.69Z" }, + { url = "https://files.pythonhosted.org/packages/83/d0/a39b2fbcde9cb17a1dac678f254b33a6336298af9df338824c685425d5e8/numpy-2.5.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e824c2acf8862052246be5a44c15da1777940c60d010dd2aab897824d9c430f9", size = 18431940, upload-time = "2026-07-04T17:06:29.521Z" }, + { url = "https://files.pythonhosted.org/packages/04/12/cff070947791c1ed425ff76413189adbdc2fbe215eba7ce7fa454a03c7f8/numpy-2.5.1-cp312-cp312-win32.whl", hash = "sha256:08d60c810432eb83360958dea0999ac4cfb94531ea8efcbf0b7f277c2068aeb2", size = 6066764, upload-time = "2026-07-04T17:06:32.571Z" }, + { url = "https://files.pythonhosted.org/packages/65/66/53f31807a48a750f9d748da273bc3fcedd12b27ff1f3e373bfec55ef2dc0/numpy-2.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:f7d60026c0bdb1380e83bfa7a0419c4577ee4b9a08880afcb6dadeb74c649fa2", size = 12430966, upload-time = "2026-07-04T17:06:34.926Z" }, + { url = "https://files.pythonhosted.org/packages/2b/2a/d1a88066b1c14186f5d3c0d18c94f17b064511982bab0578d49ee9d43c29/numpy-2.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:17a25e09640602e10bc8de0e6fa2b3fd68eedd84ba6d7842dc8f32f9ab87bd0b", size = 10350488, upload-time = "2026-07-04T17:06:37.785Z" }, +] + [[package]] name = "packaging" version = "26.2" @@ -183,6 +352,28 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" }, ] +[[package]] +name = "pandas" +version = "3.0.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, + { name = "python-dateutil" }, + { name = "tzdata", marker = "sys_platform == 'emscripten' or sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/be/4f/5f3422a2afec5ffc46308b79e53291365a93748b498ac2e58bead0197916/pandas-3.0.5.tar.gz", hash = "sha256:dca3734d6ab7c906e6730f0788b0a1dbb9f2467731f9711f77995c8e9d62d712", size = 4658219, upload-time = "2026-07-22T22:19:28.819Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1c/54/1dc810ea558d1320b597aa140a514f2fdf1d2ea09c38cf556f13ea712ec9/pandas-3.0.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fa290c16964d4963fbfbc358928239cf3bd755b20e988ce944877def2f44471d", size = 10411717, upload-time = "2026-07-22T22:18:08.307Z" }, + { url = "https://files.pythonhosted.org/packages/68/56/fbe81c09195924d8b7b8d4461a20458fe80a6a5ed6b24f0314da684277e1/pandas-3.0.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:c2e26bb46934b8a2ca0c3de1d3d606fc5f6746584791b2db264d58cf370e08dc", size = 9957095, upload-time = "2026-07-22T22:18:10.6Z" }, + { url = "https://files.pythonhosted.org/packages/e0/51/fac252f4a913ed5eabf3c11b880a9e8d5a6c10f0b2129d0462212d238b4d/pandas-3.0.5-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:73fa87b08a7ef706f8aafda39ddaccf2a99047bea62d8c88a0361bcafb2237bc", size = 10485458, upload-time = "2026-07-22T22:18:12.834Z" }, + { url = "https://files.pythonhosted.org/packages/12/98/e976540c1addf70442be7842a18cf70884a964abbf69442504f4d2939989/pandas-3.0.5-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d373ce03ffd84010ed9839fa73672a9c8256990532e158440c0085db7d914b34", size = 10998091, upload-time = "2026-07-22T22:18:15.209Z" }, + { url = "https://files.pythonhosted.org/packages/a4/8c/1f29b5be8d3fc47dd7567eb167fabba2085879b31e0287ce7cba6d3d2ff4/pandas-3.0.5-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2a29c53d85ea98c5e792c59ef82ee9fbe6ca902c0d0adb6b23f45ef894cd7bf6", size = 11499501, upload-time = "2026-07-22T22:18:17.689Z" }, + { url = "https://files.pythonhosted.org/packages/9d/e2/bd9c98ad2df7b38bde002adde4cdf353519da51881634323b126c55997f9/pandas-3.0.5-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:a5ad3b02ed6bc7d7ae9b70804b2c6aa31827489d150f8e623ce82491b82085d7", size = 12060559, upload-time = "2026-07-22T22:18:20.147Z" }, + { url = "https://files.pythonhosted.org/packages/f3/9a/ffbd852d58bd74a617fe2f8ee6a58a96982271ce41cf981eab22190b4a4b/pandas-3.0.5-cp312-cp312-pyemscripten_2024_0_wasm32.whl", hash = "sha256:b2acb4650527eec6822c3dadb2b771277b65e7dae7a267d4bccf65fd1bb3fbce", size = 7197652, upload-time = "2026-07-22T22:18:22.502Z" }, + { url = "https://files.pythonhosted.org/packages/70/b5/d2d3e9ae73362ba4229651b0ee1455cf78073a1ce585f6ff693782ce263e/pandas-3.0.5-cp312-cp312-win_amd64.whl", hash = "sha256:80a611068e8a3ac23f7398c6c14eb46dc974e5cc9997f653e2dcfd1da74edd41", size = 9831691, upload-time = "2026-07-22T22:18:24.534Z" }, + { url = "https://files.pythonhosted.org/packages/52/51/dea1e89d6a6796b9c43f85a09b484ee03edb8a4c4842e73e200a8c11301c/pandas-3.0.5-cp312-cp312-win_arm64.whl", hash = "sha256:25ff585b972a18ef1fe9ffa3ac6544d9950508aa76832e5147640b6022821e49", size = 9105796, upload-time = "2026-07-22T22:18:27.064Z" }, +] + [[package]] name = "pluggy" version = "1.6.0" @@ -192,6 +383,42 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, ] +[[package]] +name = "psycopg" +version = "3.3.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, + { name = "tzdata", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/db/2f/cb91e5502ec9de1de6f1b76cfbf69531932725361168bb06963620c77e2e/psycopg-3.3.4.tar.gz", hash = "sha256:e21207764952cff81b6b8bdacad9a3939f2793367fdac2987b3aac36a651b5bc", size = 165799, upload-time = "2026-05-01T23:31:55.179Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5c/e0/7b3dee031daae7743609ce3c746565d4a3ed7c2c186479eb48e34e838c64/psycopg-3.3.4-py3-none-any.whl", hash = "sha256:b6bbc25ccf05c8fad3b061d9db2ef0909a555171b84b07f29458a447253d679a", size = 213001, upload-time = "2026-05-01T23:20:50.816Z" }, +] + +[package.optional-dependencies] +binary = [ + { name = "psycopg-binary", marker = "implementation_name != 'pypy'" }, +] + +[[package]] +name = "psycopg-binary" +version = "3.3.4" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/95/7d/03818e13ba7f36de93573c93ee3482006d3dfa8b0f8d28df511bad0a1a92/psycopg_binary-3.3.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:5ab28a2a7649df3b72e6b674b4c190e448e8e77cf496a65bd846472048de2089", size = 4591122, upload-time = "2026-05-01T23:27:56.162Z" }, + { url = "https://files.pythonhosted.org/packages/a5/b9/11b341edf8d54e2694726b273fe9652b254d989f4f63e3ac6816ad6b55f4/psycopg_binary-3.3.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6402a9d8146cf4b3974ded3fd28a971e83dc6a0333eb7822524a3aa20b546578", size = 4669943, upload-time = "2026-05-01T23:28:04.522Z" }, + { url = "https://files.pythonhosted.org/packages/8b/18/4665bacd65e7865b4372fcd8abb8b9186ada4b0025f8c2ca691b364a556c/psycopg_binary-3.3.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:580ae30a5f95ccd90008ec697d3ed6a4a2047a516407ad904283fa42086936e9", size = 5469697, upload-time = "2026-05-01T23:28:11.337Z" }, + { url = "https://files.pythonhosted.org/packages/7c/b1/b83136c6e510593d9b0c759ba5384337bc4ad82d19fda675adc4b2703c84/psycopg_binary-3.3.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e7510c37550f91a187e3660a8cc50d4b760f8c3b8b2f89ebc5698cd2c7f2c85d", size = 5152995, upload-time = "2026-05-01T23:28:20.529Z" }, + { url = "https://files.pythonhosted.org/packages/67/8d/a9821e2a648afe6091989929982a3b0f00b2631a859cb81379728f08fb75/psycopg_binary-3.3.4-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:77df19583501ea288eaf15ac0fe7ad01e6d8091a91d5c41df5c718f307d8e31b", size = 6738180, upload-time = "2026-05-01T23:28:30.654Z" }, + { url = "https://files.pythonhosted.org/packages/7e/58/2e349e8d23905dc2317b80ac65f48fb6f821a4777a4e994a60da91c4850f/psycopg_binary-3.3.4-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:018fbed325936da502feb546642c982dcc4b9ffdea32dfef78dbf3b7f7ad4070", size = 4978828, upload-time = "2026-05-01T23:28:37.277Z" }, + { url = "https://files.pythonhosted.org/packages/45/48/57b00d03b4721878326122a1f1e6b0a90b85bcaec56b5b2f8ea6cfa45235/psycopg_binary-3.3.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:17a21953a9e5ff3a16dab692625a3676e2f101db5e40072f39dbee2250194d68", size = 4509757, upload-time = "2026-05-01T23:28:43.078Z" }, + { url = "https://files.pythonhosted.org/packages/25/37/33b47d8c007df69aec500df5889767c4d313748e8e9e27a2fef8a6dabcee/psycopg_binary-3.3.4-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:eb05ee1c2b817d27c537333224c9e83c7afb86fe7296ba970990068baf819b16", size = 4190546, upload-time = "2026-05-01T23:28:50.016Z" }, + { url = "https://files.pythonhosted.org/packages/ca/c6/32b0835dbc2122617902b649d76a91c1e75406e76bf3d595b0c3bb5ffad6/psycopg_binary-3.3.4-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:773d573e11f437ce0bdb95b7c18dc58390494f96d43f8b45b9760436114f7652", size = 3926197, upload-time = "2026-05-01T23:28:55.55Z" }, + { url = "https://files.pythonhosted.org/packages/cd/68/d190ef0c0c5b16ded07831dabc8ddd412f4cdab07ec6e30ed38d9bda0e1f/psycopg_binary-3.3.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:71e55ccbdfae79a2ed9c6369c3008a3025817ff9d7e27b32a2d84e2a4267e66e", size = 4236627, upload-time = "2026-05-01T23:29:05.336Z" }, + { url = "https://files.pythonhosted.org/packages/25/8f/81dcbc2e8454b74d14881275ea45f00791052dac531a9fa8be1730d1685b/psycopg_binary-3.3.4-cp312-cp312-win_amd64.whl", hash = "sha256:494ca54901be8cf9eb7e02c25b731f2317c378efa44f43e8f9bd0e1184ae7be4", size = 3560782, upload-time = "2026-05-01T23:29:11.967Z" }, +] + [[package]] name = "pydantic" version = "2.13.4" @@ -303,6 +530,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/9d/7a/d968e294073affff457b041c2be9868a40c1c71f4a35fcc1e45e5493067b/pytest_cov-7.1.0-py3-none-any.whl", hash = "sha256:a0461110b7865f9a271aa1b51e516c9a95de9d696734a2f71e3e78f46e1d4678", size = 22876, upload-time = "2026-03-21T20:11:14.438Z" }, ] +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, +] + [[package]] name = "python-dotenv" version = "1.2.2" @@ -330,6 +569,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" }, ] +[[package]] +name = "requests" +version = "2.34.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "idna" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz", hash = "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed", size = 142856, upload-time = "2026-05-14T19:25:27.735Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" }, +] + [[package]] name = "ruff" version = "0.16.1" @@ -355,6 +609,64 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/46/240ea004bf6dc4feb40e9832f2205a476a47dd5b8a3f8211a5fc5f95e20e/ruff-0.16.1-py3-none-win_arm64.whl", hash = "sha256:dbaadaac38c70239f056d306b7476f246b0bf000fa6b3876402acbf5b227eaf8", size = 11309414, upload-time = "2026-07-30T19:36:58.79Z" }, ] +[[package]] +name = "simplejson" +version = "4.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/0e/2a/54837395a3487c725669428d513293612a48d82b95a0642c936932e5d898/simplejson-4.1.1.tar.gz", hash = "sha256:c08eb9f7a90f77ae470e19a07472e9a79ebc0d1c2315d86a72767665bd5ba79f", size = 118860, upload-time = "2026-04-24T19:24:59.819Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/60/25/e90998fe8e480eb43b966c09e835379887d427567ebd496563d3b1e16b19/simplejson-4.1.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:19040a17154dc03d289bab68d73ce0a6a0be01de30c584bbdd93490bead14b22", size = 112414, upload-time = "2026-04-24T19:23:06.084Z" }, + { url = "https://files.pythonhosted.org/packages/9c/a0/abd4785f36c3400f1fbb21f517be39295a750a714f04b7ee175adf6ef580/simplejson-4.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:a94ebaecdbaa80d9551a3ec6bf0c9302fc8b53ab6c1b2bfd498a1df4cb28158d", size = 91120, upload-time = "2026-04-24T19:23:07.877Z" }, + { url = "https://files.pythonhosted.org/packages/b8/78/fc060d2e3b13c6ec59288574b8efac64075e316b2afba4396a56b2422f78/simplejson-4.1.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:67341c95c0a168ab4a6d1e807e50463f1c8da932c3286d81e201266c427061fa", size = 91055, upload-time = "2026-04-24T19:23:09.264Z" }, + { url = "https://files.pythonhosted.org/packages/0c/b6/156a8de1e1b47694f0e7de6675866936608d45dc68388fd017d36f8693be/simplejson-4.1.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:45ec18e337fec538b7e902d489505c450b2454653d1290f3f50385e6fd8aa607", size = 190297, upload-time = "2026-04-24T19:23:11.226Z" }, + { url = "https://files.pythonhosted.org/packages/86/1c/e4d0eab695be3eb21d0f46bce820752031f03e7113f9c80a9b3c73ee7157/simplejson-4.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:820c69a4710400e9b248d5670647d60be58824369282d3925e516b3ff1a7cd82", size = 187002, upload-time = "2026-04-24T19:23:12.982Z" }, + { url = "https://files.pythonhosted.org/packages/76/0e/7f5a59d29426b062d5928fb88b403c3f797129d53be7102f955dbe51aa44/simplejson-4.1.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2e708d373a10e4378ef2d59f8361850c7150fd907ed49efe49bc5492160476d1", size = 195146, upload-time = "2026-04-24T19:23:14.517Z" }, + { url = "https://files.pythonhosted.org/packages/78/18/9943db224dd4d5fa3c090c3e56a94c37b254338c83995ec5680285111c40/simplejson-4.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:980fc33353f81fd12d8c49d44f8c2760d1dc8192285e627c5180d141035b228a", size = 183931, upload-time = "2026-04-24T19:23:16.742Z" }, + { url = "https://files.pythonhosted.org/packages/c2/08/9a690da9a766161c06c627d805362cf159f1abe480969372b2897649b955/simplejson-4.1.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:de2ed102fff88dacf543699f53ee3a533cc11539a39baa176b7e09dd783069d6", size = 192228, upload-time = "2026-04-24T19:23:18.33Z" }, + { url = "https://files.pythonhosted.org/packages/05/88/bd8aad36b451ffb0e0a3f721d695a88befa6d1ac7d1e02ae788ca7ff4029/simplejson-4.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2785ff8edc0e28bf773a32543a6bbed46351453c997b3f6709c744e3c2f7eabb", size = 187808, upload-time = "2026-04-24T19:23:21.165Z" }, + { url = "https://files.pythonhosted.org/packages/04/ee/14f91db0d1f481533b651dafbf8cd0da088d9817f7af30c68f7f19f9c847/simplejson-4.1.1-cp312-cp312-win32.whl", hash = "sha256:2e0d5ead6d14610467ec356ec1f6b5d8a56aa216abaad8d41c8b873b16cf313f", size = 88512, upload-time = "2026-04-24T19:23:22.764Z" }, + { url = "https://files.pythonhosted.org/packages/b9/c4/90de06b2d8737c68c05ff9274113f854dbf6a5f28b7a955212111672cb57/simplejson-4.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:63a5451f557d6be48a231bae932458655c620902b868170b2f1c8afed496f6b4", size = 90748, upload-time = "2026-04-24T19:23:24.494Z" }, + { url = "https://files.pythonhosted.org/packages/ce/6a/8b74c52ffd33dbbde00fe7251fee6a0acdc8cea33f7a43805aed258fb79b/simplejson-4.1.1-py3-none-any.whl", hash = "sha256:2ce92b3748f02423e26d2bfb636fb9d7a8f67c8f5854dcae69d350d123b2eee2", size = 69195, upload-time = "2026-04-24T19:24:57.962Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "soupsieve" +version = "2.9.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d9/38/e12680bbe6b4f8f3d17adcaf38d26850aa756c85cf4a80e79fc12a018fe8/soupsieve-2.9.1.tar.gz", hash = "sha256:c33e6605bbc71dd628b00c632d58ae607c22bade247e52553928f83bbb75b4ba", size = 122261, upload-time = "2026-07-21T16:57:17.452Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0f/2c/437fe806897c2d6cfdc3ee43a18da8bf8e568530a4ae9bac781541ca9896/soupsieve-2.9.1-py3-none-any.whl", hash = "sha256:4f4477399246b7a0c720a88ca2454b11cd6bb9ae4c9d170140786e916776c14c", size = 37404, upload-time = "2026-07-21T16:57:16.421Z" }, +] + +[[package]] +name = "sqlalchemy" +version = "2.0.51" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "greenlet", marker = "platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64'" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/02/f1/a7a892f18d4d224e6b26f706531eafccc41e37594d37d304786969ee13cb/sqlalchemy-2.0.51.tar.gz", hash = "sha256:804dccd8a4a6242c4e30ad961e540e18a588f6527202f2d6791b01845d59fdc9", size = 9912201, upload-time = "2026-06-15T15:41:20.012Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d5/70/e868bc5412acd101a8280f25c95f10eeae0771c4eb806b02491142810ee8/sqlalchemy-2.0.51-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7d78702b26ba1c18b2d0fb2ea940ba7f17a9581b42e8361ff93920ebbee1235a", size = 2160291, upload-time = "2026-06-15T16:08:48.918Z" }, + { url = "https://files.pythonhosted.org/packages/e5/1c/71ee0f8a6b9d7316a1ccd30430b4c62b6c2e36adc96017a4e3a72dce49d6/sqlalchemy-2.0.51-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:581921d849d6e6f994d560389192955e80e2950e18fcdfe2ccea863e01158e6e", size = 3343835, upload-time = "2026-06-15T16:19:42.613Z" }, + { url = "https://files.pythonhosted.org/packages/2b/7c/7ab9f9aadc5944fdd06612484ed7918fe376ad871a5f50404dc1536e0194/sqlalchemy-2.0.51-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1d21ce524ab86c23046e992a5b81cb54c21079c6df6e78b8fc77d77cac70a6b9", size = 3358470, upload-time = "2026-06-15T16:26:38.011Z" }, + { url = "https://files.pythonhosted.org/packages/d0/7d/ff77169fee6186de145a7f2b87006c39638391130abbab2b1f63ac6ea583/sqlalchemy-2.0.51-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c5d98a2709840027f5a347c3af0a7c3d5f6c1ff93af2ca1c54494e23cba8f389", size = 3289874, upload-time = "2026-06-15T16:19:45.212Z" }, + { url = "https://files.pythonhosted.org/packages/6f/3b/6c505903710d781b55bc3141ee34a062bf9745a6b5bc7333305b9ed63b33/sqlalchemy-2.0.51-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:1181256e0f16479691b5616d36375dc2620ad8332b25978763c3d206ad3f3f1d", size = 3321692, upload-time = "2026-06-15T16:26:39.747Z" }, + { url = "https://files.pythonhosted.org/packages/3c/b7/c5ffe50aa2f4d947c9250e1519d939260329a07fe6272edfccd784b3d007/sqlalchemy-2.0.51-cp312-cp312-win32.whl", hash = "sha256:9f380393be5abeb6815f68fd39271b95127173511b6706b0a630a9995d53f8f5", size = 2119674, upload-time = "2026-06-15T16:23:09.543Z" }, + { url = "https://files.pythonhosted.org/packages/25/dc/46a65916af68a06ef6b972c6050ba4c8f97070fe3fb33097d34229d9bef6/sqlalchemy-2.0.51-cp312-cp312-win_amd64.whl", hash = "sha256:2cf39aabdf48e87c1c2c2ed6d20d33ffa0733b3071ce9c5f66357947dd009080", size = 2146670, upload-time = "2026-06-15T16:23:11.048Z" }, + { url = "https://files.pythonhosted.org/packages/e2/22/dbf013a12ec759e54a34a119e9e217435b3f71b2dd5c61a7ade0a25dae87/sqlalchemy-2.0.51-py3-none-any.whl", hash = "sha256:bb024d8b621d0be75f4f44ecc7c950450026e76d66dc8f791bb5331d7fed59d5", size = 1944334, upload-time = "2026-06-15T16:09:22.418Z" }, +] + [[package]] name = "starlette" version = "1.3.1" @@ -368,6 +680,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ec/bb/2799cc2ede3ed41131f8975621e7213dfc7ef4acbbaadfa440f32500c370/starlette-1.3.1-py3-none-any.whl", hash = "sha256:c7372aae11c3c3f26a42df7bd626cec2f47d03483d261d369516a615a53714c6", size = 73632, upload-time = "2026-06-12T09:23:10.017Z" }, ] +[[package]] +name = "tqdm" +version = "4.70.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/21/3b/6c24bec5be5e743ffd99576daa5cc077722fc7d5bbc00bd133fa0c698dc6/tqdm-4.70.0.tar.gz", hash = "sha256:55b0b0dbd97462d06ebee91e4dac24ed4d4702be82b24f07e6c1d27e08cea220", size = 795438, upload-time = "2026-07-27T11:33:15.271Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f9/1c/01bfd571a64e7f270e6bab5e33777debe0edc56759233ce84f27dec92d14/tqdm-4.70.0-py3-none-any.whl", hash = "sha256:7f585706bfddbdebf89daac705b2dfcc16890130727d3197ca62c732b4310953", size = 80184, upload-time = "2026-07-27T11:33:13.167Z" }, +] + [[package]] name = "truststore" version = "0.10.4" @@ -377,6 +701,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/19/97/56608b2249fe206a67cd573bc93cd9896e1efb9e98bce9c163bcdc704b88/truststore-0.10.4-py3-none-any.whl", hash = "sha256:adaeaecf1cbb5f4de3b1959b42d41f6fab57b2b1666adb59e89cb0b53361d981", size = 18660, upload-time = "2025-08-12T18:49:01.46Z" }, ] +[[package]] +name = "tushare" +version = "1.4.29" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "bs4" }, + { name = "lxml" }, + { name = "pandas" }, + { name = "requests" }, + { name = "simplejson" }, + { name = "tqdm" }, + { name = "websocket-client" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/91/a4/5ce99585209410463e4b203d13c73d356c6ac92daea48af182932af7cb61/tushare-1.4.29.tar.gz", hash = "sha256:f578fb778868c0b744ac9173a837b695fd82e8d1b1e0d39c1f882b0e4fef7ecb", size = 128623, upload-time = "2026-03-25T06:33:44.722Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/33/3e/d426a56e5feac9b0aaada1c6b0745ed03422d4a713295e0bbb44c8ea86fe/tushare-1.4.29-py3-none-any.whl", hash = "sha256:82554af953ea5ac3d8771d42330493181031c7e68dccce03a491c7356e9ba4b2", size = 142920, upload-time = "2026-03-25T06:33:43.111Z" }, +] + [[package]] name = "typing-extensions" version = "4.16.0" @@ -398,6 +740,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611, upload-time = "2025-10-01T02:14:40.154Z" }, ] +[[package]] +name = "tzdata" +version = "2026.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/92/ff/5a28bdfd8c3ebec42564ac7d0e54ca3db65044a9314a97f9564fa7a1e926/tzdata-2026.3.tar.gz", hash = "sha256:4a1518b8993086a7982523e071643f3c0e5f213e75b21318e78bcabfff9d1415", size = 198674, upload-time = "2026-07-10T08:50:37.887Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/6d/b53b99a9f2766d095985947a5782f1702cabb129a34f7a802d7197af832f/tzdata-2026.3-py2.py3-none-any.whl", hash = "sha256:dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931", size = 348168, upload-time = "2026-07-10T08:50:36.46Z" }, +] + +[[package]] +name = "urllib3" +version = "2.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, +] + [[package]] name = "uvicorn" version = "0.52.1" @@ -460,6 +820,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0a/26/88e0dc6ee3898169d7fa22bb6a69cabf2502d2ee25cb8c876d1262d204f8/watchfiles-1.2.0-cp312-cp312-win_arm64.whl", hash = "sha256:8fa585ede612ee9f9e91b18bebf9ba11b9ae29a4e3a0d0cf6fca3e382133f0d5", size = 281026, upload-time = "2026-05-18T04:30:22.23Z" }, ] +[[package]] +name = "websocket-client" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/2c/41/aa4bf9664e4cda14c3b39865b12251e8e7d239f4cd0e3cc1b6c2ccde25c1/websocket_client-1.9.0.tar.gz", hash = "sha256:9e813624b6eb619999a97dc7958469217c3176312b3a16a4bd1bc7e08a46ec98", size = 70576, upload-time = "2025-10-07T21:16:36.495Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/34/db/b10e48aa8fff7407e67470363eac595018441cf32d5e1001567a7aeba5d2/websocket_client-1.9.0-py3-none-any.whl", hash = "sha256:af248a825037ef591efbf6ed20cc5faa03d3b47b9e5a2230a529eeee1c1fc3ef", size = 82616, upload-time = "2025-10-07T21:16:34.951Z" }, +] + [[package]] name = "websockets" version = "17.0.1" @@ -494,8 +863,12 @@ name = "zhixing-server" version = "0.1.0" source = { editable = "." } dependencies = [ + { name = "alembic" }, { name = "fastapi" }, + { name = "psycopg", extra = ["binary"] }, { name = "pydantic-settings" }, + { name = "sqlalchemy" }, + { name = "tushare" }, { name = "uvicorn", extra = ["standard"] }, ] @@ -510,8 +883,12 @@ dev = [ [package.metadata] requires-dist = [ + { name = "alembic", specifier = ">=1.18.0" }, { name = "fastapi", specifier = ">=0.141.1" }, + { name = "psycopg", extras = ["binary"], specifier = ">=3.3.2" }, { name = "pydantic-settings", specifier = ">=2.14.2" }, + { name = "sqlalchemy", specifier = ">=2.0.46" }, + { name = "tushare", specifier = ">=1.4.24" }, { name = "uvicorn", extras = ["standard"], specifier = ">=0.52.1" }, ]