zkr/ocups-kafka
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0976909cc8b7fa744e48c886ed5cd07d2ae682c8
ocups-kafka/docs/采集流程时序图.md

196 lines
13 KiB
Markdown
Raw Normal View History

feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
# 增量采集流程时序图
rabbitmq
2026-01-15 21:20:57 +08:00
## 1. 核心逻辑
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
rabbitmq
2026-01-15 21:20:57 +08:00
### 采集方向（从后往前）
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
```
rabbitmq
2026-01-15 21:20:57 +08:00
offset: total-100 → total-200 → ... → last_start_offset
优势：先采集最新数据，下次只采集新增部分
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
```
rabbitmq
2026-01-15 21:20:57 +08:00
### 消息队列
- 使用 RabbitMQ，支持消息级别 TTL
- 消息过期时间：7天，过期自动删除
- 每批数据过滤后立即发送，不等待任务结束
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
rabbitmq
2026-01-15 21:20:57 +08:00
## 2. 容器启动与自动采集
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
```
rabbitmq
2026-01-15 21:20:57 +08:00
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Docker │ │ App │ │ Crawler │ │ RabbitMQ │
│ 容器 │ │ FastAPI │ │ Manager │ │ │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │ │
│ docker-compose │ │ │
│ up │ │ │
│───────────────>│ │ │
│ │ │ │
│ │ lifespan启动 │ │
│ │ auto_start=true│ │
│ │───────────────>│ │
│ │ │ │
│ │ │ 遍历enabled任务 │
│ │ │ 创建TaskCrawler │
│ │ │────────┐ │
│ │ │<───────┘ │
│ │ │ │
│ │ │ 并行启动所有任务│
│ │ │═══════════════>│
│ │ │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
```
## 3. 单任务采集流程（从后往前，实时发送）
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
rabbitmq
2026-01-15 21:20:57 +08:00
│ TaskCrawler │ │ 八爪鱼API │ │ DateFilter │ │ RabbitMQ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │ │
│ 1.获取数据总数 │ │ │
│───────────────>│ │ │
│<───────────────│ │ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ total=270000 │ │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│ │ │ │
│ 2.读取上次进度，计算采集范围 │ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ start_offset = total - 100 = 269900 │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│ end_offset = last_start_offset (上次起始位置) │
│────────┐ │ │ │
│<───────┘ │ │ │
│ │ │ │
│ ╔══════════════════════════════════════════════════════════╗
│ ║ 循环：每批请求→过滤→立即发送 ║
│ ╚══════════════════════════════════════════════════════════╝
│ │ │ │
│ 3.请求一批数据 │ │ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ offset=269900 │ │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│───────────────>│ │ │
│<───────────────│ │ │
│ 返回100条 │ │ │
│ │ │ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ 4.过滤数据(7天内有效) │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│───────────────────────────────>│ │
│<───────────────────────────────│ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ 有效95条，过期5条 │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│ │ │ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ 5.立即发送到RabbitMQ │ │
│ (消息TTL=7天，过期自动删除) │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│────────────────────────────────────────────────>│
│<────────────────────────────────────────────────│
│ 发送成功 │ │ │
│ │ │ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ 6.更新offset，继续循环 │ │
│ offset = 269900 - 100 = 269800 │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│────────┐ │ │ │
│<───────┘ │ │ │
│ │ │ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ 7.检查停止条件 │ │ │
│ offset >= end_offset ? 继续 │ │
│ offset < end_offset ? 停止 │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│────────┐ │ │ │
rabbitmq
2026-01-15 21:20:57 +08:00
│<───────┘ │ │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│ │ │ │
```
rabbitmq
2026-01-15 21:20:57 +08:00
## 4. 进度记录与增量采集
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
```
┌─────────────────────────────────────────────────────────────────────────┐
│ 进度记录与增量采集 │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ 首次采集: │
│ ┌─────────────────────────────────────────────────────────────────┐ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ │ total = 270000 │ │
│ │ start_offset = total - 100 = 269900 │ │
│ │ end_offset = 0 (首次采集，遇到连续过期数据停止) │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│ │ │ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ │ 采集完成后保存: last_start_offset = 269900 │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ 下次采集: │
│ ┌─────────────────────────────────────────────────────────────────┐ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ │ total = 270500 (新增500条) │ │
│ │ start_offset = 270500 - 100 = 270400 │ │
│ │ end_offset = 269900 (上次的起始位置) │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│ │ │ │
rabbitmq
2026-01-15 21:20:57 +08:00
│ │ 只采集 270400 → 269900 这部分新增数据 │ │
│ │ 采集完成后保存: last_start_offset = 270400 │ │
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
```
## 5. 停止条件
采集停止的条件（满足任一即停止）：
rabbitmq
2026-01-15 21:20:57 +08:00
1. `offset < end_offset` - 已采集到上次的起始位置
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
2. 连续3批数据全部过期 - 数据太旧（仅首次采集时生效）
3. 手动调用停止接口
rabbitmq
2026-01-15 21:20:57 +08:00
## 6. 消息过期机制
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
```
rabbitmq
2026-01-15 21:20:57 +08:00
┌─────────────────────────────────────────────────────────────────────────┐
│ RabbitMQ 消息TTL │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ 消息发送时设置 TTL = 7天 (604800000ms) │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ 消息A │ │ 消息B │ │ 消息C │ │ 消息D │ │
│ │ 1月8日 │ │ 1月10日 │ │ 1月14日 │ │ 1月15日 │ │
│ │ 已过期 │ │ 即将过期 │ │ 有效 │ │ 有效 │ │
│ │ 自动删除 │ │ │ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ ↓ │
│ RabbitMQ自动清理 │
│ │
│ 优势： │
│ - 消息级别TTL，精确控制每条消息的过期时间 │
│ - 过期消息自动删除，无需手动清理 │
│ - 队列中始终保持最近7天的有效数据 │
│ │
└─────────────────────────────────────────────────────────────────────────┘
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
```
rabbitmq
2026-01-15 21:20:57 +08:00
## 7. 配置说明
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
```yaml
# config.yml
rabbitmq
2026-01-15 21:20:57 +08:00
# RabbitMQ配置
rabbitmq:
host: rabbitmq # Docker内部服务名
port: 5672
username: guest
password: guest
queue: job_data
message_ttl: 604800000 # 消息过期时间：7天(毫秒)
# 采集配置
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
crawler:
filter_days: 7 # 数据有效期（天）
rabbitmq
2026-01-15 21:20:57 +08:00
max_expired_batches: 3 # 连续过期批次阈值（首次采集时生效）
feat(job_crawler): implement reverse-order incremental crawling with real-time Kafka publishing - Add comprehensive sequence diagrams documenting container startup, task initialization, and incremental crawling flow - Implement reverse-order crawling logic (from latest to oldest) to optimize performance by processing new data first - Add real-time Kafka message publishing after each batch filtering instead of waiting for task completion - Update progress tracking to store last_start_offset for accurate incremental crawling across sessions - Enhance crawler service with improved offset calculation and batch processing logic - Update configuration files to support new crawling parameters and Kafka integration - Add progress model enhancements to track crawling state and handle edge cases - Improve main application initialization to properly handle lifespan events and task auto-start This change enables efficient incremental data collection where new data is prioritized and published immediately, reducing latency and improving system responsiveness.
2026-01-15 17:46:55 +08:00
auto_start: true # 容器启动时自动开始采集
```
rabbitmq
2026-01-15 21:20:57 +08:00
## 8. API接口
| 接口 | 方法 | 说明 |
|------|------|------|
| `/status` | GET | 获取采集状态 |
| `/tasks` | GET | 获取任务列表 |
| `/crawl/start` | POST | 启动采集任务 |
| `/crawl/stop` | POST | 停止采集任务 |
| `/consume` | GET | 消费队列数据 |
| `/queue/size` | GET | 获取队列消息数量 |
## 9. 数据流向
```
八爪鱼API → 采集服务(过滤7天内数据) → RabbitMQ(TTL=7天) → 第三方消费
↓
过期自动删除
```
Reference in New Issue Copy Permalink